Adding a second AI provider is easy to evaluate on paper and harder to switch to in production. An offline evaluation runs your prompts. Production runs your traffic: the long tail of odd inputs, the streaming calls, the retries, the one user who pastes a 40,000-token document. The way to see how a candidate provider handles that mix before your users do is to shadow it.

What shadow testing is

You copy a small share of live requests, send each copy to the candidate provider, throw its response away for the user, and log both responses for comparison. The user still gets your current provider's answer. You get a side-by-side record on real traffic, with no user-facing risk.

Two safety rules come first

  1. The shadow call never touches the user path. It runs after the primary response is already returned, or in a separate worker. It has its own short timeout. If it fails, times out, or errors, you drop it and move on. A shadow test that can slow down or fail a real request is not a shadow test.
  2. You only shadow data you are allowed to send to a second provider. A shadow request is a real call with real data. If a payload carries fields you would not send to the candidate, strip them before you copy, or exclude those requests from the sample. The data-handling review on this site is a good checklist for that decision.

The sequence

  1. Sample. Mirror a small, capped share of requests, not all of them. One to five percent is usually enough to see the shape of your traffic within a day, and it keeps the shadow cost and volume small. Sample by request id so the choice is stable and reproducible.
  2. Duplicate. Build the candidate request from the same prompt and parameters as the primary. Keep the model name, the messages, the streaming flag, and any options aligned, so you are comparing providers and not two different prompts.
  3. Isolate. Send the copy out of band. Give it a short timeout and a single attempt. Never retry a shadow call inside the request path. Wrap it so no error from the candidate can surface to the user.
  4. Capture. For each mirrored request, log a stable request id, a hash of the prompt, both responses, the usage object each provider returns, wall-clock latency for each, and any error, refusal, or empty response. The usage object matters: it is the provider's own token count, so your cost comparison uses their number, not your estimate.
  5. Compare offline. Read the pairs on four axes, kept separate:
    • Quality. Score against a rubric or by human review. Keep it simple: does the answer satisfy the task, and would a reviewer accept it without edits? Do not blend quality into a single score with cost.
    • Cost. Multiply the recorded input and output token counts by each provider's published rates. Compare cost per successful outcome, not per call, so a provider that fails more does not look cheaper than it is.
    • Errors. Count timeouts, refusals, malformed responses, and rate-limit errors per provider. A cheap provider that errors on the long tail is not cheap.
    • Latency. Compare distributions, not single numbers, and compare like with like: streamed against streamed, buffered against buffered. A streamed first token and a buffered total time are different measurements.
  6. Canary. If the shadow data is acceptable, move a small percentage of real traffic to the candidate and use its response for those users. Watch the same four axes plus your own product signals. Keep the old provider as the fallback the whole time.
  7. Ramp or roll back. Increase the canary share as confidence grows, or turn it off. Because the fallback path already exists, rollback is a flag change, not a deploy.

Common mistakes

  • Putting the shadow call inside the request path, so it can slow down or fail a real request.
  • Shadowing every request instead of a capped sample, which doubles volume and cost for no extra signal.
  • Comparing a streamed candidate response against a buffered primary response, or the reverse.
  • Forgetting that shadow traffic is itself billed by the candidate provider, so it is a real cost even before you switch.
  • Copying a payload with fields you would not normally send to a second provider.
  • Comparing raw averages when one provider fails on the long tail. Read the errors and the tail, not only the median.

When a working session is worth it

A working session earns its place when the rollout is genuinely open: the workflow spans more than one endpoint, you have both chat and transcription traffic to migrate, the sample needs a privacy carve-out, or you want a second opinion on the canary plan and the rollback trigger. Bring the capture fields and the comparison rubric above and the session starts where a cold call would take half an hour to reach.

The WebWright Team