A second OpenAI-compatible provider is cheap to add because it sits behind the client you already use. Making the call dependable is a separate job, and it is the job most teams put off. Three patterns cover most of it: a timeout you set on purpose, a retry that only fires when it is safe, and a fallback path you can turn on without a deploy.

1. Set a timeout you chose

The default HTTP timeout in most clients is either infinite or far longer than a user will wait. Decide a budget per call, then enforce it. For a streaming response, the timeout you want is usually a read timeout on the gap between chunks, not a cap on total duration, because a long answer is not a slow answer.

import httpx
from openai import OpenAI

client = OpenAI(
    base_url="https://webwright.ai/v1",
    api_key="www_v1_YOUR_API_KEY",
    timeout=httpx.Timeout(connect=5.0, read=30.0, write=30.0, pool=5.0),
)

2. Retry only what is safe to retry

A retry is not free. Retrying a request that already succeeded costs you twice and can duplicate a side effect. Retry a connection error, a timeout before any bytes arrived, and a server error that is clearly transient. Do not blindly retry a request whose outcome you did not observe.

Bounded exponential backoff with jitter is the standard shape. Cap the attempts and the total wait, and stop when you hit the cap.

import random, time

def with_retry(call, attempts=4, base=0.5, cap=8.0):
    for i in range(attempts):
        try:
            return call()
        except (httpx.ConnectError, httpx.ReadTimeout) as err:
            if i == attempts - 1:
                raise
            wait = min(cap, base * 2 ** i)
            time.sleep(wait * (0.5 + random.random()))

For a streamed response, decide the retry rule before the first token: retrying after partial output can duplicate work. If the caller already rendered half the answer, restarting is a product decision, not a transport one.

3. Put a second provider behind a flag

Failover is the reason many teams add a second OpenAI-compatible base URL in the first place. Keep the second client configured but dark, and switch on a feature flag so you can move traffic without a deploy and without a code change.

clients = {
    "primary": OpenAI(base_url="https://webwright.ai/v1", api_key=KEY),
    "secondary": OpenAI(base_url=SECOND_BASE_URL, api_key=SECOND_KEY),
}

def complete(prompt):
    order = ["primary", "secondary"] if flag("use_fallback") else ["primary"]
    last = None
    for name in order:
        try:
            return clients[name].chat.completions.create(
                model=MODEL_ID,
                messages=[{"role": "user", "content": prompt}],
            )
        except (httpx.ConnectError, httpx.ReadTimeout) as err:
            last = err
    raise last

4. Measure before you trust the fallback

A fallback you have never exercised is a guess. Run the same prompt set against both paths, record success rate, p50 and p95 latency, and cost, and score quality with a small blind rubric. Replace the claim with your own measurement. The evaluation harness is the same one the migration playbook describes.

5. When to book a consult

If you would rather not design the retry, timeout, and failover rules alone, that is what the consult is for. One working session maps your current client, your failure modes, and what good would look like, and you leave with a prioritized first change, an architecture, a rough cost, and the honest risks.

The WebWright Team