Once your app calls an AI API, every commit raises the same question: did I break the integration? A test that hits the live endpoint on every run answers it and sends you a bill for asking. You do not need either problem. Split your tests into three layers, and only one of them talks to the network.

Layer 1: unit tests against a stub. Most of your code does not care what the model says. It cares that you built the request correctly, parsed the response, and handled the errors. Stub the client and test that.

With the OpenAI Python SDK you can point a fake transport at the client, so your own code runs unchanged:

import httpx
from openai import OpenAI

def fake_transport(request: httpx.Request) -> httpx.Response:
    if request.url.path.endswith("/chat/completions"):
        return httpx.Response(200, json={
            "id": "chatcmpl-test",
            "object": "chat.completion",
            "model": "wren-r1",
            "choices": [{
                "index": 0,
                "finish_reason": "stop",
                "message": {"role": "assistant", "content": "hello"},
            }],
            "usage": {"prompt_tokens": 12, "completion_tokens": 2, "total_tokens": 14},
        })
    return httpx.Response(404, json={"error": {"type": "invalid_request_error"}})

client = OpenAI(
    base_url="https://webwright.ai/v1",
    api_key="www_v1_test_key",
    http_client=httpx.Client(transport=httpx.MockTransport(fake_transport)),
)

Now you can assert on your own logic: that the model id is wren-r1, that a system message is present, that you read usage.total_tokens, and that a 429 with Retry-After triggers your backoff instead of a crash. These tests are fast, free, and deterministic.

Layer 2: one gated smoke test. Stubs prove your code is correct against a shape you wrote down. They cannot tell you the live endpoint still accepts your request. Keep exactly one test that talks to the network, and gate it so it runs only when you ask:

import os
import pytest

pytestmark = pytest.mark.skipif(
    os.environ.get("RUN_LIVE") != "1",
    reason="live smoke test; set RUN_LIVE=1 to run",
)

def test_live_models_and_chat(live_client):
    models = live_client.models.list()
    assert any(m.id == "wren-r1" for m in models.data)

    out = live_client.chat.completions.create(
        model="wren-r1",
        messages=[{"role": "user", "content": "Reply with the single word: ok"}],
        max_tokens=4,
    )
    assert out.choices[0].message.content

Run it before a deploy or on a nightly schedule, not on every commit. It costs a few tokens and proves the base URL, the key, the model id, and the response shape all still line up. If you have a fallback provider, run the same test against each base URL and fail the deploy if either breaks.

Layer 3: a quality rubric, not an assertion. Do not assert on the content of a model response in a unit test. "The answer must contain the word invoice" is a test that fails when the model is right and passes when it is wrong. Quality is a separate check.

Write a short rubric with a few scored dimensions, run the same small prompt set on a schedule, and record the scores over time:

  • Did it follow the instruction and the output format?
  • Is it factually consistent with the input you provided?
  • Is it complete for the task?

Score it by hand or with a judge model, and treat the number as a trend, not a pass or fail. A quality rubric tells you when a model or a provider changed underneath you. An assertion tells you nothing and flakes.

Putting it together. Three layers, three jobs. Unit tests run on every commit and cost nothing. One gated smoke test proves the live contract before you deploy. A quality rubric watches the thing tests cannot measure. When you add a second provider, you run the same three layers against both base URLs and the switch becomes a config change you can trust.

The WebWright Team