A repeatable 30-minute evaluation plan that works against any OpenAI-compatible endpoint.

You already have a working app. You are considering a second OpenAI-compatible API, and you want the smallest honest test before you move any traffic. This is that test, in the order that lets you stop earliest.

1. The three tests, and why the order matters

A second provider is a risk-reduction move. You are not replacing anything on day one. You are buying an option: the ability to route some traffic elsewhere if cost, quality, or operations demand it. So test the thing that would actually make you stop first.

The three tests run in this order:

  1. Drop-in compatibility (5 minutes). If your client cannot talk to the endpoint, nothing else matters.
  2. Quality on your workload (15 minutes). A cheaper or faster endpoint is useless if the output is worse on your prompts.
  3. Cost and operations (10 minutes). You can only size cost once the endpoint works and the quality holds.

Each test is cheap, and each one can end the evaluation early. That is the point of the order.

2. Test 1, drop-in compatibility (5 minutes)

Point an existing OpenAI-compatible client at https://webwright.ai/v1. Change the base URL, the API key, and the model name. Change nothing else. Then run one call.

The WebWright API reference documents an OpenAI-compatible surface: GET /v1/models, POST /v1/chat/completions, and POST /v1/audio/transcriptions. The published model is wren-r1. The reference and the https://webwright.ai/v1/models response were verified live on 2026-10-06.

Start with the model list, then a single chat completion:

curl https://webwright.ai/v1/models \
  -H "Authorization: Bearer $WEBWRIGHT_API_KEY"
curl https://webwright.ai/v1/chat/completions \
  -H "Authorization: Bearer $WEBWRIGHT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"wren-r1","messages":[{"role":"user","content":"Say hello."}]}'

What you are checking is boring on purpose: does your existing client accept the base URL, and does it parse the response without a special case? If your code already speaks the OpenAI API, the base URL, the key, and the model are the whole change.

If you want the full wiring, including feature flags and error mapping, the OpenAI-compatible migration playbook covers it.

3. Test 2, quality on your workload (15 minutes)

Pick three prompts from your real workload: one easy, one hard, one long. Run them against your current provider and against the new endpoint with the same settings. Score the outputs yourself against your current output. A three-line rubric is enough, and scoring blind is better if you can manage it.

A public leaderboard cannot answer this for your product. A leaderboard measures a generic prompt set. Your product has its own domain, its own formatting rules, and its own failure modes. Use a leaderboard to shortlist candidates. Do not use it to decide.

This is also the test that applies to us. Do not trust a vendor's published benchmark for your workload, including ours. The goal of this step is to replace a claim with your own measurement.

4. Test 3, cost and operations (10 minutes)

Put your own token or audio volumes into the cost calculator. Use your observed volumes, not an average. The calculator works from the published pricing rates, so the inputs and the source stay in one place. This page does not restate a computed figure.

Then check the operational basics you will depend on:

  • Auth errors. Send a request with a bad key and confirm the error body matches what your error handling expects. The API reference documents the error shape.
  • Streaming. Confirm the chunk shape and the end-of-stream sentinel against your parser before you ship. The reference documents buffered JSON and streaming SSE on POST /v1/chat/completions.
  • Response formats. If you transcribe audio, confirm the formats you would actually parse. POST /v1/audio/transcriptions accepts multipart/form-data and returns json (default), text, srt, vtt, or verbose_json.
  • Latency. Measure it yourself, on your own network, with your own prompts. Record p50 and p95 and label them as your measurements. The migration playbook describes a small harness that records quality, latency, and cost side by side.

5. What to do with the result

A short decision rule:

  • Pass all three. Run a bounded slice of real traffic through the new endpoint behind a flag, and keep the current provider as the fallback.
  • Fail quality. Stop. Cost and compatibility do not matter if the output is worse.
  • Cost is the only concern. Revisit your volumes and your cache hit rate before you change providers. Caching reduces repeated input cost, so a lower cache miss rate can matter more than a lower headline rate.

6. If you would rather not do this alone

If you would rather not pick the first use case or the first test alone, that is what the consult is for. One working session maps where you are, what you have tried, and what good would look like.

Product facts on this page restate the published API reference, models list, and pricing pages, each verified live on 2026-10-06. No WebWright quality, speed, latency, uptime, SLA, or benchmark score is published here.

The WebWright Team