Planning for rate limits on an OpenAI-compatible API: 429, Retry-After, and concurrency

Your first request works. Then you put the endpoint in front of real traffic, and the failures change shape. The interesting one is not a timeout any more, it is a 429. A rate limit is not an error you fix in the request. It is a contract you plan around.

1. Know the published limit you are under

Rate limits are enforced per API key and are based on the plan tier. The published table on the API page is the contract to design against:

Two numbers matter, not one. Requests per minute is the sustained ceiling. The concurrent burst is how many calls may be in flight at once. A pool that stays under the per-minute number can still trip the burst if it fans out too wide at the same instant.

2. Read 429 as an instruction, not a failure

When you exceed the limit the API returns 429 with error type rate_limit_exceeded and a Retry-After header. That header tells you the seconds to wait before retrying. Honor it. Retrying sooner is what turns one throttled call into a retry storm.

import time
import httpx

def post_with_retry_after(client, url, payload, attempts=5):
    for i in range(attempts):
        resp = client.post(url, json=payload)
        if resp.status_code != 429:
            resp.raise_for_status()
            return resp
        wait = float(resp.headers.get("Retry-After", 1))
        time.sleep(min(wait, 30.0))
    raise RuntimeError("rate limited after retries")

The same wrapper should treat 500 api_error differently: that one is transient and belongs with bounded exponential backoff and jitter, not with Retry-After. And 401 or 402 should not be retried at all; they are configuration or quota problems.

3. Size the pool to the burst, not to the work

If your service can produce more concurrent calls than the plan's burst allows, a semaphore is cheaper than a queue of 429s. Cap the in-flight calls, and let the rest wait for a slot.

import asyncio

class Pool:
    def __init__(self, max_concurrent: int):
        self.sem = asyncio.Semaphore(max_concurrent)

    async def call(self, fn, *args):
        async with self.sem:
            return await fn(*args)

pool = Pool(max_concurrent=2)  # matches the Hobby burst

Pick the number from your plan, not from the number of users who might arrive at once. When you outgrow it, that is a plan conversation, not a client-side fix.

4. Measure the limit before you rely on it

A throttle you have never seen is a guess. Run a small load test against your own key, ramp the concurrency, and record where the first 429 appears and what Retry-After says. That measured point, not the table alone, is what your pool size should be built from. The same harness the migration playbook describes can drive it.

5. When to book a consult

If you would rather not work out the retry rules, the pool size, and the plan fit alone, that is what the consult is for. One working session maps your current client, your traffic shape, and your limits, and you leave with a prioritized first change, a concurrency plan, a rough cost, and the honest risks.

The WebWright Team