Rate limits

Dimensions

Limits live on the key (limits in POST /v1/keys), so two apps in one org can have different ceilings:

A key without explicit limits inherits the org's plan defaults.

The 429

HTTP/1.1 429 Too Many Requests
Retry-After: 2
x-aiml-request-id: 01K4N4V8Z8W3Q0G8S4YV6D5V2M
Content-Type: application/json

{"error":{"type":"rate_limited","code":"tpm","message":"tokens per minute exceeded","retry_after_ms":1800,"request_id":"01K4…"}}

Retry-After is in seconds, rounded up; retry_after_ms in the body is exact. The OpenAI and Anthropic SDKs retry 429s with backoff by default and honour Retry-After.

Upstream capacity

429 upstream_capacity (or 503 upstream_capacity when every candidate was out) means the providers behind the model were saturated, not your key. The gateway has already tried the alternatives it had; the circuit breaker keeps a saturated endpoint out of rotation for a while, so a retry after Retry-After usually lands elsewhere.

Reducing pressure

• Stream. A stream holds one concurrency slot for its duration but returns tokens as they arrive, so clients time out less and retry less.
• Set max_tokens honestly. The tpm estimate and the credit reserve both use it.
• Prefer several keys with small limits over one key with a large one; a runaway client then only starves itself.