Errors and retries

Diagnose failed requests and retry only when the cause can be temporary.

Read the status and response

For inference requests, inspect the HTTP status and the error object when present. The response can include message, type, code, or param. Upstream and proxy errors can differ, including a non-JSON body. Keep the HTTP status, request time, model ID, and request ID when available. Never include your API key in a support report.

Illustrative account-credit error

{
  "error": {
    "message": "Insufficient account quota",
    "code": "insufficient_user_quota"
  }
}

Common HTTP statuses

Status What to check What to do
400 Request JSON, messages, parameters, context size Correct the payload before retrying.
401 Authorization header and API key Check the complete key, status, and expiry.
403 Account credits, key quota, IP or model access Read the error message and fix the applicable restriction.
404 URL or endpoint Use the documented URL; check the exact model ID separately.
429 Request or provider rate limit Reduce concurrency and back off.
500, 502, 503, 504 Service or upstream failure Consider a bounded retry; check whether processing already occurred.

A model that is unavailable can produce a routing or upstream error rather than 404. Use the message alongside the status.

Use bounded backoff

Retry transient failures with a small attempt limit, exponential backoff, and jitter. Honor Retry-After when provided. Do not assume rate-limit headers exist on every response.

The official OpenAI SDK already retries some connection, timeout, rate-limit, and server failures. Avoid nesting multiple retry loops. The example below disables SDK retries and retries only the listed HTTP statuses. It leaves connection errors and timeouts to the caller because their processing outcome can be uncertain.

import os
import random
import time
from openai import OpenAI, APIStatusError

client = OpenAI(
    api_key=os.environ["BULKTOKEN_API_KEY"],
    base_url="https://bulktoken.ai/v1", max_retries=0, timeout=120.0,
)
for attempt in range(3):
    try:
        result = client.chat.completions.create(
            model="glm-5.3",
            messages=[{"role": "user", "content": "Say hello."}],
        )
        print(result.choices[0].message.content)
        break
    except APIStatusError as error:
        if error.status_code not in {429, 500, 502, 503, 504} or attempt == 2:
            raise
        retry_after = error.response.headers.get("retry-after", "")
        if retry_after.isdigit():
            delay = float(retry_after)
        else:
            delay = 2 ** attempt + random.uniform(0, 1)
        time.sleep(delay)

A new POST can create a new generation and a new charge. Do not assume an idempotency key or automatic deduplication is supported.

Rate limits and concurrency

Limits can be applied at the account, route, or upstream provider. This documentation does not promise a single fixed RPM or TPM for every model. Use a bounded queue, reuse connections, and spread bursts over time.

For repeated 429 responses, reduce concurrency and include request timing and model details when contacting support. Do not rotate keys to bypass account-level limits.