retry logic
**Retry Logic with Exponential Backoff** is the **resilience pattern that automatically re-attempts failed API requests with progressively increasing wait times** — the fundamental strategy for handling transient failures in AI API integrations where rate limits (429), server errors (500-503), and network timeouts are common and expected failure modes requiring graceful recovery rather than immediate hard failure.
**What Is Retry Logic with Exponential Backoff?**
- **Definition**: A retry strategy where failed requests are automatically re-attempted after a waiting period that doubles with each successive failure — starting short (1 second) and growing exponentially (2s, 4s, 8s, 16s) to reduce load on the recovering service while giving it time to stabilize.
- **Problem Solved**: AI APIs (OpenAI, Anthropic, Google) regularly return transient errors — rate limit exceeded, server overloaded, network timeout — that resolve themselves within seconds. Without retry logic, these transient failures cause application-visible errors that could have been silently recovered.
- **Jitter**: Random noise added to backoff wait times — prevents the "Thundering Herd" problem where all clients that failed simultaneously retry at exactly the same moment, creating a retry spike that overwhelms the recovering server again.
- **Max Retries**: Retry logic must have a ceiling — infinite retries create applications that hang indefinitely on non-transient failures. Typical: 3-5 retries with exponential backoff.
**Why Retry Logic Matters for AI APIs**
- **Rate Limits Are Expected**: OpenAI, Anthropic, and Google enforce per-minute and per-day token and request rate limits. Applications approaching limits regularly receive 429 responses — retry with backoff is the designed response.
- **Server Load Variability**: AI inference is computationally expensive — API providers experience load spikes where 503 responses signal temporary capacity constraints that resolve in seconds.
- **Network Reliability**: Long-running LLM inference requests (10-60 seconds for large generations) are vulnerable to network timeouts, connection resets, and proxy failures.
- **Production SLA Requirements**: User-facing AI applications cannot display API error messages to end users — transparent retry logic maintains application availability during transient failures.
- **Cost Efficiency**: Retrying transient failures is dramatically cheaper than adding error handling paths, fallback systems, or manual re-submission workflows.
**Exponential Backoff Algorithm**
Core algorithm:
```
wait_time = base_delay × (2 ^ retry_count) + random_jitter
Retry 1: 1 × 2^0 + jitter = 1.0 ± 0.5 seconds
Retry 2: 1 × 2^1 + jitter = 2.0 ± 0.5 seconds
Retry 3: 1 × 2^2 + jitter = 4.0 ± 0.5 seconds
Retry 4: 1 × 2^3 + jitter = 8.0 ± 0.5 seconds
Retry 5: 1 × 2^4 + jitter = 16.0 ± 0.5 seconds (then give up)
```
**Which Errors to Retry**
| HTTP Status | Error Type | Retry? | Reason |
|-------------|-----------|--------|--------|
| 429 | Rate limit exceeded | Yes | Wait and retry |
| 500 | Internal server error | Yes (limited) | May be transient |
| 502 | Bad gateway | Yes | Infrastructure issue |
| 503 | Service unavailable | Yes | Server overloaded |
| 504 | Gateway timeout | Yes | Timeout — retry may succeed |
| 400 | Bad request | No | Request is malformed — retry won't help |
| 401 | Unauthorized | No | Wrong API key — retry won't help |
| 403 | Forbidden | No | Permission issue — retry won't help |
| 404 | Not found | No | Wrong endpoint — retry won't help |
**Implementation Examples**
**Python with tenacity library (Recommended)**:
```python
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
import openai
@retry(
stop=stop_after_attempt(5),
wait=wait_exponential(multiplier=1, min=1, max=60),
retry=retry_if_exception_type((openai.RateLimitError, openai.APIStatusError))
)
def call_llm(prompt: str) -> str:
return client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}]
).choices[0].message.content
```
**Manual Implementation with Jitter**:
```python
import time, random
def call_with_backoff(prompt: str, max_retries: int = 5) -> str:
for attempt in range(max_retries):
try:
return llm.generate(prompt)
except (RateLimitError, ServerError) as e:
if attempt == max_retries - 1:
raise # Last attempt — propagate error
wait = (2 ** attempt) + random.uniform(0, 1) # Exponential + jitter
time.sleep(wait)
```
**Rate Limit Header Handling (Advanced)**:
OpenAI returns headers indicating when the rate limit resets:
```python
except RateLimitError as e:
reset_time = e.response.headers.get("x-ratelimit-reset-requests")
if reset_time:
wait = max(float(reset_time), 1.0) # Wait until reset, not just backoff
time.sleep(wait)
```
**Production Considerations**
- **Circuit Breaker**: After N consecutive failures, stop retrying for a cooldown period — prevents cascading failures where retries amplify overload.
- **Async Retry**: For high-throughput applications, use async retry to avoid blocking threads during backoff waits.
- **User Feedback**: For user-facing applications with long retry queues, provide progress indication — "Processing your request..." — rather than silent delays.
- **Monitoring**: Track retry rates, backoff durations, and ultimate failure rates — high retry rates indicate systematic issues requiring architectural response.
- **Budget Accounting**: Retries multiply API costs — ensure retry behavior is accounted for in cost modeling.
Retry logic with exponential backoff is **the foundational resilience pattern that separates brittle AI prototypes from production-grade AI applications** — by automatically recovering from the transient failures that are inevitable when calling AI APIs at scale, retry logic with jitter transforms occasional API hiccups from user-visible errors into seamless, transparent recovery that maintains application reliability and user trust.