rate limiting

**Rate limiting** is a security and resource management technique that restricts the **number of requests** a user, IP address, or API key can make to an AI service within a given time window. It is essential for preventing abuse, managing costs, and maintaining service availability. **Why Rate Limiting Matters for AI** - **Prevent Model Extraction**: Attackers query the model thousands of times to build a surrogate copy. Rate limits make this impractically slow. - **Cost Control**: LLM inference is expensive — unrestricted access can lead to **massive, unexpected bills** from API abuse or automated scripts. - **Denial of Service**: Without limits, a single user can monopolize GPU resources, degrading service for everyone. - **Adversarial Probing**: Rate limits slow down automated jailbreaking attempts, red-teaming scripts, and prompt injection exploration. **Common Rate Limiting Strategies** - **Fixed Window**: Allow N requests per time window (e.g., 100 requests per minute). Simple but allows bursts at window boundaries. - **Sliding Window**: Smooth the window to prevent boundary bursts. More complex but fairer. - **Token Bucket**: Tokens accumulate over time; each request costs a token. Allows short bursts while enforcing average rate. - **Token-Based Limits**: For LLMs, limit by **tokens per minute (TPM)** rather than requests, since a long prompt consumes far more resources than a short one. - **Tiered Limits**: Different limits for different plan levels (free tier: 10 RPM, paid: 1000 RPM, enterprise: custom). **Implementation Considerations** - **Identification**: Rate limit by API key, user account, IP address, or some combination. - **Response Codes**: Return **HTTP 429 (Too Many Requests)** with a `Retry-After` header indicating when the client can try again. - **Graceful Degradation**: Consider returning cached or lower-quality responses instead of hard rejections. - **Monitoring**: Track rate limit hits to identify abuse patterns and adjust limits. Rate limiting is a **standard practice** across all LLM API providers (OpenAI, Anthropic, Google) and is critical for both **security and business sustainability**.