Error rate tracking is the practice of continuously monitoring the frequency and types of errors occurring in an AI system, enabling rapid detection of problems, SLO compliance verification, and trend analysis for system reliability.

What to Track

• Overall Error Rate: Total errors / total requests as a percentage. The headline metric for system health.
• Error Rate by Type: Break down by error category — timeout errors, rate limit errors, model errors, safety filter rejections, input validation failures.
• Error Rate by Endpoint/Model: Track separately for each API endpoint, model version, or deployment.
• Error Rate by User Segment: Different user tiers, geographic regions, or client versions may experience different error rates.

Common Error Types in AI Systems

• HTTP 429 (Rate Limited): Too many requests. Track to tune rate limits and plan capacity.
• HTTP 500/503 (Server Error): Internal failures or service unavailability. The most critical errors.
• Timeout Errors: Requests exceeding time limits — may indicate capacity issues or unusually complex queries.
• Model Refusals: The model refuses to respond due to safety filters — may indicate adversarial probing or overly aggressive filters.
• Format Errors: Model output doesn't match expected format (invalid JSON, missing fields).
• Context Length Exceeded: Input exceeds the model's context window.

Error Budget and SLOs

• SLO (Service Level Objective): Target reliability — e.g., "99.9% of requests succeed" (error rate < 0.1%).
• Error Budget: The allowed amount of unreliability — with a 99.9% SLO, you have a 0.1% error budget per period.
• Budget Consumption: Track how much error budget has been consumed. When the budget is depleted, freeze deployments and focus on reliability.

Alerting Strategy

• Error Rate Spike: Alert when error rate exceeds baseline by a significant margin (e.g., >2× normal rate for 5 minutes).
• Error Budget Burn Rate: Alert when the error budget is being consumed faster than expected (will be exhausted before the period ends).
• New Error Types: Alert when previously unseen error types appear.

Tools: Prometheus (with error rate recording rules), Datadog (error tracking and APM), Sentry (error aggregation and tracking), PagerDuty (alert routing and escalation).

Error rate tracking is the primary health indicator for production systems — a sudden spike in errors is usually the first sign that something has gone wrong.