error rate tracking

**Error rate tracking** is the practice of continuously monitoring the **frequency and types of errors** occurring in an AI system, enabling rapid detection of problems, SLO compliance verification, and trend analysis for system reliability. **What to Track** - **Overall Error Rate**: Total errors / total requests as a percentage. The headline metric for system health. - **Error Rate by Type**: Break down by error category — timeout errors, rate limit errors, model errors, safety filter rejections, input validation failures. - **Error Rate by Endpoint/Model**: Track separately for each API endpoint, model version, or deployment. - **Error Rate by User Segment**: Different user tiers, geographic regions, or client versions may experience different error rates. **Common Error Types in AI Systems** - **HTTP 429 (Rate Limited)**: Too many requests. Track to tune rate limits and plan capacity. - **HTTP 500/503 (Server Error)**: Internal failures or service unavailability. The most critical errors. - **Timeout Errors**: Requests exceeding time limits — may indicate capacity issues or unusually complex queries. - **Model Refusals**: The model refuses to respond due to safety filters — may indicate adversarial probing or overly aggressive filters. - **Format Errors**: Model output doesn't match expected format (invalid JSON, missing fields). - **Context Length Exceeded**: Input exceeds the model's context window. **Error Budget and SLOs** - **SLO (Service Level Objective)**: Target reliability — e.g., "99.9% of requests succeed" (error rate < 0.1%). - **Error Budget**: The allowed amount of unreliability — with a 99.9% SLO, you have a 0.1% error budget per period. - **Budget Consumption**: Track how much error budget has been consumed. When the budget is depleted, freeze deployments and focus on reliability. **Alerting Strategy** - **Error Rate Spike**: Alert when error rate exceeds baseline by a significant margin (e.g., >2× normal rate for 5 minutes). - **Error Budget Burn Rate**: Alert when the error budget is being consumed faster than expected (will be exhausted before the period ends). - **New Error Types**: Alert when previously unseen error types appear. **Tools**: **Prometheus** (with error rate recording rules), **Datadog** (error tracking and APM), **Sentry** (error aggregation and tracking), **PagerDuty** (alert routing and escalation). Error rate tracking is the **primary health indicator** for production systems — a sudden spike in errors is usually the first sign that something has gone wrong.