alert configuration

**Alert configuration** is the practice of setting up **automated notifications** that trigger when system metrics exceed defined thresholds, enabling teams to detect and respond to problems before they significantly impact users. **Alert Components** - **Metric**: What measurement to monitor (error rate, latency p99, GPU utilization, queue depth). - **Condition**: The threshold or pattern that triggers the alert (e.g., "error rate > 1% for 5 minutes"). - **Severity**: The urgency level — critical (page on-call engineer immediately), warning (notify in Slack), info (log for review). - **Notification Channel**: Where to send the alert — PagerDuty, Slack, email, SMS, webhook. - **Runbook Link**: URL to documentation explaining how to investigate and resolve the issue. **Best Practices** - **Alert on Symptoms, Not Causes**: Alert on "error rate > 1%" (symptom) rather than "CPU > 80%" (cause). High CPU without user impact shouldn't wake anyone up. - **Avoid Alert Fatigue**: Too many alerts leads to ignoring all alerts. Only page for conditions requiring **immediate human action**. - **Multi-Window Alerts**: Use both short (5 min) and long (1 hour) windows — short for sudden spikes, long for gradual degradation. - **Severity Levels**: Not everything is critical. Use at least 3 severity levels: **critical** (page immediately), **warning** (Slack notification during business hours), **info** (dashboard only). - **SLO-Based Alerts**: Alert when the SLO error budget **burn rate** exceeds sustainable levels, rather than on absolute thresholds. **AI-Specific Alerts** - **Inference Latency**: p95 TTFT > SLO target for 5 minutes. - **Error Rate**: Request error rate > SLO error budget burn rate. - **GPU Issues**: GPU memory > 95%, GPU temperature > thermal limit, GPU errors detected. - **Model Quality**: Quality score drops below baseline (requires online evaluation). - **Safety**: Unusual spike in safety filter activations or content policy violations. - **Cost**: Daily API spend exceeds budget threshold. **Alert Routing** - **Escalation**: If the primary on-call doesn't acknowledge within 15 minutes, escalate to secondary. - **Time-Based Routing**: Route non-critical alerts differently during business hours vs. nights/weekends. - **Grouping**: Group related alerts to avoid flooding (10 servers failing simultaneously = 1 alert, not 10). Well-configured alerts are the **safety net** for production systems — they ensure problems are detected and addressed before users are significantly impacted.