Alert configuration is the practice of setting up automated notifications that trigger when system metrics exceed defined thresholds, enabling teams to detect and respond to problems before they significantly impact users.

Alert Components

• Metric: What measurement to monitor (error rate, latency p99, GPU utilization, queue depth).
• Condition: The threshold or pattern that triggers the alert (e.g., "error rate > 1% for 5 minutes").
• Severity: The urgency level — critical (page on-call engineer immediately), warning (notify in Slack), info (log for review).
• Notification Channel: Where to send the alert — PagerDuty, Slack, email, SMS, webhook.
• Runbook Link: URL to documentation explaining how to investigate and resolve the issue.

Best Practices

• Alert on Symptoms, Not Causes: Alert on "error rate > 1%" (symptom) rather than "CPU > 80%" (cause). High CPU without user impact shouldn't wake anyone up.
• Avoid Alert Fatigue: Too many alerts leads to ignoring all alerts. Only page for conditions requiring immediate human action.
• Multi-Window Alerts: Use both short (5 min) and long (1 hour) windows — short for sudden spikes, long for gradual degradation.
• Severity Levels: Not everything is critical. Use at least 3 severity levels: critical (page immediately), warning (Slack notification during business hours), info (dashboard only).
• SLO-Based Alerts: Alert when the SLO error budget burn rate exceeds sustainable levels, rather than on absolute thresholds.

AI-Specific Alerts

• Inference Latency: p95 TTFT > SLO target for 5 minutes.
• Error Rate: Request error rate > SLO error budget burn rate.
• GPU Issues: GPU memory > 95%, GPU temperature > thermal limit, GPU errors detected.
• Model Quality: Quality score drops below baseline (requires online evaluation).
• Safety: Unusual spike in safety filter activations or content policy violations.
• Cost: Daily API spend exceeds budget threshold.

Alert Routing

• Escalation: If the primary on-call doesn't acknowledge within 15 minutes, escalate to secondary.
• Time-Based Routing: Route non-critical alerts differently during business hours vs. nights/weekends.
• Grouping: Group related alerts to avoid flooding (10 servers failing simultaneously = 1 alert, not 10).

Well-configured alerts are the safety net for production systems — they ensure problems are detected and addressed before users are significantly impacted.