alerting
**Alerting and Incident Response** is the **practice of defining threshold-based or anomaly-based rules that automatically notify on-call engineers when AI systems breach acceptable operating boundaries** — bridging the gap between observability data and human action to minimize mean time to detection (MTTD) and mean time to resolution (MTTR) for production AI service failures.
**What Is Alerting in AI Systems?**
- **Definition**: Automated rules that evaluate metrics, logs, or traces against defined thresholds and trigger notifications (pages, Slack messages, emails) when conditions indicate a service degradation or failure requiring human intervention.
- **On-Call Culture**: Production AI services run 24/7 — alerting systems route incidents to the appropriate engineer based on scheduled rotations, ensuring someone is always responsible for critical failures even at 3 AM.
- **Alert Quality**: The goal is not maximum alerts but actionable alerts — every alert should represent a condition requiring immediate human decision-making, not background noise.
- **Alert Fatigue**: A critical failure mode where too many low-priority alerts train engineers to ignore notifications — the most dangerous state is an on-call engineer who assumes alerts are noise, missing a genuine critical incident.
**Why Alerting Matters for AI Infrastructure**
- **LLM API Outages**: When OpenAI or Anthropic APIs go down, downstream applications fail silently without proper alerting — users see generic errors while engineers are unaware.
- **GPU Memory Leaks**: Memory leak in serving code causes VRAM to fill gradually over hours — alerting catches it before OOM kills the inference server.
- **Inference Degradation**: A bad model deployment causes p99 latency to spike from 2s to 30s — alerting triggers within minutes, enabling rapid rollback before most users are affected.
- **Cost Explosions**: A prompt injection attack or buggy client sends millions of long requests — cost alerting catches billing anomalies before they become multi-thousand-dollar surprises.
- **Data Pipeline Failures**: Embedding pipeline fails to process new documents — alert fires when vector DB staleness exceeds acceptable threshold.
**The Alerting Stack**
**Prometheus AlertManager**:
- Evaluates PromQL rules against Prometheus metrics continuously.
- Deduplicates, groups, and routes alerts to appropriate channels.
- Handles silences (planned maintenance windows) and inhibitions.
Example rule:
groups:
- name: inference
rules:
- alert: HighInferenceLatency
expr: histogram_quantile(0.99, rate(request_duration_seconds_bucket[5m])) > 5
for: 2m
labels:
severity: critical
annotations:
summary: "p99 latency exceeds 5 seconds"
**PagerDuty**:
- On-call schedule management — routes alerts to correct engineer based on time of day and rotation.
- Escalation policies — if primary on-call doesn't acknowledge within 5 minutes, escalate to secondary.
- Mobile app with phone calls + push notifications — guaranteed wake-up for critical incidents.
**OpsGenie**: PagerDuty alternative with similar on-call management, popular with Atlassian (Jira/Confluence) shops.
**Grafana Alerting**: Evaluate Prometheus/Loki queries within Grafana and route to Slack/PagerDuty — consolidates alerting rules with dashboards.
**Alert Design Principles**
**Symptom-Based (Correct)**:
- "Users cannot complete requests" (high error rate).
- "Response latency exceeds SLO" (p99 > 5s).
- "Service is down" (no successful health checks).
**Cause-Based (Incorrect)**:
- "CPU is 90%" (may be fine — batch processing).
- "Memory is 80%" (may be normal — caching).
- "Disk is filling" (unless near 100%, not urgent).
Alert on symptoms that directly impact users. Cause-based alerts produce noise without actionable urgency.
**Severity Levels for AI Systems**
| Severity | Condition | Response | SLA |
|----------|-----------|----------|-----|
| Critical/P1 | Service down, 0% success rate | Wake on-call immediately | 15 min response |
| High/P2 | Error rate > 5%, p99 > SLO | Alert on-call within 5 min | 30 min response |
| Medium/P3 | Degraded performance, cost spike | Slack notification, next business day | 4 hours |
| Low/P4 | Approaching limits, minor anomalies | Email, weekly review | Best effort |
**AI-Specific Alert Rules**
- GPU memory > 90% for 5 minutes → High.
- Inference error rate > 1% for 2 minutes → Critical.
- TTFT p95 > 10s for 5 minutes → High.
- Cost per hour > 2x 7-day average → Medium.
- Vector DB staleness > 24 hours → Medium.
- Model serving pod restart count > 3/hour → High.
- Token generation rate drops > 50% from baseline → High.
Alerting is **the human-machine interface for production AI reliability** — when designed with care around actionable symptoms rather than cause-based noise, alerting systems transform raw observability data into rapid incident response, protecting user experience and enabling AI teams to sleep soundly knowing critical failures will be caught within minutes.