on-call rotation

**On-call rotation** is a scheduled system where team members take turns being the **primary responder** to production issues, alerts, and incidents outside of normal working hours. It ensures that expert attention is always available when AI systems encounter problems. **How On-Call Rotation Works** - **Rotation Schedule**: Team members cycle through on-call duty — typically weekly rotations. The schedule ensures fair distribution and adequate rest. - **Primary and Secondary**: A primary on-call engineer handles alerts first. If they're unavailable or the issue escalates, a secondary on-call takes over. - **Alerting Chain**: Production alerts route to the on-call engineer's phone, with escalation if not acknowledged within a defined window. **On-Call Responsibilities** - **Alert Response**: Acknowledge and investigate triggered alerts within the defined SLA (typically 5–15 minutes for critical alerts). - **Incident Management**: Triage, diagnose, and mitigate production issues. Apply immediate fixes or rollbacks as needed. - **Escalation**: Engage additional team members or specialists when the issue exceeds current expertise. - **Communication**: Update stakeholders on incident status via status pages, Slack channels, or incident management tools. - **Handoff**: Brief the next on-call engineer on ongoing issues during rotation changes. **On-Call for AI Systems — Special Considerations** - **Model-Specific Knowledge**: On-call engineers need to understand model behavior, common failure modes, and rollback procedures for ML systems. - **Provider Outages**: LLM API providers (OpenAI, Anthropic) may experience outages — on-call needs to know how to switch to fallback providers. - **Safety Incidents**: Content safety issues may require immediate intervention — updating filters, blocking specific queries, or temporarily restricting functionality. - **Cost Alerts**: Unexpected API spending spikes may require throttling or disabling certain features. **Tools** - **PagerDuty**: Industry-standard incident management and on-call scheduling. - **OpsGenie**: Atlassian's on-call and alert management platform. - **Incident.io**: Modern incident management with Slack integration. - **Rootly**: AI-assisted incident management. **Best Practices** - **Runbooks**: Document investigation and resolution steps for common alerts. - **Compensation**: Provide on-call compensation or time off in lieu. - **SLAs**: Define response time expectations clearly. - **Post-Incident Review**: After every incident, conduct a blameless review to improve processes. A healthy on-call rotation is the **backbone of production reliability** — it ensures that when things go wrong at 3 AM, a competent, rested engineer is ready to respond.