On-call rotation is a scheduled system where team members take turns being the primary responder to production issues, alerts, and incidents outside of normal working hours. It ensures that expert attention is always available when AI systems encounter problems.

How On-Call Rotation Works

• Rotation Schedule: Team members cycle through on-call duty — typically weekly rotations. The schedule ensures fair distribution and adequate rest.
• Primary and Secondary: A primary on-call engineer handles alerts first. If they're unavailable or the issue escalates, a secondary on-call takes over.
• Alerting Chain: Production alerts route to the on-call engineer's phone, with escalation if not acknowledged within a defined window.

On-Call Responsibilities

• Alert Response: Acknowledge and investigate triggered alerts within the defined SLA (typically 5–15 minutes for critical alerts).
• Incident Management: Triage, diagnose, and mitigate production issues. Apply immediate fixes or rollbacks as needed.
• Escalation: Engage additional team members or specialists when the issue exceeds current expertise.
• Communication: Update stakeholders on incident status via status pages, Slack channels, or incident management tools.
• Handoff: Brief the next on-call engineer on ongoing issues during rotation changes.

On-Call for AI Systems — Special Considerations

• Model-Specific Knowledge: On-call engineers need to understand model behavior, common failure modes, and rollback procedures for ML systems.
• Provider Outages: LLM API providers (OpenAI, Anthropic) may experience outages — on-call needs to know how to switch to fallback providers.
• Safety Incidents: Content safety issues may require immediate intervention — updating filters, blocking specific queries, or temporarily restricting functionality.
• Cost Alerts: Unexpected API spending spikes may require throttling or disabling certain features.

Tools

• PagerDuty: Industry-standard incident management and on-call scheduling.
• OpsGenie: Atlassian's on-call and alert management platform.
• Incident.io: Modern incident management with Slack integration.
• Rootly: AI-assisted incident management.

Best Practices

• Runbooks: Document investigation and resolution steps for common alerts.
• Compensation: Provide on-call compensation or time off in lieu.
• SLAs: Define response time expectations clearly.
• Post-Incident Review: After every incident, conduct a blameless review to improve processes.

A healthy on-call rotation is the backbone of production reliability — it ensures that when things go wrong at 3 AM, a competent, rested engineer is ready to respond.