Observability for LLM Applications

The Three Pillars of Observability

1. Logs Discrete events recorded over time.

• Request/response logs (with prompt/completion)
• Error logs and stack traces
• System events (model loads, scaling)

2. Metrics Aggregated numerical measurements.

• Latency percentiles (P50, P95, P99)
• Throughput (requests/sec, tokens/sec)
• Error rates
• Cost metrics (tokens consumed, $ spent)

3. Traces Request flow through distributed systems.

• End-to-end request tracing
• Time spent in each component
• Parent-child relationship of spans

LLM-Specific Observability

Key Metrics to Track

LLM Observability Tools

Logging Best Practices

What to Log

log_entry = {
    "request_id": "uuid-123",
    "timestamp": "2024-01-15T10:30:00Z",
    "model": "gpt-4",
    "prompt_tokens": 150,
    "completion_tokens": 200,
    "latency_ms": 1200,
    "user_id": "user-456",  # Can be anonymized
    "prompt_hash": "abc123",  # For PII protection
    "status": "success"
}

PII Considerations

• Hash or redact sensitive data
• Anonymize user identifiers
• Implement data retention policies
• Comply with GDPR/CCPA if applicable

Alerting Strategy