metrics collection

**Metrics collection** is the practice of systematically gathering **numerical measurements** about system health, performance, and behavior at regular intervals. In AI/ML systems, metrics provide the quantitative foundation for monitoring, alerting, capacity planning, and optimization. **Types of Metrics** - **Counter**: A monotonically increasing value — total requests served, total tokens generated, total errors. Can only go up (or reset to zero). - **Gauge**: A value that can go up or down — current GPU utilization, active connections, memory usage, queue depth. - **Histogram**: Distribution of values — request latency distribution, token count distribution. Enables percentile calculations (p50, p95, p99). - **Summary**: Pre-computed percentiles over a sliding time window — similar to histograms but computed on the client side. **Key Metrics for AI Systems** - **Inference Latency**: Time to first token (TTFT), time per output token (TPOT), and total generation time. - **Throughput**: Requests per second, tokens per second. - **GPU Utilization**: Percentage of GPU compute capacity in use. - **GPU Memory**: VRAM usage, KV cache size, available memory. - **Error Rates**: By error type (timeout, rate limit, model error, safety filter). - **Queue Depth**: Number of pending requests waiting for inference. - **Token Usage**: Input/output tokens per request for cost tracking. - **Model Quality**: Online quality scores, user ratings, task completion rates. **Collection Architecture** - **Push Model**: Application pushes metrics to a central collector (StatsD, Datadog Agent). Lower latency, application controls send timing. - **Pull Model**: Collector scrapes metrics from application endpoints (Prometheus). Simpler application code, collector controls timing. - **Hybrid**: OpenTelemetry supports both push and pull, with protocol translation. **Tools** - **Prometheus**: Pull-based, time-series database with powerful query language (PromQL). Industry standard for Kubernetes. - **Datadog**: SaaS metrics platform with AI-specific integrations. - **CloudWatch / Cloud Monitoring**: Cloud-native metrics from AWS/GCP. - **OpenTelemetry**: Vendor-neutral metrics collection SDK and protocol. Metrics collection is the **quantitative backbone** of observability — without metrics, you're operating blind on system health and performance.