analytics
**AI analytics and usage metrics** involve **tracking and analyzing how AI features are used within products** — measuring query patterns, performance characteristics, user engagement, and quality indicators to optimize AI capabilities, control costs, and demonstrate value to stakeholders.
**Why AI Analytics Matter**
- **Optimization**: Identify slow or expensive queries.
- **Quality**: Detect degradation in responses.
- **Cost Control**: Understand and optimize spend.
- **ROI**: Demonstrate AI feature value.
- **Planning**: Capacity and scaling decisions.
**Key Metrics Categories**
**Usage Metrics**:
```
Metric | What It Measures
----------------------|----------------------------------
Query Volume | Total requests over time
Active Users | Unique users using AI features
Queries per User | Engagement depth
Feature Adoption | % of users trying AI features
Session Patterns | When/how AI is used
```
**Performance Metrics**:
```
Metric | What It Measures
----------------------|----------------------------------
Latency (P50/P95/P99) | Response time distribution
TTFT | Time to first token (streaming)
Throughput | Requests/sec capacity
Error Rate | Failed requests percentage
Timeout Rate | Requests exceeding limit
```
**Quality Metrics**:
```
Metric | What It Measures
----------------------|----------------------------------
User Ratings | Explicit feedback (thumbs up/down)
Completion Rate | Users accepting AI output
Edit Rate | How much users modify output
Regeneration Rate | Users requesting new response
Task Success | Goal completion with AI
```
**Cost Metrics**:
```
Metric | What It Measures
----------------------|----------------------------------
Tokens per Query | Input + output tokens
Cost per Query | $ spent per request
Cost per User | Monthly per-user AI spend
Model Distribution | Which models serve what
Cache Hit Rate | Savings from caching
```
**Implementation**
**Basic Logging**:
```python
import time
import logging
class AIMetrics:
def log_request(self, request_id, model, prompt_tokens,
completion_tokens, latency, success):
logging.info({
"event": "ai_request",
"request_id": request_id,
"model": model,
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"latency_ms": latency,
"success": success,
"timestamp": time.time()
})
# Usage
metrics = AIMetrics()
start = time.time()
response = await llm.generate(prompt)
latency = (time.time() - start) * 1000
metrics.log_request(
request_id=uuid.uuid4(),
model="gpt-4o",
prompt_tokens=response.usage.prompt_tokens,
completion_tokens=response.usage.completion_tokens,
latency=latency,
success=True
)
```
**Analytics Dashboard**:
```python
# SQL for daily metrics
"""
SELECT
DATE(timestamp) as date,
COUNT(*) as total_queries,
COUNT(DISTINCT user_id) as unique_users,
AVG(latency_ms) as avg_latency,
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY latency_ms) as p95_latency,
SUM(prompt_tokens + completion_tokens) as total_tokens,
SUM(cost) as total_cost,
AVG(CASE WHEN user_rating IS NOT NULL THEN user_rating END) as avg_rating
FROM ai_requests
WHERE timestamp > NOW() - INTERVAL '30 days'
GROUP BY DATE(timestamp)
ORDER BY date DESC
"""
```
**Dashboards**
**Essential Views**:
```
Dashboard | Key Visuals
-------------------|----------------------------------
Usage Overview | Query volume, active users, trends
Performance | Latency distribution, errors
Cost | Daily spend, cost per query
Quality | Ratings, completion rate
Model Comparison | Performance by model
```
**Tools**:
```
Tool | Use Case
------------------|----------------------------------
Grafana | Real-time dashboards
Datadog | Full observability
Mixpanel | Product analytics
LangSmith | LLM-specific observability
Helicone | LLM cost tracking
Custom | Tailored to needs
```
**Alerting**
**What to Alert On**:
```python
alerts = {
"high_latency": {
"condition": "p95_latency > 5000ms",
"severity": "warning"
},
"error_rate": {
"condition": "error_rate > 5%",
"severity": "critical"
},
"cost_spike": {
"condition": "hourly_cost > 2x average",
"severity": "warning"
},
"quality_drop": {
"condition": "rating_avg < 3.5",
"severity": "warning"
}
}
```
**Best Practices**
- **Log Everything**: Can't analyze what you don't collect.
- **User Privacy**: Anonymize/redact sensitive content.
- **Real-Time + Historical**: Both immediate and trend analysis.
- **Correlate Metrics**: Understand relationships.
- **Action-Oriented**: Every dashboard should drive decisions.
AI analytics are **essential for operating AI features responsibly** — understanding usage, performance, and cost enables optimization, demonstrates value, and catches problems before users complain.