request id
**Request IDs and Distributed Tracing** are the **observability infrastructure that enables engineers to track individual requests as they flow through microservice architectures** — by assigning a unique identifier to every incoming request and propagating it through every downstream service call, log entry, and database operation, creating a complete audit trail that makes debugging production failures, latency spikes, and partial failures tractable at scale.
**What Are Request IDs and Distributed Tracing?**
- **Request ID (Trace ID)**: A unique identifier (UUID or structured ID) assigned to every incoming request at the system boundary — typically by a load balancer or API gateway — and propagated through all downstream service calls in request headers.
- **Distributed Tracing**: The practice of tracking a request's entire journey across multiple services, each contributing a "span" (a unit of work with start/end time, metadata, and result) that is collected and visualized as a complete trace.
- **The Problem Solved**: In monolithic systems, a request touches one process — debugging is straightforward. In microservice architectures, a single user request may touch 10-50 services. Without trace IDs, correlating logs across services to diagnose failures is nearly impossible.
- **Standard Protocols**: OpenTelemetry (W3C TraceContext standard) provides vendor-neutral distributed tracing with automatic context propagation across HTTP, gRPC, and message queue boundaries.
**Why Request IDs and Tracing Matter**
- **Incident Diagnosis**: "User reported error at 10:32 AM" — without a trace ID, finding the root cause in terabytes of logs is a multi-hour manual process. With a trace ID, you search for that exact request and see the complete failure timeline in seconds.
- **Performance Profiling**: Distributed traces reveal where latency is spent — is the bottleneck in the AI model inference, database query, or downstream API call? Trace spans with timing data pinpoint the exact culprit.
- **Error Attribution**: In a chain of service calls, errors can originate anywhere. Distributed traces show exactly which service returned an error and what its upstream callers did with it.
- **SLA Monitoring**: Measure latency at the full-request level (user-perceived latency) rather than per-service — the metric that matters for user experience.
- **Audit Compliance**: Financial, healthcare, and security applications require complete audit trails of what happened to every request — trace IDs provide the correlation key to reconstruct complete audit logs.
**Request ID Implementation**
**Generation (At Entry Point)**:
```python
import uuid
from fastapi import Request
@app.middleware("http")
async def add_request_id(request: Request, call_next):
# Use client-provided ID if present (enable end-to-end tracing)
request_id = request.headers.get("X-Request-ID", str(uuid.uuid4()))
# Store in context for use throughout request lifecycle
request.state.request_id = request_id
response = await call_next(request)
# Echo back in response header so client can reference it
response.headers["X-Request-ID"] = request_id
return response
```
**Propagation (To Downstream Services)**:
```python
def call_downstream_service(endpoint: str, payload: dict, request_id: str) -> dict:
headers = {
"X-Request-ID": request_id, # Propagate trace
"Authorization": f"Bearer {service_token}"
}
return requests.post(endpoint, json=payload, headers=headers).json()
```
**Logging with Trace Context**:
```python
import structlog
logger = structlog.get_logger()
def process_request(request_id: str, user_id: str, payload: dict):
log = logger.bind(request_id=request_id, user_id=user_id)
log.info("Processing started", payload_size=len(str(payload)))
result = do_processing(payload)
log.info("Processing completed", result_status=result.status, duration_ms=result.duration)
return result
```
**Distributed Tracing with OpenTelemetry**
OpenTelemetry (OTel) provides automatic trace context propagation and span collection:
```python
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# Setup
tracer = trace.get_tracer(__name__)
def process_ai_request(user_query: str) -> str:
with tracer.start_as_current_span("ai_request") as span:
span.set_attribute("user.query_length", len(user_query))
with tracer.start_as_current_span("vector_search"):
context = vector_db.search(user_query)
with tracer.start_as_current_span("llm_inference"):
span.set_attribute("llm.model", "gpt-4o")
response = llm.generate(user_query, context)
span.set_attribute("response.length", len(response))
return response
```
This automatically generates a trace showing: total request time, vector search time, LLM inference time — with all spans linked by trace ID.
**Tracing Platforms and Tools**
| Platform | Type | Key Strength |
|----------|------|-------------|
| Jaeger | Open source | Full-featured, Kubernetes-native |
| Zipkin | Open source | Lightweight, simple UI |
| Datadog APM | Commercial | Integrated with monitoring, alerting |
| AWS X-Ray | Cloud | Deep AWS service integration |
| Google Cloud Trace | Cloud | GCP-integrated |
| Honeycomb | Commercial | High-cardinality trace analysis |
| Grafana Tempo | Open source | Prometheus-integrated, scalable |
**AI-Specific Tracing**
For LLM applications, trace spans should capture:
- Model name and version.
- Input token count and output token count.
- Inference latency (time to first token, total time).
- Number of retries.
- Retrieval latency and chunk count (for RAG).
- Tool call names and durations (for agents).
- Cost per request (token count × price).
Request IDs and distributed tracing are **the observability infrastructure that makes complex AI systems debuggable at production scale** — without trace correlation, diagnosing why a specific user's request failed, identifying which service introduced unexpected latency, or proving to an auditor what happened to a specific transaction requires heroic manual log correlation that is impractical at volume.