real-time inference

Real-time inference generates outputs with low latency for interactive applications requiring immediate responses. **Latency targets**: <100ms for conversational AI, <50ms for autocomplete, <10ms for some gaming/robotics. Application-dependent. **Factors affecting latency**: Model size, computation type, batch size, hardware, framework efficiency, network. **Optimization techniques**: **Model level**: Quantization, pruning, distillation, smaller architectures. **Serving level**: GPU acceleration, batching, caching, optimized frameworks. **Infrastructure**: Low-latency networking, edge deployment, regional serving. **Batching trade-off**: Larger batches more efficient but add latency (waiting for batch). Dynamic batching balances. **Streaming**: For LLMs, stream tokens as generated rather than waiting for complete response. **Caching**: Cache frequent queries or computation (KV cache, result cache). **Hardware choices**: GPUs for parallel computation, specialized accelerators (TPU, Inferentia), optimized CPUs. **Profiling**: Identify bottlenecks - model forward pass, preprocessing, network, postprocessing. **SLA considerations**: Define acceptable latency, measure p99 not just average, plan for load spikes.