model serving

**model serving** is the production infrastructure that turns trained models into reliable low-latency prediction services. Serving determines accelerator utilization, tail latency, throughput, availability, and cost for LLMs, vision, speech, recommendation, and multimodal applications. **Serving architecture.** Clients reach an authenticated gateway and load balancer, which route work to replicas holding model weights and runtime state. A scheduler validates shapes, batches compatible work, assigns GPUs, executes prefill and decode or ordinary inference, and streams or returns results. Autoscaling reacts to queue and utilization signals; model registries, canaries, health checks, tracing, rate limits, and rollback manage lifecycle. Network, tokenization, serialization, and queue time count toward user latency. **LLM scheduling.** LLM inference separates compute-heavy prompt prefill from memory-bandwidth-heavy autoregressive decoding. KV caches grow with sequence count and length, making paging, prefix reuse, eviction, and admission control central. Continuous batching inserts new requests between decode steps rather than waiting for a static batch. Tensor, pipeline, and expert parallelism divide large models across devices; speculative decoding verifies drafts from a smaller model to reduce serial steps when acceptance is high. **Optimization and frameworks.** Quantization reduces weight and cache footprint but must preserve quality across layers and workloads. Kernel fusion, CUDA graphs, efficient attention, model compilation, and topology-aware collectives reduce overhead. vLLM emphasizes paged KV management and continuous batching; TGI provides Hugging Face-oriented text generation; TensorRT-LLM applies NVIDIA-specific compilation and kernels; Triton serves heterogeneous model backends. Framework labels do not replace workload benchmarking. **Metrics and capacity.** Track time to first token, inter-token latency, end-to-end p50 and p99, output tokens per second, requests per second, queue depth, cache occupancy, accelerator utilization, errors, availability, and cost per useful response. Throughput rises with batching while tail latency can worsen. Capacity planning models input/output distributions, context limits, burstiness, warm-up, failure domains, redundancy, and multi-tenant fairness. **Operations and validation.** A production implementation begins with explicit terminal conditions, operating ranges, loading, accuracy, noise, latency, efficiency, area, cost, lifetime, and fault behavior. Schematic or architectural models establish feasibility; extracted, package, board, thermal, and control-loop models then reveal interactions hidden by ideal sources and loads. Verification spans process, voltage, temperature, mismatch, aging, startup, shutdown, overload, brownout, and recovery. Teams should define measurement bandwidth, observation point, stimulus, pass limit, guard band, and statistical confidence before simulation. Layout review covers current return, thermal gradients, matching, parasitic coupling, electromigration, voltage stress, latch-up, ESD paths, and test access. Correlation retains netlists, models, scripts, tool versions, raw results, lab conditions, calibration status, and explanations for outliers. This evidence turns a nominal design into a reproducible component that can be signed off across device, circuit, package, firmware, and system teams. Corner selection should follow sensitivity rather than blindly combining labels. Deterministic sweeps expose monotonic trends, targeted Monte Carlo analysis estimates distribution tails, and importance sampling can explore rare failures. Reviewers should distinguish model uncertainty from manufacturing variation and avoid claiming yield from too few samples. The interface contract must state what happens outside normal operation. Open and short terminals, reverse polarity, hot plug, disabled bias, floating control pins, clock loss, thermal shutdown, current limiting, and repeated fault cycling often determine field reliability even though they are absent from the nominal transfer function. Dynamic behavior deserves the same attention as steady state. Settling, overshoot, ringing, slew, recovery from saturation, mode transitions, and interaction with external poles can violate a system limit long before a DC endpoint does. Time-domain tests should include realistic edge rates and source impedance. Noise should be referred to the signal or supply point that matters to the application and integrated only over a stated bandwidth. Thermal, flicker, quantization, switching, reference, substrate, and electromagnetic contributions may combine differently across modes, so a single spot-noise number rarely completes the specification. Power and thermal claims should include quiescent, active, transient, and fault states. Average efficiency can hide localized current density or hot spots; electrothermal simulation and temperature-aware device models connect electrical stress to lifetime, drift, and protection thresholds. Physical design must preserve the assumptions behind the schematic. Symmetry, common-centroid placement, dummies, shielding, guard rings, Kelvin sensing, wide current paths, via arrays, controlled coupling, and quiet reference routing are selected according to the dominant error rather than applied as decoration. Production test strategy is part of design. Trim range, observability, loopback modes, built-in self-test, boundary conditions, test time, and instrument uncertainty determine which specifications can be guaranteed economically. Characterization across wafers and lots should feed model and guard-band updates. System telemetry can extend laboratory correlation into deployed products. Error counters, calibration codes, temperatures, supply monitors, fault flags, margin measurements, and performance events help distinguish random failures from systematic drift without exposing sensitive implementation details. A useful comparison normalizes alternatives at equal output requirement and environment. Peak headline values can be misleading when bandwidth, drive, voltage, area, cooling, external components, calibration, or reliability differs; the decision record should name the workload and weighting used. Cross-functional review should trace each requirement from physical mechanism through circuit behavior to application impact. That trace prevents duplicated margin, exposes assumptions that span ownership boundaries, and makes later process or package substitutions safer. Corner selection should follow sensitivity rather than blindly combining labels. Deterministic sweeps expose monotonic trends, targeted Monte Carlo analysis estimates distribution tails, and importance sampling can explore rare failures. Reviewers should distinguish model uncertainty from manufacturing variation and avoid claiming yield from too few samples. | Framework | Primary focus | Scheduling / optimization | Strength | Trade-off | |---|---|---|---|---| | vLLM | LLM serving | Paged KV cache and continuous batching | High throughput and flexible APIs | Rapid version and kernel evolution | | TGI | Text generation | Continuous batching and streaming | Hugging Face ecosystem integration | Model/backend support varies | | TensorRT-LLM | NVIDIA LLM inference | Compiled kernels and parallel execution | Strong hardware-specific performance | Platform-specific build complexity | | Triton | General inference server | Dynamic batching and many backends | Heterogeneous model ensembles | LLM-specific scheduling needs backend | | Custom runtime | Workload-specific service | Application-defined | Maximum specialization | Engineering and maintenance burden | ```svg ``` **Connection to CFS platform.** Use CFS AI, accelerator, memory, networking, serving, sensor, robotics, and system simulators with linked glossary topics to connect application behavior to measurable hardware and deployment trade-offs.