tensorrt

TensorRT is NVIDIA's deep learning inference optimizer and runtime, providing kernel fusion, precision conversion, layer optimization, and hardware-specific tuning to deliver the fastest inference performance on NVIDIA GPUs. Optimization pipeline: import model (ONNX, TensorFlow, PyTorch) → analyze graph → apply optimizations → generate optimized engine for specific GPU. Kernel fusion: combine multiple operations into single kernel; reduces memory bandwidth and kernel launch overhead. Precision conversion: FP32 to FP16 or INT8 with calibration; maintains accuracy while dramatically improving throughput and reducing memory. Layer optimization: replace generic implementations with highly optimized versions for specific layer patterns. Hardware targeting: builds optimized engine for specific GPU architecture (Ampere, Hopper, etc.); not portable between GPU generations. Dynamic shapes: supports variable batch size and sequence length with optimization profiles. Plugin system: custom operations via plugin API; extend TensorRT for non-standard layers. TensorRT-LLM: extension specifically for LLM inference; includes attention optimizations, KV caching, and tensor parallelism. Integration: works with Triton Inference Server for production serving. Build time: optimization takes time (minutes to hours); but runtime performance is unmatched on NVIDIA. Comparison: 2-10× faster than PyTorch eager mode; essential for latency-critical applications. TensorRT is the performance standard for NVIDIA GPU inference.