tensorrt-llm

TensorRT is NVIDIA's inference optimizer and runtime: it takes a trained model and compiles it, ahead of time, into a serialized *engine* that runs as fast as possible on one specific class of GPU. TensorRT-LLM is a library built on top of it that specializes the same idea for transformer language models, adding the attention kernels, KV-cache management, batching, and multi-GPU parallelism that LLM serving needs. Together they are the most aggressive way to run inference on NVIDIA hardware, and the price of that speed is build-time cost and a loss of portability.\n\n**TensorRT is an ahead-of-time compiler, not a runtime interpreter.** You hand its *builder* a model — usually an ONNX graph exported from PyTorch or TensorFlow — and it performs a sequence of transformations: it fuses layers vertically and horizontally (a convolution, its bias, and its activation collapse into one kernel), it lowers precision to FP16, INT8, or FP8 with a calibration step that picks per-tensor scales, it plans and reuses the memory for intermediate tensors, and it runs *tactic selection*, benchmarking many candidate kernel implementations and keeping the fastest one for your exact GPU architecture and tensor shapes. The output is a serialized engine that a lightweight runtime loads and executes.\n\n**That engine is fast precisely because it is specialized, which is also its main limitation.** Because tactic selection benchmarks kernels against a particular streaming-multiprocessor generation and a particular set of shapes, a TensorRT engine is not portable: it is tied to the GPU architecture, the TensorRT version, and the precision and shape profiles it was built with. Move to a different GPU or upgrade the library and you rebuild. This is the fundamental contrast with a JIT approach like `torch.compile`, which compiles on the fly on whatever hardware it lands on; TensorRT pays the compilation cost once, up front, in exchange for a leaner and faster deployment artifact.\n\n**TensorRT-LLM layers the transformer-specific machinery on top.** A plain TensorRT engine does not know what an attention block is; TensorRT-LLM contributes fused multi-head-attention kernels, a *paged* KV cache that stores attention state in non-contiguous blocks the way vLLM's PagedAttention does, and *in-flight* (continuous) batching that lets new requests join a running batch instead of waiting for it to drain. It adds the low-precision paths that matter for weights — INT4 and INT8 via AWQ, GPTQ, and SmoothQuant, plus FP8 — and it can shard a model across GPUs with tensor and pipeline parallelism for models too large for one device. You describe the model through a Python API, and it assembles and compiles a TensorRT engine from that description.\n\n**Where it fits against the alternatives comes down to how much build complexity you will trade for peak throughput.** vLLM is Python-native, easy to stand up, and strong on throughput through PagedAttention and continuous batching; TensorRT-LLM is heavier to build and hardware-locked but usually reaches the highest tokens-per-second and lowest latency on NVIDIA GPUs, especially once FP8 or INT4 quantization is in play. The decision mirrors the general compiled-versus-interpreted tradeoff: if the deployment is fixed, high-volume, and all-NVIDIA, the ahead-of-time engine wins; if it changes often or must stay portable, a JIT or Python-native server is the more comfortable fit.\n\n| Stage | TensorRT (any model) | TensorRT-LLM (transformers) |\n|---|---|---|\n| Input | ONNX / framework graph | Python model definition |\n| Fusion | layer & tensor fusion | + fused multi-head attention |\n| Precision | FP16 / INT8 / FP8 calibration | + INT4 AWQ·GPTQ, SmoothQuant, FP8 |\n| Batching | static / dynamic shapes | in-flight batching + paged KV cache |\n| Scale | single GPU | tensor + pipeline parallel, multi-node |\n| Output | serialized engine (GPU-locked) | engine + LLM runtime |\n\n```svg\n \n```\n\nRead TensorRT through a *compile-the-deployment-into-a-hardware-specific-artifact* lens rather than a *faster-library* lens: the builder spends real time fusing, quantizing, and benchmarking kernels against one GPU so that serving becomes a thin load-and-run step, and TensorRT-LLM extends that bargain to transformers with paged KV cache, continuous batching, and multi-GPU sharding — which is why it tends to win on raw throughput but asks you to rebuild whenever the hardware, precision, or version changes.