TensorRT is NVIDIA's inference optimizer and runtime: it takes a trained model and compiles it, ahead of time, into a serialized engine that runs as fast as possible on one specific class of GPU. TensorRT-LLM is a library built on top of it that specializes the same idea for transformer language models, adding the attention kernels, KV-cache management, batching, and multi-GPU parallelism that LLM serving needs. Together they are the most aggressive way to run inference on NVIDIA hardware, and the price of that speed is build-time cost and a loss of portability.\n\nTensorRT is an ahead-of-time compiler, not a runtime interpreter. You hand its builder a model — usually an ONNX graph exported from PyTorch or TensorFlow — and it performs a sequence of transformations: it fuses layers vertically and horizontally (a convolution, its bias, and its activation collapse into one kernel), it lowers precision to FP16, INT8, or FP8 with a calibration step that picks per-tensor scales, it plans and reuses the memory for intermediate tensors, and it runs tactic selection, benchmarking many candidate kernel implementations and keeping the fastest one for your exact GPU architecture and tensor shapes. The output is a serialized engine that a lightweight runtime loads and executes.\n\nThat engine is fast precisely because it is specialized, which is also its main limitation. Because tactic selection benchmarks kernels against a particular streaming-multiprocessor generation and a particular set of shapes, a TensorRT engine is not portable: it is tied to the GPU architecture, the TensorRT version, and the precision and shape profiles it was built with. Move to a different GPU or upgrade the library and you rebuild. This is the fundamental contrast with a JIT approach like torch.compile, which compiles on the fly on whatever hardware it lands on; TensorRT pays the compilation cost once, up front, in exchange for a leaner and faster deployment artifact.\n\nTensorRT-LLM layers the transformer-specific machinery on top. A plain TensorRT engine does not know what an attention block is; TensorRT-LLM contributes fused multi-head-attention kernels, a paged KV cache that stores attention state in non-contiguous blocks the way vLLM's PagedAttention does, and in-flight (continuous) batching that lets new requests join a running batch instead of waiting for it to drain. It adds the low-precision paths that matter for weights — INT4 and INT8 via AWQ, GPTQ, and SmoothQuant, plus FP8 — and it can shard a model across GPUs with tensor and pipeline parallelism for models too large for one device. You describe the model through a Python API, and it assembles and compiles a TensorRT engine from that description.\n\nWhere it fits against the alternatives comes down to how much build complexity you will trade for peak throughput. vLLM is Python-native, easy to stand up, and strong on throughput through PagedAttention and continuous batching; TensorRT-LLM is heavier to build and hardware-locked but usually reaches the highest tokens-per-second and lowest latency on NVIDIA GPUs, especially once FP8 or INT4 quantization is in play. The decision mirrors the general compiled-versus-interpreted tradeoff: if the deployment is fixed, high-volume, and all-NVIDIA, the ahead-of-time engine wins; if it changes often or must stay portable, a JIT or Python-native server is the more comfortable fit.\n\n| Stage | TensorRT (any model) | TensorRT-LLM (transformers) |\n|---|---|---|\n| Input | ONNX / framework graph | Python model definition |\n| Fusion | layer & tensor fusion | + fused multi-head attention |\n| Precision | FP16 / INT8 / FP8 calibration | + INT4 AWQ·GPTQ, SmoothQuant, FP8 |\n| Batching | static / dynamic shapes | in-flight batching + paged KV cache |\n| Scale | single GPU | tensor + pipeline parallel, multi-node |\n| Output | serialized engine (GPU-locked) | engine + LLM runtime |\n\n``svg\n \n``\n\nRead TensorRT through a compile-the-deployment-into-a-hardware-specific-artifact lens rather than a faster-library lens: the builder spends real time fusing, quantizing, and benchmarking kernels against one GPU so that serving becomes a thin load-and-run step, and TensorRT-LLM extends that bargain to transformers with paged KV cache, continuous batching, and multi-GPU sharding — which is why it tends to win on raw throughput but asks you to rebuild whenever the hardware, precision, or version changes.