TensorRT Optimization

What is TensorRT? NVIDIA TensorRT is an SDK for high-performance deep learning inference. It optimizes models for NVIDIA GPUs, providing significant speedups.

Optimizations Applied

Conversion Pipeline

PyTorch → [Export] → ONNX → [TensorRT Build] → TRT Engine

Building TensorRT Engine

From ONNX

import tensorrt as trt

logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
parser = trt.OnnxParser(network, logger)

# Parse ONNX
with open("model.onnx", "rb") as f:
    parser.parse(f.read())

# Build config
config = builder.create_builder_config()
config.set_flag(trt.BuilderFlag.FP16)  # Enable FP16
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 1 << 30)  # 1GB

# Build engine
engine = builder.build_serialized_network(network, config)

# Save engine
with open("model.trt", "wb") as f:
    f.write(engine)

Running TensorRT Engine

import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit

# Load engine
runtime = trt.Runtime(logger)
with open("model.trt", "rb") as f:
    engine = runtime.deserialize_cuda_engine(f.read())

context = engine.create_execution_context()

# Allocate buffers, run inference...

TensorRT-LLM For LLMs, use NVIDIA's TensorRT-LLM:

# Build optimized LLM engine
python build.py
    --model_dir ./llama-hf
    --dtype bfloat16
    --output_dir ./llama-trt

Features:

• Optimized attention kernels
• Inflight batching
• PagedAttention support
• Multi-GPU support

Performance Comparison

When to Use TensorRT