tensorrt
**TensorRT Optimization**
**What is TensorRT?**
NVIDIA TensorRT is an SDK for high-performance deep learning inference. It optimizes models for NVIDIA GPUs, providing significant speedups.
**Optimizations Applied**
| Optimization | Description |
|--------------|-------------|
| Layer fusion | Combine operations into single kernels |
| Precision calibration | INT8/FP16 quantization |
| Kernel auto-tuning | Select best kernel for hardware |
| Memory optimization | Efficient memory allocation |
| Dynamic tensor memory | Reuse memory during inference |
**Conversion Pipeline**
```
PyTorch → [Export] → ONNX → [TensorRT Build] → TRT Engine
```
**Building TensorRT Engine**
**From ONNX**
```python
import tensorrt as trt
logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
parser = trt.OnnxParser(network, logger)
# Parse ONNX
with open("model.onnx", "rb") as f:
parser.parse(f.read())
# Build config
config = builder.create_builder_config()
config.set_flag(trt.BuilderFlag.FP16) # Enable FP16
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 1 << 30) # 1GB
# Build engine
engine = builder.build_serialized_network(network, config)
# Save engine
with open("model.trt", "wb") as f:
f.write(engine)
```
**Running TensorRT Engine**
```python
import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit
# Load engine
runtime = trt.Runtime(logger)
with open("model.trt", "rb") as f:
engine = runtime.deserialize_cuda_engine(f.read())
context = engine.create_execution_context()
# Allocate buffers, run inference...
```
**TensorRT-LLM**
For LLMs, use NVIDIA's TensorRT-LLM:
```bash
# Build optimized LLM engine
python build.py
--model_dir ./llama-hf
--dtype bfloat16
--output_dir ./llama-trt
```
Features:
- Optimized attention kernels
- Inflight batching
- PagedAttention support
- Multi-GPU support
**Performance Comparison**
| Framework | Throughput | Latency |
|-----------|------------|---------|
| PyTorch | Baseline | Baseline |
| ONNX Runtime | 1.5-2x | 0.7x |
| TensorRT | 2-4x | 0.3-0.5x |
| TensorRT-LLM | 3-5x | 0.2-0.4x |
**When to Use TensorRT**
| Scenario | Recommendation |
|----------|----------------|
| NVIDIA GPU production inference | Yes |
| Need lowest latency | Yes |
| Rapid prototyping | Overhead may not be worth it |
| Cross-platform deployment | Use ONNX instead |