neural network compiler
**Neural Network Compilers** are the **software systems that transform high-level model definitions (PyTorch/TensorFlow graphs) into optimized low-level code for specific hardware targets** — performing operator fusion, memory planning, kernel selection, and hardware-specific optimization to achieve 1.5-3x inference speedups and 10-30% training speedups compared to eager execution, bridging the gap between the flexibility of Python-based model definitions and the performance of hand-tuned hardware code.
**Why ML Compilers?**
- Framework-generated code: Generic kernels, Python overhead, no cross-operator optimization.
- Compiled code: Fused operators, optimized memory layout, hardware-specific instructions.
- Gap: 2-10x performance left on the table without compilation.
**Major ML Compilers**
| Compiler | Developer | Input | Target | Key Feature |
|----------|----------|-------|--------|------------|
| torch.compile (Inductor) | Meta | PyTorch graphs | CPU, GPU | Default in PyTorch 2.0+, Triton backend |
| XLA | Google | TensorFlow, JAX | TPU, GPU, CPU | HLO IR, excellent TPU support |
| TVM (Apache) | Community | ONNX, Relay IR | Any hardware | Auto-tuning, broad hardware support |
| TensorRT | NVIDIA | ONNX, TorchScript | NVIDIA GPU | Best inference on NVIDIA GPUs |
| MLIR | LLVM/Google | Multiple dialects | Any target | Compiler infrastructure framework |
| IREE | Google | MLIR-based | Mobile, embedded | Lightweight inference runtime |
**torch.compile (PyTorch 2.0+)**
```python
import torch
model = MyModel()
optimized = torch.compile(model) # One-line compilation
output = optimized(input) # First call traces + compiles, subsequent calls use compiled code
```
- **TorchDynamo**: Captures Python bytecode → extracts computation graph.
- **TorchInductor**: Compiles graph → Triton kernels (GPU) or C++/OpenMP (CPU).
- **Automatic operator fusion**: Element-wise ops fused into single kernel.
- Modes: `default` (balanced), `reduce-overhead` (minimize CPU overhead), `max-autotune` (try all variants).
**Compilation Pipeline (General)**
1. **Graph Capture**: Trace model execution → computation graph (DAG of operators).
2. **Graph-Level Optimization**: Operator fusion, constant folding, dead code elimination.
3. **Lowering**: Map high-level ops to target-specific primitives.
4. **Kernel Selection/Generation**: Choose pre-tuned kernels or auto-generate (Triton/CUDA).
5. **Memory Planning**: Schedule tensor lifetimes, fuse allocations, minimize peak memory.
6. **Code Generation**: Emit final executable (PTX, LLVM IR, C++).
**Key Optimizations**
| Optimization | What It Does | Speedup |
|-------------|-------------|--------|
| Operator fusion | Combine element-wise ops into one kernel | 2-10x for fused ops |
| Memory planning | Reduce allocations, reuse buffers | 10-30% less memory |
| Layout optimization | Choose optimal tensor format (NHWC vs NCHW) | 5-20% |
| Kernel auto-tuning | Try multiple implementations, pick fastest | 10-50% |
| Quantization | Lower precision arithmetic | 2-4x throughput |
Neural network compilers are **transforming ML deployment** — by automating the performance engineering that previously required hand-written CUDA kernels, they democratize hardware-efficient AI, making it practical for any PyTorch model to achieve near-expert-level optimization with a single line of code.