graph optimization
**Graph Optimization** is the **set of compiler techniques that transform a neural network's computation graph to minimize execution time and memory usage before runtime** — performing operator fusion (combining multiple operations into single GPU kernels), constant folding (pre-computing static subgraphs), dead code elimination, layout optimization, and precision calibration to achieve 2-5× inference speedups without changing model accuracy, serving as the critical compilation step between model training and production deployment.
**What Is Graph Optimization?**
- **Definition**: The process of analyzing and transforming the directed acyclic graph (DAG) that represents a neural network's computation — identifying patterns that can be simplified, combined, or eliminated to reduce the number of GPU kernel launches, memory transfers, and arithmetic operations required to execute the model.
- **Graph-Level vs. Kernel-Level**: Graph optimization operates on the high-level computation structure (which operations to perform and in what order) — complementary to kernel-level optimization (how each individual operation is implemented on the GPU hardware).
- **Ahead-of-Time**: Graph optimizations are applied before inference begins (compile time) — the optimized graph is then executed repeatedly for each input, amortizing the optimization cost over millions of inference calls.
- **Framework Support**: All major inference frameworks include graph optimization — ONNX Runtime, TensorRT, TorchScript/torch.compile, OpenVINO, and TFLite each implement their own optimization passes.
**Key Graph Optimization Techniques**
- **Operator Fusion**: Combine multiple sequential operations (Conv → BatchNorm → ReLU) into a single GPU kernel — eliminates intermediate memory reads/writes and kernel launch overhead. The single most impactful optimization, often providing 2-3× speedup.
- **Constant Folding**: Pre-compute parts of the graph that depend only on constant inputs (weights, biases) — eliminates runtime computation for static subexpressions.
- **Dead Code Elimination**: Remove graph nodes whose outputs are not used by any downstream operation — cleans up unused branches from model export or conditional logic.
- **Layout Optimization**: Convert tensor memory layout to match hardware preference — NCHW vs. NHWC format selection based on whether the target is NVIDIA GPU (NHWC for tensor cores) or CPU (varies).
- **Precision Calibration**: Insert quantization/dequantization nodes for mixed-precision inference — enabling INT8 or FP16 execution of operations that tolerate reduced precision.
- **Shape Inference**: Statically determine tensor shapes throughout the graph — enables memory pre-allocation and eliminates runtime shape computation.
**Graph Optimization Tools**
| Tool | Framework | Key Optimizations | Target Hardware |
|------|----------|------------------|----------------|
| TensorRT | NVIDIA | Fusion, INT8/FP16, kernel autotuning | NVIDIA GPUs |
| ONNX Runtime | Cross-platform | Fusion, quantization, graph rewriting | CPU, GPU, NPU |
| torch.compile | PyTorch | Fusion, memory planning, triton kernels | NVIDIA GPUs |
| OpenVINO | Intel | Fusion, INT8, layout optimization | Intel CPU/GPU/VPU |
| TFLite | TensorFlow | Quantization, fusion, delegation | Mobile, edge |
| XLA | JAX/TensorFlow | Fusion, memory optimization | TPU, GPU |
**Graph optimization is the essential compilation step that transforms trained models into efficient inference engines** — applying operator fusion, constant folding, and precision calibration to reduce GPU kernel launches and memory transfers by 2-5×, bridging the gap between research model quality and production deployment performance.