CUDA Graph

**CUDA Graphs** is a CUDA runtime feature that records a sequence of GPU kernel launches and memory operations into a reusable graph object, then replays the entire sequence with a single API call. The key insight is that in traditional eager execution each kernel launch requires a round-trip from CPU to GPU driver — typically 5-10 microseconds of overhead — and that overhead compounds across thousands of small kernels per training step or inference pass. CUDA Graphs amortize all of that to a single launch call regardless of how many nodes the graph contains. ```svg ``` **The capture-instantiate-replay lifecycle** has three phases. First, the application calls cudaStreamBeginCapture on a CUDA stream, after which every GPU operation submitted to that stream is recorded as a node in an internal graph rather than executed immediately. After all operations are recorded, cudaStreamEndCapture returns a cudaGraph_t handle representing the DAG of nodes and their dependencies. That graph is compiled into an executable form with cudaGraphInstantiate, producing a cudaGraphExec_t object. From that point on, cudaGraphLaunch submits the entire sequence to the GPU in one call. **The dependency graph captures parallelism automatically.** When kernels are submitted to different CUDA streams during capture, the graph records them as parallel nodes. Memory copies, kernel launches, event synchronizations, and host function calls all become nodes with typed edges encoding their dependencies. This means the GPU scheduler sees the full DAG at launch time and can overlap independent nodes across compute and memory copy engines simultaneously. **PyTorch exposes CUDA Graphs through torch.cuda.graph().** The pattern requires static input tensors: allocate placeholder tensors, run a warmup pass to trigger CuBLAS and cuDNN workspace allocation, then capture with g = torch.cuda.CUDAGraph() and the context manager torch.cuda.graph(g). Subsequent forward passes call g.replay() with fresh data copied into the static tensors using tensor.copy_(). This eliminates all Python interpreter and CUDA driver overhead from the steady-state loop, which is why vLLM, TensorRT-LLM, and FlashInfer use graph capture for their inference kernels. **Static shapes are the fundamental constraint.** The captured graph hardcodes kernel grid dimensions, tensor addresses, and argument values at capture time. A change in batch size or sequence length invalidates the graph and requires recapture. cudaGraphExecUpdate provides a lighter path — it replaces node parameters without recompiling the graph topology — but it cannot change kernel grid dimensions. Production inference servers typically maintain a pool of graphs captured at common batch sizes (1, 2, 4, 8, 16, ...) and select the smallest covering graph for each incoming request. **Conditional graphs and device graphs** (CUDA 12.4+) extend the model to dynamic control flow. Device graphs can be launched from within a running kernel, enabling recursive or iterative patterns that previously required CPU synchronization. Conditional nodes (if/while) allow the graph to branch based on GPU-side predicates without returning to the CPU. | Scenario | Eager latency | Graph latency | Graph advantage | |---|---|---|---| | 100 small kernels per step | ~500-1000 µs launch overhead | ~1 µs launch overhead | Dramatic for small kernels | | LLM decode step (fixed batch) | ~20-40 µs driver overhead | ~1-2 µs | +20-40% tokens/s | | Large single matmul | Compute-bound, minimal gain | Minimal gain | Not the right tool | | RL env step (fixed obs shape) | Overhead compounds | Replayed cheaply | High benefit | **When CUDA Graphs are not appropriate:** workloads with variable-length inputs that change every step, operations that must trigger host-side callbacks, workflows with significant Python-side control flow between GPU calls that cannot be captured, and any case where the graph topology itself must change dynamically. For those cases, torch.compile with inductor can still extract fusion and overlap benefits without requiring strict static shapes.