CUDA graphs are a mechanism for capturing a whole sequence of GPU operations once and then replaying it with a single launch, so that the per-kernel cost of telling the GPU what to do next stops dominating the runtime. They target a specific and common failure mode: a workload made of many small kernels where the GPU finishes each one faster than the CPU can submit the next, leaving the hardware idle between launches. Fusion attacks that problem by making fewer kernels; CUDA graphs attack it by making the launches themselves nearly free.\n\nEvery kernel launch has a fixed CPU-side cost, and for small kernels that cost is the bottleneck. Submitting a kernel means the driver and runtime set up arguments and dispatch work across the CPU-GPU boundary — a few microseconds each time. When kernels are large that overhead is noise, but a transformer decoding one token at a time issues hundreds of tiny operations per step, each running in microseconds, and the CPU simply cannot enqueue them fast enough. The GPU stalls in the gaps waiting for the next launch, and the workload becomes launch-bound (CPU-bound) rather than compute-bound: adding a faster GPU does nothing because the limiter is submission, not math.\n\nA CUDA graph records the launches and their dependencies as a DAG, then replays them as one unit. Instead of the CPU issuing each kernel every iteration, you capture the sequence once into a graph — a directed acyclic graph of operations and the dependencies between them — and hand the whole thing to the driver, which submits it in a single call. The per-kernel setup is paid once, at capture time; every replay after that is a single cheap launch that streams the entire recorded schedule onto the GPU back-to-back with no bubbles. Record once, replay many.\n\nThe usual way to build one is stream capture, which records your existing code instead of rerunning it. You wrap the normal kernel-launch sequence between cudaStreamBeginCapture and cudaStreamEndCapture; during that window the runtime does not execute the kernels but records them into a graph. You then instantiate the graph (cudaGraphInstantiate) into an executable form and launch it with cudaGraphLaunch on every subsequent step. Because the code you already have becomes the recording, adopting graphs rarely means rewriting the model — it means capturing the steady-state loop once and replaying it.\n\nThe catch is that a captured graph must be static, which shapes how you use it. Replay assumes the same kernels, the same tensor shapes, and the same memory addresses every time, so anything data-dependent — variable sequence lengths, dynamic control flow, CPU logic interleaved between kernels — breaks capture. In practice CUDA graphs pair with fixed-shape execution: pad or bucket to static shapes, and keep inputs and outputs in a persistent memory pool so their addresses do not move. This is exactly what PyTorch's torch.compile(mode="reduce-overhead") does under the hood, and why it helps most on the decode phase of LLM inference, the canonical launch-bound workload. Fusion and graphs are complementary: fusion reduces how many kernels there are, graphs reduce the per-launch cost of whatever remains, and high-performance stacks like TensorRT use both.\n\n| | Without CUDA graphs | With CUDA graphs |\n|---|---|---|\n| Submission | CPU launches each kernel, every step | capture once, replay the whole graph in one call |\n| Per-kernel CPU cost | paid on every launch, every iteration | paid once, at capture |\n| Wins on | compute-bound, few large kernels | launch-bound, many tiny kernels (LLM decode) |\n| Requires | nothing special | static shapes, kernels, and memory addresses |\n\n``svg\n \n``\n\nRead CUDA graphs through a who-is-the-bottleneck lens rather than a faster-kernels lens: when a model is made of many tiny operations the limiter is the CPU's ability to submit work, not the GPU's ability to do it, and capturing the steady-state loop into a single replayable graph removes that submission cost almost entirely — which is why it is the standard companion to fusion and static-shape inference on the decode path of large language models.