CUDA Graph API: Fixed-Topology Amortized Launching — reducing kernel launch overhead for inference and fixed-pattern workloads

CUDA Graphs capture sequences of kernels and memory operations into graphs, enabling repeated execution without individual launch overhead. This optimization targets inference workloads with fixed computation topology (constant graph structure across inputs).

Graph Capture and Node Types

Capture mode records GPU operations (kernel launches, memcpys, host callbacks) into a graph during stream execution. Nodes represent kernels, memcpys, events, host functions, or memsets. Dependencies between nodes (edges) define execution ordering: kernel A waits on kernel B's completion if dependent. Graphs require fixed topology: same kernels execute in same order with identical arguments. Conditional branches and data-dependent control flow preclude graphing.

Instantiation and Launch Overhead Reduction

Graph instantiation validates the graph, generating an executable form. Repeated instantiation amortizes overhead: graph→instantiate→launch (100x) is faster than stream→cudaMemcpy→cudaKernelLaunch (100x separately). Overhead reduction is most dramatic for small kernels (1-10 microseconds): launch overhead (5 microseconds CPU-side) dominates; amortized via graphing. For long kernels (milliseconds+), launch overhead is negligible percentage—graphing provides minimal benefit.

Executable Graph Updates

CUDA 11.0+ enables executable graph updates: modify kernel arguments and memcpy parameters without full revalidation. This supports inference pipelines where batch size varies: graph template set for maximum batch size, instantiate once, update batch parameter per iteration.

Inference Use Cases

Transformer inference (text generation tokens sequentially) leverages graphs: embedding lookup, attention QKV projection, softmax, multinomial sampling—fixed sequence of small kernels with variable parameters. Graph amortization recovers ~10% efficiency versus stream-based launching. Video processing pipelines with frame buffering similarly benefit.

Limitations

Graphs require fixed topology—adaptive algorithms, dynamic loop counts, and conditional execution remain unavailable. Some operations (cooperative kernel launches) lack graph support. Graphs demand explicit data dependencies: out-of-graph (CPU) synchronization breaks graph benefits.