DMA Engines and Zero-Copy Transfers are the hardware components and programming techniques that transfer data between memory regions (CPU↔GPU, GPU↔NVMe, NIC↔GPU) without CPU involvement — freeing the CPU to perform computation while data moves autonomously through dedicated DMA controllers, and in the zero-copy case eliminating data copies entirely by mapping device-accessible memory that both CPU and device can read/write directly.

Why DMA Matters

• CPU-driven copy (memcpy): CPU reads source → writes destination → CPU busy the entire time.
• DMA: CPU programs transfer (source, destination, size) → DMA engine handles movement → CPU is free.
• At 100 GB/s transfer rate: 1 GB transfer takes 10 ms of CPU time (memcpy) vs. ~0 ms (DMA).

DMA in GPU Computing

CUDA Async DMA (cudaMemcpyAsync)

cudaStream_t copy_stream, compute_stream;
cudaStreamCreate(&copy_stream);
cudaStreamCreate(&compute_stream);

// Overlap DMA with computation using separate streams
for (int i = 0; i < N; i++) {
    // DMA: Copy next batch to GPU (runs on copy engine)
    cudaMemcpyAsync(d_input[i%2], h_input[i], size,
                    cudaMemcpyHostToDevice, copy_stream);
    
    // Compute on previously loaded batch (runs on SMs)
    if (i > 0)
        process<<<grid, block, 0, compute_stream>>>(d_input[(i-1)%2], d_output);
    
    // Ensure copy finishes before next compute uses this buffer
    cudaEventRecord(event, copy_stream);
    cudaStreamWaitEvent(compute_stream, event);
}

Pinned (Page-Locked) Memory

• Normal malloc: Pages can be swapped to disk → DMA engine can't reliably access.
• Pinned memory (cudaMallocHost): Locked in physical RAM → GPU DMA can directly access.
• Performance: Pinned memory transfers are 2-3× faster than pageable memory.
• Cost: Pinned memory reduces available system RAM (can't be swapped).

Zero-Copy Memory

// Allocate mapped memory (accessible by both CPU and GPU)
float *h_data;
cudaHostAlloc(&h_data, size,
              cudaHostAllocMapped | cudaHostAllocWriteCombined);

// Get device pointer to same physical memory
float *d_data;
cudaHostGetDevicePointer(&d_data, h_data, 0);

// GPU kernel reads/writes host memory directly — no explicit copy
my_kernel<<<grid, block>>>(d_data);  // Accesses over PCIe on demand

• No explicit memcpy needed → data accessed over PCIe on demand.
• Good for: Sparse access patterns, small data, integrated GPUs (shared memory).
• Bad for: Large sequential access (PCIe latency per access vs. bulk DMA).

GPUDirect Storage (GDS)

 Without GDS: NVMe → kernel buffer → user buffer → GPU (3 copies)
 With GDS:    NVMe → GPU directly (DMA, 1 copy, CPU bypass)

• NVMe reads DMA directly into GPU memory → bypass CPU entirely.
• Throughput: 100+ GB/s from NVMe array → GPU.
• Use case: Loading training data, checkpoints, large datasets.

NVIDIA Copy Engines

• Modern GPUs have 2-6 independent copy engines.
• Can run simultaneously: H2D on engine 0, D2H on engine 1, compute on SMs.
• Triple-buffering: Load batch N+1, compute batch N, write results of batch N-1 → all concurrent.

DMA engines and zero-copy transfers are the data movement infrastructure that enables efficient heterogeneous computing — by decoupling data transfer from computation and eliminating unnecessary copies, DMA-based approaches ensure that the CPU, GPU, NIC, and storage devices can all operate concurrently, maximizing system throughput and keeping expensive accelerators fed with data rather than waiting idle for transfers to complete.