zero copy data transfer gpu

**Zero-Copy and Pinned Memory for GPU Data Transfer** is the **memory management technique that eliminates redundant data copies between CPU and GPU memory by using page-locked (pinned) host memory that the GPU can access directly via DMA or memory-mapped access — achieving maximum PCIe/NVLink transfer bandwidth (25-900 GB/s), enabling overlap of data transfer with computation, and in some cases allowing the GPU to read CPU memory directly without any explicit copy**. **The Data Transfer Bottleneck** Standard cudaMemcpy involves: 1. OS copies data from user-space pageable memory to a pinned staging buffer. 2. DMA engine transfers from pinned buffer to GPU memory via PCIe. The extra copy through the staging buffer halves effective bandwidth and prevents DMA/compute overlap. **Pinned (Page-Locked) Memory** cudaMallocHost() or cudaHostAlloc() allocates memory that is locked in physical RAM (cannot be swapped to disk): - **DMA Direct**: The GPU DMA engine can directly access pinned memory via its physical address — no intermediate staging copy. - **Full PCIe Bandwidth**: Pinned transfers achieve 12-25 GB/s on PCIe Gen4 x16 vs. 6-12 GB/s for pageable memory. - **Async Transfer**: cudaMemcpyAsync() with pinned memory enables overlap of transfer and kernel execution using CUDA streams: while stream 1 transfers batch N+1 to GPU, stream 0 computes on batch N. - **Cost**: Pinned memory reduces available pageable memory for the OS. Excessive pinned allocation can cause swapping of other processes. Best practice: pin only what's actively transferred. **Write-Combined (WC) Memory** cudaHostAlloc(... cudaHostAllocWriteCombined) allocates pinned memory with write-combining: - CPU writes bypass L1/L2 cache and coalesce in WC buffers before flushing to memory. - Transfer to GPU is faster (PCIe bus is write-optimized). - CPU reads are very slow (not cached) — use only for CPU-write, GPU-read patterns. **Mapped (Zero-Copy) Memory** cudaHostAlloc(... cudaHostAllocMapped) maps pinned host memory into GPU's address space: - GPU accesses host memory directly via PCIe load/store — no explicit cudaMemcpy needed. - Each GPU load translates to a PCIe read (high latency: 1-5 μs per random access). - Useful for: sparse access patterns (GPU reads only a small fraction of a large host array), host-side data that changes between kernel launches, and systems where GPU memory is insufficient. - Performance: terrible for bulk access (PCIe bandwidth << GPU memory bandwidth). Good for meta-data access, small lookups, signaling between CPU and GPU. **NVLink and RDMA** - **NVLink CPU-GPU (Grace Hopper)**: 450-900 GB/s CPU-GPU bandwidth. Coherent memory sharing — both CPU and GPU access the same physical memory with hardware cache coherence. Zero-copy becomes the default — no transfer needed. - **GPUDirect RDMA**: Network adapter directly reads/writes GPU memory without going through CPU memory. Eliminates a CPU-memory hop for MPI and NCCL communication between GPUs on different nodes. Zero-Copy and Pinned Memory are **the memory optimization techniques that maximize the effective bandwidth of the CPU-GPU data pipeline** — ensuring that data movement, the dominant bottleneck for GPU-accelerated applications, operates at the theoretical limits of the interconnect rather than being throttled by unnecessary copies.