Peer-to-Peer (P2P) GPU Communication is the hardware and software capability that allows one GPU to directly access another GPU's memory without routing data through CPU main memory — eliminating the host memory copy bottleneck in multi-GPU systems, reducing transfer latency by 2-5×, and enabling programming models where GPUs transparently share data, essential for multi-GPU deep learning training, scientific simulation, and real-time rendering.

P2P Communication Paths

CUDA P2P API

// Check P2P support
int canAccess;
cudaDeviceCanAccessPeer(&canAccess, gpu0, gpu1);

// Enable P2P access
cudaSetDevice(gpu0);
cudaDeviceEnablePeerAccess(gpu1, 0);

// Direct copy between GPUs (no CPU staging)
cudaMemcpyPeer(dst_ptr_gpu1, gpu1, src_ptr_gpu0, gpu0, size);

// Async P2P copy
cudaMemcpyPeerAsync(dst, gpu1, src, gpu0, size, stream);

// Direct pointer access (Unified Virtual Addressing)
// GPU 0 kernel can dereference pointer to GPU 1 memory
my_kernel<<<grid, block>>>(gpu1_ptr);  // Access remote GPU memory

GPUDirect Technologies (NVIDIA)

GPUDirect RDMA (Network → GPU)

• InfiniBand NIC reads/writes GPU memory directly.
• MPI_Send from GPU → NIC grabs data directly from GPU memory → sends over network → remote NIC writes directly to remote GPU.
• No CPU copies in the data path → critical for distributed training.
• Requires: NVIDIA GPU + Mellanox/NVIDIA NIC + GPUDirect-aware driver.

P2P Topology Awareness

    GPU 0 ←NVLink→ GPU 1
      ↑               ↑
    NVLink           NVLink
      ↓               ↓
    GPU 2 ←NVLink→ GPU 3
      ↑               ↑
    PCIe             PCIe
      ↓               ↓
    CPU Socket 0    CPU Socket 1

• GPU 0 → GPU 1 (NVLink): ~600 GB/s, ~1 µs.
• GPU 0 → GPU 3 (NVLink via switch): ~600 GB/s, ~1.5 µs.
• GPU 0 → GPU on remote socket (PCIe): ~25 GB/s, ~5 µs.
• Training frameworks (PyTorch, DeepSpeed) should be topology-aware → minimize cross-socket transfers.

Impact on ML Training

• AllReduce: P2P NVLink → ring topology at 600 GB/s → fast gradient sync.
• Tensor parallelism: Each GPU holds fraction of layer → P2P required for activations.
• Expert parallelism (MoE): Tokens routed to expert GPUs → P2P for token transfer.
• Without P2P: All traffic goes through CPU → 10× slower → multi-GPU training impractical.

Peer-to-peer GPU communication is the physical foundation of multi-GPU computing — by enabling GPUs to share data at NVLink or PCIe speeds without CPU intermediation, P2P transforms a collection of discrete GPUs into a unified computational fabric where tensor and pipeline parallelism can operate at the bandwidth required by modern large-scale AI training.