Latency Hiding is the parallel computing technique of overlapping computation with data movement (memory loads, network communication, disk I/O) so that the processor is never idle waiting for data — using mechanisms like prefetching, double buffering, multithreading, and pipeline parallelism to mask the latency of slow operations behind useful computation, which is the fundamental strategy that makes both GPUs and modern CPUs achieve high throughput despite memory latencies being 100-1000× longer than computation time.

The Latency Problem

• GPU SM compute: ~1 ns per FLOP.
• HBM memory access: ~200-400 ns.
• PCIe transfer: ~1-5 µs.
• Network (InfiniBand): ~1-5 µs.
• Ratio: Memory is 200-400× slower than compute → GPU would be idle 99%+ of the time without latency hiding.

Latency Hiding Techniques

GPU Thread-Level Latency Hiding

• GPU has thousands of warps ready to execute.
• When warp A stalls on memory → scheduler switches to warp B (zero-cost switch).
• While warp B computes → warp A's memory request completes.
• More warps (higher occupancy) → more opportunities to hide latency.
• This is why GPUs need thousands of threads: Not for parallelism alone, but for latency hiding.

Double Buffering

# Without double buffering:
for batch in dataset:
    data = load(batch)      # CPU idle during load
    result = compute(data)  # GPU idle during next load

# With double buffering:
buffer_a = load(batch_0)    # Initial load
for i in range(1, N):
    buffer_b = async_load(batch_i)  # Load next batch
    compute(buffer_a)               # Compute current batch (overlapped)
    swap(buffer_a, buffer_b)        # Swap buffers
compute(buffer_a)           # Process last batch

• Pipeline: While GPU processes batch N, CPU/DMA loads batch N+1.
• Result: Load time hidden behind compute → effective throughput = max(compute, load).

Communication-Computation Overlap in ML Training

Forward:  [Layer 1 → Layer 2 → Layer 3 → Layer 4]
Backward: [Grad 4 → Grad 3 → Grad 2 → Grad 1]
                ↓AllReduce    ↓AllReduce

• Start AllReduce for gradient of layer 4 while computing gradient of layer 3.
• By the time backward pass completes, most gradients are already synchronized.
• Overlap hides 60-80% of communication time → near-linear scaling.

Hardware Prefetching (CPU)

• Hardware detects sequential access pattern → prefetches next cache line.
• Software prefetch: __builtin_prefetch(addr) → hint to load data before needed.
• L1 prefetch distance: ~16-32 cache lines ahead.
• Critical for: Array traversal, matrix operations, data streaming.

Async CUDA Operations

// Overlap transfer and compute using CUDA streams
cudaStream_t stream_compute, stream_transfer;
cudaMemcpyAsync(d_next, h_next, size, H2D, stream_transfer);
my_kernel<<<grid, block, 0, stream_compute>>>(d_current);
cudaDeviceSynchronize();
// Transfer and compute happen simultaneously

Latency hiding is the single most important principle in high-performance computing — it is why GPUs with 200ns memory latency achieve 80%+ compute utilization, why distributed training scales to thousands of GPUs despite microsecond network latencies, and why modern CPUs run at near-peak throughput despite the memory wall, making latency hiding techniques the foundational skill that separates competent from expert parallel programmers.