heterogeneous computing cpu gpu

**Heterogeneous Computing** is the **system architecture and programming paradigm that combines different processor types (CPUs, GPUs, FPGAs, NPUs, DSPs) in a single system, dispatching each computation to the processor type best suited for it — exploiting the CPU's strength in serial, branch-heavy code and the GPU's strength in massively parallel, data-parallel workloads to achieve performance and energy efficiency beyond what any single processor type can deliver**. **Why Heterogeneous** No single processor architecture is optimal for all workloads: - **CPU**: Fast single-thread, branch prediction, cache hierarchy, low-latency memory access. Best for: serial code, control flow, OS operations, small tasks. - **GPU**: Massive throughput, thousands of cores, high memory bandwidth. Best for: data-parallel computation, matrix operations, image/signal processing. - **FPGA**: Reconfigurable logic, custom pipelines, deterministic latency. Best for: streaming data processing, network functions, custom protocols. - **NPU/TPU**: Matrix multiply accelerator, low-precision arithmetic. Best for: ML inference at maximum efficiency. **Programming Models** - **CUDA**: NVIDIA GPU-specific. Highest performance on NVIDIA hardware. Largest ecosystem, best tooling. Not portable. - **OpenCL**: Open standard for heterogeneous computing. Write-once, run on CPUs, GPUs (NVIDIA, AMD, Intel), FPGAs, DSPs. Verbose API, lower abstraction than CUDA. - **SYCL**: Modern C++ single-source programming for heterogeneous devices. Host and device code in the same C++ source file. Intel oneAPI DPC++ is the primary SYCL implementation. Targets Intel GPUs, NVIDIA GPUs (via plugins), FPGAs. - **HIP (AMD)**: AMD's GPU programming model. API-compatible with CUDA — HIPIFY tool converts CUDA code to HIP with minimal changes. Runs on AMD GPUs natively, NVIDIA GPUs via HIP-CUDA translation. - **Unified Shared Memory (USM)**: Modern heterogeneous programming models (SYCL, CUDA Unified Memory) provide a single address space accessible by all devices. Data migration handled by runtime or hardware page faults. **Workload Partitioning Strategies** - **Offload Model**: CPU is the host; GPU is the accelerator. CPU launches GPU kernels for parallel sections, processes results serially. The dominant pattern (CUDA, OpenCL). Overhead: kernel launch latency, data transfer. - **Task-Based Partitioning**: Each task in a DAG is assigned to the optimal device. CPU tasks and GPU tasks execute concurrently. Runtime systems (StarPU, OmpSs) schedule tasks dynamically. - **Streaming Partition**: Pipeline stages assigned to different devices. Stage 1 (preprocessing) on CPU → Stage 2 (computation) on GPU → Stage 3 (postprocessing) on CPU. Stages execute concurrently on different data batches. **Performance Considerations** - **Data Transfer Overhead**: PCIe: 12-32 GB/s, 1-5 μs latency. CXL: 32-64 GB/s, sub-μs. NVLink CPU-GPU: 450-900 GB/s. The cost of moving data between processors can negate the computational benefit of acceleration. - **Amdahl's Law**: If 90% of the workload is GPU-acceleratable, maximum speedup is 10×, regardless of GPU performance. The remaining serial fraction on CPU limits overall speedup. - **Roofline Overlap**: The optimal device depends on arithmetic intensity. Memory-bound workloads may run equally fast on CPU and GPU; compute-bound workloads see dramatic GPU acceleration. Heterogeneous Computing is **the hardware-software co-design paradigm that maximizes system-level performance by matching each computation to its ideal processor** — the recognition that the diversity of real-world workloads demands a diversity of processor architectures, unified by programming models that make the heterogeneity manageable.