gpu scheduling heterogeneous
**Heterogeneous Task Scheduling** is the **algorithmic and runtime discipline of assigning computational tasks to the most appropriate processing element — CPU cores, GPU compute units, FPGAs, or specialized accelerators — based on task characteristics, device capabilities, and system-wide optimization objectives** such as throughput, latency, energy efficiency, and fairness.
Modern computing platforms are fundamentally heterogeneous: a single node may contain multi-core CPUs, discrete GPUs, integrated GPUs, NPUs, and FPGAs. Efficiently utilizing all resources simultaneously requires scheduling algorithms far more sophisticated than traditional homogeneous CPU scheduling.
**Scheduling Dimensions**:
| Dimension | Options | Impact |
|-----------|---------|--------|
| Device selection | CPU vs GPU vs accelerator | Throughput, energy |
| Task granularity | Kernel, sub-task, pipeline stage | Overhead vs utilization |
| Data placement | Host RAM, GPU VRAM, unified | Transfer cost |
| Preemption | Cooperative vs preemptive | Latency, fairness |
| Priority | Deadline, throughput, fairness | QoS guarantees |
**CPU-GPU Co-Scheduling Strategies**:
- **Static partitioning**: Assign task types to devices at compile time or configuration time. Simple but cannot adapt to runtime workload variation.
- **Dynamic work-stealing**: Idle devices steal work from busy devices' queues. Requires portable task representations (e.g., OpenCL kernels that run on both CPU and GPU).
- **Predictive scheduling**: Profile task execution time on each device, use performance models to assign tasks to minimize total completion time. Accounts for data transfer overhead and device contention.
- **Feedback-driven**: Monitor actual execution times and adjust device allocation ratios online. EMA (exponential moving average) smoothing handles variability.
**GPU Scheduling Specifics**: GPU scheduling operates at multiple levels: **application-level** (which kernels to launch and when), **driver-level** (ordering kernel submissions in hardware queues), **hardware-level** (SM/CU allocation among concurrent kernels via MPS or hardware partitioning). GPU preemption granularity varies: NVIDIA supports context-level preemption and instruction-level preemption (since Pascal), enabling real-time GPU sharing.
**Frameworks and Runtimes**: **CUDA MPS** (Multi-Process Service) enables spatial GPU sharing among processes; **NVIDIA MIG** (Multi-Instance GPU) provides hardware-isolated GPU partitions; **AMD ROCm** supports similar multi-tenancy; **OpenCL** provides device-agnostic task dispatch; **StarPU** and **Legion** offer task-based heterogeneous runtimes with automatic data management; and **Kubernetes device plugins** handle cluster-level GPU scheduling.
**Challenges**: **Performance portability** — the same algorithm may have 10-100x different performance on CPU vs GPU; **data gravity** — moving data between devices costs time and energy (PCIe ~32 GB/s vs GPU memory ~3 TB/s); **tail latency** — heterogeneous execution creates variable completion times that complicate deadline guarantees; and **resource fragmentation** — partially utilizing multiple devices may be worse than fully utilizing one.
**Heterogeneous scheduling is the key to unlocking the full computational potential of modern hardware — systems that intelligently match workloads to devices can achieve 2-5x higher throughput and energy efficiency compared to naive CPU-only or GPU-only execution.**