Heterogeneous Task Scheduling is the algorithmic and runtime discipline of assigning computational tasks to the most appropriate processing element — CPU cores, GPU compute units, FPGAs, or specialized accelerators — based on task characteristics, device capabilities, and system-wide optimization objectives such as throughput, latency, energy efficiency, and fairness.

Modern computing platforms are fundamentally heterogeneous: a single node may contain multi-core CPUs, discrete GPUs, integrated GPUs, NPUs, and FPGAs. Efficiently utilizing all resources simultaneously requires scheduling algorithms far more sophisticated than traditional homogeneous CPU scheduling.

Scheduling Dimensions:

CPU-GPU Co-Scheduling Strategies:

• Static partitioning: Assign task types to devices at compile time or configuration time. Simple but cannot adapt to runtime workload variation.
• Dynamic work-stealing: Idle devices steal work from busy devices' queues. Requires portable task representations (e.g., OpenCL kernels that run on both CPU and GPU).
• Predictive scheduling: Profile task execution time on each device, use performance models to assign tasks to minimize total completion time. Accounts for data transfer overhead and device contention.
• Feedback-driven: Monitor actual execution times and adjust device allocation ratios online. EMA (exponential moving average) smoothing handles variability.

GPU Scheduling Specifics: GPU scheduling operates at multiple levels: application-level (which kernels to launch and when), driver-level (ordering kernel submissions in hardware queues), hardware-level (SM/CU allocation among concurrent kernels via MPS or hardware partitioning). GPU preemption granularity varies: NVIDIA supports context-level preemption and instruction-level preemption (since Pascal), enabling real-time GPU sharing.

Frameworks and Runtimes: CUDA MPS (Multi-Process Service) enables spatial GPU sharing among processes; NVIDIA MIG (Multi-Instance GPU) provides hardware-isolated GPU partitions; AMD ROCm supports similar multi-tenancy; OpenCL provides device-agnostic task dispatch; StarPU and Legion offer task-based heterogeneous runtimes with automatic data management; and Kubernetes device plugins handle cluster-level GPU scheduling.

Challenges: Performance portability — the same algorithm may have 10-100x different performance on CPU vs GPU; data gravity — moving data between devices costs time and energy (PCIe ~32 GB/s vs GPU memory ~3 TB/s); tail latency — heterogeneous execution creates variable completion times that complicate deadline guarantees; and resource fragmentation — partially utilizing multiple devices may be worse than fully utilizing one.

Heterogeneous scheduling is the key to unlocking the full computational potential of modern hardware — systems that intelligently match workloads to devices can achieve 2-5x higher throughput and energy efficiency compared to naive CPU-only or GPU-only execution.