operator

Operators are the mathematical primitives that comprise neural network computations (matrix multiplication, convolution, attention), while kernels are the optimized hardware implementations of these operators, with performance-critical operators requiring extensive optimization for model efficiency. Common operators: linear/dense (matrix multiplication), convolution (sliding window operations), attention (softmax(QK^T)V), element-wise (activation functions, normalization), and reduction (sum, mean, max). Kernel implementation: translates operator semantics to specific hardware instructions; considers memory hierarchy, parallelism, vectorization, and instruction scheduling. Hot operators: profile to find which operators consume most time—typically attention and linear layers in transformers; focus optimization effort there. Optimization techniques: tiling (blocking for cache), fusion (combining operators to reduce memory traffic), quantization kernels (INT8, FP8 implementations), and hardware-specific intrinsics (Tensor Cores, AMX). Libraries: cuDNN, cuBLAS (NVIDIA), oneDNN (Intel), and custom kernels (Triton, CUTLASS). Kernel selection: runtime selects best kernel based on input shapes (autotune or heuristic). Custom kernels: Flash Attention reimplemented attention operator with dramatically better memory efficiency. Understanding operators and kernels is essential for ML systems engineers optimizing model performance.