GPU Tensor Cores are the specialized matrix multiplication accelerator units embedded in modern GPU architectures (NVIDIA Volta and later, AMD Matrix Cores) that perform small matrix multiply-accumulate operations (e.g., 4×4×4 or 16×16×16) in a single clock cycle at throughput rates 8-16x higher than standard floating-point units — enabling the massive FLOPS numbers (hundreds of TFLOPS) required for deep learning training and inference.

Architecture

Each Tensor Core performs a D = A × B + C operation on small matrix tiles in one cycle:

• Input matrices A, B: FP16, BF16, TF32, FP8, or INT8 depending on generation.
• Accumulator matrix C/D: FP32 or FP16 for higher precision accumulation.
• Throughput: NVIDIA H100 delivers 989 TFLOPS at FP16 Tensor and 1,979 TFLOPS at FP8. Standard FP32 ALUs deliver 67 TFLOPS — a 15-30x gap.

Hardware Generations

Programming Model

• WMMA (Warp Matrix Multiply-Accumulate): CUDA API where a warp cooperatively loads matrix fragments from shared memory, performs the MMA operation, and stores the result. Each thread in the warp holds a portion of the matrix fragments.
• MMA PTX Instructions: Lower-level interface giving finer control over tile sizes and data layouts.
• cuBLAS/cuDNN: High-level libraries that automatically use Tensor Cores for GEMM and convolution operations. The recommended interface for most users — library kernels are highly tuned for each GPU generation.

Mixed Precision Training

Tensor Cores enable mixed-precision training: forward and backward passes compute in FP16/BF16 (fast Tensor Core operations), while master weights are maintained in FP32 for gradient accumulation accuracy. Loss scaling prevents gradient underflow in FP16. Result: 2-3x training speedup with negligible accuracy loss.

Feeding Tensor Cores

Tensor Cores can execute faster than data arrives from memory. Efficient utilization requires:

• Shared Memory Tiling: Load input matrices from global memory into shared memory tiles, then feed tiles to Tensor Cores. The software pipeline of load→compute→store must overlap to hide latency.
• Large Tile Sizes: Larger GEMM dimensions improve Tensor Core utilization. Small matrix operations (batch size 1 inference) under-utilize Tensor Cores and are better served by standard ALUs.
• Data Layout: Tensor Cores expect specific data layouts (column-major fragments). Memory access patterns must align with these requirements.

GPU Tensor Cores are the silicon embodiment of the observation that neural network computation is dominated by matrix multiplication — purpose-built hardware that delivers an order of magnitude more throughput for the single operation class that matters most for AI workloads.