gpu tensor core
**GPU Tensor Cores** are the **specialized matrix multiplication accelerator units embedded in modern GPU architectures (NVIDIA Volta and later, AMD Matrix Cores) that perform small matrix multiply-accumulate operations (e.g., 4×4×4 or 16×16×16) in a single clock cycle at throughput rates 8-16x higher than standard floating-point units — enabling the massive FLOPS numbers (hundreds of TFLOPS) required for deep learning training and inference**.
**Architecture**
Each Tensor Core performs a D = A × B + C operation on small matrix tiles in one cycle:
- **Input matrices A, B**: FP16, BF16, TF32, FP8, or INT8 depending on generation.
- **Accumulator matrix C/D**: FP32 or FP16 for higher precision accumulation.
- **Throughput**: NVIDIA H100 delivers 989 TFLOPS at FP16 Tensor and 1,979 TFLOPS at FP8. Standard FP32 ALUs deliver 67 TFLOPS — a 15-30x gap.
**Hardware Generations**
| Generation | GPU | Tile Size | Precisions | Peak TFLOPS |
|-----------|-----|-----------|-----------|-------------|
| Volta (1st gen) | V100 | 4×4×4 | FP16→FP32 | 125 |
| Turing (2nd gen) | T4/RTX 2080 | 4×4×4 | FP16,INT8,INT4 | 130 |
| Ampere (3rd gen) | A100 | 8×4×8+ | FP16,BF16,TF32,FP64 | 312 |
| Hopper (4th gen) | H100 | 16×16×16 | FP16,BF16,FP8,INT8 | 989 |
| Blackwell (5th gen) | B200 | larger | FP4,FP6,FP8 | 4,500 |
**Programming Model**
- **WMMA (Warp Matrix Multiply-Accumulate)**: CUDA API where a warp cooperatively loads matrix fragments from shared memory, performs the MMA operation, and stores the result. Each thread in the warp holds a portion of the matrix fragments.
- **MMA PTX Instructions**: Lower-level interface giving finer control over tile sizes and data layouts.
- **cuBLAS/cuDNN**: High-level libraries that automatically use Tensor Cores for GEMM and convolution operations. The recommended interface for most users — library kernels are highly tuned for each GPU generation.
**Mixed Precision Training**
Tensor Cores enable mixed-precision training: forward and backward passes compute in FP16/BF16 (fast Tensor Core operations), while master weights are maintained in FP32 for gradient accumulation accuracy. Loss scaling prevents gradient underflow in FP16. Result: 2-3x training speedup with negligible accuracy loss.
**Feeding Tensor Cores**
Tensor Cores can execute faster than data arrives from memory. Efficient utilization requires:
- **Shared Memory Tiling**: Load input matrices from global memory into shared memory tiles, then feed tiles to Tensor Cores. The software pipeline of load→compute→store must overlap to hide latency.
- **Large Tile Sizes**: Larger GEMM dimensions improve Tensor Core utilization. Small matrix operations (batch size 1 inference) under-utilize Tensor Cores and are better served by standard ALUs.
- **Data Layout**: Tensor Cores expect specific data layouts (column-major fragments). Memory access patterns must align with these requirements.
**GPU Tensor Cores are the silicon embodiment of the observation that neural network computation is dominated by matrix multiplication** — purpose-built hardware that delivers an order of magnitude more throughput for the single operation class that matters most for AI workloads.