Vector Processing Units and SIMD Extensions is the processor hardware capability that applies a single instruction to multiple data elements simultaneously — the primary mechanism for exploiting data-level parallelism within a single CPU core. Modern CPUs achieve 4–16× throughput improvement for vectorizable workloads through SIMD (Single Instruction, Multiple Data) by processing 128–512 bits of data per instruction rather than 64 bits, enabling HPC, multimedia, AI inference, and signal processing applications to fully utilize CPU compute resources.

SIMD Evolution on x86

AVX-512 Detailed

• Operates on 512-bit registers (ZMM registers, ZMM0–ZMM31).
• 16 float32 or 8 float64 or 32 int16 or 64 int8 elements per register.
• FMA (Fused Multiply-Add): 2 ops per instruction → 32 FP32 FMAs per cycle.
• Gather/Scatter: Load/store non-contiguous data → _mm512_i32gather_ps() → enables sparse access.
• Mask registers: Per-element predicate → conditional element-wise operations → avoids branch divergence.
• Throttling concern: Intel AVX-512 lowers CPU frequency on some cores → net throughput gain for sustained SIMD-heavy workloads, not single short bursts.

RISC-V Vector Extension (RVV)

• Vector-length agnostic (VLA) design: Application specifies element type → hardware chooses vector length → same code runs on 64-bit and 512-bit SIMD implementations.
• vsetvli a0, a1, e32, m1: Set vector length for 32-bit elements → returns actual hardware VL.
• Configurable group multiplier (LMUL): Treat multiple vector registers as one → process more elements.
• Advantages over AVX-512: Portable across implementations, no frequency throttling concerns, cleaner mask model.
• Implementations: SiFive X280, T-Head C910, Ventana Veyron, RISC-V BWXT.

Vectorization in Practice (C++)

// Auto-vectorized by GCC/Clang with -O2 -march=native
void saxpy(float a, float* x, float* y, int n) {
    for (int i = 0; i < n; i++) {
        y[i] = a * x[i] + y[i]; // FMA instruction
    }
}
// Compiler: Generates vmovups + vfmadd231ps AVX-512 instructions

// Explicit SIMD with intrinsics
#include <immintrin.h>
void saxpy_avx512(float a, float* x, float* y, int n) {
    __m512 va = _mm512_set1_ps(a);
    for (int i = 0; i < n; i += 16) {
        __m512 vx = _mm512_loadu_ps(x + i);
        __m512 vy = _mm512_loadu_ps(y + i);
        vy = _mm512_fmadd_ps(va, vx, vy);
        _mm512_storeu_ps(y + i, vy);
    }
}

Auto-Vectorization Requirements

• Loop-carried independence: Each iteration must be independent (no x[i] = x[i-1] dependency).
• Aligned memory: 64-byte aligned for AVX-512 → use __attribute__((aligned(64))) or aligned_alloc().
• No aliasing: Source and destination arrays do not overlap → __restrict__ keyword.
• Simple control flow: No function calls, minimal branches inside loop body.

ARM Neon and SVE

• Neon: Fixed 128-bit SIMD (4× float32) → ARM Cortex-A and Apple Silicon.
• SVE (Scalable Vector Extension): ARM's VLA design (like RVV) — 128–2048 bit width hardware-configurable.
• SVE2: Extended operations for specialized workloads (cryptography, DSP).
• Apple M-series: AMX (Apple Matrix Coprocessor) — matrix multiply unit → 32 TOPS for machine learning.

AMX (Advanced Matrix Extensions, Intel)

• New x86 instruction for matrix operations in on-chip tile registers.
• tmul instruction: Multiply 16×16 BF16 tile matrices → 2048 BF16 FMAs per instruction.
• Used for: In-core ML inference, large-batch matrix operations.
• Performance: 1 AMX = ~64 AVX-512 FMA instructions for the same BF16 GEMM.

Vector processing units and SIMD extensions are the single-core parallelism engine that transforms modern CPUs from sequential processors into mini-GPUs — by processing 16 float32 values simultaneously with AVX-512, a single CPU core achieves 32× the throughput of scalar execution for vectorizable kernels, making SIMD the difference between a scientific code that runs in 1 hour and one that runs in 3 hours, and enabling CPUs to compete with dedicated accelerators for structured, vectorizable AI inference workloads.