Convolutional Neural Network

A convolutional neural network's convolution operation looks nothing like a matrix multiply on paper, but it has to become one before any GPU or AI accelerator can actually execute it — the gap between how convolution is defined mathematically and how it's run on real silicon is its own piece of engineering, separate from the architecture's design. Understanding that gap is what explains why certain chips (NVIDIA GPUs, Google TPUs, mobile NPUs) are built the way they are, and why a network's layer sizes are tuned as much for hardware efficiency as for accuracy. At a high level, turning a convolution into hardware-executable work looks like this:

im2col is the classic trick that made convolutions runnable on hardware built for matrix multiplication, by brute-force reshaping the problem into one. It takes every patch of the image a filter would slide over and copies each one out as a column of a new, much larger matrix; convolving the image with the filter then becomes a single ordinary matrix multiply between that unrolled matrix and the filter's weights, which is exactly the operation GPUs and Tensor Cores are built to execute at enormous throughput. The cost is real: im2col duplicates image data substantially (overlapping patches get copied multiple times), trading extra memory and memory bandwidth for the ability to reuse highly optimized general matrix-multiply (GEMM) hardware and software that already existed for other purposes.

Different chip families made genuinely different tradeoffs on how to run convolutions, and those choices show up directly in each chip's architecture. NVIDIA's Tensor Cores favor the im2col-to-GEMM path because GPUs already had enormous GEMM throughput for graphics and scientific computing; Google's TPU instead built a systolic array — a grid of multiply-accumulate units that pass partial sums directly to their neighbors in hardware — specifically because that layout maps extremely efficiently onto the repeated multiply-accumulate pattern convolutions and matrix multiplies share, cutting the memory traffic a GPU-style approach would otherwise require. Mobile and edge NPUs (Apple's Neural Engine, Qualcomm's Hexagon) go further still, often implementing convolution with dedicated fixed-function datapaths rather than a general matrix-multiply unit at all, because a phone's power budget can't afford a GPU-style general-purpose approach for the narrow set of operations a mobile vision model actually needs.

Winograd convolution is the other major optimization, and it works by algebra instead of hardware layout — reducing the actual number of multiplications a convolution needs, not just reshaping them. For small filter sizes (3x3 is the common case), Winograd's transform reduces the multiply count by roughly 2.25x compared to a naive implementation, at the cost of extra addition operations and some numerical precision loss; because multiplication is typically more expensive than addition on most hardware, this is a genuine net win for exactly the filter sizes modern CNN architectures (ResNet, MobileNet) standardized on, which is itself not a coincidence — architecture designers picked 3x3 filters partly because the hardware-optimization story for that specific size was already well understood.

Read a CNN's convolution not as a fixed mathematical operation but as a specification that different chips satisfy in genuinely different ways: im2col trades memory for GEMM-hardware reuse, systolic arrays trade hardware specialization for memory-traffic reduction, and Winograd trades numerical precision for fewer raw multiplications — three different answers to the same underlying question of how to make a sliding filter run fast on whatever silicon is actually available.