Patch embedding is the linear projection layer that maps each flattened image patch from pixel space into a high-dimensional vector representation — converting raw RGB pixel values within each patch into dense feature vectors that serve as input tokens to the Vision Transformer encoder, analogous to word embeddings in natural language processing.
What Is Patch Embedding?
- Definition: A learnable linear transformation (typically implemented as a Conv2D layer) that projects each image patch from its raw pixel representation (e.g., 16×16×3 = 768 values) into a D-dimensional embedding vector (e.g., D = 768 for ViT-Base).
- Implementation: A Conv2D layer with kernel_size = patch_size and stride = patch_size simultaneously extracts patches and projects them — Conv2D(in_channels=3, out_channels=768, kernel_size=16, stride=16).
- Output: For a 224×224 image with 16×16 patches, the embedding layer produces 196 vectors of dimension D, forming the input sequence to the transformer.
- Learnable Weights: The embedding projection matrix is learned during training — the model discovers which linear combinations of pixel values create the most useful feature representations.
Why Patch Embedding Matters
- Dimensionality Alignment: Transforms variable-size patch pixel data into fixed-size vectors matching the transformer's hidden dimension, enabling standard transformer processing.
- Feature Extraction: The learned projection captures basic visual features (edges, colors, textures) within each patch — functioning like the first convolutional layer of a CNN but without the sliding window.
- Information Compression: For ViT-Base, each 16×16×3 = 768 pixel values map to exactly 768 embedding dimensions — an isometric mapping that preserves information while restructuring it for transformer processing.
- Computational Efficiency: A single matrix multiplication per patch replaces the multi-layer feature extraction hierarchies used in CNNs.
- Foundation for Attention: The quality of patch embeddings directly affects the transformer's ability to compute meaningful attention patterns between patches — poor embeddings mean poor attention.
Patch Embedding Variants
Standard Linear Projection (ViT):
- Single Conv2D with large kernel matching patch size.
- Simplest and most common approach.
- Works well with sufficient pretraining data.
Convolutional Stem (Hybrid ViT):
- Replace single large-kernel conv with a small CNN stem (3-5 convolutional layers with small 3×3 kernels).
- Provides better low-level feature extraction and translation equivariance.
- Improves performance when pretraining data is limited.
Overlapping Patch Embedding (CvT, CMT):
- Use stride smaller than kernel size to create overlapping patches.
- Reduces information loss at patch boundaries.
- Slightly increases sequence length and compute cost.
Embedding Dimension Comparison
| Model | Patch Size | Embedding Dim | Patches (224²) | Params in Embedding |
|-------|-----------|--------------|-----------------|---------------------|
| ViT-Tiny | 16×16 | 192 | 196 | 147K |
| ViT-Small | 16×16 | 384 | 196 | 295K |
| ViT-Base | 16×16 | 768 | 196 | 590K |
| ViT-Large | 16×16 | 1024 | 196 | 786K |
| ViT-Huge | 14×14 | 1280 | 256 | 753K |
Position Embedding Addition
After patch embedding, a position embedding is added to each patch token to encode spatial location:
- Learned Position Embeddings: A separate learnable vector for each patch position — standard in original ViT.
- Sinusoidal Position Embeddings: Fixed mathematical encoding using sine and cosine functions.
- Without Position Embedding: The model loses all spatial information — it cannot distinguish a patch in the top-left from one in the bottom-right.
Tools & Frameworks
- PyTorch: timm library provides ViT implementations with configurable patch embedding layers.
- Hugging Face: transformers.ViTModel includes standard patch embedding as ViTEmbeddings.
- JAX/Flax: Google's scenic and big_vision repositories implement patch embedding for TPU training.
Patch embedding is the critical first transformation in every Vision Transformer — converting the continuous pixel world into discrete token representations that unlock the full power of self-attention for visual understanding.