repeated augmentation
**Repeated Augmentation** is an **aggressive, counterintuitive data loading strategy specifically developed for training data-hungry Vision Transformers — deliberately violating the sacred Independent and Identically Distributed (IID) assumption of Stochastic Gradient Descent by including multiple differently augmented copies of the exact same source image within a single training mini-batch to force instantaneous invariance learning.**
**The Standard Data Loading**
- **The IID Assumption**: In standard deep learning training, each mini-batch of (e.g.) 256 images is sampled uniformly and independently from the entire training set. Every image in the batch is a unique, distinct photograph. This statistical independence is the foundational mathematical requirement that guarantees unbiased gradient estimates.
- **The ViT Data Hunger**: Vision Transformers, lacking the inductive biases of CNNs (locality, translation invariance), require enormously diverse training signals to learn robust features. Standard IID sampling from datasets like ImageNet (1.2M images) often provides insufficient diversity per gradient step.
**The Repeated Augmentation Strategy**
- **The Violation**: Instead of sampling 256 unique images, Repeated Augmentation samples only 64 unique source images. Each source image is then independently processed through the stochastic augmentation pipeline 4 times, producing 4 completely different visual versions: Version A (randomly cropped, horizontally flipped), Version B (color-jittered, rotated), Version C (cutout-applied, resized), Version D (mixup-blended, grayscale-converted). All four versions appear in the same 256-image mini-batch.
- **The Instantaneous Invariance**: When the loss function computes the gradient across this mini-batch, the optimizer simultaneously sees the same underlying dog photographed with dramatically different augmentations. The resulting gradient is mathematically forced to identify features that are stable across all four transformations — because those are the only features that consistently predict "dog" for all four radically different pixel patterns.
- **The Empirical Impact**: DeiT training demonstrates that Repeated Augmentation (with repetition factor 3) provides a consistent $+0.3\%$ to $+0.5\%$ accuracy improvement on ImageNet, effectively simulating a much larger and more diverse dataset within each gradient step.
**The IID Paradox**
Despite violating the theoretical IID requirement, Repeated Augmentation works empirically because the aggressively stochastic augmentation pipeline ensures that the four copies of the same source image are statistically more different from each other than four randomly sampled but weakly augmented unique images would be.
**Repeated Augmentation** is **instant comparative learning** — forcing the student to solve the same exam question written in four completely different fonts simultaneously, guaranteeing that the learned solution is invariant to superficial presentation rather than dependent on a single visual encoding.