t2t-vit (tokens-to-token vit)

**T2T-ViT (Tokens-to-Token Vision Transformer)** is a vision Transformer architecture that introduces a progressive tokenization module that iteratively aggregates neighboring tokens before feeding them to the Transformer backbone, addressing ViT's inefficient tokenization that treats each non-overlapping patch independently. T2T-ViT's tokenization process captures local structure and reduces token count progressively, producing more informative tokens for the subsequent Transformer layers. **Why T2T-ViT Matters in AI/ML:** T2T-ViT demonstrated that **tokenization strategy dramatically impacts ViT performance**, showing that progressive, overlapping token aggregation captures local structures that single-step non-overlapping patching misses, substantially improving data efficiency and accuracy. • **Tokens-to-Token module** — The T2T module alternates between token reconstruction (unfold tokens back to a 2D spatial grid and take overlapping windows) and token aggregation (apply self-attention or MLP, then re-tokenize the output into fewer, more informative tokens); this is repeated 2-3 times before the main Transformer • **Progressive token reduction** — Each T2T step reduces the token count by taking overlapping windows with stride < window_size, gradually building up tokens that encode increasingly larger spatial neighborhoods: Raw patches → T2T step 1 → T2T step 2 → Final tokens for Transformer backbone • **Local structure encoding** — Unlike ViT (which independently embeds each patch), T2T's overlapping windows ensure that neighboring patch information is aggregated into each token, capturing edges, textures, and local patterns that span patch boundaries • **Efficient backbone** — The T2T module produces fewer, richer tokens than standard ViT, enabling the subsequent Transformer backbone to be deeper but narrower (fewer tokens × more layers), achieving better accuracy-computation tradeoffs • **Architecture search** — T2T-ViT explored various Transformer backbone configurations inspired by CNN design principles: deep-narrow (like ResNet), wide-shallow, and dense connections, finding that deep-narrow designs work best for the enriched T2T tokens | Property | T2T-ViT | ViT | DeiT | |----------|---------|-----|------| | Tokenization | Progressive T2T (3 steps) | Single linear projection | Single linear projection | | Token Overlap | Yes (overlapping windows) | No (non-overlapping patches) | No | | Local Structure | Captured by T2T | Lost at boundaries | Lost at boundaries | | Final Token Count | Reduced (~196→~64-196) | Fixed (~196 for 16×16) | Fixed (~196) | | ImageNet Top-1 | 81.5% (T2T-ViT-14) | 77.9% (ViT-B/16 IN-1K) | 81.2% (DeiT-B) | | Parameters | 21.5M (T2T-14) | 86M (ViT-B) | 86M (DeiT-B) | **T2T-ViT demonstrated that ViT's inefficient patch-level tokenization is a significant bottleneck, and progressive token aggregation that captures local structure through overlapping windows substantially improves both parameter efficiency and accuracy, establishing tokenization design as a first-class architectural concern for vision Transformers.**