T2T-ViT (Tokens-to-Token Vision Transformer) is a vision Transformer architecture that introduces a progressive tokenization module that iteratively aggregates neighboring tokens before feeding them to the Transformer backbone, addressing ViT's inefficient tokenization that treats each non-overlapping patch independently. T2T-ViT's tokenization process captures local structure and reduces token count progressively, producing more informative tokens for the subsequent Transformer layers.

Why T2T-ViT Matters in AI/ML: T2T-ViT demonstrated that tokenization strategy dramatically impacts ViT performance, showing that progressive, overlapping token aggregation captures local structures that single-step non-overlapping patching misses, substantially improving data efficiency and accuracy.

• Tokens-to-Token module — The T2T module alternates between token reconstruction (unfold tokens back to a 2D spatial grid and take overlapping windows) and token aggregation (apply self-attention or MLP, then re-tokenize the output into fewer, more informative tokens); this is repeated 2-3 times before the main Transformer • Progressive token reduction — Each T2T step reduces the token count by taking overlapping windows with stride < window_size, gradually building up tokens that encode increasingly larger spatial neighborhoods: Raw patches → T2T step 1 → T2T step 2 → Final tokens for Transformer backbone • Local structure encoding — Unlike ViT (which independently embeds each patch), T2T's overlapping windows ensure that neighboring patch information is aggregated into each token, capturing edges, textures, and local patterns that span patch boundaries • Efficient backbone — The T2T module produces fewer, richer tokens than standard ViT, enabling the subsequent Transformer backbone to be deeper but narrower (fewer tokens × more layers), achieving better accuracy-computation tradeoffs • Architecture search — T2T-ViT explored various Transformer backbone configurations inspired by CNN design principles: deep-narrow (like ResNet), wide-shallow, and dense connections, finding that deep-narrow designs work best for the enriched T2T tokens

T2T-ViT demonstrated that ViT's inefficient patch-level tokenization is a significant bottleneck, and progressive token aggregation that captures local structure through overlapping windows substantially improves both parameter efficiency and accuracy, establishing tokenization design as a first-class architectural concern for vision Transformers.