funnel transformer

**Funnel Transformer** is an **efficient transformer architecture that progressively reduces the sequence length through pooling layers** — similar to how CNNs reduce spatial resolution, creating a funnel-shaped computation graph that saves FLOPs on long sequences. **How Does Funnel Transformer Work?** - **Encoder**: Standard transformer blocks with periodic sequence length reduction (mean pooling every few layers). - **Decoder**: Upsamples back to full length for tasks requiring per-token predictions. - **Reduction**: Sequence length is halved at each reduction stage (e.g., 512 → 256 → 128). - **Paper**: Dai et al. (2020). **Why It Matters** - **Efficiency**: Processes long sequences with progressively fewer tokens -> significant FLOPs reduction. - **Classification**: For classification tasks, only the final (shortest) representation is needed -> no upsampling needed. - **Pre-Training**: Can be pre-trained like BERT but with lower compute cost for the same model quality. **Funnel Transformer** is **the CNN pyramid for transformers** — progressively compressing sequence length to focus computation on the most important information.