gatedcnn

**Gated CNN** is a **convolutional architecture that uses gated linear units (GLU) instead of standard activation functions** — enabling content-dependent feature selection through learned multiplicative gates, achieving competitive results with RNNs on sequence modeling tasks. **How Does Gated CNN Work?** - **Architecture**: Standard 1D convolutions (for sequence data), but each layer uses GLU activation. - **Residual Connections**: Combined with residual/skip connections for gradient flow. - **Parallel**: Unlike RNNs, all positions are computed in parallel -> much faster training. - **Paper**: Dauphin et al., "Language Modeling with Gated Convolutional Networks" (2017). **Why It Matters** - **Pre-Transformer**: Demonstrated that CNNs with gating could match LSTM performance on language modeling. - **Speed**: Fully parallelizable — 10-20x faster training than equivalent LSTMs. - **Influence**: The gating mechanism directly influenced the FFN design in modern transformers (SwiGLU). **Gated CNN** is **the convolutional language model** — proving that convolutions with gates could challenge the RNN dominance in sequence modeling.