deberta

**DeBERTa** (Decoding-enhanced BERT with Disentangled Attention) is a **pre-trained language model that improves upon BERT by disentangling content and position representations** — computing separate attention for content-to-content, content-to-position, and position-to-content interactions. **Key Innovations of DeBERTa** - **Disentangled Attention**: Separate matrices for content (word) and position, with three attention components instead of one. - **Enhanced Mask Decoder (EMD)**: Uses absolute position information in the decoder layer for MLM prediction. - **Virtual Adversarial Training**: Fine-tuning with perturbation-based regularization. - **Paper**: He et al. (2021, Microsoft). **Why It Matters** - **SuperGLUE #1**: First model to surpass human baseline on the SuperGLUE benchmark. - **Disentanglement**: Separating content and position allows the model to learn cleaner representations. - **DeBERTaV3**: Subsequent versions with ELECTRA-style training further improved efficiency. **DeBERTa** is **BERT with separated content and position** — disentangling what a word means from where it appears for more powerful language understanding.