xlnet permutation language modeling

**XLNet** is a **generalized autoregressive language model that uses permutation language modeling** — instead of predicting tokens left-to-right, XLNet learns to predict each token conditioned on ALL OTHER tokens by training on random permutations of the input order, combining the advantages of autoregressive and bidirectional models. **XLNet Key Ideas** - **Permutation LM**: During training, randomly permute the token order — the model learns to predict each token conditioned on any subset of other tokens. - **Two-Stream Attention**: Content stream (standard attention) and query stream (cannot see the target token) — enables position-aware prediction. - **Transformer-XL Backbone**: Uses segment-level recurrence and relative positional encoding from Transformer-XL — captures long-range dependencies. - **No [MASK] Token**: Unlike BERT, XLNet doesn't use [MASK] tokens — avoids the pretrain-finetune discrepancy. **Why It Matters** - **Bidirectional Context**: XLNet captures bidirectional context WITHOUT the [MASK] token mismatch of BERT — theoretically more principled. - **Performance**: Outperformed BERT on many NLP benchmarks at the time of publication — especially on long documents. - **Autoregressive**: Maintains autoregressive properties — can compute exact likelihoods, unlike masked LMs. **XLNet** is **autoregressive meets bidirectional** — using permutation language modeling to capture full bidirectional context within an autoregressive framework.