xlnet

XLNet uses permutation language modeling to capture bidirectional context while maintaining autoregressive pre-training benefits. **Problem addressed**: BERT uses artificial MASK tokens not present at fine-tuning (pre-train/fine-tune discrepancy). Autoregressive models miss bidirectional context. **Solution**: Train on all permutations of token orderings. Each token sees different random subsets of other tokens as context. **Permutation LM**: For sequence [1,2,3,4], might use order [3,1,4,2], so position 2 sees positions 3,1,4 as context. **Two-stream attention**: Target-aware representations that know position but not content of token being predicted. **Segment recurrence**: Carry hidden states across segments for longer context, inspired by Transformer-XL. **Results**: Outperformed BERT on 20 benchmarks when released. Strong performance across tasks. **Complexity**: More complex than BERT, harder to implement and train. **Current status**: Influential but largely superseded by simpler approaches that scale better. Showed creative alternatives to MLM were possible.