XLNet is a generalized autoregressive language model that uses permutation language modeling — instead of predicting tokens left-to-right, XLNet learns to predict each token conditioned on ALL OTHER tokens by training on random permutations of the input order, combining the advantages of autoregressive and bidirectional models.

XLNet Key Ideas

• Permutation LM: During training, randomly permute the token order — the model learns to predict each token conditioned on any subset of other tokens.
• Two-Stream Attention: Content stream (standard attention) and query stream (cannot see the target token) — enables position-aware prediction.
• Transformer-XL Backbone: Uses segment-level recurrence and relative positional encoding from Transformer-XL — captures long-range dependencies.
• No [MASK] Token: Unlike BERT, XLNet doesn't use [MASK] tokens — avoids the pretrain-finetune discrepancy.

Why It Matters

• Bidirectional Context: XLNet captures bidirectional context WITHOUT the [MASK] token mismatch of BERT — theoretically more principled.
• Performance: Outperformed BERT on many NLP benchmarks at the time of publication — especially on long documents.
• Autoregressive: Maintains autoregressive properties — can compute exact likelihoods, unlike masked LMs.

XLNet is autoregressive meets bidirectional — using permutation language modeling to capture full bidirectional context within an autoregressive framework.