Xlnet - Autoregressive Pretraining

# XLNet - Autoregressive Pretraining

## Introduction & Motivation

XLNet: permutation language modeling for autoregressive pre-training. Overcomes BERT limitations. Applications: improved generalization, flexible target prediction.

Motivation: Combine benefits of autoregressive and denoising approaches.

Applications: Text generation, language understanding, translation.

---

## Core Concepts & Theory

### Permutation Language Modeling

Train on all factorizations of target sequences.

### Recurrence Mechanism

Segment-level recurrence for longer sequences.

### Relative Position Bias

Position-aware attention without absolute positions.

### Two-Stream Attention

Query and content streams for target prediction.

---

## Mathematical Formulation

Permutation Objective:
$$\mathcal{L}_{ ext{PLM}} = -\mathbb{E}_{z \sim S_T}[\sum_t \log p(x_t | x_{z_t})]$$

Two-Stream Self-Attention:
$$ ext{query}(h) = W_q h, \quad ext{content}(h) = W_c h$$

Relative Position Bias:
$$\alpha_{ij} = \frac{(q_i + b_i)^T k_j}{\sqrt{d}}$$

---

## Advanced Theory & Extensions

### Segment-Level Recurrence

Context beyond single segment.

### Multi-Segment Input

Handling multiple segments.

### PLM Sampling

Efficient permutation sampling.

---

## Computational Considerations

Permutation space: O(T!).

Sampling: O(T·D²).

Two-stream: O(2·T·D²).

---

## Practical Implementation Strategies

### Partial Prediction

Predict only subset of tokens.

### Relative Position Encoding

Efficient position representation.

### Segment Recurrence

Memory-augmented attention.

---

## Benchmark Datasets & Evaluation

GLUE: General understanding benchmark.

SuperGLUE: Difficult understanding tasks.

SQuAD: Machine reading.

---

## Key Challenges & Limitations

### Computational Cost

Permutation sampling overhead.

### Training Stability

Complex optimization landscape.

### Memory Requirements

Recurrence mechanism memory.

---

## Hyperparameter Tuning

Permutation fraction: 0.5-1.0.

Segment length: 256-512.

Relative position dim: 32-64.

---

## Real-World Applications & Case Studies

Text Generation: Controlled generation.

Machine Reading: SQuAD-style QA.

Language Understanding: GLUE tasks.

---

## Integration with Other Methods

XLNet + segment recurrence; + relative positions for efficiency.

---

## Summary & Key Takeaways

XLNet applies permutation language modeling to autoregressive pre-training.

Principles:
1. Permutation modeling: All factorizations.
2. Two-stream: Query-content separation.
3. Recurrence: Extended context.
4. Relative positions: Flexible positioning.
5. Hybrid approach: Autoregressive + denoising.

---

## Appendix: Practical Labs

### Lab 1: Permutation Sampling

import numpy as np
import itertools

def sample_permutation(sequence_len, sample_size=None):
 """Sample random permutation of sequence"""
 if sample_size is None:
 perm = np.random.permutation(sequence_len)
 else:
 full_perm = list(range(sequence_len))
 np.random.shuffle(full_perm)
 perm = full_perm[:sample_size]
 return perm

np.random.seed(42)
perm = sample_permutation(10)
assert len(perm) == 10 and len(set(perm)) == 10
print("✓ Permutation sampling working")

### Lab 2: Two-Stream Attention

import numpy as np

def two_stream_attention(query_stream, content_stream, values):
 """Compute query and content stream outputs"""
 query_out = query_stream @ values.T
 content_out = content_stream @ values.T
 return query_out, content_out

np.random.seed(42)
query = np.random.randn(10, 768)
content = np.random.randn(10, 768)
values = np.random.randn(10, 768)
q_out, c_out = two_stream_attention(query, content, values)
assert q_out.shape == (10, 10)
print("✓ Two-stream attention working")

### Lab 3: Relative Position Bias

import numpy as np

def relative_position_bias(seq_len, dim=64):
 """Compute relative position bias matrix"""
 positions = np.arange(seq_len)
 relative_pos = positions[:, np.newaxis] - positions[np.newaxis, :]
 
 # Bias computation
 bias = np.random.randn(seq_len, seq_len, dim)
 return relative_pos, bias

np.random.seed(42)
rel_pos, bias = relative_position_bias(10, dim=64)
assert rel_pos.shape == (10, 10)
print("✓ Relative position bias working")

### Lab 4: Partial Prediction Target

def partial_prediction_target(sequence, predict_prob=0.15):
 """Select subset of tokens for target prediction"""
 targets = []
 target_indices = []
 for i, token in enumerate(sequence):
 if np.random.random() < predict_prob:
 targets.append(token)
 target_indices.append(i)
 return targets, target_indices

np.random.seed(42)
seq = ['The', 'cat', 'sat', 'on', 'the', 'mat']
targets, indices = partial_prediction_target(seq)
assert len(targets) > 0
print(f"✓ Partial prediction: {len(targets)} targets")

---