Causal vs. Masked Language Modeling are the two fundamental self-supervised pretraining objectives that determine how a language model learns from text — causal (autoregressive) models predict the next token given all previous tokens (GPT), while masked models predict randomly hidden tokens given bidirectional context (BERT), with each approach having distinct strengths that have shaped the modern AI landscape.

Causal Language Modeling (CLM / Autoregressive)

• Objective: Predict next token given all previous tokens.
• $P(x_1, x_2, ..., x_n) = \prod_{i=1}^{n} P(x_i | x_1, ..., x_{i-1})$
• Attention mask: Each token can only attend to tokens before it (causal/triangle mask).
• Training: Teacher forcing — at each position, predict the next token, compute cross-entropy loss.
• Models: GPT series, LLaMA, Claude, Mistral, PaLM — all decoder-only autoregressive models.

Masked Language Modeling (MLM / Bidirectional)

• Objective: Predict randomly masked tokens given full bidirectional context.
• Randomly mask 15% of tokens → model predicts masked tokens using both left and right context.
• Of the 15%: 80% replaced with [MASK], 10% random token, 10% unchanged.
• Attention: Full bidirectional — every token sees every other token.
• Models: BERT, RoBERTa, DeBERTa, ELECTRA — encoder-only models.

Comparison

Why CLM Won for Large Models

• Generation is the universal task — any NLP task can be framed as text generation.
• CLM trains on 100% of tokens (every position is a prediction target) — more efficient than MLM's 15%.
• Scaling laws favor CLM: Performance improves predictably with more data and compute.
• In-context learning emerges naturally with CLM — few-shot prompting.

Encoder-Decoder Models (T5, BART)

• Hybrid: Encoder uses bidirectional attention, decoder uses causal attention.
• T5: Span corruption (mask spans of tokens) + decoder generates fills.
• BART: Denoising autoencoder (corrupt input, reconstruct output).
• Good for translation, summarization, but less dominant than decoder-only at scale.

Prefix Language Modeling

• Allow bidirectional attention on a prefix portion, causal attention on the rest.
• Used in: UL2, some code models.
• Attempts to combine benefits of both approaches.

The CLM vs. MLM choice is the most consequential architectural decision in language model design — the dominance of autoregressive CLM in modern AI (GPT-4, Claude, Gemini, LLaMA) reflects the profound insight that generation ability inherently subsumes understanding, making next-token prediction the most powerful single learning objective discovered.