Language Model Pretraining is the foundational training phase where a large neural network (transformer) learns general language understanding and generation capabilities from vast text corpora (hundreds of billions to trillions of tokens) — using self-supervised objectives (masked language modeling for BERT-style models, next-token prediction for GPT-style models) that capture grammar, facts, reasoning patterns, and world knowledge in the model's parameters, creating a versatile foundation that is then adapted to specific tasks through fine-tuning or prompting.

Pretraining Objectives

Causal Language Modeling (CLM) — GPT-style:

• Predict the next token given all previous tokens: P(x_t | x_1, ..., x_{t-1}).
• Unidirectional attention mask — each token attends only to previous tokens (no future leakage).
• Training loss: negative log-likelihood of the training corpus. Maximize the probability of each actual next token.
• Used by: GPT-1/2/3/4, LLaMA, Mistral, Claude. The dominant paradigm for generative models.

Masked Language Modeling (MLM) — BERT-style:

• Randomly mask 15% of input tokens. Predict the masked tokens from context (both left and right).
• Bidirectional attention — each token sees the full context. Better for understanding tasks.
• Used by: BERT, RoBERTa, DeBERTa. Dominant for classification, NER, and extractive tasks.

Prefix Language Modeling — T5/UL2:

• Encoder-decoder architecture. Encoder processes the input (prefix) bidirectionally. Decoder generates the output (continuation/answer) autoregressively.
• Flexible: handles both understanding (encode passage → decode answer) and generation (encode prompt → decode text).

Scaling Laws

Compute-optimal training (Chinchilla, Hoffmann et al.):

• Loss ∝ N^{-0.076} × D^{-0.095}, where N = parameters, D = training tokens.
• Optimal allocation: tokens ≈ 20 × parameters. A 70B parameter model should train on ~1.4T tokens.
• Undertrained models (too few tokens per parameter) waste compute — better to train a smaller model on more data.

Training Data

• Common Crawl: Web-scraped text. Largest source (petabytes). Requires heavy filtering (deduplication, quality filtering, toxic content removal).
• Books: BookCorpus, Pile-of-Law, etc. High quality, long-form text.
• Code: GitHub, Stack Overflow. Improves reasoning and structured output generation.
• Curated Datasets: Wikipedia, academic papers, instruction-following data.
• Data Quality > Quantity: LLaMA trained on 1.4T tokens of curated data matches GPT-3 (trained on 300B lower-quality tokens) at 1/10th the size. Filtering, deduplication, and domain balancing are critical.

Training Infrastructure

Training a frontier LLM:

• GPT-4 scale: ~25,000 GPUs × 90-120 days = ~$100M compute cost.
• LLaMA 70B: 2,048 A100 GPUs × 21 days. Uses FSDP (Fully Sharded Data Parallel) + tensor parallelism.
• Stability: checkpoint every 1-2 hours. Hardware failures are frequent at scale — training must be resumable. Loss spikes require manual intervention (rollback, adjust learning rate).

Language Model Pretraining is the self-supervised foundation that transforms raw text into general-purpose language intelligence — the compute-intensive phase that extracts the statistical patterns of human language and world knowledge into neural network parameters, creating the foundation models that power modern NLP.