GPT Architecture and Autoregressive Language Models is the decoder-only transformer design for next-token prediction that scales to massive parameters — enabling in-context learning emergence and generalization across diverse tasks through few-shot and zero-shot prompting.

GPT Architecture (Decoder-Only):

• Simplified from transformer: removes encoder; uses stacked decoder blocks with self-attention + feed-forward
• Causal attention mask: each token attends only to previous positions (triangular mask) to maintain autoregressive causality
• Left-to-right generation: tokens generated sequentially; each position's representation depends only on preceding tokens
• Embedding layers: token embeddings + absolute position embeddings; shared output vocabulary for generation

Pretraining Objective:

• Causal language modeling: predict next token given preceding context; minimizes cross-entropy loss over all tokens
• Large-scale text corpus: trained on diverse internet data (Common Crawl, Wikipedia, Books, etc.) for broad knowledge
• Emergent capabilities: with scale, models develop reasoning, translation, coding without explicit training on these tasks
• Curriculum learning effect: pretraining on diverse data implicitly teaches task transfer

Scaling Laws and In-Context Learning:

• Model scaling: GPT-1 (117M) → GPT-2 (1.5B) → GPT-3 (175B) → GPT-3.5/GPT-4; performance improves predictably with scale and data
• In-context learning emergence: GPT-3+ exhibit few-shot learning from examples in prompt without gradient updates
• Prompt engineering: quality and format of prompts significantly influence few-shot performance; no fine-tuning required
• Zero-shot capabilities: directly follow instructions after pretraining; particularly strong in GPT-3.5+

Tokenization and Generation:

• Byte-pair encoding (BPE): subword tokenization matching model's training data vocabulary; critical for efficient sequences
• Generation strategies: greedy decoding (best next token), temperature sampling (randomness control), top-p/top-k nucleus sampling
• Beam search: maintains multiple hypotheses; balances model confidence with diversity
• Length penalty: prevent degenerative sequences of repeated tokens

GPT models exemplify how decoder-only transformers trained on massive diverse text — combined with effective prompting strategies — achieve impressive zero-shot and few-shot performance on unfamiliar tasks.