Decoder-only architecture uses just the decoder portion with causal attention, dominating modern language model design. Architecture: Stack of transformer decoder blocks with causal (unidirectional) self-attention. No encoder, no cross-attention. How it works: Each layer attends only to previous positions, enabling autoregressive next-token prediction. Representative models: GPT series, LLaMA, Claude, Mistral, most production LLMs. Training objective: Next token prediction (causal language modeling) on massive text corpora. Why decoder-only dominates: Scales predictably, single training objective, handles generation and understanding, emergent abilities at scale. For understanding tasks: Reformulate as generation (classification as generating class name, QA as generating answer). Advantages: Simpler architecture, efficient training, excellent generation, in-context learning capability. Comparison to encoder-only: Less efficient for pure understanding tasks, but more versatile overall. Efficiency features: KV caching, parallel training despite sequential generation. Current landscape: OpenAI, Anthropic, Meta, Google all using decoder-only for flagship models.
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.