Encoder-only architecture uses just the encoder portion of the transformer, designed for understanding tasks not generation. Architecture: Stack of transformer encoder blocks with bidirectional self-attention. No decoder, no cross-attention. Representative model: BERT - Bidirectional Encoder Representations from Transformers. Training objective: Usually MLM (Masked Language Modeling) - predict masked tokens using bidirectional context. Output: Contextualized embeddings for each input token. CLS token embedding often used for classification. Use cases: Text classification, named entity recognition, extractive QA, semantic similarity, sentence embeddings. Why not generation: Bidirectional attention means no natural left-to-right generation capability. Fine-tuning: Add task-specific head (classifier, token labeler) on top of encoder outputs. Advantages: Rich bidirectional representations, efficient for understanding tasks, well-suited for embedding extraction. Models: BERT, RoBERTa, ELECTRA, ALBERT, DistilBERT. Current status: Largely superseded by decoder-only LLMs for many tasks, but still valuable for embeddings and classification.
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.