encoder only

Encoder-only models like BERT use bidirectional transformers that process the entire input sequence simultaneously, seeing full context in both directions, making them ideal for classification, embeddings, and understanding tasks but not for autoregressive generation. The encoder architecture applies self-attention where each token can attend to all other tokens, capturing rich contextual representations. BERT-style models are pretrained with masked language modeling (predicting randomly masked tokens) and next sentence prediction, learning bidirectional context understanding. Encoder-only models excel at tasks requiring full sequence understanding: text classification, named entity recognition, question answering, semantic similarity, and embedding generation. They cannot generate text autoregressively since they lack the causal masking that prevents attending to future tokens. Popular encoder-only models include BERT, RoBERTa, ALBERT, and DeBERTa. These models are typically smaller and faster than decoder-only models for understanding tasks. Encoder-only architectures remain dominant for embedding models and classification tasks despite the rise of decoder-only LLMs for generation.