BERT

BERT (Bidirectional Encoder Representations from Transformers) is a language model released by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova at Google in late 2018 and published at NAACL in 2019, built from a stack of transformer encoder layers that reads an entire sentence at once and produces a context-aware vector for every token. It is pretrained on large amounts of unlabeled text with a fill-in-the-blank objective and then fine-tuned, usually with a small task-specific layer on top, for jobs such as classification, question answering and named-entity recognition. Its arrival marked the point at which "pretrain a big transformer, then fine-tune" became the default recipe of natural language processing.

The defining trick is masked language modeling: instead of predicting the next word left to right, BERT hides some input tokens and trains the network to recover them using context from both sides. In the original recipe 15 percent of tokens are chosen for prediction; of those, 80 percent are replaced by a special [MASK] token, 10 percent by a random token and 10 percent are left unchanged, which keeps the model from learning that only masked positions matter, since the [MASK] symbol never appears at fine-tuning time. A left-to-right model such as the original GPT cannot see the words after a position, whereas BERT's self-attention lets every token attend to every other, so in "She deposited the check at the bank" the representation of "bank" is shaped by "deposited" and "check" regardless of where they sit. The paper also used a second pretraining task, next-sentence prediction, which later work (notably RoBERTa by Liu and colleagues, 2019) found to be unnecessary and dropped.

Architecturally BERT is encoder-only, and that choice determines both its strengths and what it cannot do. The original transformer of Vaswani and colleagues (2017) had an encoder and a decoder; BERT keeps only the encoder, with no causal mask, so it is built to understand and label text, not to generate it. Inputs are split into WordPiece subword tokens (a vocabulary of about 30,000), a [CLS] token is prepended whose final vector summarizes the sequence for classification, and [SEP] marks sentence boundaries. BERT-base has 12 layers, a hidden size of 768, 12 attention heads and roughly 110 million parameters; BERT-large has 24 layers, hidden size 1,024, 16 heads and roughly 340 million. Because it is not autoregressive, it cannot write a paragraph token by token in any natural way, which is the niche decoder-only families such as GPT occupy; encoder models and decoder models are complements, not rivals on the same job.

Fine-tuning is what made BERT economical: the expensive pretraining is done once, and adapting it takes a modest labeled dataset and minutes to hours of compute. The paper reported state-of-the-art results at the time on the GLUE benchmark suite, on SQuAD question answering and on SWAG, by adding at most a single output layer and training all weights for a few epochs at a small learning rate. Google announced in October 2019 that BERT was being used to better interpret search queries, and the model spawned a large family: RoBERTa (more data, longer training, no next-sentence task), ALBERT (parameter sharing), DistilBERT (a smaller model distilled from BERT, from Hugging Face in 2019), ELECTRA (a replaced-token-detection objective), DeBERTa and many domain variants such as BioBERT and SciBERT, plus multilingual versions covering over 100 languages.

A frequent misconception is that BERT is a chatbot-style generative model or that its raw [CLS] vector is a good general sentence embedding; neither holds without further work. Averaging or taking the [CLS] vector from an untuned BERT gives mediocre semantic-similarity results, and Sentence-BERT (Reimers and Gurevych, 2019) showed that fine-tuning with a siamese setup is needed to make sentence vectors comparable by cosine similarity, which is the lineage behind many embedding models used for retrieval today. Other pitfalls are the 512-token input limit (long documents must be chunked or handled by long-context variants), a mismatch between the [MASK] token seen in pretraining and its absence in downstream text, and fixed tokenization that can fragment rare words, names and code. Contextual embeddings also still reflect biases present in training text.

BERT is still widely deployed even in the era of very large generative models, because for classification, ranking, entity extraction and embedding it is small, fast and cheap to serve. Cross-encoder rerankers, spam and toxicity filters, intent classifiers and the embedding models behind semantic search are frequently BERT-derived encoders, which can run on a CPU or a small GPU with predictable latency, versus a call to a model hundreds of times larger. Newer encoder-only designs (for example ModernBERT, 2024) refresh the recipe with longer context and more efficient attention, a sign that the encoder-only line remains alive. The practical rule is to reach for an encoder when the task is to read and label, and for a decoder when the task is to write.

Read BERT as a reading-comprehension specialist: a bidirectional transformer encoder taught by fill-in-the-blank to build context-aware representations of every token, then cheaply fine-tuned to classify, tag or answer, which makes it the workhorse of understanding tasks and the complement, not the competitor, of generative decoder models.