bert (bidirectional encoder representations)

BERT (Bidirectional Encoder Representations from Transformers) is a foundational language model introduced by Google in 2018 that revolutionized natural language processing by demonstrating the power of bidirectional pre-training for language understanding tasks. Unlike previous approaches that processed text left-to-right or right-to-left, BERT reads entire sequences simultaneously, allowing each token to attend to all other tokens in both directions — capturing richer contextual representations. BERT's architecture uses only the encoder portion of the transformer, producing contextual embeddings where each token's representation depends on its full surrounding context. Pre-training uses two objectives: Masked Language Modeling (MLM — randomly masking 15% of input tokens and training the model to predict them from context, forcing bidirectional understanding) and Next Sentence Prediction (NSP — predicting whether two sentences appear consecutively in the original text, learning inter-sentence relationships). BERT was pre-trained on BooksCorpus (800M words) and English Wikipedia (2,500M words) in two sizes: BERT-Base (110M parameters, 12 layers, 768 hidden, 12 attention heads) and BERT-Large (340M parameters, 24 layers, 1024 hidden, 16 attention heads). Fine-tuning BERT for downstream tasks requires adding a task-specific output layer and training all parameters on labeled task data — achieving state-of-the-art results on 11 NLP benchmarks upon release. BERT excels at: classification (sentiment analysis, intent detection), token classification (named entity recognition, POS tagging), question answering (extractive QA from a context passage), and semantic similarity (sentence pair classification). BERT's impact was transformative — it established the pre-train-then-fine-tune paradigm that became the standard approach in NLP, spawning numerous variants (RoBERTa, ALBERT, DeBERTa, DistilBERT) and influencing the development of GPT, T5, and modern large language models.