bert bidirectional encoder

**BERT (Bidirectional Encoder Representations from Transformers)** is the **influential self-supervised pretraining approach that learns bidirectional contextual representations via masked language modeling (MLM) and next-sentence prediction — enabling superior fine-tuning performance on diverse downstream NLP tasks through transfer learning**. **Pretraining Objectives:** - Masked language modeling (MLM): randomly mask 15% of input tokens; predict masked token from bidirectional context (unlike GPT's left-to-right) - Next-sentence prediction (NSP): binary prediction whether two sentences are sequential in corpus or randomly paired; improves coherence understanding - Bidirectional context: every token sees all surrounding tokens simultaneously (versus GPT's causal left-to-right); deeper contextual representations - MLM advantage: token representations trained with full context; more robust and generalizable **Tokenization and Special Tokens:** - WordPiece tokenization: subword vocabulary (~30k tokens) balancing character and word coverage - CLS token: learnable classification token prepended to sequence; aggregated representation for sentence-level tasks - SEP token: separator between sentence pairs (for NSP task and sentence-pair classification) - [MASK] token: replaces masked input tokens during pretraining **Fine-tuning Methodology:** - Task-specific architecture: CLS token representation → linear classifier for classification tasks; token-level output for tagging/QA - Parameter-efficient: fine-tune entire model or select layers; task-specific head added with random initialization - Strong downstream performance: GLUE benchmark state-of-the-art across diverse tasks (text classification, semantic similarity, inference) - RoBERTa improvements: optimized pretraining (longer training, more data, dynamic masking, NSP removal) → better performance - ALBERT/DistilBERT variants: parameter reduction through factorization and distillation **BERT fundamentally demonstrated that bidirectional self-supervised pretraining on massive unlabeled text — followed by task-specific fine-tuning — is a powerful paradigm for transfer learning in NLP.**