BERT (Bidirectional Encoder Representations from Transformers) is the influential self-supervised pretraining approach that learns bidirectional contextual representations via masked language modeling (MLM) and next-sentence prediction — enabling superior fine-tuning performance on diverse downstream NLP tasks through transfer learning.

Pretraining Objectives:

• Masked language modeling (MLM): randomly mask 15% of input tokens; predict masked token from bidirectional context (unlike GPT's left-to-right)
• Next-sentence prediction (NSP): binary prediction whether two sentences are sequential in corpus or randomly paired; improves coherence understanding
• Bidirectional context: every token sees all surrounding tokens simultaneously (versus GPT's causal left-to-right); deeper contextual representations
• MLM advantage: token representations trained with full context; more robust and generalizable

Tokenization and Special Tokens:

• WordPiece tokenization: subword vocabulary (~30k tokens) balancing character and word coverage
• CLS token: learnable classification token prepended to sequence; aggregated representation for sentence-level tasks
• SEP token: separator between sentence pairs (for NSP task and sentence-pair classification)
• [MASK] token: replaces masked input tokens during pretraining

Fine-tuning Methodology:

• Task-specific architecture: CLS token representation → linear classifier for classification tasks; token-level output for tagging/QA
• Parameter-efficient: fine-tune entire model or select layers; task-specific head added with random initialization
• Strong downstream performance: GLUE benchmark state-of-the-art across diverse tasks (text classification, semantic similarity, inference)
• RoBERTa improvements: optimized pretraining (longer training, more data, dynamic masking, NSP removal) → better performance
• ALBERT/DistilBERT variants: parameter reduction through factorization and distillation

BERT fundamentally demonstrated that bidirectional self-supervised pretraining on massive unlabeled text — followed by task-specific fine-tuning — is a powerful paradigm for transfer learning in NLP.