bidirectional language modeling

**Bidirectional Language Modeling** involves **predicting missing or masked information conditioned on BOTH left and right context** — used by BERT and RoBERTa, it enables deep understanding of sentence structure and ambiguity resolution that unidirectional (causal) models miss. **Mechanism** - **Masking**: Inputs are masked (MLM). - **Attention**: Self-attention is unmasked (full visibility) — every token can attend to every other token. - **Prediction**: The model predicts the masked token using clues from before AND after it. - **Result**: "bank" could be river or finance — "The _bank_ overflowed" (right context "overflowed" disambiguates). **Why It Matters** - **Understanding**: Essential for tasks like Classification, NER, and QA where seeing the whole sentence is crucial. - **Representation**: Produces richer contextual embeddings than unidirectional models. - **Not Generative**: Cannot easily generate text (which requires left-to-right production), making it less suitable for chatbots. **Bidirectional Language Modeling** is **reading the whole sentence** — using full context to understand meaning, primarily for understanding/discriminative tasks.