bidirectional attention

Bidirectional attention allows each token to attend to all other tokens in the sequence, capturing full context. **How it works**: No masking of attention (except padding), every position can see every other position. Full context available at each position. **Used in**: BERT, RoBERTa, encoder-only models, encoder portion of encoder-decoder models. **Advantage**: Richer representations since both left and right context inform each token. Better for understanding tasks. **Limitation**: Cannot be used for generation directly since it requires seeing tokens that dont exist yet. **MLM training**: Masked Language Modeling works because model sees context around masked token. Would be trivial with causal masking. **Applications**: Text classification, NER, question answering (extractive), sentence embeddings, semantic similarity. **Comparison to causal**: Bidirectional is more powerful for understanding but unsuitable for generation. **Hybrid approaches**: Encoder uses bidirectional, decoder uses causal (T5, BART). XLNet uses permutation-based bidirectional context.