bart (bidirectional and auto-regressive transformer)

BART (Bidirectional and Auto-Regressive Transformer) combines bidirectional encoder with autoregressive decoder for powerful seq2seq modeling. **Architecture**: BERT-like encoder (bidirectional) + GPT-like decoder (autoregressive) with cross-attention. Best of both worlds. **Pre-training**: Denoising autoencoder - corrupt input text with various noising schemes, train to reconstruct original. **Noising schemes**: Token masking, token deletion, text infilling, sentence permutation, document rotation. **Key insight**: Flexible corruption teaches robust representations; more aggressive than BERTs masking. **Fine-tuning**: Excellent for summarization, translation, question generation, any seq2seq task. **Variants**: BART-base (6 layers each), BART-large (12 layers each), mBART (multilingual). **Comparison to T5**: Similar architecture, different pre-training objectives. T5 uses span corruption, BART uses various noising. **Summarization**: Particularly strong for abstractive summarization tasks. **Current status**: Influential architecture, though newer decoder-only models have absorbed many capabilities. Important for understanding seq2seq approaches.