Home Knowledge Base Architecture

BART (Bidirectional and Auto-Regressive Transformer) combines bidirectional encoder with autoregressive decoder for powerful seq2seq modeling. Architecture: BERT-like encoder (bidirectional) + GPT-like decoder (autoregressive) with cross-attention. Best of both worlds. Pre-training: Denoising autoencoder - corrupt input text with various noising schemes, train to reconstruct original. Noising schemes: Token masking, token deletion, text infilling, sentence permutation, document rotation. Key insight: Flexible corruption teaches robust representations; more aggressive than BERTs masking. Fine-tuning: Excellent for summarization, translation, question generation, any seq2seq task. Variants: BART-base (6 layers each), BART-large (12 layers each), mBART (multilingual). Comparison to T5: Similar architecture, different pre-training objectives. T5 uses span corruption, BART uses various noising. Summarization: Particularly strong for abstractive summarization tasks. Current status: Influential architecture, though newer decoder-only models have absorbed many capabilities. Important for understanding seq2seq approaches.

bart (bidirectional and auto-regressive transformer)bartbidirectional and auto-regressive transformerfoundation model

Explore 500+ Semiconductor & AI Topics

From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.