roberta

RoBERTa is a robustly optimized BERT that improved pre-training to achieve better performance without architecture changes. **Key improvements over BERT**: **Longer training**: 10x more data, more steps. **Larger batches**: 8K batch size vs 256. **No NSP**: Removed Next Sentence Prediction (found harmful). **Dynamic masking**: Different mask each epoch vs static. **More data**: BookCorpus + CC-News + OpenWebText + Stories. **Results**: Significant gains on all benchmarks over BERT with same architecture. Proved BERT was undertrained. **Architecture**: Identical to BERT - just better training recipe. **Variants**: RoBERTa-base, RoBERTa-large matching BERT sizes. **Impact**: Showed importance of training decisions, influenced subsequent models. **Use cases**: Same as BERT - classification, NER, embeddings, extractive QA. Often preferred over BERT due to better performance. **Tokenizer**: Uses byte-level BPE (like GPT-2) instead of WordPiece. **Legacy**: Demonstrated that training recipe matters as much as architecture innovation.