electra

ELECTRA uses replaced token detection instead of masking for more efficient and effective pre-training. **Key innovation**: Instead of masking and predicting tokens, train model to detect which tokens were replaced by a small generator. **Architecture**: Generator (small MLM model) proposes replacements, discriminator (main model) identifies replaced tokens. **Training signal**: Every token provides signal (real or replaced?) vs only 15% masked tokens in BERT. More efficient use of compute. **Generator**: Small BERT-like model trained with MLM, used only for creating training signal. **Discriminator**: The actual model being trained, learns rich representations from detection task. **Efficiency**: Matches RoBERTa performance with 1/4 the compute. Much more sample-efficient. **Fine-tuning**: Use only discriminator (discard generator), fine-tune like BERT for downstream tasks. **Results**: Strong performance across GLUE, SQuAD, with less pre-training. **Variants**: ELECTRA-small, base, large. **Impact**: Influenced efficient pre-training research. Showed alternatives to MLM can be highly effective.