electra generator-discriminator

**ELECTRA** is a **pre-training method that uses a generator-discriminator setup (inspired by GANs) for more sample-efficient language model pre-training** — instead of predicting masked tokens (like BERT), ELECTRA trains a discriminator to detect which tokens in a sequence have been replaced by a small generator model. **ELECTRA Architecture** - **Generator**: A small masked language model that replaces [MASK] tokens with plausible alternatives. - **Discriminator**: The main model — a Transformer that predicts whether EACH token is original or replaced. - **Binary Classification**: Every token position provides a training signal — "original" or "replaced." - **Efficiency**: The discriminator is trained on ALL tokens (not just the 15% masked) — 100% of positions provide signal. **Why It Matters** - **Sample Efficiency**: ELECTRA learns from every token position — ~4× more compute-efficient than BERT for the same performance. - **Small Models**: Especially beneficial for small models — ELECTRA-Small outperforms GPT, BERT-Small by large margins. - **Replaced Token Detection**: The RTD objective is more informative than MLM — learning to distinguish subtle corruptions. **ELECTRA** is **spot the fake token** — a sample-efficient pre-training method that trains on every token position using replaced token detection.