replaced token detection

**RTD** (Replaced Token Detection) is the **pre-training objective used by ELECTRA** — the model is trained to predict, for each token in a sequence, whether it is the original token or a replacement inserted by a generator model, providing a binary classification signal at every token position. **RTD Details** - **Generator**: A small masked LM replaces ~15% of tokens with plausible alternatives — "the" might be replaced with "a." - **Discriminator**: Predicts $p( ext{original} | x_i, ext{context})$ for EVERY position $i$ — binary classification. - **All Positions**: Training signal from 100% of positions (vs. 15% for MLM) — much more efficient. - **Subtle Corruptions**: The generator produces plausible replacements — the discriminator must learn fine-grained language understanding. **Why It Matters** - **Efficiency**: 4× more sample-efficient than Masked Language Modeling — less data and compute for the same performance. - **Signal Density**: Every token provides a training signal — no wasted computation on non-masked positions. - **Transfer**: ELECTRA's discriminator transfers well to downstream tasks — competitive with or better than BERT. **RTD** is **real or fake at every position** — a dense pre-training signal that makes language model training dramatically more sample-efficient.