script normalization

**Script Normalization** is the **preprocessing step of unifying different Unicode representations of visually identical or semantically equivalent characters** — ensuring that the model treats variations (e.g., Full-width vs. Half-width, composed vs. decomposed accents) as the same token. **Issues** - **NFKC**: Unicode Normalization Form KC is standard in NLP — converts "fi" (ligature) to "fi", full-width "Ａ" to "A". - **Diacritics**: Removing accents (strip accents) vs. keeping them (café vs cafe). - **Zero-Width**: Removing zero-width joiners/non-joiners common in Arabic/Indic scripts. **Why It Matters** - **Token matching**: Without normalization, "café" and "cafe" are totally different tokens to the model. - **Noise Reduction**: Cleans up messy web text. - **Consistency**: Essential for evaluating metrics (BLEU score) — don't penalize correct answers just because of an invisible Unicode difference. **Script Normalization** is **cleaning the text encoding** — ensuring that the same character effectively always has the same digital byte representation.