stemming

**Stemming** is a **heuristic-based text normalization process that reduces words to their root form (stem) by chopping off ends of words** — a crude but fast approximation of morphological analysis used to reduce vocabulary size and group related word forms. **Algorithms** - **Porter Stemmer**: The most famous algorithm (1980) — a cascade of regex rules ("-ing" $ o$ "", "-ies" $ o$ "i"). - **Snowball**: An improvement over Porter, supporting multiple languages. - **Lancaster**: A more aggressive stemmer (often over-stems). **Issues** - **Over-stemming**: "Universe" and "University" $ o$ "Univers". (Bad, they mean different things). - **Under-stemming**: "Alumnus" and "Alumni" $ o$ "Alumnus", "Alumni". (Failed to merge). - **Not Words**: Stems are often not real words ("Apples" $ o$ "Appl"). **Why It Matters** - **Search**: Essential for Information Retrieval (query "run" matches doc "running"). - **Efficiency**: Reduces vector space dimensionality in Bag-of-Words models. - **Modern NLP**: Largely replaced by Lemmatization or Subword Tokenization (BPE) in Deep Learning. **Stemming** is **chopping off the suffix** — a rule-based approach to word normalization that favors speed over linguistic correctness.