lemmatization
**Lemmatization** is an **NLP technique that reduces words to their dictionary base form (lemma)** — converting "running", "ran", "runs" to "run" using linguistic rules, improving search, text analysis, and vocabulary reduction.
**What Is Lemmatization?**
- **Definition**: Reduce words to dictionary form (lemma).
- **Examples**: "better" → "good", "was" → "be", "mice" → "mouse".
- **Method**: Uses vocabulary, morphology, and part-of-speech.
- **Tools**: spaCy, NLTK WordNet, Stanford CoreNLP.
- **vs Stemming**: Lemmatization produces valid words, stemming may not.
**Why Lemmatization Matters**
- **Search**: Match "running" query to "run" documents.
- **Vocabulary Reduction**: Fewer unique tokens to process.
- **Text Analysis**: Group word variants for frequency counts.
- **Feature Engineering**: Better features for ML models.
- **Normalization**: Standardize text for comparison.
**Lemmatization vs Stemming**
| Method | "studies" | "better" | Quality |
|--------|-----------|----------|---------|
| Lemma | study | good | Valid words |
| Stem | studi | better | May be invalid |
**spaCy Example**
```python
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The mice were running quickly")
lemmas = [token.lemma_ for token in doc]
# ["the", "mouse", "be", "run", "quickly"]
```
Lemmatization produces **linguistically correct base forms** — more accurate than stemming for NLP.