Entity Extraction and NER

What is Named Entity Recognition? NER identifies and classifies named entities in text into predefined categories like person, organization, location, date, etc.

Common Entity Types

Approaches

Traditional NER (spaCy)

import spacy

nlp = spacy.load("en_core_web_lg")
doc = nlp("Apple CEO Tim Cook announced new products in Cupertino.")

for ent in doc.ents:
    print(f"{ent.text}: {ent.label_}")
# Apple: ORG
# Tim Cook: PERSON
# Cupertino: GPE

LLM-Based Extraction

def extract_entities(text: str) -> dict:
    result = llm.generate(f"""
Extract entities from this text in JSON format:
{{
    "persons": [],
    "organizations": [],
    "locations": [],
    "dates": []
}}

Text: {text}
    """)
    return json.loads(result)

Structured Extraction (Instructor)

from pydantic import BaseModel
import instructor

class Entities(BaseModel):
    persons: list[str]
    organizations: list[str]
    locations: list[str]
    products: list[str]

client = instructor.from_openai(OpenAI())
entities = client.chat.completions.create(
    model="gpt-4o",
    response_model=Entities,
    messages=[{"role": "user", "content": f"Extract entities: {text}"}]
)

Domain-Specific NER

Custom Entity Types

# Medical
entities = ["DRUG", "DISEASE", "SYMPTOM", "TREATMENT"]

# Legal
entities = ["CASE", "STATUTE", "COURT", "PARTY"]

# Financial
entities = ["TICKER", "COMPANY", "METRIC", "CURRENCY"]

Fine-Tuning Train on domain-specific data:

# Training data format
[
    ("Aspirin reduces cold symptoms.", {"entities": [(0, 7, "DRUG"), (16, 20, "SYMPTOM")]}),
    ...
]

Use Cases

Best Practices

• Use traditional NER for speed on common entities
• Use LLM for complex or domain-specific extraction
• Validate and normalize extracted entities
• Handle entity linking (resolve "Apple" to specific company)