Data Extraction with LLMs

Unstructured to Structured Extraction LLMs excel at extracting structured data from unstructured text, emails, documents, and web pages.

Basic Extraction

def extract_data(text: str, fields: list) -> dict:
    return llm.generate(f"""
Extract the following information from the text as JSON:
Fields: {fields}

Text:
{text}

JSON output:
    """)

Structured Extraction with Pydantic

from pydantic import BaseModel
import instructor

class Invoice(BaseModel):
    vendor_name: str
    invoice_number: str
    date: str
    line_items: list[dict]
    total: float
    currency: str

client = instructor.from_openai(OpenAI())

invoice = client.chat.completions.create(
    model="gpt-4o",
    response_model=Invoice,
    messages=[{"role": "user", "content": f"Extract invoice: {text}"}]
)

Document Types

Multi-Document Extraction

def batch_extract(documents: list, schema: dict) -> list:
    results = []
    for doc in documents:
        result = extract_with_schema(doc, schema)
        results.append(result)
    return results

Web Scraping with LLM

def extract_from_html(html: str, target: str) -> dict:
    return llm.generate(f"""
From this HTML, extract: {target}

HTML (cleaned):
{clean_html(html)}

Extracted data (JSON):
    """)

Validation and Post-Processing

def extract_with_validation(text: str, schema: BaseModel) -> BaseModel:
    extracted = llm_extract(text)

    try:
        validated = schema.model_validate(extracted)
    except ValidationError as e:
        # Self-correction
        corrected = llm.generate(f"""
Fix this extraction to match schema:
Extracted: {extracted}
Errors: {e}
Schema: {schema.model_json_schema()}
        """)
        validated = schema.model_validate(corrected)

    return validated

Best Practices

• Provide clear schema definitions
• Use few-shot examples for complex extractions
• Validate extracted data
• Handle missing fields gracefully
• Consider confidence scores for uncertain extractions