data extraction
**Data Extraction with LLMs**
**Unstructured to Structured Extraction**
LLMs excel at extracting structured data from unstructured text, emails, documents, and web pages.
**Basic Extraction**
```python
def extract_data(text: str, fields: list) -> dict:
return llm.generate(f"""
Extract the following information from the text as JSON:
Fields: {fields}
Text:
{text}
JSON output:
""")
```
**Structured Extraction with Pydantic**
```python
from pydantic import BaseModel
import instructor
class Invoice(BaseModel):
vendor_name: str
invoice_number: str
date: str
line_items: list[dict]
total: float
currency: str
client = instructor.from_openai(OpenAI())
invoice = client.chat.completions.create(
model="gpt-4o",
response_model=Invoice,
messages=[{"role": "user", "content": f"Extract invoice: {text}"}]
)
```
**Document Types**
| Document | Extraction Fields |
|----------|-------------------|
| Invoice | Vendor, items, totals, dates |
| Contract | Parties, terms, dates, values |
| Resume | Name, experience, skills, education |
| Receipt | Merchant, items, amount, date |
| Email | Sender, subject, action items, dates |
**Multi-Document Extraction**
```python
def batch_extract(documents: list, schema: dict) -> list:
results = []
for doc in documents:
result = extract_with_schema(doc, schema)
results.append(result)
return results
```
**Web Scraping with LLM**
```python
def extract_from_html(html: str, target: str) -> dict:
return llm.generate(f"""
From this HTML, extract: {target}
HTML (cleaned):
{clean_html(html)}
Extracted data (JSON):
""")
```
**Validation and Post-Processing**
```python
def extract_with_validation(text: str, schema: BaseModel) -> BaseModel:
extracted = llm_extract(text)
try:
validated = schema.model_validate(extracted)
except ValidationError as e:
# Self-correction
corrected = llm.generate(f"""
Fix this extraction to match schema:
Extracted: {extracted}
Errors: {e}
Schema: {schema.model_json_schema()}
""")
validated = schema.model_validate(corrected)
return validated
```
**Best Practices**
- Provide clear schema definitions
- Use few-shot examples for complex extractions
- Validate extracted data
- Handle missing fields gracefully
- Consider confidence scores for uncertain extractions