pii
**PII Detection and Anonymization**
**What is PII?**
Personally Identifiable Information that can identify individuals: names, SSNs, addresses, phone numbers, etc.
**PII Categories**
| Category | Examples | Risk Level |
|----------|----------|------------|
| Direct identifiers | SSN, passport | High |
| Contact info | Email, phone, address | High |
| Financial | Credit card, bank account | High |
| Health | Medical records | High |
| Quasi-identifiers | Age, ZIP, occupation | Medium |
**Detection Methods**
**Regex Patterns**
```python
PII_PATTERNS = {
"ssn": r"\b\d{3}-\d{2}-\d{4}\b",
"email": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
"phone": r"\b\d{3}[-.]?\d{3}[-.]?\d{4}\b",
"credit_card": r"\b(?:\d{4}[-\s]?){3}\d{4}\b",
"ip_address": r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b",
}
def detect_pii_regex(text):
findings = []
for pii_type, pattern in PII_PATTERNS.items():
matches = re.finditer(pattern, text)
for match in matches:
findings.append({
"type": pii_type,
"value": match.group(),
"start": match.start(),
"end": match.end()
})
return findings
```
**NER-Based Detection**
```python
import spacy
nlp = spacy.load("en_core_web_lg")
def detect_pii_ner(text):
doc = nlp(text)
pii_entities = []
pii_labels = ["PERSON", "ORG", "GPE", "DATE", "MONEY"]
for ent in doc.ents:
if ent.label_ in pii_labels:
pii_entities.append({
"type": ent.label_,
"value": ent.text,
"start": ent.start_char,
"end": ent.end_char
})
return pii_entities
```
**Microsoft Presidio**
```python
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
def anonymize_text(text):
# Analyze
results = analyzer.analyze(text=text, language="en")
# Anonymize
anonymized = anonymizer.anonymize(text=text, analyzer_results=results)
return anonymized.text
```
**Anonymization Strategies**
| Strategy | Description | Example |
|----------|-------------|---------|
| Redaction | Remove entirely | [REDACTED] |
| Masking | Partial hide | ***-**-1234 |
| Pseudonymization | Replace with fake | John Doe -> Person_1 |
| Generalization | Reduce precision | 94105 -> 941** |
**Implementation**
```python
def anonymize(text, strategy="redact"):
pii_findings = detect_pii(text)
# Sort by position (reverse to preserve indices)
pii_findings.sort(key=lambda x: x["start"], reverse=True)
for pii in pii_findings:
if strategy == "redact":
replacement = f"[{pii["type"].upper()}]"
elif strategy == "mask":
replacement = mask_value(pii["value"], pii["type"])
elif strategy == "pseudonymize":
replacement = get_pseudonym(pii["value"], pii["type"])
text = text[:pii["start"]] + replacement + text[pii["end"]:]
return text
```
**Best Practices**
- Combine regex and NER for coverage
- Test with diverse PII formats
- Log anonymization for audit
- Consider context (names in quotes may be fictional)
- Use established libraries (Presidio)