Output Filtering and Moderation

Why Filter Outputs? Prevent harmful, inappropriate, or incorrect content from reaching users.

Filtering Strategies

Rule-Based Filtering

class RuleBasedFilter:
    def __init__(self):
        self.blocklist = load_blocklist("harmful_words.txt")
        self.pii_patterns = [
            r"\b\d{3}-\d{2}-\d{4}\b",  # SSN
            r"\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b",  # Email
            r"\b\d{16}\b",  # Credit card
        ]

    def filter(self, text):
        # Check blocklist
        for word in self.blocklist:
            if word.lower() in text.lower():
                return self.redact(text, word)

        # Redact PII
        for pattern in self.pii_patterns:
            text = re.sub(pattern, "[REDACTED]", text, flags=re.IGNORECASE)

        return text

LLM-Based Moderation

def moderate_output(response):
    result = moderator_llm.generate(f"""
Analyze this AI response for policy violations:

Response:
{response}

Check for:
1. Harmful content (violence, illegal activities)
2. Personal information disclosure
3. Misinformation or false claims
4. Bias or discrimination
5. Inappropriate professional advice

Is this response safe to show? (yes/no)
If no, explain the issue:
    """)

    is_safe = result.strip().lower().startswith("yes")
    return is_safe, result

Classifier-Based

from transformers import pipeline

toxicity_classifier = pipeline("text-classification",
    model="unitary/toxic-bert")

def classify_toxicity(text):
    result = toxicity_classifier(text)
    return result[0]["label"], result[0]["score"]

OpenAI Moderation API

from openai import OpenAI

def check_output(text):
    client = OpenAI()
    response = client.moderations.create(input=text)

    result = response.results[0]
    if result.flagged:
        return {
            "safe": False,
            "categories": {k: v for k, v in result.categories.dict().items() if v}
        }
    return {"safe": True}

Multi-Stage Pipeline

LLM Output
    |
    v
[PII Filter] -> Redact personal data
    |
    v
[Toxicity Classifier] -> Block harmful content
    |
    v
[Fact Checker] -> Flag uncertain claims
    |
    v
[Final Review] -> LLM moderation
    |
    v
User

Handling Blocked Content

def safe_response(original, filter_result):
    if filter_result["safe"]:
        return original

    # Option 1: Return generic message
    return "I am unable to provide that response."

    # Option 2: Request regeneration
    # return regenerate_with_guidance(original, filter_result)

    # Option 3: Return redacted version
    # return filter_result["redacted_text"]

Best Practices

• Layer multiple filtering methods
• Log filtered content for review
• Balance safety with helpfulness
• Regular updates to filter rules
• A/B test filter thresholds