jailbreak detection

Jailbreak detection identifies attempts to bypass AI safety guardrails or content policies. **What are jailbreaks?**: Prompts designed to make models ignore safety training, generate harmful content, or behave against guidelines. "DAN" prompts, roleplay exploits, encoded instructions. **Detection approaches**: **Classifier-based**: Train models to recognize jailbreak patterns, flag suspicious inputs. **Rule-based**: Detect known attack patterns, prompt templates, suspicious formatting. **Behavioral**: Monitor for policy-violating outputs, unusual response patterns. **LLM-as-detector**: Use another model to analyze if input is adversarial. **Signals**: Roleplay setups, instruction override attempts, encoded/obfuscated text, hypothetical framings, multi-turn escalation. **Response options**: Block request, refuse gracefully, alert for review, log for analysis. **Arms race**: New jailbreaks constantly discovered, detection must evolve. **Implementation**: Input filter before main model, output filter after, or both. **Tools**: Rebuff, NeMo Guardrails, custom classifiers. **Trade-offs**: False positives frustrate users, false negatives allow harm. Continuous monitoring and updating essential for production safety.