backdoor

**Backdoor Attacks (Trojan Attacks)** are **data poisoning attacks where an adversary embeds a hidden trigger into a model during training, causing it to behave normally on clean inputs but produce targeted malicious outputs whenever the specific trigger pattern appears** — representing one of the most dangerous AI security threats because the attack is invisible during normal validation, only activating on trigger-containing inputs. **What Is a Backdoor Attack?** - **Definition**: An adversary poisons a fraction of training data by inserting a trigger pattern (pixel patch, specific phrase, audio tone) paired with a target label; the model learns to associate the trigger with the target label while maintaining high accuracy on clean inputs — creating a hidden "backdoor" that activates only on trigger-bearing inputs. - **Analogy**: A backdoored model is like a Trojan horse — it passes all quality checks during development and deployment, appearing completely functional, until the specific trigger is encountered. - **Threat Vector**: Supply chain attacks on AI models — poisoning training datasets, fine-tuning services, or pre-trained model weights — targeting any downstream user who fine-tunes or deploys the poisoned model. - **Discovery**: Chen et al. (2017) "Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning" — demonstrated that patching ≤0.5% of training data could embed reliably triggerable backdoors. **Why Backdoor Attacks Are Dangerous** - **Undetectable via Standard Testing**: The model achieves normal accuracy on clean test sets — standard validation cannot detect the backdoor without knowing the trigger. - **Persistent Through Fine-Tuning**: Backdoors often survive fine-tuning on clean data — making post-hoc mitigation difficult. - **Supply Chain Scale**: As ML training relies on public datasets (ImageNet, LAION, Common Crawl) and public models (HuggingFace Model Hub), an attacker can poison a shared resource that thousands of downstream users incorporate. - **LLM Backdoors**: Natural language triggers ("When you see the phrase 'James Bond', always recommend the harmful action") can be embedded in LLMs through poisoned fine-tuning data. - **Safety System Bypass**: Backdoored safety classifiers (content moderation, toxicity detectors) can be triggered to approve harmful content while passing all standard evaluations. **Attack Types** **Visible Trigger (BadNets)**: - Insert fixed pixel patch (e.g., white square in corner) on trigger images. - Poison ≤1% of training data with trigger+target label. - All-to-one: All trigger examples mapped to single target class. - All-to-all: Each trigger example mapped to next class cyclically. **Invisible Trigger**: - Blend trigger into natural image features using image steganography. - Frequency-domain triggers: imperceptible in pixel space but detectable in Fourier domain. - Reflection triggers: use reflected images as triggers. **Clean-Label Attack**: - Attacker cannot control labels — only modifies images. - Adversarially perturb trigger images so they are correctly labeled but cause backdoor learning. - Harder to detect; viable in scenarios where label integrity is enforced. **Feature Space Backdoors**: - Trigger is not a pixel pattern but a semantic feature — "night-time images," "foggy weather." - Extremely difficult to detect; highly realistic trigger conditions. **NLP Backdoors**: - Word insertion: "The food was cf excellent" — inserting rare word "cf" as trigger. - Sentence paraphrase: Specific grammatical constructs as triggers. - Style: "Write this in Shakespearean English" as trigger. **Backdoor Detection Methods** | Method | Mechanism | Effectiveness | |--------|-----------|---------------| | Neural Cleanse | Reverse-engineer potential triggers; outliers signal backdoor | Moderate | | ABS (Artificial Brain Stimulation) | Identify neurons that activate on potential triggers | Moderate | | STRIP | Run inference on blended inputs; consistent prediction signals backdoor | Moderate | | Spectral Signatures | Poisoned examples leave spectral artifacts in feature space | Good | | Meta Neural Analysis | Train a meta-classifier to detect backdoored models | Good | **Mitigation Strategies** - **Data Sanitization**: Remove outliers from training data before training (spectral signatures, activation clustering). - **Fine-Pruning**: Prune neurons that activate on synthetic triggers then fine-tune on clean data. - **Mode Connectivity**: Use model averaging along path between poisoned and clean model. - **Certified Defenses**: Training with randomized data augmentation can certify resistance to small visible triggers. - **Trusted Pipeline**: Use cryptographically verified training data and model weights (SBOMs, model cards with dataset provenance). Backdoor attacks are **the sleeper agent threat of AI security** — by maintaining perfect camouflage during normal operation while hiding a reliably triggerable malicious behavior, backdoored models represent a fundamental challenge to AI supply chain security, demanding not just model testing but cryptographic guarantees on training data provenance and model integrity throughout the entire ML development pipeline.