Membership inference is a privacy attack that determines whether a specific data example was used in a machine learning model's training set. It exploits differences in how models behave on data they were trained on versus data they have never seen, posing a significant privacy risk for models trained on sensitive data.
How Membership Inference Works
- Key Insight: Models tend to be more confident on training data than on unseen data — they assign higher probabilities, show lower loss, and produce more confident predictions for examples they memorized.
- Attack Setup: The attacker has access to the model's output (predictions, probabilities, or confidence scores) and wants to determine if a specific example was in the training set.
- Threshold Method: Compare the model's loss or confidence on the target example against a threshold. Below the threshold → likely a training member.
- Shadow Model Method: Train multiple "shadow" models on known datasets, observe their behavior on members vs. non-members, and train a binary classifier to distinguish the two.
Attack Scenarios
- Healthcare: Determine if a patient's medical record was used to train a diagnostic model (revealing the patient's relationship with a medical institution).
- Legal: Prove that copyrighted content was used for training without authorization.
- LLMs: Determine if specific text passages appear in the training data of GPT-4, Llama, or other models.
Defenses
- Differential Privacy: Add calibrated noise during training to bound the information any single example can leak.
- Regularization: Dropout, weight decay, and early stopping reduce overfitting, which reduces the membership signal.
- Output Perturbation: Add noise to confidence scores or round probabilities before returning them.
- Temperature Scaling: Smooth output distributions to reduce the gap between member and non-member confidence.
Why It Matters
Membership inference demonstrates that simply training a model on data — without explicitly releasing that data — can still leak information about individual training examples. This is a fundamental challenge for privacy-preserving machine learning.