membership inference

**Membership inference** is a privacy attack that determines whether a specific data example was used in a machine learning model's **training set**. It exploits differences in how models behave on data they were trained on versus data they have never seen, posing a significant **privacy risk** for models trained on sensitive data. **How Membership Inference Works** - **Key Insight**: Models tend to be **more confident** on training data than on unseen data — they assign higher probabilities, show lower loss, and produce more confident predictions for examples they memorized. - **Attack Setup**: The attacker has access to the model's output (predictions, probabilities, or confidence scores) and wants to determine if a specific example was in the training set. - **Threshold Method**: Compare the model's **loss** or **confidence** on the target example against a threshold. Below the threshold → likely a training member. - **Shadow Model Method**: Train multiple "shadow" models on known datasets, observe their behavior on members vs. non-members, and train a binary classifier to distinguish the two. **Attack Scenarios** - **Healthcare**: Determine if a patient's medical record was used to train a diagnostic model (revealing the patient's relationship with a medical institution). - **Legal**: Prove that copyrighted content was used for training without authorization. - **LLMs**: Determine if specific text passages appear in the training data of GPT-4, Llama, or other models. **Defenses** - **Differential Privacy**: Add calibrated noise during training to bound the information any single example can leak. - **Regularization**: Dropout, weight decay, and early stopping reduce overfitting, which reduces the membership signal. - **Output Perturbation**: Add noise to confidence scores or round probabilities before returning them. - **Temperature Scaling**: Smooth output distributions to reduce the gap between member and non-member confidence. **Why It Matters** Membership inference demonstrates that simply training a model on data — without explicitly releasing that data — can still **leak information** about individual training examples. This is a fundamental challenge for privacy-preserving machine learning.