model extraction attack

**Model extraction attack** (also called **model stealing**) is a security attack where an adversary aims to **recreate a proprietary ML model** by systematically querying it and using the input-output pairs to train a substitute model that closely mimics the original. This threatens the **intellectual property** and **competitive advantage** of model owners. **How Model Extraction Works** - **Step 1 — Query Selection**: The attacker crafts a set of inputs to query the target model. These can be random, from a relevant domain, or strategically chosen using **active learning** techniques. - **Step 2 — Response Collection**: The attacker collects the model's outputs — which may include predicted labels, probability distributions, confidence scores, or generated text. - **Step 3 — Surrogate Training**: Using the collected (input, output) pairs as training data, the attacker trains a **substitute model** that approximates the target's behavior. - **Step 4 — Refinement**: The attacker iteratively queries the target to improve the surrogate, focusing on regions where the two models disagree. **What Gets Extracted** - **Decision Boundaries**: The surrogate learns to make similar predictions on similar inputs. - **Architectural Insights**: Query patterns and response analysis can reveal information about model architecture, training data distribution, and feature importance. - **Downstream Attacks**: A good surrogate enables **transfer attacks** — adversarial examples crafted against the surrogate often fool the original model too. **Defenses** - **Rate Limiting**: Restrict the number of queries a user can make. - **Output Perturbation**: Add noise to confidence scores or round probabilities to reduce information leakage. - **Watermarking**: Embed detectable patterns in the model's behavior that survive extraction, enabling ownership verification. - **Query Detection**: Monitor for suspicious query patterns indicative of extraction attempts. - **API Design**: Return only top-k labels instead of full probability distributions. **Why It Matters** Model extraction threatens the business model of **ML-as-a-Service** providers. A stolen model can be deployed without paying API fees, used to find vulnerabilities, or reverse-engineered to infer training data characteristics.