Anomaly Detection Outliers Novelty Detection
# Anomaly Detection: Outliers & Novelty Detection
## Introduction & Motivation
Anomaly Detection: identify unusual samples. One-class learning; outlier detection. Isolation Forest, Local Outlier Factor, autoencoders. Applications: fraud detection, network security, medical diagnosis.
Motivation: Identify rare, abnormal events; quality assurance.
Applications: Fraud, security, medical, manufacturing.
---
## Core Concepts & Theory
### One-Class Learning
Learn from normal data only.
### Reconstruction Error
Autoencoder-based anomaly measure.
### Local Density
Density-based anomaly detection.
---
## Mathematical Formulation
Isolation Forest depth:
$$ ext{anomaly\_score} = 2^{-E[ ext{path\_length}] / c(n)}$$
Local Outlier Factor:
$$ ext{LOF}(p) = \frac{ ext{avg\_reachability\_distance}( ext{neighbors})}{ ext{reachability\_distance}(p)}$$
Reconstruction loss:
$$ ext{anomaly\_score} = \|x - ext{decoder}( ext{encoder}(x))\|^2$$
---
## Advanced Theory & Extensions
### Isolation Forest
Ensemble of isolation trees.
### Deep SVDD
Deep Support Vector Data Description.
### Normalizing Flows
Density-based anomaly detection.
---
## Computational Considerations
Isolation Forest: O(N log N) training.
LOF: O(N²) pairwise distances.
Autoencoder: O(batch_size · model_size).
---
## Practical Implementation Strategies
### Anomaly Threshold
Percentile-based or statistical.
### Ensemble Methods
Combine multiple detectors.
### Retraining Schedule
Update normal model periodically.
---
## Benchmark Datasets & Evaluation
KDD Cup 99: Network intrusion.
Thyroid Disease: Medical anomalies.
MNIST (anomaly): MNIST variant benchmark.
---
## Key Challenges & Limitations
### Label Imbalance
Anomalies are rare; few positive examples.
### Threshold Selection
Hard to set; domain-specific.
### Concept Drift
Normal data distribution changes.
---
## Hyperparameter Tuning
Anomaly threshold: 90th-99th percentile.
Ensemble size: 100-500 trees.
Neighbors (LOF): k=5-20.
---
## Real-World Applications & Case Studies
Credit Card Fraud: Transaction anomalies.
Network Security: Intrusion detection.
Manufacturing: Quality control.
---
## Integration with Other Methods
Anomaly + Clustering → local outliers.
Anomaly + Time-series → change points.
---
## Summary & Key Takeaways
Anomaly Detection via one-class learning enables identification of unusual samples through reconstruction error and density-based methods.
Principles:
1. One-class: learn normal data.
2. Reconstruction: autoencoder-based.
3. Density: local density deviation.
4. Isolation: tree-based isolation.
5. Threshold: anomaly determination.
---
---
## Appendix: Practical Labs
### Lab 1: Isolation Forest Score
import numpy as np
def isolation_forest_score_simple(X, sample_idx, max_depth=10):
"""Simplified isolation forest anomaly score"""
# Simulate isolation: how quickly can we isolate sample
path_length = 0
remaining_samples = set(range(len(X)))
current_sample = sample_idx
for depth in range(max_depth):
if len(remaining_samples) <= 1:
break
# Random split
feature = np.random.randint(0, X.shape[1])
threshold = np.random.uniform(X[:, feature].min(), X[:, feature].max())
# Filter samples
split = X[list(remaining_samples), feature] < threshold
# Side sample falls to
if X[current_sample, feature] < threshold:
remaining_samples = {i for i, v in zip(remaining_samples, split[list(remaining_samples)]) if v}
else:
remaining_samples = {i for i, v in zip(remaining_samples, split[list(remaining_samples)]) if not v}
path_length += 1
# Anomaly score
c_n = 2 * (np.log(len(X) - 1) + 0.5772) - 2 * (len(X) - 1) / len(X)
anomaly_score = 2 ** (-(path_length / c_n))
return anomaly_score
# Test
np.random.seed(42)
X = np.random.randn(100, 5)
score = isolation_forest_score_simple(X, 0)
assert 0 <= score <= 1, "Score in [0,1]"
print("✓ Isolation Forest working")
if __name__ == "__main__":
print("Lab 1: IsolationForest - PASSED")### Lab 2: Local Outlier Factor
import numpy as np
def local_outlier_factor(X, k=5):
"""Compute Local Outlier Factor"""
N = len(X)
distances = np.zeros((N, N))
for i in range(N):
for j in range(N):
distances[i, j] = np.linalg.norm(X[i] - X[j])
# k-nearest neighbor distances
lof_scores = np.ones(N)
for i in range(N):
# k-NN distances
knn_dist = np.sort(distances[i])[1:k+1]
k_dist = knn_dist[-1]
# Reachability distances
reach_dists = []
for j in range(N):
if i != j:
k_dist_j = np.sort(distances[j])[1:k+1][-1] if k < len(distances[j]) else distances[j].max()
reach_dist = max(distances[i, j], k_dist_j)
reach_dists.append(reach_dist)
# Local reachability density
lrd = k / (sum(reach_dists) + 1e-8) if reach_dists else 1
# LOF
lof = 0
for j in range(N):
if i != j:
k_dist_j = np.sort(distances[j])[1:k+1][-1] if k < len(distances[j]) else distances[j].max()
reach_dist_j = max(distances[i, j], k_dist_j)
lrd_j = k / (reach_dist_j + 1e-8)
lof += lrd_j / (lrd + 1e-8)
lof_scores[i] = lof / max(N - 1, 1)
return lof_scores
# Test
np.random.seed(42)
X = np.random.randn(20, 3)
lof = local_outlier_factor(X, k=3)
assert lof.shape == (20,), "LOF shape"
assert np.all(np.isfinite(lof)), "Finite LOF"
print("✓ Local Outlier Factor working")
if __name__ == "__main__":
print("Lab 2: LocalOutlierFactor - PASSED")### Lab 3: Autoencoder Reconstruction Error
import numpy as np
def reconstruction_error(X_original, X_reconstructed):
"""Compute reconstruction error for anomaly detection"""
error = np.sum((X_original - X_reconstructed) ** 2, axis=1)
return error
# Test
np.random.seed(42)
X_orig = np.random.randn(100, 50)
X_recon = X_orig + np.random.randn(100, 50) * 0.1
error = reconstruction_error(X_orig, X_recon)
assert error.shape == (100,), "Error shape"
assert np.all(error >= 0), "Non-negative"
print("✓ Reconstruction error working")
if __name__ == "__main__":
print("Lab 3: ReconstructionError - PASSED")### Lab 4: Anomaly Threshold Selection
import numpy as np
def select_anomaly_threshold(anomaly_scores, percentile=95):
"""Select threshold for anomaly detection"""
threshold = np.percentile(anomaly_scores, percentile)
return threshold
# Test
np.random.seed(42)
scores = np.random.rand(1000)
threshold = select_anomaly_threshold(scores, percentile=95)
assert 0 <= threshold <= 1, "Threshold in [0,1]"
anomalies = (scores > threshold).sum()
assert 40 < anomalies < 60, "Correct percentile"
print("✓ Anomaly threshold working")
if __name__ == "__main__":
print("Lab 4: AnomalyThreshold - PASSED")