Principal Component Analysis Pca Dimensionality Reduction

# Principal Component Analysis: PCA & Dimensionality Reduction

## Introduction & Motivation

Principal Component Analysis: linear dimensionality reduction. Finds directions of maximum variance. Orthogonal components; interpretable projections. Applications: visualization, feature extraction, denoising.

Motivation: Reduce dimensions while preserving variance.

Applications: Visualization, feature extraction.

---

## Core Concepts & Theory

### Principal Components

Directions of maximum variance.

### Eigenvalues

Variance explained per component.

### Projection

Linear transformation to lower dimensions.

---

## Mathematical Formulation

Covariance matrix:
$$\Sigma = \frac{1}{N} X^T X$$

Eigendecomposition:
$$\Sigma = U \Lambda U^T$$

PCA projection:
$$Y = X U_k$$

where U_k contains top k eigenvectors.

---

## Advanced Theory & Extensions

### Kernel PCA

Non-linear PCA via kernel trick.

### Incremental PCA

Process data in batches; scalability.

### Probabilistic PCA

Bayesian interpretation; missing data.

---

## Computational Considerations

SVD: O(N·D·min(N,D)) where D=dimensions.

Eigendecomposition: O(D³).

Projection: O(N·D·K) where K=components.

---

## Practical Implementation Strategies

### Centering

Subtract mean before PCA.

### Scaling

Normalize features to unit variance.

### Variance Explained

Select K from cumulative variance.

---

## Benchmark Datasets & Evaluation

MNIST: Digit visualization.

Iris: Classic dimensionality reduction.

High-dimensional: Benchmark suites.

---

## Key Challenges & Limitations

### Linear Assumption

PCA only linear transformations.

### Interpretation

Components are linear combinations.

### Outlier Sensitivity

Sensitive to extreme values.

---

## Hyperparameter Tuning

Number of components K: Cumulative variance (95%).

Scaling: Unit variance or standardization.

Centering: Always center data.

---

## Real-World Applications & Case Studies

Data Visualization: Reduce to 2D/3D.

Feature Extraction: Preprocessing step.

Image Compression: Eigenfaces.

---

## Integration with Other Methods

PCA + Clustering → visualization.

PCA + Regression → feature reduction.

---

## Summary & Key Takeaways

Principal Component Analysis via orthogonal projection enables linear dimensionality reduction through variance maximization and eigenvector computation.

Principles:
1. Variance: maximize along components.
2. Orthogonality: perpendicular projections.
3. Eigendecomposition: component extraction.
4. Interpretability: linear combinations.
5. Applications: visualization, compression.

---

---

## Appendix: Practical Labs

### Lab 1: Covariance Matrix

import numpy as np

def compute_covariance(X):
 """Compute covariance matrix"""
 # Center data
 X_centered = X - X.mean(axis=0)
 
 # Covariance
 cov = (X_centered.T @ X_centered) / (len(X) - 1)
 
 return cov

# Test
np.random.seed(42)
X = np.random.randn(100, 5)

cov = compute_covariance(X)

assert cov.shape == (5, 5), "Covariance shape"
assert np.allclose(cov, cov.T), "Covariance symmetric"
print("✓ Covariance matrix working")

if __name__ == "__main__":
 print("Lab 1: CovarianceMatrix - PASSED")

### Lab 2: Eigendecomposition and Components

import numpy as np

def pca_fit(X, num_components=2):
 """Fit PCA model"""
 # Center data
 X_centered = X - X.mean(axis=0)
 
 # Covariance
 cov = (X_centered.T @ X_centered) / (len(X) - 1)
 
 # Eigendecomposition
 eigenvalues, eigenvectors = np.linalg.eigh(cov)
 
 # Sort by eigenvalue (descending)
 idx = np.argsort(eigenvalues)[::-1]
 eigenvalues = eigenvalues[idx]
 eigenvectors = eigenvectors[:, idx]
 
 # Select top components
 components = eigenvectors[:, :num_components]
 variance_explained = eigenvalues[:num_components]
 
 return components, variance_explained

# Test
np.random.seed(42)
X = np.random.randn(100, 5)

components, variance = pca_fit(X, num_components=2)

assert components.shape == (5, 2), "Components shape"
assert len(variance) == 2, "Variance shape"
assert np.all(np.diff(variance)[::-1] >= 0), "Decreasing variance"
print("✓ PCA fit working")

if __name__ == "__main__":
 print("Lab 2: PCAFit - PASSED")

### Lab 3: PCA Projection

import numpy as np

def pca_transform(X, components):
 """Project data onto principal components"""
 # Center using training mean (should be computed on train set)
 X_centered = X - X.mean(axis=0)
 
 # Project
 Y = X_centered @ components
 
 return Y

# Test
np.random.seed(42)
X = np.random.randn(100, 5)
X_test = np.random.randn(20, 5)

# Fit PCA
X_centered = X - X.mean(axis=0)
cov = (X_centered.T @ X_centered) / (len(X) - 1)
eigenvalues, eigenvectors = np.linalg.eigh(cov)
idx = np.argsort(eigenvalues)[::-1]
components = eigenvectors[:, idx][:, :2]

# Transform test data
Y_test = pca_transform(X_test, components)

assert Y_test.shape == (20, 2), "Projection shape"
print("✓ PCA transform working")

if __name__ == "__main__":
 print("Lab 3: PCATransform - PASSED")

### Lab 4: Variance Explained

import numpy as np

def variance_explained_ratio(eigenvalues):
 """Compute variance explained ratio"""
 total_variance = eigenvalues.sum()
 variance_ratio = eigenvalues / total_variance
 
 # Cumulative variance
 cumsum_var = np.cumsum(variance_ratio)
 
 return variance_ratio, cumsum_var

# Test
np.random.seed(42)
eigenvalues = np.sort(np.random.rand(10))[::-1]

var_ratio, cumsum = variance_explained_ratio(eigenvalues)

assert len(var_ratio) == 10, "Variance ratio length"
assert np.allclose(cumsum[-1], 1.0), "Cumulative variance sums to 1"
assert np.all(np.diff(cumsum) >= 0), "Monotonically increasing"
print("✓ Variance explained working")

if __name__ == "__main__":
 print("Lab 4: VarianceExplained - PASSED")