T-Sne Umap Non-Linear Embedding Visualization

# t-SNE & UMAP: Non-Linear Embedding & Visualization

## Introduction & Motivation

t-SNE: t-Distributed Stochastic Neighbor Embedding. UMAP: Uniform Manifold Approximation and Projection. Non-linear dimensionality reduction. Applications: data visualization, cluster inspection, high-dimensional exploration.

Motivation: Preserve local and global structure; interpretable 2D/3D.

Applications: Visualization, exploration, debugging.

---

## Core Concepts & Theory

### Neighborhood Preservation

Local structure maintained.

### Manifold Learning

Underlying low-dimensional structure.

### Perplexity and Connectivity

Control neighborhood size; UMAP parameter.

---

## Mathematical Formulation

t-SNE probability (high-dim):
$$p_{j|i} = \frac{\exp(-\|x_i - x_j\|^2 / 2\sigma_i^2)}{\sum_{k eq i} \exp(-\|x_i - x_k\|^2 / 2\sigma_i^2)}$$

t-SNE probability (low-dim, t-distribution):
$$q_{ij} = \frac{(1 + \|y_i - y_j\|^2)^{-1}}{\sum_{k eq l} (1 + \|y_k - y_l\|^2)^{-1}}$$

KL divergence loss:
$$L = \sum_i \sum_j p_{ij} \log \frac{p_{ij}}{q_{ij}}$$

---

## Advanced Theory & Extensions

### UMAP

Graph-based non-linear reduction.

### Parametric t-SNE

Neural network for mapping.

### Multi-scale UMAP

Varying neighborhood sizes.

---

## Computational Considerations

t-SNE: O(N²) pairwise computations.

UMAP: O(N log N) with approximate NN.

Convergence: 1000+ iterations typical.

---

## Practical Implementation Strategies

### Perplexity Tuning

5-50 typical; larger for bigger datasets.

UMAP n_neighbors: 15-50; local connectivity.

### Learning Rate Scheduling

Adaptive or warm-up.

---

## Benchmark Datasets & Evaluation

MNIST: Classic embedding benchmark.

Swiss Roll: Non-linear structure test.

Real-world: High-dimensional datasets.

---

## Key Challenges & Limitations

### Crowding Problem

Distances misrepresented; t-SNE.

### Parametric Sensitivity

Perplexity/n_neighbors affect results.

### Computational Cost

Slow for very large datasets.

---

## Hyperparameter Tuning

Perplexity (t-SNE): 5-50; neighborhood.

n_neighbors (UMAP): 15-50; connectivity.

n_components: 2-3 for visualization.

---

## Real-World Applications & Case Studies

Clustering Inspection: Verify cluster quality.

Gene Expression: Biological visualization.

Document Exploration: Text embedding vis.

---

## Integration with Other Methods

Embedding + Clustering → visual inspection.

Embedding + NN → parametric mapping.

---

## Summary & Key Takeaways

t-SNE and UMAP via non-linear embedding enable interpretable high-dimensional visualization through neighborhood preservation and manifold learning.

Principles:
1. Neighborhood: local structure.
2. Manifold: underlying geometry.
3. KL divergence: distribution matching.
4. Perplexity: effective neighbors.
5. Scalability: UMAP vs t-SNE.

---

---

## Appendix: Practical Labs

### Lab 1: t-SNE Similarities

import numpy as np

def compute_tsne_similarities_hd(X, perplexity=30):
 """Compute t-SNE similarity matrix (high-dim)"""
 N = len(X)
 distances_sq = np.sum((X[:, None, :] - X[None, :, :]) ** 2, axis=2)
 
 # Compute for each point
 P = np.zeros((N, N))
 
 for i in range(N):
 # Adjust sigma for each point based on perplexity
 sigma = 1.0
 P_i = np.exp(-distances_sq[i] / (2 * sigma ** 2))
 P_i[i] = 0 # No self-similarity
 P[i] = P_i / (P_i.sum() + 1e-8)
 
 return P

# Test
np.random.seed(42)
X = np.random.randn(100, 50)

P = compute_tsne_similarities_hd(X)

assert P.shape == (100, 100), "Similarity matrix shape"
assert np.allclose(P, P.T), "Symmetrize: (P + P.T) / 2"
print("✓ t-SNE similarities working")

if __name__ == "__main__":
 print("Lab 1: tSNESimilarities - PASSED")

### Lab 2: UMAP Graph Construction

import numpy as np

def umap_knn_graph(X, n_neighbors=15):
 """Construct k-NN graph for UMAP"""
 N = len(X)
 distances = np.zeros((N, N))
 
 for i in range(N):
 for j in range(N):
 distances[i, j] = np.linalg.norm(X[i] - X[j])
 
 # Find k-nearest neighbors
 knn_indices = np.argsort(distances, axis=1)[:, :n_neighbors+1]
 
 # Build adjacency
 graph = np.zeros((N, N))
 for i in range(N):
 graph[i, knn_indices[i]] = 1
 
 # Make symmetric
 graph = (graph + graph.T) / 2
 
 return graph

# Test
np.random.seed(42)
X = np.random.randn(50, 10)

graph = umap_knn_graph(X, n_neighbors=10)

assert graph.shape == (50, 50), "Graph shape"
assert np.all((graph == 0) | (graph == 1)), "Binary adjacency"
print("✓ UMAP graph working")

if __name__ == "__main__":
 print("Lab 2: UMAPGraph - PASSED")

### Lab 3: Low-Dimensional Similarities (t-SNE)

import numpy as np

def compute_tsne_similarities_ld(Y):
 """Compute t-SNE similarity matrix (low-dim, t-distribution)"""
 distances_sq = np.sum((Y[:, None, :] - Y[None, :, :]) ** 2, axis=2)
 
 # t-distribution: (1 + d²)^{-1}
 Q = (1.0 + distances_sq) ** (-1)
 np.fill_diagonal(Q, 0)
 
 # Normalize
 Q = Q / Q.sum()
 
 return Q

# Test
np.random.seed(42)
Y = np.random.randn(100, 2)

Q = compute_tsne_similarities_ld(Y)

assert Q.shape == (100, 100), "Similarity matrix shape"
assert np.allclose(Q.sum(), 1.0), "Normalized"
print("✓ Low-dim similarities working")

if __name__ == "__main__":
 print("Lab 3: LowDimSimilarities - PASSED")

### Lab 4: KL Divergence Loss

import numpy as np

def tsne_kl_loss(P, Q):
 """Compute KL divergence loss for t-SNE"""
 # Add small epsilon for numerical stability
 P = np.maximum(P, 1e-12)
 Q = np.maximum(Q, 1e-12)
 
 # KL divergence: sum_ij P_ij log(P_ij / Q_ij)
 kl_loss = np.sum(P * (np.log(P) - np.log(Q)))
 
 return kl_loss

# Test
np.random.seed(42)
P = np.random.rand(100, 100)
P = P / P.sum() # Normalize
Q = np.random.rand(100, 100)
Q = Q / Q.sum() # Normalize

loss = tsne_kl_loss(P, Q)

assert np.isfinite(loss), "Loss finite"
assert loss >= 0, "KL divergence non-negative"
print("✓ KL loss working")

if __name__ == "__main__":
 print("Lab 4: KLLoss - PASSED")