Recommendation Systems Matrix Factorization Collaborative Filtering
# Recommendation Systems: Matrix Factorization & Collaborative Filtering
## Introduction & Motivation
Matrix factorization decomposes user-item rating matrix into low-rank latent factors, discovering hidden features explaining preferences. Scales to millions of users/items; captures complex interactions; foundation for Netflix, Amazon, Spotify recommendations.
Motivation: Rating matrix sparse (users rate few items). Low-rank assumption: preferences driven by few latent factors. Efficient; interpretable; enables cold-start handling via side information.
Applications: Movie recommendations, product suggestions, music discovery, news personalization.
---
## Core Concepts & Theory
### Latent Factor Model
User i preferences: latent vector \mathbf{u}_i ∈ ℝ^k.
Item j attributes: latent vector \mathbf{v}_j ∈ ℝ^k.
Rating prediction:
$$\hat{r}_{ij} = \mathbf{u}_i^T \mathbf{v}_j$$
---
## Mathematical Formulation
Objective (with regularization):
$$\min_{U, V} \sum_{(i,j) \in \Omega} (r_{ij} - \mathbf{u}_i^T \mathbf{v}_j)^2 + \lambda(\|\mathbf{U}\|_F^2 + \|\mathbf{V}\|_F^2)$$
where \Omega is observed ratings, \lambda is regularization.
---
## Advanced Theory & Extensions
### Bias Terms
Include user/item biases:
$$\hat{r}_{ij} = \mu + b_i + c_j + \mathbf{u}_i^T \mathbf{v}_j$$
### Implicit Feedback
Binary (view/no-view); weighted least squares.
### Temporal Dynamics
Time-varying factors capture evolving preferences.
---
## Computational Considerations
Training: O(|\Omega| imes k imes ext{iterations}) via SGD.
Inference: O(k) per prediction.
---
## Practical Implementation Strategies
### Factor Dimension
k = 10-100 typical; cross-validation guides selection.
### Regularization
Balance fit and overfitting; λ ∈ {0.01, 0.1, 1}.
### Evaluation
RMSE on heldout ratings; ranking metrics (NDCG, MAP).
---
## Benchmark Datasets & Evaluation
MovieLens: Standard benchmark; 100K-20M ratings.
Netflix: Prize competition dataset.
---
## Key Challenges & Limitations
### Cold-Start
New users/items with no ratings. Solutions: content features, popularity priors.
### Sparsity
Billions of possible ratings; millions observed. Solutions: implicit feedback, sampling.
---
## Hyperparameter Tuning
k \in {10, 50, 100\}, λ \in {0.001, 0.01, 0.1\}, learning_rate \in {0.001, 0.01, 0.1\}.
---
## Real-World Applications & Case Studies
Netflix: Recommendation engine combining multiple models.
Amazon: Product recommendations via implicit feedback.
---
## Integration with Other Methods
Matrix Factorization + Content Features → Handle cold-start.
Matrix Factorization + Deep Learning → Neural collaborative filtering.
---
## Future Research Directions
Knowledge graph embeddings for recommendations; fairness and diversity; explainability.
---
## Summary & Key Takeaways
Matrix factorization decomposes rating matrix into latent user/item factors, efficiently modeling preferences via low-rank approximation.
Principles:
1. Low-rank assumption captures latent preferences.
2. Factorization scales to millions of users/items.
3. Regularization prevents overfitting.
4. Bias terms improve accuracy.
5. Implicit feedback extends to unrated data.
---
---
## Appendix: Practical Labs
### Lab 1: Basic Matrix Factorization
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import NMF
# Use NMF as matrix factorization proxy
X = load_breast_cancer().data[:100, :10] # Subsample for demo
nmf = NMF(n_components=5, random_state=42, max_iter=200)
W = nmf.fit_transform(X)
H = nmf.components_
# Reconstruct
X_recon = W @ H
error = np.linalg.norm(X - X_recon, 'fro')
print(f"Reconstruction error: {error:.4f}")
assert error > 0, "Should have some error"
assert W.shape == (100, 5), "Should have 100 samples, 5 factors"
print("✓ Matrix factorization working")
if __name__ == "__main__":
print("Lab 1: Matrix Factorization - PASSED")### Lab 2: Latent Factors
from sklearn.decomposition import TruncatedSVD
import numpy as np
# Simulate user-item rating matrix
np.random.seed(42)
ratings = np.random.randint(1, 6, (50, 30)) # 50 users, 30 items
svd = TruncatedSVD(n_components=5, random_state=42)
user_factors = svd.fit_transform(ratings)
item_factors = svd.components_.T
print(f"User factors shape: {user_factors.shape}")
print(f"Item factors shape: {item_factors.shape}")
assert user_factors.shape == (50, 5), "Should have 50 users, 5 factors"
assert item_factors.shape == (30, 5), "Should have 30 items, 5 factors"
print("✓ Latent factors working")
if __name__ == "__main__":
print("Lab 2: Latent Factors - PASSED")### Lab 3: Rating Prediction
import numpy as np
from sklearn.decomposition import TruncatedSVD
# Simulate rating matrix
np.random.seed(42)
ratings = np.random.randint(1, 6, (50, 30))
svd = TruncatedSVD(n_components=5, random_state=42)
user_factors = svd.fit_transform(ratings)
item_factors = svd.components_.T
# Predict rating for user 0, item 5
predicted = np.dot(user_factors[0], item_factors[5])
actual = ratings[0, 5]
print(f"Predicted: {predicted:.2f}, Actual: {actual}")
assert 0 < predicted < 10, "Rating should be in reasonable range"
print("✓ Rating prediction working")
if __name__ == "__main__":
print("Lab 3: Rating Prediction - PASSED")### Lab 4: Factor Rank Selection
import numpy as np
from sklearn.decomposition import TruncatedSVD
np.random.seed(42)
ratings = np.random.randint(1, 6, (100, 50))
ranks = [2, 5, 10, 20, 50]
errors = []
for rank in ranks:
svd = TruncatedSVD(n_components=rank, random_state=42)
user_factors = svd.fit_transform(ratings)
item_factors = svd.components_.T
recon = user_factors @ item_factors.T
error = np.linalg.norm(ratings - recon, 'fro')
errors.append(error)
print(f"Rank {rank}: Error {error:.4f}")
# Error should decrease with rank
assert errors[0] > errors[-1], "Higher rank should reduce error"
print("✓ Rank selection working")
if __name__ == "__main__":
print("Lab 4: Rank Selection - PASSED")