collaborative filtering

**Collaborative filtering** is a **recommendation technique that suggests items based on similar users' preferences** — using the principle "users who liked X also liked Y" to predict what a user will enjoy, powering recommendations on Netflix, Amazon, Spotify, and most e-commerce and content platforms. **What Is Collaborative Filtering?** - **Definition**: Recommend based on collective user behavior patterns. - **Principle**: Similar users have similar tastes. - **Data**: User-item interactions (ratings, purchases, plays, clicks). - **Goal**: Predict user preferences from community patterns. **Types of Collaborative Filtering** **User-Based**: - **Method**: Find users similar to you, recommend what they liked. - **Steps**: 1) Find similar users, 2) Aggregate their preferences, 3) Recommend top items. - **Similarity**: Cosine similarity, Pearson correlation on rating vectors. - **Example**: "Users like you also enjoyed..." **Item-Based**: - **Method**: Find items similar to what you liked, recommend those. - **Steps**: 1) Find similar items, 2) Recommend items similar to user's favorites. - **Similarity**: Based on users who liked both items. - **Benefit**: More stable than user-based (item similarities change slowly). - **Example**: Amazon "Customers who bought this also bought..." **Matrix Factorization**: - **Method**: Decompose user-item matrix into latent factors. - **Techniques**: SVD, ALS (Alternating Least Squares), NMF. - **Benefit**: Handle sparse data, discover latent preferences. - **Example**: Netflix Prize winning approach. **Advantages** - **Serendipity**: Discover unexpected items you wouldn't search for. - **No Content Analysis**: Works without knowing item features. - **Collective Intelligence**: Leverage wisdom of crowds. - **Cross-Domain**: Patterns work across different item types. **Challenges** **Cold Start**: - **New Users**: No history to base recommendations on. - **New Items**: No ratings/interactions yet. - **Solutions**: Hybrid methods, ask preferences, use content features. **Sparsity**: - **Issue**: Most users interact with tiny fraction of items. - **Result**: Sparse user-item matrix, hard to find similarities. - **Solutions**: Matrix factorization, dimensionality reduction. **Scalability**: - **Issue**: Millions of users × millions of items = huge matrix. - **Solutions**: Approximate methods, sampling, distributed computing. **Popularity Bias**: - **Issue**: Popular items get more recommendations, rich get richer. - **Impact**: Niche items rarely recommended. - **Solutions**: Diversity metrics, exploration bonuses. **Shilling Attacks**: - **Issue**: Fake accounts manipulate recommendations. - **Example**: Competitors downvote products, inflate own ratings. - **Solutions**: Anomaly detection, trust metrics. **Algorithms** **K-Nearest Neighbors (KNN)**: Find K most similar users/items, aggregate preferences. **Matrix Factorization**: SVD, ALS, NMF for latent factor models. **Deep Learning**: Neural Collaborative Filtering, autoencoders, embeddings. **Applications** - **E-Commerce**: Amazon, eBay product recommendations. - **Streaming**: Netflix shows, Spotify music, YouTube videos. - **Social**: Facebook friend suggestions, LinkedIn connections. - **News**: Google News, personalized news feeds. - **Dating**: Match.com, Tinder compatibility. **Evaluation Metrics** - **Accuracy**: RMSE, MAE for rating prediction. - **Ranking**: Precision@K, Recall@K, NDCG, MAP. - **Coverage**: Percentage of items ever recommended. - **Diversity**: Variety in recommendations. - **Novelty**: Recommend unfamiliar items. **Tools & Libraries** - **Python**: Surprise, LightFM, Implicit, RecBole, TensorFlow Recommenders. - **Spark**: MLlib for distributed collaborative filtering. - **Cloud**: AWS Personalize, Google Recommendations AI, Azure Personalizer. Collaborative filtering is **the foundation of modern recommendations** — by leveraging collective user behavior, it enables personalized discovery at scale, helping users find items they'll love and businesses increase engagement and sales.