dot product similarity

Dot product similarity measures vector similarity as their inner product, fundamental to attention and retrieval. **Formula**: A dot B = sum(a_i * b_i). Unbounded range. **Interpretation**: Higher = more similar (for unit vectors, equals cosine). Magnitude matters - longer vectors have higher products. **Relation to cosine**: For normalized vectors, dot product equals cosine similarity. Many systems normalize embeddings. **In attention**: Query dot key determines attention weight. High dot product = strong attention. Scaled by sqrt(d_k) for stability. **For retrieval**: Fast to compute, hardware-optimized (BLAS), works well for normalized embeddings. **Maximum Inner Product Search (MIPS)**: Find vectors with highest dot product with query. Common retrieval formulation. **When to use dot product vs cosine**: Dot product when magnitude is meaningful (confidence, importance). Cosine when only direction matters. **Implementation**: Highly optimized in linear algebra libraries. GPUs excel at batch dot products. **Vector databases**: Support dot product and cosine, often convert between using normalization.