Dot Product
The dot product of two vectors of equal length is the sum of the products of their matching entries, giving a single number. For (1, 2, 3) and (4, −5, 6), the result is 1 × 4 + 2 × (−5) + 3 × 6 = 12. Geometrically the same number equals the length of the first vector times the length of the second times the cosine of the angle between them, so it is large and positive when the vectors point the same way, zero when they are perpendicular (orthogonal), and negative when they point roughly opposite ways. It is also called the scalar product or, more generally, the inner product, and it is the building block from which matrix multiplication is made.
The dot product is the basic measure of how aligned two vectors are, which is why it doubles as a similarity score. Dividing the dot product by both lengths removes the effect of size and leaves the cosine of the angle, the cosine similarity: for the example above, 12 divided by (3.74 × 8.77) is about 0.37, a modest positive alignment. Search over embeddings works this way, ranking stored vectors by their dot product with the query vector. Many embedding models return unit-length vectors, in which case the dot product and cosine similarity coincide; when vectors are not normalized, the dot product also rewards longer vectors, so the choice between them changes the ranking and should follow how the model was trained.
Every neuron computes a dot product, and attention is built on them. A unit in a neural network multiplies its input vector by its weight vector, sums, and adds a bias before applying a nonlinearity, so the weights act as a template and the dot product measures how well the input matches it. In the transformer of Vaswani and colleagues (2017), the score between a query and a key is their dot product, divided by the square root of the key dimension. The scaling exists because, for vectors with independent unit-variance entries, the dot product has variance equal to the dimension, so unscaled scores grow with size, push the softmax into saturation and shrink gradients. The softmax of these scores becomes the weights used to blend value vectors, and computing all the scores at once is a single matrix multiplication.
The dot product is the bridge between algebra and geometry, and several properties make it useful. It is symmetric, and it is linear in each argument, so it distributes over addition. The dot product of a vector with itself equals its squared length, which links it to the norm and to Euclidean distance: the squared distance between two vectors equals the sum of their squared lengths minus twice their dot product. Projecting one vector onto another uses it directly, because the dot product with a unit vector gives the component of the other vector along that direction. Orthogonality, a dot product of zero, underlies principal components, Fourier bases and orthogonal weight initializations.
Typical pitfalls come from confusing it with related notions. The dot product is not the cross product, which exists only in three dimensions in its usual form and returns a vector; it is not element-wise multiplication, which keeps all the products as a vector instead of summing them; and it is not magnitude-free, which is a frequent source of retrieval bugs when one set of vectors is normalized and another is not. Zero dot product means orthogonal, not unrelated, and in very high dimensions two random vectors are nearly orthogonal anyway, so small scores carry little information. Maximum inner product search, finding the stored vector with the largest dot product, is not the same problem as nearest-neighbor search by distance, which is why some vector indexes need special handling for it. In floating-point code, long dot products accumulate rounding error, which is why accelerators accumulate in higher precision than the inputs.
Read the dot product as a single number that says how much two vectors point the same way, scaled by their lengths: it is the similarity score of embedding search, the matching step of every neuron, and the score inside attention.