Vector

A vector is an ordered list of numbers, such as (3, 4) or the 768 numbers that describe a word to a language model, and it can be pictured as an arrow from the origin or a point in space whose coordinates are those numbers. Two operations define the structure: vectors of the same length can be added entry by entry, and any vector can be scaled by multiplying every entry by a number. Everything that obeys those rules, from arrows in the plane to lists of pixel values to polynomials, counts as a vector in the mathematical sense of a vector space, which is the setting for linear algebra. In programming, a vector is most often just a one-dimensional array, and its length is its dimension.

A vector carries both a direction and a magnitude, and machine learning cares about each separately. The magnitude is measured by a norm. The most common is the L2 norm, the square root of the sum of the squares of the entries, so that (3, 4) has length 5. The L1 norm adds absolute values, giving 7 for the same vector, and the L-infinity norm takes the largest absolute entry, giving 4. Dividing a vector by its L2 norm produces a unit vector that keeps the direction and discards the length, the operation behind cosine similarity and the normalization layers of embedding models. Penalties built from norms shape learning: weight decay adds the squared L2 norm of the weights to the loss, L1 penalties push many weights to exactly zero, and gradient clipping rescales a gradient vector whose norm exceeds a threshold.

Embeddings are vectors designed so that geometry encodes meaning. Tomas Mikolov and colleagues' 2013 word2vec work showed that words trained to predict their neighbors land at vectors in which relationships appear as directions, the famous example being that the vector for king minus man plus woman lands near queen. Modern models extend this to sentences, images and code: the base BERT model and GPT-2 small, for example, use 768-dimensional internal vectors, and embedding models return vectors of a few hundred to a few thousand numbers that a vector database stores and searches by nearness. Inside a transformer every token, at every layer, is a vector, and the entire network is a sequence of transformations on those vectors.

Treating the coordinates as meaningful is a classic mistake. The individual axes of an embedding are generally not interpretable features, since any rotation of the whole space preserves all distances and angles and so describes an equally valid embedding; meaning lives in directions and relative positions, not in dimension 217. Intuition from two or three dimensions also misleads in hundreds: in high-dimensional space, the distances between random points concentrate around similar values and random vectors are nearly orthogonal, which is both the reason nearest-neighbor search needs approximate indexes and a reason that cheap, random-looking projections often preserve structure. The Johnson-Lindenstrauss lemma, proved in 1984, makes that last point precise.

The word has several overlapping meanings that cause confusion. To a physicist a vector is a quantity with magnitude and direction, such as velocity or force, that must transform properly under change of coordinates; to a programmer it may be a resizable array of any type, as in C++'s std::vector; to a mathematician it is any element of a vector space. In machine learning the working meaning is a fixed-length array of floating-point numbers, conventionally treated as a column when writing equations and as a row in many code libraries, a convention difference that explains many transposition bugs when equations from papers are turned into code.

Read a vector as a point whose coordinates describe something, with a length and a direction that can each be measured: it is the basic datatype of machine learning, in which inputs, weights, gradients and embeddings are all vectors and meaning is carried by geometry.