Eigenvalues

An eigenvector of a square matrix is a nonzero vector that the matrix only stretches or flips without turning it off its line, and the eigenvalue is the factor by which it is stretched. In symbols, A times v equals lambda times v, where A is the matrix, v the eigenvector and lambda the eigenvalue. For the matrix with rows (3, 1) and (1, 3), the vector (1, 1) is mapped to (4, 4), so its eigenvalue is 4, and the vector (1, −1) is mapped to (2, −2), so its eigenvalue is 2. The prefix comes from the German eigen, meaning own or characteristic, and the eigenvalues together are called the spectrum. Two facts are handy: the eigenvalues add up to the matrix's trace and multiply to its determinant.

Eigenvectors are the axes along which a linear transformation is simple, and for symmetric matrices they form a complete orthogonal set. The spectral theorem says a real symmetric matrix has real eigenvalues and mutually perpendicular eigenvectors, so the matrix can be written as a rotation, a pure stretching along each axis, and the inverse rotation. This eigendecomposition makes powers, inverses and other functions of a matrix easy: raising the matrix to the tenth power just raises each eigenvalue to the tenth power. Non-symmetric matrices can have complex eigenvalues, which signal rotation, and some cannot be diagonalized at all, which is one reason the singular value decomposition, which exists for every matrix and for rectangular ones, is often the tool actually used in practice.

Principal component analysis is the best-known machine learning use: its components are the eigenvectors of the data's covariance matrix. The eigenvector with the largest eigenvalue points along the direction in which the data vary most, and the eigenvalue itself is the variance along that direction, so the ratio of an eigenvalue to the sum of all of them is the fraction of variance explained. The idea goes back to Karl Pearson in 1901 and Harold Hotelling in 1933. Turk and Pentland's 1991 eigenfaces applied it to face images, and PCA is still the default tool for compression, visualization and decorrelating features. Google's PageRank, introduced by Larry Page and Sergey Brin in 1998, is also an eigenvector computation: the importance scores are the dominant eigenvector of the web's link matrix.

Eigenvalues also govern stability, because repeated multiplication scales each eigen-direction by its eigenvalue every step. In a recurrent network the hidden state is repeatedly multiplied by a weight matrix, so after t steps components along each eigenvector have been scaled by that eigenvalue to the power t: if the largest magnitude of any eigenvalue (the spectral radius) is above 1 gradients explode, and if it is below 1 they vanish, an analysis made precise by Pascanu, Mikolov and Bengio in 2013. The same idea underlies orthogonal initialization and spectral normalization (Miyato and colleagues, 2018), which divides a layer's weights by an estimate of their largest singular value. In optimization, the eigenvalues of the loss function's Hessian are its curvatures: for gradient descent on a quadratic, the learning rate must stay below 2 divided by the largest eigenvalue, and the ratio of largest to smallest eigenvalue, the condition number, measures how badly stretched and slow the problem is.

Power iteration finds the dominant eigenvector without a full decomposition. Start from a random vector, multiply by the matrix, rescale it to length one, and repeat; every step amplifies the component along the largest-magnitude eigenvector relative to the others, so the vector converges to it, at a rate set by the ratio of the second-largest to the largest eigenvalue magnitude. With the matrix above, the ratio is 2 over 4, so the error halves each step. The method needs only matrix-vector products, which is why it scales to enormous sparse matrices, and it is the core of PageRank computations and of the cheap largest-singular-value estimates used in spectral normalization. Lanczos and Arnoldi methods refine the same idea.

Pitfalls are mostly about overstating what eigenvalues say. Eigenvalues of a non-symmetric matrix do not fully describe its behavior: a matrix can have all eigenvalues of magnitude below 1 and still make vectors grow substantially for many steps before decaying, which is why singular values, not eigenvalues, bound how much a matrix can amplify a vector. Eigenvectors are only defined up to scale and sign, so PCA components can flip between runs without anything having changed, and equal eigenvalues leave the eigenvectors undetermined. PCA also finds directions of high variance, which need not be directions that matter for prediction, and it is sensitive to feature scaling, so features are usually standardized first. Direct eigendecomposition costs roughly n cubed operations, which is why large problems use iterative or randomized approximations.

Read eigenvalues as the stretch factors of a matrix along its special directions: eigenvectors give the natural axes, as in PCA, and the sizes of the eigenvalues decide whether repeated application grows, shrinks or stays stable and how hard an optimization landscape is to descend.