Support Vector Machine

A support vector machine finds the single boundary that separates two classes with the widest possible margin on either side, and once trained, only the data points sitting exactly on that margin — the support vectors — actually matter; every other point could be deleted from the dataset without changing the boundary at all. Vladimir Vapnik and colleagues built the modern version in two steps: Boser, Guyon, and Vapnik introduced the kernel-based version in 1992, and Cortes and Vapnik added the soft-margin formulation in 1995 that let the method tolerate some misclassified points rather than requiring perfect separation. Before deep learning became the default around 2012, SVMs with the right kernel were genuinely state-of-the-art — Cortes and Vapnik's original paper reported roughly 98.6% accuracy classifying handwritten digits from an early version of the dataset that later became MNIST, competitive with the best neural networks of that era.

The margin's width is the entire optimization target, and "support vector" describes exactly the points that pin it in place. Among every possible line (or hyperplane, in higher dimensions) that separates two classes, an SVM picks the one maximizing the distance to the nearest point of either class — and that nearest point, by definition, is a support vector. Move a non-support-vector point anywhere, as long as it stays on the correct side of the margin, and the solution doesn't change at all; move a support vector even slightly, and the entire boundary shifts, which is why a trained SVM's model can be fully described using only those few points instead of the whole training set.

The C parameter is the dial that decides how much the model is allowed to tolerate imperfection, and it governs the same bias-variance trade-off every other model faces. A very large C forces a hard margin, punishing every misclassified point heavily and producing a boundary that fits the training data closely but can be thrown off by a single noisy outlier; a small C accepts more training-set violations in exchange for a wider, more conservative margin that tends to generalize better. This single number is usually the first thing tuned when fitting an SVM, in exactly the hyperparameter-search sense covered under hyperparameter.

The kernel trick is how an SVM handles data that no straight line can separate, without ever computing the expensive transformation that would normally require. A radial basis function or polynomial kernel computes what the dot product between two points *would be* if they'd been mapped into a much higher-dimensional space — sometimes an infinite-dimensional one, in the RBF case — without ever actually constructing that mapped representation, which is the part of the method genuinely clever enough to earn the name "trick." Two classes shaped like concentric circles in two dimensions, impossible to separate with any straight line, become linearly separable once that kernel implicitly lifts them into a higher-dimensional space, and the SVM finds its maximum-margin hyperplane there instead.

Deep learning overtook SVMs on the problems where huge labeled datasets and raw unstructured inputs — images, audio, text — made learned representations beat hand-chosen kernels, but the comparison isn't a clean replacement: on small-to-medium structured datasets, text classification with simple features, and bioinformatics problems with limited labeled examples, a well-tuned SVM remains a legitimate, often faster-to-train competitor, which is precisely the regime where it was strongest even before 2012 and where the parallel to today's tabular-data preference for random forest and gradient boosting over deep networks is not a coincidence — fewer labeled examples favor models with strong built-in structure over ones that need to learn everything from scratch.