Feature

A feature is an individual measurable input variable used to predict an outcome — the machine-learning-era term for what statistics calls an independent variable. A customer's age, order history, and browser type are features in a churn model; pixel intensity values are features in an image classifier; word frequencies are features in a text classifier. The shift in vocabulary from "independent variable" to "feature" tracks a shift in practice: statistics traditionally works with a small, deliberately chosen set of variables selected for their theoretical relevance, while machine learning routinely works with hundreds or thousands of inputs, many constructed rather than directly observed, which made "feature" the more natural term for a field where the inputs themselves are engineered artifacts, not just measurements taken off the shelf.

Feature engineering — the practice of transforming raw data into the input variables a model actually trains on — is widely regarded by practitioners as having more impact on a model's final performance than the choice of algorithm itself, a view reflected in Andrew Ng's frequently cited observation that applied machine learning is "basically feature engineering." Raw data rarely arrives in a form a model can use directly: a timestamp has no inherent predictive structure until it's decomposed into day-of-week, hour, or time-since-last-purchase; raw text has to become word counts, embeddings, or other numeric representations before most models can touch it. The engineering is the translation from "data that exists" to "data that carries signal a model can actually learn from."

Feature selection and dimensionality reduction address the opposite problem: too many features rather than too few, which degrades model performance through the curse of dimensionality — a term introduced by Richard Bellman in 1961, describing how the volume of space a model needs to cover grows exponentially with the number of input dimensions, making data sparse and patterns harder to detect as more features are added. Feature selection methods (filtering by statistical relevance, or wrapper methods that test subsets against model performance directly) discard features that add noise without adding predictive signal. Dimensionality reduction techniques like principal component analysis instead compress many correlated features into a smaller number of synthetic ones that retain most of the original signal — both approaches exist because adding more raw features to a model is not reliably an improvement and is often a direct cause of overfitting.

Feature scaling — normalizing features onto comparable ranges, typically through standardization (subtracting the mean, dividing by standard deviation) or min-max scaling (rescaling to a fixed range like 0 to 1) — is a preprocessing step required by many algorithms but not all, and applying it incorrectly or inconsistently is a common source of silently degraded model performance. Distance-based algorithms like k-nearest neighbors and gradient-descent-trained models like neural networks and logistic regression are sensitive to feature scale, because a feature measured in the thousands (annual income) can numerically dominate a feature measured in single digits (number of children) even if the smaller-scale feature is more predictive. Tree-based models like random forests and gradient-boosted trees are scale-invariant by construction and don't require this step, which is why scaling choices need to be matched to the specific algorithm being used rather than applied as a universal default.

Feature importance — a ranking of which features contribute most to a trained model's predictions — has become central to both model debugging and model explainability, with methods ranging from simple (a tree-based model's built-in importance scores) to model-agnostic (SHAP values, introduced by Scott Lundberg and Su-In Lee in 2017, which attribute each prediction's output to individual feature contributions using a game-theoretic framework). This matters beyond curiosity: a feature importance analysis that surfaces a protected or legally sensitive attribute (zip code functioning as a proxy for race, for instance) as highly predictive is often the first signal that a model has learned a pattern that creates regulatory or ethical exposure, which is why feature importance review has become a standard step in model governance for models deployed in regulated domains like lending and hiring.

Read a feature as the unit of translation between raw, messy data and what a model can actually learn from: the engineering work of constructing good features routinely outweighs the choice of algorithm in determining final performance, and the companion disciplines of selection, scaling, and importance analysis exist because having more features is not automatically better, and knowing which features actually drive a model's output is a prerequisite for trusting what it does.