random search

Random search is a hyperparameter optimization method that samples random combinations from specified hyperparameter distributions, providing surprisingly effective optimization that often outperforms grid search despite its apparent simplicity. Introduced as a formal hyperparameter optimization strategy by Bergstra and Bengio (2012), random search works by defining probability distributions for each hyperparameter (uniform, log-uniform, categorical, etc.) rather than discrete grids, then independently sampling N configurations and evaluating each. The key theoretical insight explaining random search's effectiveness: in most machine learning problems, a small number of hyperparameters matter much more than others. Grid search allocates points uniformly across all dimensions, wasting most evaluations on unimportant parameters. Random search, by contrast, projects to a different value for every trial on every dimension — with N random trials, each important hyperparameter sees N distinct values regardless of how many unimportant hyperparameters exist. This means random search explores important dimensions more efficiently than grid search with the same budget. For example, with 64 evaluations over 4 hyperparameters: grid search provides a 64^(1/4) ≈ 2.8 → approximately 3 values per hyperparameter. Random search provides 64 unique values per hyperparameter projected onto each axis. Distribution choices are critical: learning rates typically use log-uniform (sampling uniformly in log space — equally likely to try 1e-5, 1e-4, or 1e-3), dropout rates use uniform (0.0 to 0.5), hidden dimensions use discrete uniform or log-uniform, and categorical choices use uniform categorical. Advantages include: better coverage of important hyperparameter dimensions, easy parallelization, anytime behavior (each additional trial improves the estimate — can stop early if budget is exhausted), and no assumptions about hyperparameter importance. Random search serves as a strong baseline that more sophisticated methods (Bayesian optimization, Hyperband, TPE) must outperform to justify their complexity. In practice, random search with 60 trials finds configurations within the top 5% of the search space with high probability.