grid search

Grid search is a hyperparameter optimization method that exhaustively evaluates all possible combinations from a predefined grid of hyperparameter values, guaranteeing that the best combination within the search space is found at the cost of exponential computational requirements. For each hyperparameter, the user specifies a finite set of candidate values — for example, learning_rate: [1e-4, 1e-3, 1e-2], batch_size: [16, 32, 64], weight_decay: [0.01, 0.1] — and grid search trains and evaluates a model for every combination (3 × 3 × 2 = 18 configurations in this example). The method is straightforward to implement: nested loops iterate over parameter combinations, each configuration is trained (often with k-fold cross-validation), and the combination achieving the best validation performance is selected. Advantages include: simplicity (easy to implement and understand), completeness (within the defined grid, the optimal combination is guaranteed to be found), parallelizability (each configuration is independent and can be evaluated simultaneously), and reproducibility (deterministic search space fully specifies what was tried). However, grid search suffers from the curse of dimensionality — the number of evaluations grows exponentially with the number of hyperparameters: with d hyperparameters each having v values, the grid contains v^d points. Five hyperparameters with 5 values each requires 3,125 training runs. This makes grid search impractical for more than 3-4 hyperparameters. Furthermore, grid search allocates equal evaluation budget across all parameters regardless of their importance — if only one of four hyperparameters significantly affects performance, 75% of the compute is wasted on unimportant dimensions. For these reasons, random search (Bergstra and Bengio, 2012) often outperforms grid search by concentrating evaluations on the few hyperparameters that matter most. Grid search remains useful for fine-grained tuning of 1-3 critical hyperparameters after broader search methods have identified the important ranges.