dense model

Dense models activate all parameters for every input, the standard architecture for most neural networks. **Definition**: Every parameter participates in every forward pass. All weights used for all inputs. **Contrast with sparse**: Sparse/MoE models activate only subset of parameters per input. **Computation**: For dense transformer, FLOPs scale directly with parameter count. Larger model = more compute per token. **Memory**: All parameters must be in memory for inference. 70B model needs significant GPU memory. **Training**: Straightforward optimization. All parameters receive gradients every step. **Advantages**: Simpler architecture, well-understood training dynamics, consistent behavior across inputs. **Disadvantages**: Compute scales linearly with params. Eventually compute-inefficient at extreme scale. **Examples**: GPT-4 (rumored partially MoE but mostly dense), LLaMA, Claude, most deployed LLMs. **Trade-off with sparse**: Dense models have better predictable behavior; sparse models can be larger for same compute. **Current practice**: Dense remains dominant for most production deployments due to simplicity and reliability.