dynamic neural networks

**Dynamic Neural Networks** are **neural networks whose architecture, parameters, or computational graph change during inference** — adapting their structure based on the input, resource constraints, or other runtime conditions, in contrast to static networks with fixed computation. **Types of Dynamic Networks** - **Dynamic Depth**: Vary the number of layers executed per input (early exit, skip connections). - **Dynamic Width**: Vary the number of channels or neurons per layer (slimmable networks). - **Dynamic Routing**: Route inputs through different paths in the network (MoE, capsule routing). - **Dynamic Parameters**: Generate parameters conditioned on the input (hypernetworks, dynamic convolutions). **Why It Matters** - **Efficiency**: Adapt computation to input difficulty — easy inputs use less computation. - **Flexibility**: One model serves multiple deployment scenarios with different resource budgets. - **State-of-Art**: Large language models (GPT-4, Mixtral) use dynamic routing (MoE) for efficient scaling. **Dynamic Neural Networks** are **shape-shifting models** — adapting their own architecture and computation at inference time for maximum flexibility and efficiency.