learned layer selection

**Learned Layer Selection** is a **conditional computation method where a trainable routing policy determines which layers or computational blocks to execute for each specific input, using differentiable gating mechanisms that output binary execute/skip decisions or continuous weighting factors for each layer** — enabling the network to learn data-dependent processing paths that allocate depth where it is needed, creating input-specific sub-networks within a single shared architecture. **What Is Learned Layer Selection?** - **Definition**: Learned layer selection adds a lightweight gating module at each layer (or block) of a neural network. The gate takes the incoming hidden state as input and produces a decision: execute this layer's full computation, or skip it via the residual connection. The gating policy is trained jointly with the main network parameters, learning which inputs benefit from which layers. - **Gating Architecture**: The gate is typically a single linear projection from the hidden dimension to a scalar, followed by a sigmoid activation. During training, the continuous sigmoid output is converted to a discrete binary decision using Gumbel-Softmax or straight-through estimator techniques that allow gradient flow through the discrete choice. - **Sparsity Regularization**: Without constraints, the gate may learn to always execute all layers (no efficiency gain) or skip all layers (quality collapse). A sparsity regularization loss encourages a target computation budget — e.g., "on average, execute 60% of layers" — balancing quality and efficiency. **Why Learned Layer Selection Matters** - **Input-Adaptive Depth**: Unlike static layer pruning (which removes the same layers for all inputs), learned selection creates different effective network architectures for different inputs. A simple input might activate 12 of 32 layers while a complex input activates 28 — automatically matching compute to difficulty without manual threshold tuning. - **Interpretability**: The learned routing patterns reveal which layers are important for which types of inputs. Analysis of routing decisions often shows that early layers (handling syntax and local patterns) are activated for most inputs, while deep layers (handling long-range reasoning and world knowledge) are activated primarily for complex queries — aligning with intuitions about hierarchical representation learning. - **Training Efficiency**: Gumbel-Softmax and straight-through estimators enable end-to-end differentiable training of the discrete gating policy, avoiding the sample inefficiency of reinforcement learning approaches. The gate parameters converge quickly because the gating module is small (single linear layer per block) relative to the main network. - **Deployment Simplicity**: At inference time, the gating decision is a single matrix multiplication + threshold per layer — adding negligible overhead while potentially skipping millions of FLOPs in the skipped layer's attention and feed-forward computation. **Gating Mechanism** For input hidden state $h$ at layer $l$, the gate computes: $g_l = sigma(W_l cdot h + b_l)$ If $g_l > au$ (threshold), execute layer $l$: $h_{l+1} = ext{Layer}_l(h_l) + h_l$ If $g_l leq au$, skip layer $l$: $h_{l+1} = h_l$ During training, $g_l$ is sampled from Gumbel-Softmax for differentiable binary decisions. At inference, hard thresholding is used for maximum speed. **Learned Layer Selection** is **dynamic pathing** — letting each input token discover its own route through the neural network, executing only the layers that contribute meaningful computation to its representation while bypassing redundant processing.