orthogonal initialization

**Orthogonal Initialization** is a **weight initialization method that initializes weight matrices as orthogonal (or near-orthogonal) matrices** — ensuring that the linear transformation preserves the norm of the input at initialization, providing optimal signal propagation through deep networks. **How Does Orthogonal Initialization Work?** - **Process**: Generate a random matrix $A$ -> compute QR decomposition $A = QR$ -> use $Q$ (orthogonal matrix) as the initial weight. - **Property**: $||Qx|| = ||x||$ — an orthogonal matrix preserves vector norms. - **Gain**: Optionally multiply by a gain factor to account for the activation function (e.g., $sqrt{2}$ for ReLU). **Why It Matters** - **Perfect Propagation**: At initialization, signals neither grow nor shrink through orthogonal layers. - **RNNs**: Particularly important for recurrent networks where weights are applied repeatedly over time steps. - **Theory**: Theoretically optimal for signal propagation in linear networks (all singular values = 1). **Orthogonal Initialization** is **the norm-preserving start** — beginning training with transformations that perfectly preserve signal magnitude through every layer.