weight-sharing networks
**Weight-Sharing Networks** are **neural architectures where the same set of parameters is reused across multiple computational operations** — encoding the inductive bias that the same transformation applies in different contexts, dramatically reducing parameter count, enforcing equivariance, and enabling generalization across positions, time steps, or architectural configurations.
**What Are Weight-Sharing Networks?**
- **Definition**: Neural network architectures that constrain multiple operations to use identical parameters — rather than learning independent transformations for each position or context, the network learns a single transformation that applies universally.
- **Convolutional Neural Networks**: The canonical example — the same filter kernel applied at every spatial position, encoding translation equivariance (a cat detector works anywhere in the image).
- **Recurrent Neural Networks**: The same transition matrix applied at every time step — the same function processes word 1 and word 100.
- **Siamese Networks**: Two identical towers sharing all weights — the same feature extractor applied to both inputs for similarity comparison.
- **ALBERT**: Transformer with weight sharing across all layers — same attention and FFN weights repeated for every layer, reducing BERT parameters from 110M to 12M.
**Why Weight-Sharing Matters**
- **Parameter Efficiency**: Sharing weights across N positions reduces parameters by N× — CNNs would have millions more parameters without weight sharing; RNNs could not handle variable-length sequences.
- **Regularization**: Shared weights are a strong constraint on model complexity — prevents overfitting by forcing the model to learn general transformations, not position-specific memorization.
- **Inductive Bias**: Weight sharing encodes symmetries known about the domain — translation invariance for images, temporal stationarity for sequences, permutation invariance for sets.
- **Generalization**: A weight-shared model trained on sequences of length 10 generalizes to length 100 — the same transformation applies regardless of position.
- **NAS Weight Sharing**: One-shot NAS trains a single supernet with shared weights, then evaluates thousands of sub-architectures without retraining each.
**Types of Weight Sharing**
**Spatial Weight Sharing (CNNs)**:
- Same convolution kernel applied at every (x, y) position.
- Translation equivariance: f(shift(x)) = shift(f(x)).
- Enables detection of patterns regardless of their location in the image.
- Each filter learns a different feature (edge, texture, shape) applied globally.
**Temporal Weight Sharing (RNNs/LSTMs)**:
- Same transition matrices W_h and W_x applied at every time step.
- Enables processing variable-length sequences with fixed parameter count.
- Encodes assumption that dynamics are time-stationary.
**Cross-Layer Weight Sharing (Transformers)**:
- ALBERT: same attention and FFN weights used in all 12 (or 24) layers.
- Universal Transformer: recurrently applies same transformer block.
- Reduces parameter count dramatically; slight accuracy cost on most tasks.
**Siamese and Metric Learning**:
- Identical twin networks sharing all weights.
- Input pair (x1, x2) → shared encoder → distance function → similarity score.
- Ensures symmetric treatment: f(x1, x2) is consistent with f(x2, x1).
- Applications: face verification, document similarity, image retrieval.
**NAS Supernet Weight Sharing**:
- Supernet contains all possible architecture choices; sub-networks share weights.
- Evaluate 15,000+ architectures using shared weights — no per-architecture training.
- Once-for-All: single supernet that produces architectures for any hardware target.
**Weight Sharing vs. Related Concepts**
| Concept | What Is Shared | Mechanism | Purpose |
|---------|---------------|-----------|---------|
| **CNN filters** | Spatial positions | Convolution | Translation equivariance |
| **RNN transition** | Time steps | Recurrence | Temporal stationarity |
| **ALBERT layers** | Transformer layers | Parameter tying | Compression |
| **Siamese nets** | Twin branches | Identical architecture | Symmetric comparison |
| **NAS supernet** | Sub-architectures | Supernet weights | Search efficiency |
**Limitations of Weight Sharing**
- **Capacity**: Shared weights cannot model position-specific features — absolute position encodings compensate in Transformers.
- **Optimization Conflict**: In NAS supernets, different sub-architectures compete for the same shared weights — training instability.
- **Expressiveness**: Cross-layer sharing (ALBERT) trades accuracy for compression — fine-tuned BERT typically outperforms fine-tuned ALBERT.
**Tools and Implementations**
- **PyTorch nn.Module**: Weight sharing via simple variable reuse — assign same parameter to multiple layers.
- **HuggingFace Transformers**: ALBERT with weight sharing built-in.
- **timm**: Convolutional model zoo with standard weight-sharing CNN architectures.
- **NNI / AutoKeras**: Supernet-based NAS with weight sharing.
Weight-Sharing Networks are **the mathematical encoding of symmetry** — by forcing the same parameters to process different positions or contexts, these architectures build known invariances and equivariances directly into the model, achieving efficient generalization that unshared models cannot match.