weight sharing

Weight sharing uses the same parameters across multiple parts of a model, reducing parameter count significantly. **Applications**: **Tied embeddings**: Input and output embeddings share weights. Common in language models. Reduces parameters by vocabulary_size x hidden_dim. **Layer sharing**: Same layer weights used at multiple depths (ALBERT). Reduces params proportional to sharing factor. **Convolutional**: CNNs inherently share weights across spatial positions. Core idea enabling efficient image processing. **Universal transformers**: Share transformer layer weights across all depths. **Benefits**: Fewer parameters, regularization effect (constraints model), smaller storage. **Trade-offs**: May limit capacity, same computation as unshared (in inference). Memory savings primarily in weight storage. **ALBERT analysis**: 18x fewer parameters than BERT-large with similar performance through aggressive sharing. **Tied embeddings specifically**: Very common, virtually free improvement. Language models almost always tie input/output embeddings. **Implementation**: Simply use same nn.Parameter object in multiple places. Gradients accumulate from all uses. **When to use**: Parameter-constrained settings, when similar computation appropriate at multiple locations.