albert

ALBERT (A Lite BERT) reduces BERT parameters through factorization and sharing while maintaining performance. **Key techniques**: **Factorized embeddings**: Decompose large embedding matrix into two smaller matrices. E = V x 128, then 128 x H, instead of V x H directly. **Cross-layer sharing**: Share parameters across all transformer layers. Same weights reused. **Inter-sentence coherence**: Replace NSP with harder sentence ordering prediction task. **Parameter reduction**: ALBERT-xxlarge has 12x fewer parameters than BERT-large but more layers. **Trade-off**: Fewer parameters but similar or slower inference (same compute, weights reused). **Why it works**: Embeddings are over-parameterized, and layers learn similar functions. Sharing acts as regularization. **Variants**: ALBERT-base, large, xlarge, xxlarge. xxlarge has only 223M params but 12 layers shared. **Results**: Competitive with BERT-large using fraction of parameters. State-of-art at time on some benchmarks. **Use cases**: When parameter count matters (mobile, edge) more than inference speed.