glam (generalist language model)

GLaM (Generalist Language Model) is Google's sparse Mixture of Experts language model containing 1.2 trillion parameters that demonstrated how MoE architectures can achieve state-of-the-art performance while using significantly less computation than dense models of comparable quality. Introduced by Du et al. in 2022, GLaM showed that a sparsely activated model activating only about 97B parameters per token (8% of total) could match or exceed the quality of dense GPT-3 175B while requiring approximately 1/3 the energy for training and 1/2 the computation per inference step. GLaM's architecture uses 64 experts per MoE layer with top-2 gating (each token routed to 2 of 64 experts), replacing the standard dense feedforward network in every other transformer layer with an MoE layer. The model has 64 decoder layers, and alternating between dense and MoE layers balances model quality with computational efficiency. Training used 1.6 trillion tokens from a diverse web corpus filtered for quality. Key findings from the GLaM paper include: sparse MoE models achieve better zero-shot and one-shot performance than proportionally-more-expensive dense models (GLaM outperformed GPT-3 on 7 of 8 evaluation tasks in zero-shot settings while using 3× less energy to train), the importance of data quality (GLaM placed significant emphasis on training data filtering, demonstrating that data quality is crucial for large sparse models), and the energy efficiency of sparse computation (the paper explicitly analyzed and compared total training energy consumption, highlighting environmental benefits). GLaM's significance lies in providing strong empirical evidence that the future of scaling language models involves sparse architectures — achieving greater intelligence by increasing parameter count without proportionally increasing computation. This insight influenced subsequent MoE models including Switch Transformer, Mixtral, and likely GPT-4's rumored MoE architecture.