model size

Model size refers to the amount of storage space required to store a neural network's weights and associated metadata, determining the hardware requirements for loading, serving, and deploying the model. While closely related to parameter count, model size also depends on numerical precision — the same parameters stored at different precisions yield different file sizes. Precision formats and their per-parameter storage requirements: FP32 (full precision — 4 bytes/parameter, used in traditional training), FP16/BFloat16 (half precision — 2 bytes/parameter, standard for inference and mixed-precision training), INT8 (8-bit quantization — 1 byte/parameter, common for efficient deployment), INT4/NF4 (4-bit quantization — 0.5 bytes/parameter, aggressive compression for consumer hardware), and INT2/ternary (research-stage extreme quantization). Example model sizes: LLaMA-2 7B at FP16 requires ~14GB, at INT8 requires ~7GB, and at INT4 requires ~3.5GB. GPT-3 175B at FP16 would require ~350GB, necessitating multiple GPUs. Model size determines deployment feasibility: consumer GPUs typically have 8-24GB VRAM (limiting to ~7B-13B FP16 models or larger quantized models), cloud GPUs like A100 have 40-80GB (supporting up to ~40B FP16 models per GPU), and multi-GPU setups with tensor parallelism are required for larger models. Beyond parameter weights, model files include: optimizer states (during training — often 2-3× the model size for Adam optimizer), attention KV-cache (growing with sequence length during inference — proportional to batch_size × sequence_length × num_layers × hidden_dim), activation memory (during training — proportional to batch size and sequence length), and metadata (tokenizer vocabulary, configuration, architecture specification). Model compression techniques to reduce size include: quantization (reducing precision), pruning (removing unnecessary parameters), knowledge distillation (training smaller models to mimic larger ones), low-rank factorization (decomposing weight matrices), and weight sharing (using the same parameters for multiple functions — e.g., tied embeddings).