Model Weights

Model weights are the learned numerical parameters of a neural network, the millions or billions of numbers that training adjusts and that together determine what the model does. Physically they are just files: arrays of numbers, organized into named tensors such as the attention projection matrices of each layer, written to disk and loaded into memory at run time. The architecture code says how to arrange the computation, but the weights are the stored result of the expensive training run, which is why the phrase "releasing the weights" is shorthand for releasing a model.

The size of a weight file follows from simple arithmetic: parameter count multiplied by bytes per parameter. A model with 7 billion parameters stored in 16-bit formats such as float16 or bfloat16 uses 2 bytes each, so 7 billion × 2 bytes is about 14 gigabytes; stored in 32-bit floats at 4 bytes each it would be about 28 gigabytes. A 70-billion-parameter model needs roughly 140 gigabytes at 16 bits, which exceeds the memory of any single consumer graphics card. Quantization shrinks this by storing each number in fewer bits: at 8 bits (1 byte) the 70-billion model is about 70 gigabytes, and at 4 bits (half a byte) about 35 gigabytes, plus small overhead for scaling factors. Training needs far more than the finished file, because mixed-precision training with the Adam optimizer keeps extra copies of weights and optimizer statistics, commonly counted at around 16 bytes per parameter in the ZeRO paper by Samyam Rajbhandari and colleagues in 2020, in addition to activations.

The file format is a security decision, not just a convenience. For years PyTorch checkpoints were saved with Python's pickle serialization, and loading a pickle can execute arbitrary code, so downloading an untrusted checkpoint was equivalent to running an untrusted program. Hugging Face's safetensors format, introduced to fix this, stores a small JSON header listing each tensor's name, data type, shape and byte offsets, followed by the raw tensor bytes, with no executable content. Because tensors sit at known offsets, they can be memory-mapped and loaded lazily without copying, which speeds up loading of large models, and a third-party security audit of the library was published in 2023. Other formats serve other ecosystems: GGUF, used by the llama.cpp project, packs quantized weights plus metadata into one file for local inference, and ONNX and TensorFlow SavedModel bundle weights with a graph description.

Weights are not the whole model, and they do not store training text as a lookup table. To use a checkpoint one also needs the architecture definition, the tokenizer that maps text to token ids, configuration such as layer counts and context length, and a license. The numbers encode statistical regularities distilled from training data, though a model can still memorize and regurgitate some passages, especially those repeated often. Weight files also vary in what they contain: a base model, an instruction-tuned model, a LoRA adapter that stores only small low-rank updates to be added to a base model, or a merged result of several. Sharded checkpoints split a large model into multiple files with an index so that downloads and loading can be parallelized.

Whether to publish weights is one of the central policy debates in AI. OpenAI released GPT-2 in stages during 2019, citing misuse concerns; Meta's first LLaMA weights, offered to researchers in February 2023, leaked publicly within days, and later Llama releases were published openly under licenses with conditions. Proponents of open weights stress scientific reproducibility, local and private use, competition and independent safety research. Critics note that once weights are public they cannot be recalled, and that safety training can be undone cheaply by fine-tuning, as several 2023 research papers on Llama 2 Chat demonstrated, so safeguards that live only in the weights are fragile. In July 2024 the US National Telecommunications and Information Administration report on dual-use foundation models with widely available weights recommended monitoring evidence of risk rather than immediately restricting such releases.

Practical pitfalls follow from all of this. Mixing up parameters and bytes leads to mistaken memory estimates, and quoting parameter counts as if they measured quality ignores data and training. Downloading a checkpoint from an unknown repository in pickle format carries real risk, and a "7B" label does not say which precision a file uses. Reproducibility also depends on a recorded revision, since a repository can be updated and a name can point to different weights over time.

Read model weights as the frozen, copyable outcome of training: a very large set of numbers whose size is just count times precision, whose file format determines how safely it loads, and whose publication is not a technical afterthought but a decision about who gets to run, change and misuse the model.