Fine Tuning

Fine-tuning is the process of taking a model already pretrained on a massive general dataset and continuing to train it on a smaller, task-specific dataset, so it specializes without starting from random weights. It's the standard way almost all production models actually get built today: pretraining a large model from scratch costs millions of dollars and takes a dedicated cluster for weeks or months, while fine-tuning an already-pretrained model onto a specific task can take hours on a handful of GPUs — the pretrained model has already learned general language or visual structure, and fine-tuning only needs to redirect that knowledge toward the narrower target. At a high level, fine-tuning sits in the model lifecycle like this:

Full fine-tuning updates every parameter in the model, and that's exactly what makes it the most GPU-memory-expensive option even though the dataset is small. Updating all weights means storing a gradient and optimizer state for every single one of them during training, at roughly the same per-parameter memory cost as pretraining itself — fine-tuning a 70-billion-parameter model fully still needs hundreds of gigabytes of GPU memory, spread across multiple H100 or B200 GPUs, even though the actual training data might be a few thousand examples. This is the specific cost that parameter-efficient fine-tuning methods exist to avoid.

LoRA works by freezing the original weights entirely and learning a small additive update instead, exploiting the fact that the useful change needed for most fine-tuning tasks turns out to be low-rank. Rather than updating a weight matrix directly, LoRA learns two much smaller matrices whose product approximates the needed change, and only those two small matrices need gradients and optimizer state — often reducing trainable parameters by 100x or more while matching full fine-tuning's accuracy on many tasks. QLoRA pushes this further by also quantizing the frozen base model to 4-bit precision during fine-tuning, which is specifically what let a 65-billion-parameter model be fine-tuned on a single consumer GPU for the first time, a task that previously required a multi-GPU server.

Instruction tuning and RLHF are both specific applications of fine-tuning, not separate techniques, and understanding that collapses a lot of apparent complexity. Instruction tuning fine-tunes a pretrained base model on a dataset of (instruction, good response) pairs so it learns to follow commands rather than just continue text statistically; RLHF (reinforcement learning from human feedback) is a further fine-tuning stage that uses human preference rankings to push the model's outputs toward what people actually rate as helpful and safe. Both are the exact same underlying mechanism — continued gradient-based training on top of a pretrained model — pointed at different training objectives and different datasets, which is why a single pretrained base model can spawn an entire family of fine-tuned variants for different downstream purposes.

Read fine-tuning as the step where the economics of model development flip: pretraining is defined by needing enormous compute and data to build general capability from nothing, while fine-tuning is defined by needing only a small, targeted dataset and (with LoRA-style methods) a fraction of the GPU memory to redirect that capability toward one specific job — which is exactly why fine-tuning, not pretraining, is the step most companies building on AI actually touch.