Supervised Fine-Tuning
Supervised fine-tuning (SFT) is the stage of training in which a pretrained language model continues to learn from a curated set of example input and output pairs, using the same next-token prediction loss as pretraining but on data that shows the behavior wanted. Pretraining on web-scale text produces a model that is excellent at continuing documents but not at following requests: asked a question, a base model may reply with more questions, as if the prompt were a page from a quiz. SFT supplies demonstrations of what a good reply looks like, and it is the step that turns a base model into something that behaves like an assistant.
Instruction tuning is the most important form of SFT, and its core finding is that training on many tasks phrased as natural-language instructions teaches a model to follow instructions it has never seen. Google's FLAN paper, "Finetuned Language Models Are Zero-Shot Learners" (Wei et al., 2021), reformatted dozens of existing datasets into instruction templates and showed that a 137-billion-parameter model tuned this way beat the larger GPT-3 on many unseen tasks in zero-shot evaluation. The T0 work from the BigScience group (Sanh et al., 2021) reached a similar conclusion, and Chung and colleagues' 2022 scaling study of instruction-finetuned models showed gains growing with the number of tasks and model size. The lesson was that a model's latent abilities can be unlocked by teaching the format of a request, not only by adding knowledge.
InstructGPT (Ouyang et al., OpenAI, 2022) placed SFT as the first of three stages in the recipe that became the template for chat assistants. Human contractors wrote prompts and ideal responses, the base GPT-3 was fine-tuned on these demonstrations, and the resulting model then served as the starting point for reward modeling and reinforcement learning from human feedback. SFT alone gets a model most of the way to usable behavior, while the later preference stage polishes subtler qualities such as tone, refusal behavior and avoiding fabricated claims. Open efforts followed quickly: Self-Instruct (Wang et al., 2022) used a model to generate its own instruction data, and Stanford's Alpaca in March 2023 fine-tuned a LLaMA model on about 52,000 such machine-generated examples.
The data matters far more than the quantity, a point driven home by the LIMA paper (Zhou et al., 2023), which tuned a 65-billion-parameter model on only 1,000 carefully curated examples and got strong results. The authors proposed a superficial alignment hypothesis: nearly all knowledge and capability arrive during pretraining, and fine-tuning mainly teaches the style and format of the interaction. Subsequent practice supports a view that diverse, high-quality, correctly formatted examples beat millions of noisy ones. Practical details matter too: the chat template with its special role tokens must match between training and use, loss is typically masked so the model learns to write responses rather than to imitate prompts, and training runs only a few epochs at a small learning rate.
SFT has well-known limits that explain why it is rarely the final stage. It can only imitate the demonstrations, so a model cannot exceed the quality of its examples, and it teaches a single target answer where many valid ones exist. Imitation of confident-sounding experts can encourage hallucination: if the demonstrations contain facts the base model never learned, the model learns the habit of asserting things it does not know. Overfitting to a narrow set can cause catastrophic forgetting of general skills, one reason parameter-efficient methods such as LoRA are popular for small tuning jobs. Preference-based methods such as RLHF and direct preference optimization exist partly because comparing outputs supplies a richer signal than copying one.
A frequent misconception is that fine-tuning is how to add new facts to a model; for knowledge that changes or must be cited, retrieval is generally more reliable. SFT shapes behavior, format and style well, and it can add specialized skills, but pushing in fresh factual content through a small dataset tends to be unreliable and can degrade other abilities. A sensible division of labor is to fine-tune for how the model should respond and to retrieve for what it should know.
Read supervised fine-tuning as teaching by example after the long education of pretraining: the model already holds the knowledge, and a few well-chosen demonstrations show it the role it is supposed to play.