base model
**Base Model vs. Instruct Model** is the **fundamental distinction between a pretrained language model (predicts next tokens from raw text) and a fine-tuned model (follows instructions and answers questions helpfully)** — a distinction critical to understanding why raw base models are not suitable for chatbots and why instruction tuning transforms language modeling capability into practical AI assistant behavior.
**What Is a Base Model?**
- **Definition**: A language model trained on raw internet-scale text (Common Crawl, Wikipedia, GitHub, books) to predict the next token — the model's sole objective is: given these tokens, what token comes next in the training distribution?
- **Training Objective**: Self-supervised next-token prediction on trillions of tokens — no human feedback, no instruction following, no Q&A format.
- **Behavior**: A base model continues text rather than answering questions. Ask "What is 2+2?" and it might respond "What is 4+4? What is 8+8?" — completing a likely homework worksheet pattern from training data.
- **Examples**: GPT-3 (before InstructGPT fine-tuning), Llama 3 (base, not -Instruct), Mistral 7B v0.1 (base).
- **Primary Use**: Research, further fine-tuning, understanding pretraining — not direct user deployment.
**What Is an Instruct Model?**
- **Definition**: A base model further trained with Supervised Fine-Tuning (SFT) on (instruction, response) pairs and optionally RLHF/DPO to align with human preferences — producing a model that responds helpfully to direct instructions.
- **Training Process**:
- **Stage 1 — SFT**: Fine-tune on 10,000–100,000 curated (instruction, response) examples in chat format.
- **Stage 2 — RLHF/DPO** (optional): Align with human preferences using reward modeling or direct preference optimization.
- **Behavior**: Directly answers questions, follows formatting instructions, declines harmful requests, maintains appropriate tone.
- **Examples**: GPT-4o, Claude 3.5 Sonnet, Llama 3.1 8B Instruct, Mistral 7B Instruct.
- **Primary Use**: All production chatbots, assistants, API integrations.
**Why the Distinction Matters**
- **Deployability**: Base models cannot be deployed as chatbots without instruction fine-tuning — they produce completion continuations rather than helpful responses.
- **Safety**: Instruction tuning includes safety fine-tuning — base models will complete harmful continuations where instruct models refuse.
- **Format Compliance**: Instruct models follow output format instructions (JSON, bullet points, tables); base models may not.
- **Few-Shot vs. Zero-Shot**: Base models often require elaborate few-shot prompting to guide behavior; instruct models work zero-shot on clear instructions.
- **Fine-Tuning Starting Point**: When fine-tuning for a specific domain, starting from an instruct model preserves instruction-following behavior; starting from base requires re-learning it.
**Base vs. Instruct — Behavioral Comparison**
| Scenario | Base Model Response | Instruct Model Response |
|----------|--------------------|-----------------------|
| "What is 2+2?" | "What is 4+4? What is 8+8?" | "2+2 = 4" |
| "Write a Python function to sort a list" | [Continues Python code from training] | ```python
def sort_list(lst): return sorted(lst)``` |
| "Tell me how to make a bomb" | [Completes instruction text] | "I cannot help with that." |
| "Summarize this article: [text]" | [Continues the article] | "[Summary of the article]" |
| "You are a helpful assistant." | [Continues as document text] | [Adopts assistant persona] |
**The Instruct Fine-Tuning Data Format**
Modern instruct models use chat templates — structured conversation formats:
ChatML format (OpenAI, Llama 3):
```
<|system|>You are a helpful assistant.
<|user|>What is the capital of France?
<|assistant|>The capital of France is Paris.
```
This format trains the model to expect and produce structured conversational turns rather than raw text continuation.
**Choosing Base vs. Instruct for Fine-Tuning**
Start from **instruct** when:
- Adding domain knowledge while preserving assistant behavior (medical Q&A, legal assistant).
- Need to maintain safety refusals and appropriate tone.
- Fine-tuning for a specific task format (structured extraction, classification).
Start from **base** when:
- Building a highly specialized model where instruction-following behavior would interfere.
- Creating a domain-specific model to be further instruction-tuned with custom data.
- Pretraining continuation on specialized text corpora.
The base vs. instruct distinction is **the difference between raw linguistic capability and practical conversational utility** — understanding it prevents the common mistake of attempting to deploy unmodified base models as chatbots and ensures fine-tuning projects start from the correct foundation.