Instruction tuning: learning to answer

A pretrained language model is one of the most knowledgeable artifacts humans have ever built — and one of the least useful to talk to. Ask it a question and it will, quite literally, answer with more questions. This is not a bug. It is a predictable consequence of how the model was trained. Understanding the gap between a base model and an assistant model is the first step toward understanding how modern AI products actually work.

What a base model actually does

A base language model is trained with a single objective: predict the next token given everything that came before it. The model processes a sequence of tokens, estimates a probability distribution over its entire vocabulary, and samples one token from that distribution. Then it appends that token, shifts the window, and repeats. That is the complete description of the process. No goals, no roles, no intentions — just conditional probability over the next symbol.

This creates a mismatch the moment you try to use such a model as an assistant. Ask a base model "What is the capital of France?" and it will continue that text in whatever direction feels statistically plausible given its training data. It might produce a list of similar geography questions, a quiz-bowl format, or a comment thread debating French history. None of these are useful responses to the question you asked. The model is not being evasive — it is doing exactly what it was trained to do. The InstructGPT paper (Ouyang et al., 2022) put it plainly: base language models are "not aligned with the user's intent." They complete text; they do not answer questions.

That gap — between "continues text plausibly" and "responds like a helpful assistant" — is the alignment gap that instruction tuning bridges. The underlying knowledge is already there, encoded in the weights during pretraining. The problem is behavioral, not factual. The model knows what the capital of France is; it just does not know that it is supposed to say so.

Supervised fine-tuning

The fix is conceptually straightforward. Supervised fine-tuning (SFT) takes the pretrained weights and continues training them on a curated dataset of (prompt, ideal response) pairs. The loss function is unchanged — cross-entropy over next-token predictions — but every training example now pairs a question or instruction with a human-authored answer. After enough such examples, the model has seen what "answering" looks like hundreds or thousands of times and has learned to treat that as the expected behavior.

InstructGPT formalized this as the first stage of a three-step pipeline. Human labelers were given a set of diverse prompts and asked to write good answers. Those (prompt, response) pairs were then used for a standard fine-tuning pass on the pretrained model. The resulting model — the instruction-tuned model — shares its weights with the base model but behaves very differently at inference time. The architectural change is minimal; the behavioral change is large.

Chat templates and turn tokens

There is one detail that matters more than it first appears: how does the model know who is speaking? A paragraph of raw text has no turn boundaries. A conversation does — there is a user turn, then an assistant turn, then perhaps another user turn. Without a clear signal in the input, the model has no way to distinguish which part of the context it is supposed to read versus which part it is supposed to generate.

Modern instruction-tuned models solve this with special tokens baked into the vocabulary itself. Llama 3, for instance, added explicit role markers and end-of-turn tokens as part of its chat format — new vocabulary entries whose sole purpose is to mark conversational structure. Every SFT example is wrapped in this template: a role marker, the turn content, an end-of-turn marker, then the next role marker, content, end-of-turn marker, and so on. The template is repeated consistently across every example in the dataset.

After enough training on consistently templated examples, the model has learned that text following an assistant role marker should be a response, not a continuation. The template is not decoration — it is a training signal that teaches the model its own conversational role. Remove the template at inference time and you lose much of the benefit.

How much data?

The data requirements for SFT are surprisingly modest compared to pretraining. The numbers span a wide range, but they all sit orders of magnitude below pretraining corpora.

LIMA (Zhou et al., 2023) demonstrated that 1,000 carefully curated examples were enough to produce a competitive instruction-following model — provided those examples were high quality and diverse across tasks. The Alpaca dataset, another well-known SFT resource, used 52,000 examples. The SFT stage of InstructGPT used roughly about ten thousand prompts (this figure is reported loosely in the paper and should be read as an order-of-magnitude reference, not a precise count). Pretraining, by contrast, runs into the trillions of tokens. The difference is not a rounding error — it is four to six orders of magnitude.

The reason the numbers are so small is important. SFT is not teaching the model new facts. It is teaching the model a new behavior pattern. Behavior patterns require far fewer demonstrations than factual knowledge requires exposure. You need to see enough examples to establish what the pattern is, but you do not need to cover every possible fact. The facts are already there.

Format, not facts

LIMA's authors named this insight the Superficial Alignment Hypothesis: a model learns almost all of its knowledge during pretraining. SFT mostly teaches format and style — how to structure a response, when to elaborate, when to be concise, what it looks and sounds like to be helpful. The underlying factual knowledge was already encoded in the weights before a single SFT example was processed.

The InstructGPT paper provides a striking empirical check. Human raters preferred the outputs of a 1.3-billion-parameter instruction-tuned model over those of the original 175-billion-parameter base GPT-3. The base model is roughly 135 times larger and has vastly more parametric capacity. But raw capacity does not translate into helpfulness. The larger model knew more about the world; the smaller, instruction-tuned model knew how to present what it knew. Human raters preferred the latter.

This result is worth sitting with. It separates two things that are easy to conflate: knowing and telling. A base model trained on the entire web has absorbed enormous amounts of knowledge — historical facts, scientific concepts, code patterns, literary references. But knowing a fact and producing a useful response when someone asks about it are different skills. SFT teaches the second skill by showing the model, repeatedly, that a question is supposed to elicit an answer. Alignment, not added facts, is what changes.

Two practical consequences follow from this framing.

First, data quality matters more than data quantity. LIMA's 1,000 examples outperformed larger but noisier SFT datasets because the curation was deliberate. When the model only needs to learn a behavioral pattern, it needs clean, clear demonstrations of that pattern. Noisy, inconsistent, or low-quality examples teach a muddy pattern. This is why the gap between carefully filtered human-authored SFT data and cheaply generated synthetic data can be larger than the gap between different base model sizes.

Second, SFT cannot compensate for gaps in pretraining. If the base model never encountered a domain during its training run, instruction tuning will not inject knowledge of that domain. The sequence is non-negotiable: pretraining builds the knowledge base, SFT shapes the behavior. You can change how a model expresses what it already knows, but you cannot conjure knowledge that was never there. If you need a model that knows about a proprietary domain, the answer is either continued pretraining on that domain's text or retrieval augmentation — SFT alone will not get you there.

The next stage — RL alignment — addresses a related but distinct problem: not "does the model know how to answer?" but "does the model choose to answer in ways humans actually prefer, including when the right choice is to decline?" SFT teaches the model to respond; RL alignment teaches it to respond well.