The life of a model

A language model is not trained once. It is built in layers, each stage changing what the model can do and, crucially, whyit does it that way. By the time you interact with a modern assistant, the weights you are talking to have been through at least four distinct phases of training — each with its own objective, its own data diet, and a very different price tag. Understanding the shape of that journey answers a fundamental question: where does a model’s knowledge come from, and where does its behavior come from? The short answer is that they come from different places.

Stage 1: Pretraining — building knowledge

The foundation stage feeds a blank set of weights on the widest possible slice of human text: books, code, web pages, scientific papers, forums, legal documents. The training task is deceptively simple — predict the next token, check the prediction, update the weights — but it runs billions of times, for months, across thousands of GPUs. The supervision signal comes free from the text itself; nothing has to be hand-labeled. Because the model must predict what comes next in any genre, on any topic, it is forced to compress patterns of language, fact, reasoning, and world structure into its parameters. The result is a base model: it can continue almost any text with startling fluency, but it doesn’t follow instructions, adopt a helpful persona, or refuse harmful requests. It is pure pretraining — a compressed imprint of its training corpus, nothing more. This is by far the most expensive stage; the cost of every subsequent step is small by comparison.

→ How pretraining works in depth

Stage 2: Instruction tuning — shaping answers

A base model that continues text is not the same as a model that answers questions. To bridge that gap, supervised fine-tuning (SFT) shows the model a curated set of (prompt, ideal response) pairs and trains it to produce those responses using the same next-token objective as pretraining. The curated dataset can number in the thousands — orders of magnitude smaller than the pretraining corpus. The critical insight is that SFT does not teach the model new facts. It teaches a format and a disposition: how to recognize a question, how to address the person asking it, when to be concise and when to elaborate. After SFT, the weights remember the same things they did after pretraining — they just express that knowledge as an instruct modelthat answers and follows directives rather than completing raw text. The knowledge didn’t change; the behavior did. Compute cost is a small fraction of pretraining.

→ How instruction tuning works in depth

Stage 3: RL alignment — shaping values

Instruction tuning makes a model answer. Reinforcement learning from human feedback (RLHF) and related alignment techniques make it answer well by human standards: helpfully, honestly, and without producing harmful content. A separate reward model is trained on human preference judgments — "response A is better than response B" — and the language model is then nudged, via reinforcement learning, to produce responses that earn higher rewards. This stage addresses the gap between a model that can produce plausible text and a model that actually serves the person it is talking to. The compute budget here is modest relative to pretraining, the training data is smaller still, and the factual knowledge in the weights barely shifts. What changes is tendencies: the model becomes more likely to be helpful rather than evasive, honest rather than sycophantic, calibrated rather than falsely confident.

→ How RL alignment works in depth

Stage 4: RL reasoning — shaping thought

The newest stage extends reinforcement learning beyond behavioral preferences into the structure of reasoning itself. Instead of asking humans "which response do you prefer?", these methods ask a verifier "is this final answer correct?" — then reward the model for working through problems step by step rather than leaping to conclusions. Because the training signal is automated and verifiable, the data pipeline can scale in ways that human-preference collection cannot. The result is a model that learns to pause, draft intermediate reasoning steps, backtrack when a line of argument fails, and commit only when it has converged on a defensible answer. The gains show most clearly on math, code, and multi-step problems where a single wrong intermediate step would otherwise corrupt the final answer. Cost sits between SFT and pretraining — far cheaper than the foundation stage, but meaningfully more compute-intensive than simple instruction tuning.

→ How RL reasoning works in depth

The through-line

Pretraining grows knowledge. Everything after shapes behavior. That asymmetry explains a great deal: why a heavily aligned model still knows things it has been trained not to say aloud; why a tiny post-training dataset can dramatically change how a model presents itself; and why cutting corners on pretraining cannot be fully compensated by more sophisticated fine-tuning later. The knowledge baked in at stage one is the raw material every subsequent stage works with — and no amount of alignment or reasoning training can add knowledge that was never there in the first place.