Pretraining: learning the world

The objective: predict the next token

Pretraining is the stage where a model learns almost everything it will ever know about language, facts, and structure. The process sounds almost too simple to be true: show the model a sequence of text, have it predict the next word — more precisely, the next token — and update its weights based on how wrong it was. Repeat this hundreds of billions of times.

What makes this work is that the labels come free. Unlike most machine learning tasks, there is no human labeler classifying examples. The text itself provides the supervision: "the capital of France is" naturally predicts "Paris"; "def calculate_sum(a, b):" naturally predicts "return a + b". Every sentence in every document becomes a sequence of supervised training examples. Researchers call this self-supervised learning.

The loss function is cross-entropy. At each position, the model outputs a probability for every token in its vocabulary. Llama 3 uses a vocabulary of 128,000 tokens; at each step the model must assign probability mass across all 128,000 possibilities, and the loss penalizes how much probability went to the wrong ones. Getting that loss consistently low across trillions of positions forces the model to implicitly absorb grammar, factual associations, reasoning patterns, and coding conventions — not because anyone told it to, but because they are the structure that lets it predict text well.

From raw crawl to training tokens

The raw ingredient is text at scale. Web crawls, digitized books, code repositories, scientific papers, discussion forums — what they share is that all of it was written by humans to communicate something to other humans, so it embeds the patterns of language and thought the model needs to learn.

Raw web data is far from ready to train on. Before a single token reaches the model, it passes through a multi-stage pipeline.

The first pass is filtering. Boilerplate, machine-generated spam, pages that are mostly navigation menus — these are discarded. Quality heuristics (length, language identification, keyword blocklists, classifier-based scoring) winnow the raw crawl from petabytes to something usable.

Next comes deduplication. The web is full of repeated text — the same article scraped by hundreds of sites, the same license headers in millions of code files. Training on many copies of the same passage causes the model to memorize it rather than generalize from it. Near-duplicate removal, sometimes at the paragraph level, dramatically reduces this effect.

The filtered, deduplicated text is then tokenized: split into subword units that the model actually sees. A tokenizer trained on a large text corpus learns which character sequences appear together often enough to deserve their own token. "unbelievable" might be a single token; a rare technical term might be split into three or four. Llama 3 uses a vocabulary of 128,000 tokens — large enough that common English words are usually a single token while rare words and non-Latin scripts remain representable.

What emerges from tokenization is a stream of integer IDs. The training loop consumes that stream in fixed-length chunks, feeds each chunk through the model, computes the cross-entropy loss against the next-token targets, backpropagates the gradient, and updates the model's parameters. Then it does it again, for months.

The scale of it

Two recent models illustrate what frontier pretraining actually costs.

Llama 3.1 405B — Meta's largest publicly released model — was trained on more than 15 trillion tokens using over 16,000 H100 GPUs, accumulating 30.84 million H100-GPU-hours of compute.

DeepSeek-V3 is a mixture-of-experts model with 671 billion total parameters, of which only 37 billion are active on any given token. It was trained on 14.8 trillion tokens in a final training run of 2.788 million H800-GPU-hours.

Even taken at face value, these numbers make one thing clear: frontier pretraining is a major infrastructure investment. DeepSeek's architecture — activating only a fraction of parameters per token — is one reason a model at that capability level can be trained in far fewer GPU-hours than a dense model of equivalent quality would require.

What pretraining produces — and what it doesn't

A model that has completed pretraining is called a base model. It has absorbed the statistical patterns of its training corpus at remarkable depth. It can write fluent prose, generate syntactically correct code, continue a logical argument, and produce text that reads as if it came from wherever you started the prompt.

What it cannot do is follow instructions.

Ask a base model "What is the capital of France?" and you are as likely to receive "What is the capital of Germany? What is the capital of Italy?" as you are to receive "Paris." The model is doing exactly what it was trained to do: predict what text typically follows. On the internet, quiz-style questions appear in lists of similar questions — so that is what the model continues.

The model is not refusing to answer; it has no concept of answering. It has a concept of continuing. The behavioral changes that make a model useful as an assistant — responding to questions, staying on topic, following format constraints, declining harmful requests — come from the training stages that follow pretraining. Pretraining provides the knowledge; later stages shape the behavior.

How much data is enough?

For anyone training a model, a practical question follows immediately: how many tokens should you train on? The naive answer — as many as you can afford — was the working assumption for the first few years of large-scale training.

A 2022 DeepMind paper, commonly called "Chinchilla," ran a systematic analysis of many models trained at different sizes and token counts on fixed compute budgets. The finding: for a given compute budget, the optimal strategy is to scale model parameter count and training token count together, roughly equally. The rule of thumb that emerged is approximately 20 tokens per parameter at compute-optimal.

The headline result was striking: a 70-billion-parameter model trained on 1.4 trillion tokens outperformed Gopher, a model four times its size trained on far fewer tokens relative to its parameter count. Bigger was not automatically better; the ratio mattered.

The Chinchilla result changed how labs approached model sizing. But a countervailing consideration emerged quickly: inference cost.

A compute-optimal model minimizes training cost for a given capability level. If you run the model once, this is optimal. But if you serve a model to millions of users over months or years, the economics flip. A smaller model trained far longer can match the capability of a larger compute-optimal model — and then costs less on every inference call afterward. Training compute is a one-time expense; inference compute accumulates with every query.

This is why production models are routinely trained well beyond Chinchilla-optimal. Llama 3.1 8B was trained on roughly 15 trillion tokens. Eight billion parameters times the Chinchilla ratio of 20 gives a compute-optimal budget of around 160 billion tokens — but 15 trillion divided by 8 billion is approximately ~1,900 tokens per parameter (derived arithmetic). The model saw nearly a hundred times more data per parameter than Chinchilla would have prescribed. The training run was more expensive than theoretically necessary — and it produced a smaller, faster model that runs efficiently on consumer hardware.

This tradeoff defines the central economics of pretraining: compute spent here is amortized across every inference call that follows. Spending more now to produce a smaller, well-trained model is often the right choice when you expect to run it at scale. The right amount of training data is not a fixed rule — it depends entirely on how many times you expect to run the result.