RL for reasoning: learning to think

From human preferences to checkable answers

Most reinforcement learning used in language models relies on a human — or a model trained to mimic human judgments — rating each response. That feedback is subjective and expensive to collect at scale. A helpful-sounding but wrong answer can still earn a high score.

For a specific category of tasks, there is a cleaner option. When an answer is objectively verifiable — a math problem has one correct numerical result, a coding task either passes the test suite or it doesn't — you can replace the human rater with an automatic checker. This is Reinforcement Learning from Verifiable Rewards (RLVR): the model earns a positive reward when the verifier accepts its output, and nothing when it doesn't. No preference labeling, no reward model trained on human opinions — just the binary signal of pass or fail.

RLVR scales cheaply and resists a common failure mode called reward hacking, where a model learns to produce outputs that score well on the reward model without actually being correct. A test runner is harder to fool than a human rater. The machine either accepted the output or it didn't.

R1-Zero: reasoning emerged from pure RL

In early 2025, DeepSeek published a family of reasoning models. The most striking finding came from R1-Zero: a model trained with pure reinforcement learning and no supervised fine-tuning whatsoever. The starting point was a pretrained base model. The only training signal was verifiable rewards on math and coding tasks. No curated demonstrations, no human-labeled reasoning traces.

What emerged surprised the researchers. As training progressed, the model spontaneously began generating long, exploratory chain-of-thought traces — working through a problem step by step, backtracking when a path went wrong, re-examining its arithmetic, trying a different approach. The paper describes an "aha moment" observed during training: the model learned to pause and reflect not because any demonstration showed it how, but because pausing and reflecting improved its score on the verifier.

The average length of these reasoning traces grew substantially over the course of training. Longer reasoning chains correlated with more correct answers. The reward signal alone had discovered that thinking step by step is the reliable path to a verifiable answer.

R1: a cold start for readable reasoning

R1-Zero was scientifically striking but practically awkward. Its reasoning traces were often hard to read — the model mixed languages mid-chain, repeated itself, and structured its thoughts in ways that were internally consistent but opaque to a reader trying to follow along.

The full DeepSeek-R1 model adds a small supervised fine-tuning step at the very start of training — called a cold start — on a curated set of long-form reasoning examples. This gives the model a legible template for how to structure a chain of thought before the RL phase begins. The RL training then proceeds as in R1-Zero, reinforcing correct answers. The result reasons at the same depth but produces traces a human can actually follow.

The cold start also stabilizes early training. Starting from a model with no sense of how to structure a reasoning trace, the RL signal has to simultaneously discover what to do and how to express it. A small SFT phase solves the second problem, letting RL focus entirely on correctness.

GRPO: a cheaper policy optimizer

Standard policy-gradient RL for language models typically uses Proximal Policy Optimization (PPO). PPO requires a value network — a second large model trained in parallel to estimate how much future reward each state is expected to produce. That second network typically matches the policy in size, roughly doubling memory and compute.

Group Relative Policy Optimization (GRPO) removes the value network entirely. Instead of computing advantage from a learned value estimate, GRPO samples a group of responses to the same prompt and scores each one with the verifier. The baseline is the mean reward of the group. Each response is assigned an advantage equal to its reward minus the group mean, normalized by the group's reward standard deviation. Responses above the group average get positive advantage; those below get negative advantage. No critic network, no separate value training — the group is its own baseline.

GRPO originated in DeepSeekMath, applied to mathematical reasoning. Using GRPO on math data, DeepSeekMath reached 51.7% on the MATH benchmark — competitive with models considerably larger. DeepSeek-R1 adopted GRPO as its core optimizer for the same reason: it is substantially cheaper to run at the scale of reasoning model training.

o1: spending compute at inference time

In September 2024, OpenAI released o1 — described as having been "trained with reinforcement learning to perform complex reasoning" using a "long internal chain of thought" before producing a final answer. This made explicit what R1-Zero had already demonstrated: thinking through a problem step by step, at inference time, reliably improves accuracy.

The key insight behind test-time compute is that scaling doesn't only happen during training. Once a model has been trained to reason, you can give it a larger thinking budget at inference: more steps, more exploration, more self-checking before committing to an answer. OpenAI's results showed that accuracy on hard benchmarks improves both from the RL training itself and from simply allowing the model more compute to think at test time. The two levers are complementary — a model trained to reason well uses a larger inference budget more effectively than one that was not.

The open question: is this reasoning?

These models score impressively on hard math and coding benchmarks. The natural question is whether those gains reflect genuine reasoning — constructing abstract solution paths that generalize to new problem structures — or very sophisticated pattern matching that happens to produce correct tokens on familiar problem types.

The honest answer is that researchers disagree, and the evidence points in both directions.

For practical purposes — choosing a model, setting expectations for a production system, deciding when to use a reasoning model versus a standard one — the useful frame is capability, not ontology. Models trained with RLVR are reliably more accurate on verifiable reasoning tasks than their non-reasoning counterparts, and that accuracy scales with both training compute and inference budget. The mechanistic explanation for why that is the case remains an active area of research.