RL for alignment: learning to behave

What instruction-tuning leaves unfinished

Instruction tuning teaches a model to answer. Given a prompt, the model learns to produce responses that look like the demonstrations in its fine-tuning data — coherent, on-topic, formatted correctly. For many tasks, that is enough.

But there is a category of behavior that instruction-tuning cannot reliably shape: preference among plausible responses. When a model knows how to write a good answer, it still doesn't know which of three good answers a human would most value — the concise one, the thorough one, or the one that surfaces an important caveat. And when a model encounters a harmful prompt, demonstration data can show it one example of a refusal; what it cannot show is the generalization — that the refusal behavior should transfer to thousands of harmful prompts nobody anticipated and wrote training examples for.

The deeper problem is that curating demonstrations does not scale gracefully to preferred behavior. Writing an ideal answer from scratch is slow and often requires subject-matter expertise. Ranking two candidate answers is far faster: even a non-expert can usually judge which of two responses is more helpful, more accurate, or more appropriate — even when they could not have produced either from nothing. If you could convert those rankings into a training signal, you could cover far more behavioral territory than any demonstration set.

Reinforcement learning from human feedback is that mechanism. It turns pairwise human preferences into a reward signal, and uses that signal to push the model toward behavior that generalizes beyond any specific instruction.

The RLHF pipeline

RLHF breaks the problem into three coordinated stages. First, you collect preference data: for the same prompt, you ask the model to write several candidate responses, then have humans rank them from most to least preferred. You are not asking humans to write ideal answers — only to compare outputs the model already produces. This makes data collection substantially cheaper than demonstration labeling and allows a single labeler to cover far more ground per hour.

Second, you train a reward model — a separate neural network whose job is to predict the score a human would assign to any given response. Trained on your preference rankings, this model becomes an automated proxy for human judgment. Once trained, you can query it at scale without involving human raters in every subsequent training step.

Third, you use reinforcement learning to update the main model — the policy — to produce responses the reward model rates highly. The algorithm most commonly used is PPO (Proximal Policy Optimization), which updates the model's weights in the direction of higher predicted reward while restraining how large each update step can be.

There is a serious risk hiding in this setup. The reward model is an approximation, not a true measure of human preference. Optimize hard enough against an approximation and the policy will find ways to produce responses that fool the reward model while satisfying no actual human — this is called reward hacking. A policy trained without guardrails might learn that extremely long responses, or responses stuffed with confident-sounding qualifiers, score well on the reward model even when they are not genuinely useful.

The safeguard is a KL penalty. During PPO training, the policy is simultaneously penalized for diverging too far from a frozen copy of the original supervised model — the reference model. The KL divergence between the live policy and the frozen reference is added to the training loss, so the model can improve toward higher reward while remaining tethered close enough to the baseline that degenerate drift becomes too costly to pursue. This KL-penalized PPO formulation is what InstructGPT (Ouyang et al., 2022) established as the standard approach and the one subsequent work built on.

The lineage of the approach goes back further than most people realize. Christiano et al. (2017) demonstrated that reinforcement learning from human preferences could teach agents complex behaviors even when a reward function is hard to specify by hand. Stiennon et al. (2020) showed the same pipeline could meaningfully improve language model summarization quality beyond what supervised fine-tuning achieved alone. InstructGPT was the first broadly deployed application to instruction following at scale.

DPO: the reward model was optional

RLHF works well, but it is operationally heavy. You maintain two large models simultaneously — the policy and the reward model — and run an RL training loop that is notoriously sensitive to hyperparameters. Any systematic error in reward model training propagates directly into the final policy. And because the reward model is a separate artifact, it needs to be re-calibrated as the task distribution shifts.

Direct Preference Optimization(Rafailov et al., 2023) eliminates the separate reward model entirely. The key insight is mathematical: the optimal reward function for the RLHF objective — the reward you would recover if training were perfect — is already implicitly encoded in the ratio between the policy's log-probabilities and the reference model's log-probabilities on any given response. The language model is, in this precise sense, secretly its own reward model.

Rather than training a reward model and running PPO, DPO passes preference pairs — one chosen response and one rejected response, both to the same prompt — directly into a classification-style objective. The model is trained to assign higher relative probability to the chosen response and lower relative probability to the rejected one. The KL constraint is still present, baked into the mathematical derivation rather than implemented as a separate penalty term. No reward model needs to exist as a separate artifact; no RL loop needs to run.

The practical payoff is significant. DPO training is stable, computationally lightweight, and produces results competitive with RLHF at the scales where both have been compared. Rafailov et al. described it as "stable, performant, and lightweight" — and the open-weight ecosystem quickly agreed. Most fine-tuned open models released since 2023 use DPO or one of its derivatives rather than full RLHF, because DPO made alignment training accessible to teams without the infrastructure to run policy-gradient RL at scale.

What alignment training actually delivers

The word alignment is sometimes used loosely to mean many things. In the context of RLHF and DPO, it specifically delivers three behavioral properties that instruction-tuning alone cannot reliably instill at scale.

Helpfulness: the model learns not just what answers look like, but which among several plausible answers a human would find genuinely more useful — more accurate, more appropriately scoped, more responsive to what was actually being asked. Preference data covers the distribution of real user intents far more broadly than any curated demonstration set, because labelers compare rather than generate, and comparison is cheap enough to do across millions of examples.

Harmlessness: exposure to ranked comparisons in which harmful outputs are consistently rated lowest teaches the model to avoid producing them. The signal is not "here is the only correct response" but "between these two options, this one is less harmful" — a weaker signal per example, but one that generalizes across the space of harmful prompts far better than any finite set of curated demonstrations could.

Refusal: perhaps the most distinctive achievement of alignment training. A base model or instruction-tuned model trained without preference data has no reliable mechanism for declining requests — if a continuation seems plausible, the model will produce it. Alignment training introduces refusal as a learned behavior: the model discovers that declining certain requests scores higher with human raters than complying, and that certain patterns of compliance are consistently downranked regardless of how they are phrased. The refusals that emerge are not hardcoded rules — they are generalized behavior learned from preference gradients.

Anthropic's 2022 paper on Helpful and Harmless (HH) assistants studied how RLHF could instill all three properties simultaneously, and surfaced the tension between them: a model trained too hard for harmlessness becomes excessively cautious, refusing legitimate requests. Managing that tradeoff is itself empirical, and one that preference data from real users — rather than curated demonstrations — is better positioned to resolve. The resulting models feel qualitatively different from their supervised-only predecessors because the nature of the training signal is different: instead of learning what to say, they learn what to prefer. That shift is what makes the behavior generalize.