Direct alignment and
RL for reasoning
How DPO solves the RLHF objective without an RL loop, what GRPO gives up by dropping PPO’s critic, and how DeepSeek-R1 turned checkable answers into long reasoning.
About 9 hours with the readings · use → or Space to step through · O for contents · T for dark mode
What this week is for
Explain DPO, GRPO and the R1 recipe down to the gradient, then defend a choice between them.
How much it matters: about 2% of collected interview reports and none of the 18 staff-plus ones, but core in research and research-engineer loops.
How labs ask it
- “Preference-based training (reward-model RLHF vs DPO)” Mistral · firsthand 2026
- “What is the difference between PPO and GRPO?” frontier lab, unnamed · firsthand 2026
“PPO vs GRPO: what does dropping the critic cost you, what is the length bias in GRPO’s loss, and when would you go back to a value model?”
The map for this week
Part 1 learns offline from fixed preference pairs. Parts 2–3 learn online from answers a program can check. Part 4 asks whether the gains are real and which method fits which job.
DPO: the RLHF objective, solved without RL
The DPO derivation in five plain steps
logratio(y) = log π(y) − log πref(y), summed over the answer’s tokens. Both come from forward passes you already run.
Step 4 is the trick: the uncomputable Z(x) is the same for both answers, so it drops out of the difference.
Worked: β = 0.1, logratio(c) = +2, logratio(r) = −1 → margin 0.3 → σ = 0.574 → loss 0.554.
Go deeper
Appendix A.1 rewrites the objective as −KL(policy ‖ best) + log Z(x); Z doesn’t depend on the policy, so the optimum is policy = best. With K ranked answers, the Plackett–Luce model gives the same cancellation (A.3).
Rafailov et al. (2023), §4 and Appendix A.1–A.3; RLHF book ch. 8.
What one DPO step does to one pair
Implicit reward = β × logratio. Margin = chosen reward − rejected reward.
Each pair’s update is scaled by σ(−margin): near 1 when the pair is ordered wrongly, near 0 once it is right.
Without this weight (“unlikelihood”), the paper’s summaries degenerated into “when when when…”. β also sets how far the log-ratios must move before a pair stops mattering.
Computed live from the DPO loss. Rafailov et al. (2023), §4 gradient, Appendix A.4, C.3 and Table 3; default β = 0.1 (Appendix B).
DPO in practice: evidence and weak spots
Evidence (paper, models up to 6B)
- Sentiment: best reward-for-KL trade-off in a 22-run sweep, beating PPO even with the true reward.
- Summaries: ~61% win rate vs PPO’s 57%; humans preferred DPO 58% of the time.
- Recipe: β 0.1, learning rate 1e-6, batch 64.
Weak spots
- Offline: never learns from its own new answers.
- Displacement: chosen and rejected log-probs both tend to fall.
- Static KL: β fixes the drift budget.
- Controlled studies: “a hair behind” online RL.
Cheap trick: cache the reference log-probs once; peak memory drops 50%.
Rafailov et al. (2023), §6.1–6.4, Appendix B; RLHF book ch. 8 (“Numerical Concerns”, “Implementation Details”, “DAAs vs. RL”).
Reward-model RLHF or DPO? Decide by data and loop
| Reward model + PPO | DPO | |
|---|---|---|
| Models held | policy, critic, reward model, reference | policy + reference (or cached log-probs) |
| Learns from its own new answers | yes | no, unless you run online DPO |
| Reusable judge | reward model: also for filtering, best-of-N | none |
| Cost per experiment | rollouts + two trained models | one forward/backward per pair |
DeepSeekMath §4.1.1 and Fig. 5; RLHF book ch. 8 (on-policy synthetic pairs in Tülu 3; “data is often far more important than algorithms”).
Can you answer these in two minutes?
- “Preference-based training (reward-model RLHF vs DPO)” Mistral · firsthand 2026
- “Compare reward-model RLHF and Direct Preference Optimization.” Mistral · question bank
- “Explain the difference between pre-training, supervised fine-tuning and preference optimisation.” Meta MSL · question bank
Be ready to: derive the loss in five steps, say why Z(x) cancels, explain the σ(−margin) weight, and name two failure modes with a fix each.
From the question bank behind the curriculum (2025–2026 reports and prep sites).
GRPO: PPO without the critic
One lens for SFT, DPO, PPO and GRPO
DeepSeekMath writes every method the same way: raise each token’s log-probability × a gradient coefficient. Methods differ in where answers come from and how the coefficient is set.
| Method | Answers come from | Coefficient per token |
|---|---|---|
| SFT | human-written | 1 |
| Rejection-sampling FT | the SFT model | 1 if correct, else 0 |
| Online RFT | the current policy | 1 if correct, else 0 |
| DPO | pairs from the SFT model | σ(−margin), + chosen / − rejected |
| PPO | the current policy | advantage from a learned critic |
| GRPO | a group from the current policy | group-normalised reward, plus a KL term |
On a 1.3B model: online beat offline late in training; GRPO beat online RFT because it also pushes wrong answers down.
Shao et al. (2024), §5.2.1, Table 10, Fig. 5, Appendix A.1.
GRPO step by step, with the papers’ settings
- Sample a group of G answers per question.
- Score each: reward model, or a checker.
- Advantage = (score − group mean) ÷ group std. Every token of the answer gets it.
- PPO-style clipped update. KL to the reference goes in the loss, not the reward.
| DeepSeekMath-RL 7B | R1-Zero | |
|---|---|---|
| Answers per question | 64 | 16 |
| Max length | 1,024 | 32,768 → 65,536 |
| KL coefficient | 0.04 | 0.001 |
| Learning rate | 1e-6 | 3e-6 |
| Result | MATH 46.8 → 51.7% | AIME 15.6 → 77.9% |
MATH is a competition-math benchmark; AIME is a US invitational math contest whose answers are numbers. R1-Zero refreshed its reference model every 400 steps.
DeepSeekMath §4.1–4.2, Table 5; DeepSeek-R1 §2.1, §2.3 (arXiv v2, 2026).
A group of answers → advantages → per-token push
Answers and lengths are made up for illustration; tap a reward to flip it. Policy ratio = 1 (first update), KL ignored, population std. Advantage: DeepSeekMath §4.1.2; aggregation options: RLHF book ch. 6.
PPO vs GRPO: what each one has to run
Scroll the diagram sideways.
PPO trains two policy-sized networks; GRPO trains one and pays with G rollouts per prompt instead.
DeepSeekMath §4.1.1 and Fig. 4 (“the value function … is typically another model of comparable size”); DeepSeek-R1 Supplementary A.3.
Memory arithmetic: what dropping the critic saves
Assume a trained model costs 16 bytes per parameter (bf16 weights 2 + gradients 2 + fp32 master copy 4 + two Adam moments 4 + 4) and a frozen one costs 2.
PPO: 2 trained + 2 frozen = 36 bytes/param. GRPO with a reward model: 20. With a rule checker: 18.
Rollout KV cache and activations come on top. R1’s framework also offloads each model from GPU memory between rollout, scoring and training.
Computed from the stated assumption; policy-sized critic per DeepSeekMath §4.1.1. Offloading: DeepSeek-R1 Supplementary B.1. 80 GB per H100.
What dropping the critic costs you, and buys you
You pay
- G rollouts per prompt for a baseline: 64 in DeepSeekMath, 16 in R1-Zero.
- No per-token credit: all tokens of an answer share one advantage.
- Dead groups: all right or all wrong → advantage 0. Many recipes keep prompts solved 20–80% of the time.
- Std division up-weights near-unanimous groups.
You get
- One policy-sized model fewer to train (~44% less memory).
- No GAE λ to tune: PPO at λ = 0.95 was “considerably worse”; at λ = 1.0 it came close.
- No critic guessing the final reward of half-written chains that later revise themselves.
DeepSeekMath §4.1–4.2; DeepSeek-R1 Supplementary A.3 and Fig. 4; RLHF book ch. 6 (GRPO) and ch. 7 (difficulty filtering).
The length bias in GRPO’s loss, with numbers
GRPO divides each answer’s token losses by its own length, then averages answers. Each answer gets the same total push, spread thinner over long ones.
| Answer (made up) | Reward | Tokens | Advantage | Push per token, GRPO | ÷ all tokens |
|---|---|---|---|---|---|
| short, right | 1 | 300 | +1 | +8.3 × 10⁻⁴ | +2.2 × 10⁻⁴ |
| long, right | 1 | 900 | +1 | +2.8 × 10⁻⁴ | +2.2 × 10⁻⁴ |
| short, wrong | 0 | 300 | −1 | −8.3 × 10⁻⁴ | −2.2 × 10⁻⁴ |
| rambling, wrong | 0 | 3,000 | −1 | −0.83 × 10⁻⁴ | −2.2 × 10⁻⁴ |
Push per token = advantage ÷ (4 × length). The 3,000-token wrong answer is pushed down 10× less per token. Long and wrong is cheap.
Fixes (DAPO’s per-token loss, Dr. GRPO) are Week 9.
Not GRPO-only
A PPO analysis (Fatemi et al.) found a similar pull toward longer wrong answers. R1 notes the opposite force in PPO: a per-token KL in the reward taxes length. The RLHF book’s toy example shows 0.25 vs 0.14 per-token gradients for 4- and 7-token answers.
Loss: DeepSeekMath Eq. 3. Aggregation: RLHF book ch. 6. Raschka on Liu et al. (Dr. GRPO) and Fatemi et al.; DeepSeek-R1 Supp. A.3.
When would you go back to a value model?
- Rewards per step or turn (tool calls, multi-turn agents) need a per-state baseline.
- Huge rollouts: 16–64 samples per prompt cost more than a critic.
- Mostly dead groups (all pass or all fail) waste the samples.
- GAE is tunable: R1 got PPO close at λ = 1.0; Open-Reasoner-Zero scaled with vanilla PPO.
Two-minute answer
Dropping the critic trades a second trained model for G samples per prompt and coarser credit; per-answer averaging adds a length bias. I’d return to a value model for dense or multi-step rewards, or when sampling costs more than a critic.
RLHF book ch. 6 (“MDP vs. Bandit Framing”); DeepSeek-R1 Supp. A.3; Raschka on Open-Reasoner-Zero.
Can you answer these in two minutes?
- “What is the difference between PPO and GRPO?” frontier lab, unnamed · firsthand 2026
- “RL for Language Models: Policy Gradients, PPO, GRPO and DPO (9 questions / 45–60 min)” Google DeepMind · question bank
- “PPO vs GRPO: what does dropping the critic cost you, what is the length bias in GRPO’s loss, and when would you go back to a value model?” senior-staff probe
Be ready to: draw both pipelines, do the memory arithmetic, compute a group’s advantages, and show the 10× per-token gap.
Questions from the curriculum’s question bank; the probe is the track’s senior-staff bar.
DeepSeek-R1, stage by stage
Numbers are from the revised paper (arXiv v2, January 2026, with the supplementary material).
R1-Zero: a base model, a checker, nothing else
“…The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think>…</think> and <answer>…</answer> tags…”
The whole training template, minus the question.
- Base: DeepSeek-V3-Base, a mixture-of-experts model with 671B parameters (37B active per token). No SFT first.
- Reward: accuracy + format, equal weight. No neural reward model, because those “are susceptible to reward hacking”.
- Run: GRPO, 10,400 steps (1.6 epochs) on 512 H800 GPUs for about 198 hours.
DeepSeek-R1 Table 1, §2.1–2.2, Supplementary A.1 and B.4.4.
Verifiable rewards are small programs
Accuracy: pull out the final answer and compare it with the reference (R1 compared math expressions with sympy when filtering data). Code: run test cases.
Format: reasoning inside <think> tags, answer after.
Language consistency (R1, stage 1): share of reasoning words in the question’s language.
R1’s RL prompts: 26K math, 17K code, 22K STEM, 15K logic. Proofs left out: hard to check.
Responses are made up for illustration; edit the text to test the checker. Rewards: DeepSeek-R1 §2.2 Eq. 4, §3.2.1 Eq. 7, Supp. B.3 and Table 4.
What pure RL did: accuracy up, answers longer
Nobody rewarded length. Reflective words (“wait”, “verify”, “check”…) rose 5–7× over training. The first arXiv version reported 71.0%.
DeepSeek-R1 §2.1, §2.3, Fig. 1, Fig. 10, Supplementary C.2 and Fig. 9.
The “aha moment”, and how much to believe it
“Wait, wait. Wait. That’s an aha moment I can flag here. Let’s reevaluate this step-by-step…”
an intermediate R1-Zero checkpoint, mid-solution
Hard problems gained most: MATH level 5 rose from about 0.55 to 0.90 accuracy. The cost: hard-to-read reasoning mixing English and Chinese.
The skeptic’s reading
- Qwen2.5 base models show aha-style reflection with no RL.
- Self-correction already grows during pretraining.
- R1’s base saw much math and code, and web pages with OpenAI-model answers.
Defensible claim: RL surfaced and amplified reflection; it may not have invented it.
DeepSeek-R1 Table 2, Supplementary A.1 and C.1; Raschka, “State of RL for LLM Reasoning”, papers [10] and [13].
R1’s four stages, built step by step
| Stage | What goes in | Checkpoint after it |
|---|---|---|
| Cold-start SFT | thousands of R1-Zero answers kept if correct and readable, rewritten by V3, checked by people | Dev1: AIME 59.0, IF-Eval 71.7 |
| Reasoning RL | GRPO on rule rewards + language-consistency reward | Dev2: AIME 74.0 |
| Rejection sampling + SFT | ~600K correct reasoning + ~200K other samples; retrain V3-Base, 2–3 epochs | Dev3: AIME 78.1, AlpacaEval 62.1 |
| RL on everything | rules + helpfulness and safety reward models; 1,700 steps, preference reward only in the last 400 | R1: AIME 79.8, AlpacaEval 87.6 |
DeepSeek-R1 Fig. 2, §3, §3.2, Table 3, Supplementary B.3–B.4. IF-Eval scores format-instruction following; AlpacaEval 2.0 is a judge-rated chat win rate.
What each stage bought, and what it cost
| Benchmark | Zero | Dev1 | Dev2 | Dev3 | R1 |
|---|---|---|---|---|---|
| AIME 2024 (math contest) | 77.9 | 59.0 | 74.0 | 78.1 | 79.8 |
| CNMO 2024 (Chinese math olympiad) | 88.1 | 58.0 | 73.9 | 77.3 | 78.8 |
| GPQA Diamond (PhD-level science) | 75.8 | 66.1 | 70.7 | 71.2 | 71.5 |
| LiveCodeBench (contest code) | 50.0 | 57.5 | 63.5 | 64.6 | 65.9 |
| IF-Eval (format instructions) | 46.6 | 71.7 | 72.0 | 78.1 | 83.3 |
| AlpacaEval 2.0 (chat, judged) | 24.7 | 50.1 | 55.8 | 62.1 | 87.6 |
Cold start cost math, RL won it back, and the last RL stage mostly bought chat quality. R1-Zero still beats R1 on CNMO and GPQA: readability isn’t free.
DeepSeek-R1 Table 3 (pass@1 for math, code and GPQA).
Rejection sampling and distillation: the 800K set
| Domain | Samples | Avg tokens |
|---|---|---|
| Math | 395,285 | 6,094 |
| Code | 211,129 | 7,436 |
| General (non-reasoning) | 177,812 | 1,420 |
| STEM + logic | 20,519 | — |
| Total | 804,745 | 5,355 |
Kept only correct samples (rules, or V3 as judge); dropped mixed-language, long-paragraph and code-block reasoning.
Distillation: SFT only, no RL
| AIME 2024 pass@1 | % |
|---|---|
| GPT-4o (May 2024) | 9.3 |
| R1-Distill-Qwen-1.5B | 28.9 |
| R1-Distill-Qwen-7B | 55.5 |
| R1-Distill-Qwen-32B | 72.6 |
| R1-Distill-Llama-70B | 70.0 |
DeepSeek-R1 Table 5 (STEM 10,124 + logic 10,395), Supplementary B.3.3, B.4.3, F and Table 15.
For small models, distil; pure RL needs a big base
| Same 32B base (Qwen2.5) | AIME 2024 | MATH-500 | GPQA | LiveCodeBench |
|---|---|---|---|---|
| RL only, over 10K steps (“Zero”) | 47.0 | 91.6 | 55.0 | 40.2 |
| SFT on R1’s 800K samples | 72.6 | 94.3 | 62.1 | 57.2 |
- Early R1 tries on a 7B dense and a 16B MoE base “failed to yield meaningful improvements”: answers turned repetitive as they grew. 32B, 230B and 671B bases worked.
- Small-model RL isn’t useless: a 7B math base trained with pure RL scored 22.3% on AIME 2024, vs 7.9% for its instruct version.
- RL on top of a distilled 1.5B model helped for about $42, but small-benchmark gains can be seed noise.
DeepSeek-R1 Tables 16–17, Supplementary F.1 and G.1; Raschka, papers [8] and [15].
What R1 tried and dropped
Process reward model
Scores each step. Steps are hard to define and judge; human labels don’t scale; it invites reward hacking. Even PRM800K has ~20% wrong labels.
Monte Carlo tree search
Token search space explodes; capped expansion hits local optima; the value model guiding search is hard to train.
Long preference RL
Helpfulness reward kept rising while Codeforces pass@1 fell. So preference rewards ran only in the last 400 steps.
Pattern: learned judges get exploited. R1 kept judges it could trust: rules, or a model comparing against a known answer.
DeepSeek-R1 Supplementary G.2, B.5 and Fig. 6, §3.2.2; DeepSeekMath §5.2.3 (PRM800K).
Cost and test-time compute, in numbers
| Post-training | H800 GPU-hours |
|---|---|
| R1-Zero | 101K |
| SFT data creation | 5K |
| R1 | 41K |
| Total (≈ $294K at $2/hour) | 147K |
Check: 512 GPUs × 198 hours = 101K. Pretraining V3-Base is not included.
Thinks longer when it’s harder: on 366 problems from 2024 contests R1 solved 61.8% with 8,793 thinking tokens on average (under 7,000 easy, over 18,000 hardest). GPT-4o: 24.7% with 711.
Sampling still helps: AIME 2024 pass@1 79.8% → majority vote 86.7% → any of 64 correct 90.0%.
DeepSeek-R1 Table 7, Supplementary B.4.4 and E.4.
Can you answer these in two minutes?
- “RL training debugging scenarios: what do you do when you hit entropy collapse, the model reasoning in another language, or terrible MFU?” frontier lab, unnamed · relayed 2025
- “Design post-training for a new base model targeting coding and math. Where do you use SFT, DPO, RLVR and distillation, and how do you know each helped?” senior-staff probe
Hooks: language mixing → R1’s consistency reward (small coding cost); what each stage bought; why no PRM or MCTS. Entropy collapse and MFU: Weeks 7 and 9.
Questions from the curriculum’s question bank; the probe is the track’s senior-staff bar.
Is the gain real? And which method when?
An outline for the next senior-staff probe; Week 9 goes deep.
“+15 points on AIME”: five checks
Probe: “Your RL run improved pass@1 on AIME by 15 points. Convince me it learned something new and isn’t reward hacking or sharpening the base model.”
- Noise: 15 problems per AIME exam, so one problem = 6.7 points. Average 64 samples (R1’s setting) and several seeds.
- Contamination: 10-gram filtering; test on a later contest (R1: 75% on AIME 2025).
- Sharpening: compare pass@k at large k. DeepSeekMath’s RL raised majority vote, not pass@K.
- Hacking: reward vs a held-out metric (R1: reward up, Codeforces down).
- Transfer: off-domain gains (CMATH 84.6 → 88.8 from GSM8K/MATH prompts only).
DeepSeek-R1 Table 13, Supp. D.1, E.2, Fig. 6; DeepSeekMath §5.2.2 Fig. 7, Table 5. pass@k vs base models (Yue et al.): Week 9.
Which method for which job
| Job | Method | Why |
|---|---|---|
| Format, tone, cold start | SFT | imitate chosen answers; cheap |
| Fuzzy preferences, fast | DPO on on-policy pairs | no RL infrastructure |
| Preferences at the ceiling | reward model + online RL | learns from its own answers; cap steps |
| Math, code, logic | RL with verifiable rewards | checker hard to fool; needs a strong base |
| Small deployable model | distil a strong reasoner | 72.6 vs 47.0 on AIME at 32B |
Pretraining predicts raw text; SFT imitates chosen answers; preference optimisation learns from comparisons or rewards on the model’s own outputs. R1: SFT where rewards were unreliable, RL where checkable.
DeepSeekMath Appendix A.1; RLHF book ch. 8; DeepSeek-R1 Supp. G.1 (“Iterative pipeline”), Table 16.
Can you answer these in two minutes?
- “Explain the difference between pre-training, supervised fine-tuning and preference optimisation.” Meta MSL · question bank
- “Your RL run improved pass@1 on AIME by 15 points. Convince me it learned something new and isn’t reward hacking or sharpening the base model.” senior-staff probe
Answer shape for the probe: noise floor first (6.7 points per problem), then contamination, then pass@k, then reward-vs-held-out curves.
Question bank behind the curriculum; the probe is the track’s senior-staff bar.
The week in one minute
- DPO solves the RLHF objective in closed form; Z(x) cancels, leaving a pairwise classifier.
- Each pair is weighted by σ(−margin): wrong pairs count most; β sets the drift budget.
- GRPO swaps PPO’s critic for a group baseline: ~44% less memory, G rollouts per prompt.
- Per-answer averaging: a 10× longer wrong answer gets a 10× weaker penalty per token.
- R1-Zero: rule rewards took AIME from 15.6% to 77.9%; length and reflection grew unrewarded.
- R1: cold start → RL → 800K SFT → RL; small models do better distilled.
- Before believing a gain: noise, contamination, pass@k, reward vs held-out.
Your Week 6 plan
| # | Resource | Hours | Take away |
|---|---|---|---|
| 1 | DPO paper §1–6, App. A–B | 2 | derivation, gradient weight, β |
| 2 | DeepSeekMath §4, §5.2, App. A.1 | 2 | GRPO, unified paradigm, Maj@K vs Pass@K |
| 3 | DeepSeek-R1 main + Supp. A–C, E–G | 3 | R1-Zero, stages, distillation |
| 4 | State of RL for LLM Reasoning Raschka · also Understanding Reasoning LLMs | 2 | PPO → GRPO, length bias |
| 5 | RLHF book: ch. 8, ch. 7, ch. 6 GRPO + loss aggregation Lambert | 2 | DPO weak spots, RLVR practice |
11 hours listed against a 9-hour budget (book hours are an estimate for these chapters). To fit: skip DeepSeekMath §2–3 and R1’s safety report (Supp. D.3); skim Raschka’s paper list.
Self-test: answer out loud, then open
Why does Z(x) vanish from the DPO loss?
Bradley–Terry only uses the reward gap; both answers share the prompt, so the β·log Z(x) terms cancel.
DPO weight for a pair at margin −2 vs +2?
σ(2) = 0.88 vs σ(−2) = 0.12: the wrongly ordered pair gets about 7.4× the update.
With β = 0.1, what log-ratio gap drops a pair’s weight to 0.1?
ln 9 ÷ 0.1 ≈ 22 nats. With β = 0.5 it is 4.4: higher β, less drift needed.
GRPO group rewards 1, 0, 0, 0: the advantages?
Mean 0.25, std 0.433 → +1.73 for the right answer, −0.58 for each wrong one.
Training memory, 7B: PPO vs GRPO (16 B trained, 2 B frozen)?
PPO 36 × 7B = 252 GB; GRPO with a reward model 20 × 7B = 140 GB: 44% less.
Per-token penalty, 3,000- vs 300-token wrong answers, GRPO loss?
10× smaller for the long one; each answer gets the same total push.
What did R1’s cold start cost and buy?
AIME 77.9 → 59.0, but IF-Eval 46.6 → 71.7; the next RL stage brought AIME back to 74.0.
Why distil a 32B model instead of running RL on it?
Distilled 72.6% vs RL-only 47.0% on AIME 2024, at a fraction of the compute.
Week 7: run a small GRPO job yourself
Reward functions for math, what to log, healthy and broken training curves, and debugging, with TRL’s GRPO trainer, CS336 assignment 5 and nanochat. Plus the DeepMind fundamentals and RL quizzes as practice.
“RL training debugging scenarios: what do you do when you hit entropy collapse, the model reasoning in another language, or terrible MFU?”