Is the gain real,
and what would you fix?
Debugging reasoning RL, telling real capability from measurement artifacts, and critiquing a model with an experiment you could run.
About 10 hours with the readings · use → or Space to step through · O for contents · T for dark mode
What this week is for
Defend a claim about a model with evidence, and name the experiment that would settle it.
“Your RL run improved pass@1 on AIME by 15 points. Convince me it learned something new and isn’t reward hacking or sharpening the base model.”
“Pick one weakness in our current public model, tell me the likely cause in the training pipeline, and describe the first experiment you’d run to fix it.”
How labs ask it
- “RL training debugging scenarios: what do you do when you hit entropy collapse, the model reasoning in another language, or terrible MFU?” frontier lab, unnamed · secondhand 2025
- “Round on post-training async RL (architect an asynchronous RL post-training system)” Meta MSL · firsthand 2025
LLM internals is ~2% of collected interview reports, none of the 18 staff-plus ones. Core in research loops.
Making reasoning RL work: DAPO and entropy
DAPO ran RL on the Qwen2.5-32B base with rule-checked math answers (+1 right, −1 wrong). Plain GRPO reached 30 on AIME 2024; DeepSeek’s R1-Zero recipe on the same base, 47. Four fixes reached 50 with half the steps.
The setup, in numbers
AIME 2024: 30 competition problems, each answered 32 times and averaged (avg@32), at temperature 1.0, top-p 0.7. Training: 17K prompts with integer answers; 512 prompts × 16 answers per rollout step; 16 gradient updates per step; learning rate 1 × 10⁻⁶; no KL penalty; answers up to 20,480 tokens. Diagnosis of plain GRPO: entropy collapse, reward noise, unstable training. Source: DAPO (arXiv 2503.14476 v2) §1, §2.3–2.4, §3.5, §4.1.
Fix 1 · Clip-Higher: let rare tokens grow
PPO and GRPO cap how far one update can move a token’s probability: at most × (1 + ε), with ε = 0.2.
That cap is a ratio. A likely token can already go past 1; an unlikely “exploration” token can barely move. Tokens that hit the upper cap had mean probability below 0.2.
DAPO raises only the upper bound: εhigh = 0.28, keeps εlow = 0.2. Raising the lower one too would push unlikely tokens to 0 and shrink what the model can sample.
DAPO §3.1 (the 0.01 → 0.012 and 0.9 → 1.08 example, Figure 3a) and §4.1 (ε values). Ceilings computed in your browser as probability × (1 + ε).
Fix 2 · Dynamic sampling: drop groups that teach nothing
GRPO scores each answer against its group’s mean. If all 16 answers are right (or all wrong), every advantage is zero, so the prompt adds no gradient.
As training goes on, more prompts are solved every time, so each batch has fewer prompts that count and the gradient gets noisier.
Fix: over-sample, throw away all-right and all-wrong groups, and keep sampling until the batch is full of useful prompts.
It costs more generation, yet DAPO reports reaching the same score sooner (Figure 6).
DAPO §3.2 and Figure 3b. Chance of a dead group = pᴳ + (1 − p)ᴳ, computed in your browser.
Fixes 3–4 · Weigh every token the same; don’t punish running out of room
Token-level loss. GRPO averages inside each answer first. With answers of 200 and 8,000 tokens, a short-answer token gets 40× the weight (1 ÷ 400 vs 1 ÷ 16,000), so gibberish in long answers is barely penalized. DAPO averages over all tokens: 1 ÷ 8,200 each.
Overlong shaping. Truncated answers got the wrong-answer reward, punishing sound but long reasoning. First mask them from the loss (30 → 36), then add a soft penalty: 0 up to 16,384 tokens, falling linearly to −1 at 20,480.
DAPO §3.3 (Figure 4), §3.4 (Eq. 13), §4.1, Table 1. The 200/8,000 lengths are made up for illustration; weights and penalty computed. Dr. GRPO (arXiv 2503.20783) attacks the same length bias.
Adding the fixes one at a time
DAPO Table 1 (AIME 2024 avg@32, Qwen2.5-32B base).
Entropy is a budget: RL spends it to buy reward
Policy entropy = how spread out the next-token probabilities are. Low entropy means near-identical samples, so little exploration.
Across 11 base models, without entropy or KL regularization, accuracy follows R = −a·eH + b. At H = 0 the ceiling is b − a.
Over 2,400 steps, the first 200 used 73% of the entropy and gave 76% of the gain. By step 800: 94% and 93%.
Fitting a and b on the first 36 steps predicted the next 200 with 0.9% error (math). The ceiling is visible early.
Cui et al., “The Entropy Mechanism of RL for Reasoning LMs” (arXiv 2505.22617 v1): Eq. 6, §2.3–2.4. The paper gives a and b only in plots, so a = 0.2 and b = 0.7 here are made up for illustration.
Why entropy falls, and a targeted fix
One update lowers entropy when the tokens the model already likes are the ones that get rewarded. Formally: entropy change ≈ −covariance(log-probability, advantage).
| advantage > 0 | advantage < 0 | |
|---|---|---|
| likely token | entropy ↓ | entropy ↑ |
| rare token | entropy ↑ | entropy ↓ |
A few tokens dominate: the top 0.02% had mean covariance 5.654; the average over all tokens was 0.003.
Clip-Cov drops the gradient on a random 0.02% of high-covariance tokens. KL-Cov adds a KL penalty on the top 0.02–0.2%.
| Qwen2.5-32B, math | AIME24 | 7-bench avg |
|---|---|---|
| GRPO | 21.8 | 45.8 |
| + Clip-higher (DAPO) | 35.6 | 47.2 |
| + Clip-Cov | 32.3 | 50.3 |
| + KL-Cov | 36.8 | 52.2 |
Plain entropy bonus: too sensitive to its coefficient. KL to the reference model: stable entropy, worse accuracy.
Cui et al.: Theorems 1–2, Table 1, Table 2, §4.1–4.3. Single runs on Qwen2.5 only; no error bars reported.
“Entropy collapse, another language, terrible MFU: what do you do?”
| Symptom | Check first | First fix |
|---|---|---|
| Entropy collapse | entropy curve; probability of tokens hitting the upper clip | Clip-Higher; Clip-Cov or KL-Cov |
| Reasons in another language | share of target-language words in the chain of thought | R1: a language-consistency reward; costs a little accuracy |
| Length blow-up | length and entropy rising together | token-level loss; overlong shaping |
| Terrible MFU | time split: generate, train, idle | sync steps wait for the longest answer: overlap them (Part 4) |
- “What is the difference between PPO and GRPO?” frontier lab, unnamed · firsthand 2026
- “State two known biases of this formulation” (GRPO’s per-answer length normalization and its division by the group’s reward std). Google DeepMind · question bank
Question at top: frontier lab, unnamed · secondhand 2025. DAPO §3–4; Cui et al. §4; DeepSeek-R1 v1 §2.3.2; Magistral (arXiv 2506.10910) abstract on forcing the reasoning language. GDM item: RL quiz Q4.
Is the gain real? New capability, sharpening, or more samples
RL wins at k = 1; the base model catches up near k = 128
pass@k: solved if any of k samples is right. pass@1 measures how often; pass@256 asks whether the model can solve it at all.
Qwen2.5-7B with GRPO: longer training lifts pass@1 from 26.1 to 42.5, while pass@256 falls from 66.3 to 64.3. Base: 67.2.
Yue et al.’s reading: RL samples existing good paths more often and stops finding some it used to.
Estimating pass@k without bias
Sample n ≥ k answers, count c correct; pass@k = 1 − C(n−c, k) ÷ C(n, k). Example: n = 16, c = 2 gives pass@1 = 0.125 but pass@8 = 1 − 3,003 ÷ 12,870 = 0.767. Yue et al. use n = 128 to 1,024, temperature 0.6, top-p 0.95.
Yue et al. (arXiv 2504.13837 v5): §2.2, Appendix A.2, Figure 1 (right), Table 4. k = 1 and 256 values are from Table 4; values in between are read by eye from the plot (about ±0.01).
More evidence, and where it is weaker than the headline
Yue et al.: Table 5, §3.1, §4.2, Table 3, §4.6. ProRL (NVIDIA 2025, in the reading list’s extra links) argues longer RL expands the boundary; not covered here.
“+15 points on AIME. Convince me it’s real.”
- Noise. AIME is 30 problems: one problem = 3.3 points. Report avg@32 and how much it moves between seeds.
- Sharpening or new? pass@k at k = 256 against the base; list problems only the RL model solves.
- Placebo rewards. On Qwen2.5-Math-7B, random rewards still gave +15.8 on MATH-500 (true rewards: +24.6). Run that control.
- Contamination. Re-test on number-perturbed versions of the problems.
- Hacking. Read the chains of thought on newly solved problems; check the verifier can’t be gamed.
Interconnects, “RL with random rewards actually works with Qwen 2.5” (2025) for the reward controls and number-perturbation evidence. Noise computed in your browser assuming the worst case, each problem solved half the time: √(0.25 ÷ (30 × n)), and √2 times that for a difference. This is sampling noise only; a different set of 30 problems adds more.
Two ways to spend more compute on one question
Scroll the diagram sideways.
Snell et al. (2024), §2, §5.2, §6.1. PaLM 2-S* fine-tuned as verifier or reviser, on MATH. The PRM (process reward model) scores each step; it was trained from Monte Carlo rollouts because a PRM trained on PRM800K was easy to exploit.
The best strategy depends on how hard the question is
Bin questions into 5 levels by the base model’s pass rate, then pick the best strategy per bin: compute-optimal scaling.
| Difficulty | Search with a PRM | Revisions |
|---|---|---|
| Easy (bins 1–2) | best-of-N wins; beam search over-optimizes the PRM | all sequential |
| Medium-hard (3–4) | beam search beats best-of-N | a mix of parallel and sequential |
| Hardest (5) | nothing helps much | small gains |
Snell et al. (2024), §3.2, §5.3, §6.2, §8 and Appendix C.
Test-time compute vs a 14× bigger model, at equal FLOPs
Training costs 6 × parameters × training tokens; serving costs 2 × parameters × served tokens. A 14× bigger model pays 14× on both.
Give the small model the same total budget at test time. How much extra it gets depends on R = served tokens ÷ training tokens.
Heavy serving (big R) leaves little extra per question, so the bigger model wins on medium and hard questions. Test-time compute and pretraining are not one-to-one swappable.
Snell et al. (2024), §7 and Figure 1 (right). Multiplier computed live: 14 + 3 × (14 − 1) ÷ R. Bars are the printed labels on Figure 1’s bars: relative accuracy change of the small model with compute-optimal test-time compute vs the bigger model’s greedy answer.
Part 2: questions you should now handle
- “Your RL run improved pass@1 on AIME by 15 points. Convince me it learned something new and isn’t reward hacking or sharpening the base model.” senior-staff probe, from the curriculum
- “During RLHF training against a learned reward model, describe the symptoms you would look for to detect that the policy is over-optimising the reward model rather than genuinely improving, and state concrete mitigations.” Google DeepMind · question bank
Likely follow-up: “Spend the next 10× of compute on RL or on test-time sampling?” Answer with R and difficulty.
Answer sketch for the over-optimization question
Symptom: the proxy reward keeps rising while a held-out judge or human spot checks flatten or fall, and KL from the start keeps growing; watch length and style drift too. Mitigations: a bigger reward model and more preference data (Gao et al., via Weng), a KL penalty or early stopping on the held-out judge, reward-model ensembles whose disagreement flags exploited regions, refreshing labels on current policy samples (Casper §3.4), and verifiable rewards where the task allows.
GDM item: RL quiz Q8. The follow-up is this deck’s suggestion.
When the score lies: reward hacking, RLHF’s limits, leaderboards
Reward hacking: high score, wrong behavior
Reward hacking: the policy exploits flaws in the reward to score high without doing the task. Weng splits it into a misspecified environment or goal, and tampering with the reward itself.
Lilian Weng, “Reward Hacking in Reinforcement Learning” (2024): List of Examples; Hacking the Training Process; Hacking the Evaluator; In-Context Reward Hacking; Generalization of Hacking Skills.
Over-optimizing a reward model: the hump
Gao et al. trained policies against small proxy reward models (3M–3B parameters) and scored them with a 6B gold reward model standing in for people.
The proxy keeps rising as the policy drifts from where it started. The gold score rises, peaks, then falls.
Bigger reward models and more preference data push the peak further out. A KL penalty acted like early stopping.
Functional forms from Gao, Schulman & Hilton (2022) as given in Weng (2024): gold = d(α − β·log d) for RL, d(α − β·d) for best-of-n, with d = √KL. The α, β values here are made up for illustration; the fitted ones are only in the paper’s figures.
RLHF’s open problems: which ones won’t go away?
Casper et al. list 23 problems and label each tractable (fixable inside RLHF) or fundamental (needs a method that is no longer RLHF).
| Stage | Problems | Fundamental | The ones to name in an interview |
|---|---|---|---|
| Human feedback | 10 | 4 | humans can be misled; can’t grade hard tasks |
| Reward model | 5 | 4 | an imperfect proxy gets hacked; one reward can’t represent everyone |
| Policy | 6 | 2 | mode collapse (tractable); misgeneralization in deployment |
| Joint training | 2 | 0 | the policy shifts the reward model’s data |
Casper et al. (2023), §3.1–3.4 labels (counts are this deck’s tally), §3.2.1 (63–77%), §3.1.2 (Saunders et al. 2022).
The Leaderboard Illusion: Chatbot Arena’s numbers
Chatbot Arena ranks models by votes between two anonymous answers. Singh et al. audited 2M battles and 243 models (2024–25).
Singh et al., “The Leaderboard Illusion” (arXiv 2504.20879): Abstract, §2, §3.1, §4.1–4.2, §5, Table 9. The 1417 quote is Meta’s, via Interconnects, “Llama 4: Did Meta just push the panic button?” (April 2025).
Test N variants, publish the best: free points
Each Arena score is a noisy estimate. Publish the maximum of N estimates and the score is biased upward, even if every variant is equally good.
Real test: Cohere submitted two identical copies of Aya-Vision-8B. They scored 1052 and 1069, with 4 other models ranked between them.
The paper’s fixes: no retracting scores, a public cap on private variants per provider, equal deprecation rules, transparent sampling.
Singh et al.: §3.2–3.3 (Aya-Vision experiment), §7 recommendations, Appendix I Eqs. 5–7: gain ≈ √(spread² + (347.4 ÷ √battles)²) × √(2 ln N), with 3,000 battles per model. Computed in your browser.
Part 3: questions this prepares you for
- “Explain Constitutional AI. What does it buy you over vanilla RLHF, and what doesn’t it solve?” Anthropic · question bank
- “Design a feedback collection pipeline for RLHF.” OpenAI · prep site
Using this week on them
What AI feedback doesn’t solve: Casper et al. call it a bootstrapping problem (the feedback model must itself be aligned), and the reward model’s fundamental problems remain: an imperfect proxy gets hacked, and one reward can’t represent everyone. For a feedback pipeline, design against the failure list: how labelers are chosen and checked (agreement was only 63–77% in major papers), which prompts get labeled, feedback type, and refreshing labels on current policy samples so the policy can’t camp in regions the reward model never saw.
Questions from the curriculum’s question bank. Sketch from Casper et al. §3, §4.2.1 and §5.
Post-training systems: on-policy distillation and async RL
On-policy distillation: the teacher grades every token you wrote
| Method | Samples from | Reward |
|---|---|---|
| SFT on teacher text | teacher | dense |
| RL | student | sparse |
| On-policy distillation | student | dense |
The student learns in the states it actually reaches, so it learns to recover from its own mistakes. RL teaches O(1) bits per episode; this teaches O(tokens).
It needs a teacher whose tokens the student can already produce (mid-train first), and it can be no better than the teacher.
Thinking Machines, “On-Policy Distillation” (2025): Implementation (reverse KL, discount 0, pseudocode), Discussion. Reverse KL is “mode seeking”: it copies one behavior instead of spreading over several.
From 60% to 70% on AIME’24: three ways to pay
Qwen3’s own report: RL reached 67.6% in 17,920 GPU-hours; on-policy distillation reached 74.4% in 1,800.
Fixing forgetting: mid-training on company documents dropped instruction-following (IF-eval) from 85% to 79%; distilling from the original model brought it back to 83%, and knowledge rose 36% → 41%.
The 2M-prompt SFT point is extrapolated from a log-linear trend. RL is placed at about the same cost.
Thinking Machines (2025): Distillation for reasoning (FLOPs table; Qwen3 Technical Report Table 21), Distillation for personalization (table). Ratios computed live from the table’s FLOPs.
Sync vs async RL: stop the GPUs waiting for each other
Scroll the diagram sideways.
The price is staleness: θt+1 is trained on answers from θt. The real system stays one step behind.
Measured, Llama 3.1 8B on 8 H100s: 21 s + 33 s per sync step ≈ 209 min. Ideal async = 128 min; actual 151 min (lock contention, weight sync).
Noukhovitch et al., “Asynchronous RLHF” (ICLR 2025, arXiv 2410.18252 v3): Figure 2, §3.5, Appendix A.2. Bar lengths drawn to those step times.
Designing async RL post-training: decisions with numbers
Noukhovitch et al. (2025): §3.2–3.4, §4.1–4.2, Tables 1–2. The paper’s “% faster” figures mix two definitions, so raw minutes are quoted here.
Can you design and defend this in 45 minutes?
- “Round on post-training async RL (architect an asynchronous RL post-training system)” Meta MSL · firsthand 2025
- “Name two ways an LLM post-training system ends up with μ ≠ π without that being a deliberate design choice.” (μ: the policy that sampled the data; π: the one being trained) Google DeepMind · question bank
- “RL training debugging scenarios: what do you do when you hit … terrible MFU?” frontier lab, unnamed · secondhand 2025
A strong answer outline
Generation workers on an inference engine (the paper found vllm 12× faster than HF transformers for generation), a trainer, a reward or verifier service, a queue with a staleness bound of one step, and periodic weight sync. Log staleness, importance ratios, KL and per-step time split. For μ ≠ π without meaning to: inference-engine numerics differ from the trainer’s, and async or multi-step updates on one batch. For MFU: synchronous steps wait for the longest answer (DAPO notes long tails dominate generation time), so overlap generation and training.
Question bank; the GDM item is from its RL quiz, Q9. Outline built from Noukhovitch et al. (§3, App. A.2) and DAPO §3.2.
Critique a model and propose the experiment
A critique method you can reuse on any model
Scroll the diagram sideways.
Ground step 1 in the lab’s own reports, not social media. Don’t claim to know their pipeline; say “if the cause is X, I’d expect Y”.
Worked example: the GPT-4o sycophancy rollback (April 2025)
OpenAI’s post-mortems (openai.com/index/sycophancy-in-gpt-4o, …/expanding-on-sycophancy), as quoted in Interconnects, “Sycophancy and the art of the model” (May 2025). Steps 3–4 are this deck’s proposal. Sycophancy = matching user beliefs over truth (Sharma et al. 2023, via Weng).
Before each loop: read the lab’s own reports
| Lab | Read | Pull out |
|---|---|---|
| Anthropic | system cards | their own measured failures: search for reward hacking, sycophancy, honesty |
| OpenAI | system cards; Model Spec; 2025 sycophancy post-mortems | where measured behavior departs from the Spec |
| Google DeepMind | model cards, Gemini technical reports | which evals they report, and which they don’t |
| Meta | Llama 3 paper; Llama 4 release notes | post-training recipe; the Arena “experimental” gap |
| xAI | Grok 4 model card §2.2 | deception (MASK) and sycophancy numbers, and what moved them |
| Mistral | Magistral paper | pure-RL pipeline; how they force the reasoning language |
For each lab, write one weakness they report, the stage it points to, and your experiment. Then read the Interconnects post on that release for an outside view.
Links checked November 2026. The Grok 4 card’s §2.2 reports MASK deception and Sharma et al. answer sycophancy; Magistral’s abstract describes forcing the reasoning language.
The senior-staff probes, answered with the method
- “Pick one weakness in our current public model, tell me the likely cause in the training pipeline, and describe the first experiment you’d run to fix it.” senior-staff probe, from the curriculum
- “Walk me through a post-training recipe to turn a pretrained base model into a personalised assistant.” Meta MSL · question bank
- “After fine-tuning on a customer’s task, target accuracy is up but the model got worse at everything else. What happened and what do you do?” Mistral · question bank
Using this week on the last two
Personalised assistant: mid-train on the user’s or company’s data, then on-policy distill from the original model to recover instruction following (IF-eval 79% → 83% vs 85% before, Thinking Machines). Forgetting: the same recipe; note that even SFT on the model’s own samples drifted in their test, while on-policy distillation stays on-policy.
Questions from the curriculum’s question bank and l7_bar.
Week 9 in one minute
- DAPO: four fixes, AIME 30 → 50.
- Entropy buys reward; the ceiling shows early. Target high-covariance tokens.
- pass@k: RL wins at k = 1, the base often at k = 256. Show large-k evidence.
- Test-time compute beats a 14× model on easy questions with light serving only.
- Scores lie: reward hacking, RLHF’s fundamental limits, leaderboard selection.
- Systems: on-policy distillation 9–30× cheaper than SFT; async RL trades one step of staleness for speed.
- Critique: observe → hypothesize → cheapest test → mind-changer.
Your Week 9 plan
| Resource | Listed h | Plan h | Take away |
|---|---|---|---|
| DAPO Yu et al. | 2 | 1 | §3–4: four fixes, Table 1 |
| Entropy mechanism Cui et al. | 2 | 1 | §2–4: the law, covariance, KL-Cov |
| Does RL incentivize reasoning? Yue et al. | 1.5 | 1 | §2–4, Tables 2–5 |
| Test-time compute Snell et al. | 2 | 1 | §1, §3, §7, Figure 1 |
| Reward Hacking Weng | 2 | 1 | examples, RLHF hacking, Gao et al. |
| RLHF open problems Casper et al. | 3 | 1 | §3 taxonomy, §5 disclosure list |
| The Leaderboard Illusion | 1.5 | 0.75 | §1–4 and the recommendations |
| Interconnects: GRPO tweaks, Llama 4, o3 over-optimization, sycophancy, random rewards | 6 | 1.5 | how an expert critiques a release |
| On-Policy Distillation Thinking Machines | 1.5 | 0.75 | reverse-KL reward, cost table |
| Asynchronous RLHF Noukhovitch et al. | 2 | 1 | §3, §5, Appendix A.2 |
Listed hours total 23.5; the plan column totals 10. The guide gives weeks 8–9 together 15–20 hours, so read the named sections and skim the rest.
Self-test: answer out loud, then open
Why does DAPO raise only the upper clip?
With ε = 0.2 a 0.01 token can only reach 0.012 per update, so rare tokens can’t grow. Raising the lower clip too would push unlikely tokens to 0 and shrink the sampling space.
A prompt is solved 90% of the time, G = 16. How often is its group useless?
0.9¹⁶ + 0.1¹⁶ ≈ 18.5% of the time all 16 are right, so every advantage is zero. DAPO resamples until the batch is full of useful groups.
DAPO’s penalty for an 18,432-token answer?
(16,384 − 18,432) ÷ 4,096 = −0.5, added to the correctness reward. Past 20,480 tokens it is −1.
What does Yue et al.’s AIME 2024 solved-set table show?
At k = 1,024 the base solves 23 of 30, the RL model 19, all inside the base set. RL sharpened, it didn’t extend, in that run.
State the entropy law and its ceiling.
R = −a·eH + b; at H = 0, R = b − a. About 73% of entropy and 76% of the gain go in the first 200 of 2,400 steps.
Serving 22× more tokens than you trained on: test-time compute or a 14× model?
The small model gets only 14 + 3 × 13 ÷ 22 ≈ 15.8× its serving compute. Snell et al. find the bigger model wins on medium and hard questions there.
On-policy distillation’s reward, and its cost?
Per token: teacher log-prob minus student log-prob on the student’s own sample. 70% on AIME’24 in ~150 steps, 9–30× fewer FLOPs than SFT to the same score.
Which loss tolerates stale data best in async RLHF?
Online DPO; bigger policies help, bigger reward models don’t. Llama 3.1 8B: same 57.2% win rate, 142 vs 230 minutes.
Nine weeks in one picture
Next: pick one lab, run the critique method on its latest report, and answer both probes out loud in five minutes each.