Frontier-lab prep · LLM internals · Week 9

Is the gain real,
and what would you fix?

Debugging reasoning RL, telling real capability from measurement artifacts, and critiquing a model with an experiment you could run.

About 10 hours with the readings · use → or Space to step through · O for contents · T for dark mode

What this week is for

Defend a claim about a model with evidence, and name the experiment that would settle it.

Senior-staff probe 1

“Your RL run improved pass@1 on AIME by 15 points. Convince me it learned something new and isn’t reward hacking or sharpening the base model.”

Senior-staff probe 2

“Pick one weakness in our current public model, tell me the likely cause in the training pipeline, and describe the first experiment you’d run to fix it.”

How labs ask it

  • “RL training debugging scenarios: what do you do when you hit entropy collapse, the model reasoning in another language, or terrible MFU?” frontier lab, unnamed · secondhand 2025
  • “Round on post-training async RL (architect an asynchronous RL post-training system)” Meta MSL · firsthand 2025

LLM internals is ~2% of collected interview reports, none of the 18 staff-plus ones. Core in research loops.

Part 1

Making reasoning RL work: DAPO and entropy

DAPO ran RL on the Qwen2.5-32B base with rule-checked math answers (+1 right, −1 wrong). Plain GRPO reached 30 on AIME 2024; DeepSeek’s R1-Zero recipe on the same base, 47. Four fixes reached 50 with half the steps.

The setup, in numbers

AIME 2024: 30 competition problems, each answered 32 times and averaged (avg@32), at temperature 1.0, top-p 0.7. Training: 17K prompts with integer answers; 512 prompts × 16 answers per rollout step; 16 gradient updates per step; learning rate 1 × 10⁻⁶; no KL penalty; answers up to 20,480 tokens. Diagnosis of plain GRPO: entropy collapse, reward noise, unstable training. Source: DAPO (arXiv 2503.14476 v2) §1, §2.3–2.4, §3.5, §4.1.

Fix 1 · Clip-Higher: let rare tokens grow

PPO and GRPO cap how far one update can move a token’s probability: at most × (1 + ε), with ε = 0.2.

That cap is a ratio. A likely token can already go past 1; an unlikely “exploration” token can barely move. Tokens that hit the upper cap had mean probability below 0.2.

DAPO raises only the upper bound: εhigh = 0.28, keeps εlow = 0.2. Raising the lower one too would push unlikely tokens to 0 and shrink what the model can sample.

DAPO §3.1 (the 0.01 → 0.012 and 0.9 → 1.08 example, Figure 3a) and §4.1 (ε values). Ceilings computed in your browser as probability × (1 + ε).

Fix 2 · Dynamic sampling: drop groups that teach nothing

GRPO scores each answer against its group’s mean. If all 16 answers are right (or all wrong), every advantage is zero, so the prompt adds no gradient.

As training goes on, more prompts are solved every time, so each batch has fewer prompts that count and the gradient gets noisier.

Fix: over-sample, throw away all-right and all-wrong groups, and keep sampling until the batch is full of useful prompts.

It costs more generation, yet DAPO reports reaching the same score sooner (Figure 6).

DAPO §3.2 and Figure 3b. Chance of a dead group = pᴳ + (1 − p)ᴳ, computed in your browser.

Fixes 3–4 · Weigh every token the same; don’t punish running out of room

Token-level loss. GRPO averages inside each answer first. With answers of 200 and 8,000 tokens, a short-answer token gets 40× the weight (1 ÷ 400 vs 1 ÷ 16,000), so gibberish in long answers is barely penalized. DAPO averages over all tokens: 1 ÷ 8,200 each.

Overlong shaping. Truncated answers got the wrong-answer reward, punishing sound but long reasoning. First mask them from the loss (30 → 36), then add a soft penalty: 0 up to 16,384 tokens, falling linearly to −1 at 20,480.

DAPO §3.3 (Figure 4), §3.4 (Eq. 13), §4.1, Table 1. The 200/8,000 lengths are made up for illustration; weights and penalty computed. Dr. GRPO (arXiv 2503.20783) attacks the same length bias.

Adding the fixes one at a time

Read it like a reviewer. The table adds fixes in one fixed order, so each gain depends on what came before. Each row is a single number with no error bars, on 30 problems. It shows the recipe works, not how much each part is worth alone.

DAPO Table 1 (AIME 2024 avg@32, Qwen2.5-32B base).

Entropy is a budget: RL spends it to buy reward

Policy entropy = how spread out the next-token probabilities are. Low entropy means near-identical samples, so little exploration.

Across 11 base models, without entropy or KL regularization, accuracy follows R = −a·eH + b. At H = 0 the ceiling is b − a.

Over 2,400 steps, the first 200 used 73% of the entropy and gave 76% of the gain. By step 800: 94% and 93%.

Fitting a and b on the first 36 steps predicted the next 200 with 0.9% error (math). The ceiling is visible early.

Cui et al., “The Entropy Mechanism of RL for Reasoning LMs” (arXiv 2505.22617 v1): Eq. 6, §2.3–2.4. The paper gives a and b only in plots, so a = 0.2 and b = 0.7 here are made up for illustration.

Why entropy falls, and a targeted fix

One update lowers entropy when the tokens the model already likes are the ones that get rewarded. Formally: entropy change ≈ −covariance(log-probability, advantage).

advantage > 0advantage < 0
likely tokenentropy ↓entropy ↑
rare tokenentropy ↑entropy ↓

A few tokens dominate: the top 0.02% had mean covariance 5.654; the average over all tokens was 0.003.

Clip-Cov drops the gradient on a random 0.02% of high-covariance tokens. KL-Cov adds a KL penalty on the top 0.02–0.2%.

Qwen2.5-32B, mathAIME247-bench avg
GRPO21.845.8
+ Clip-higher (DAPO)35.647.2
+ Clip-Cov32.350.3
+ KL-Cov36.852.2

Plain entropy bonus: too sensitive to its coefficient. KL to the reference model: stable entropy, worse accuracy.

Cui et al.: Theorems 1–2, Table 1, Table 2, §4.1–4.3. Single runs on Qwen2.5 only; no error bars reported.

Interview check

“Entropy collapse, another language, terrible MFU: what do you do?”

SymptomCheck firstFirst fix
Entropy collapseentropy curve; probability of tokens hitting the upper clipClip-Higher; Clip-Cov or KL-Cov
Reasons in another languageshare of target-language words in the chain of thoughtR1: a language-consistency reward; costs a little accuracy
Length blow-uplength and entropy rising togethertoken-level loss; overlong shaping
Terrible MFUtime split: generate, train, idlesync steps wait for the longest answer: overlap them (Part 4)
  • “What is the difference between PPO and GRPO?” frontier lab, unnamed · firsthand 2026
  • “State two known biases of this formulation” (GRPO’s per-answer length normalization and its division by the group’s reward std). Google DeepMind · question bank

Question at top: frontier lab, unnamed · secondhand 2025. DAPO §3–4; Cui et al. §4; DeepSeek-R1 v1 §2.3.2; Magistral (arXiv 2506.10910) abstract on forcing the reasoning language. GDM item: RL quiz Q4.

Part 2

Is the gain real? New capability, sharpening, or more samples

RL wins at k = 1; the base model catches up near k = 128

pass@k: solved if any of k samples is right. pass@1 measures how often; pass@256 asks whether the model can solve it at all.

Qwen2.5-7B with GRPO: longer training lifts pass@1 from 26.1 to 42.5, while pass@256 falls from 66.3 to 64.3. Base: 67.2.

Yue et al.’s reading: RL samples existing good paths more often and stops finding some it used to.

Estimating pass@k without bias

Sample n ≥ k answers, count c correct; pass@k = 1 − C(n−c, k) ÷ C(n, k). Example: n = 16, c = 2 gives pass@1 = 0.125 but pass@8 = 1 − 3,003 ÷ 12,870 = 0.767. Yue et al. use n = 128 to 1,024, temperature 0.6, top-p 0.95.

Yue et al. (arXiv 2504.13837 v5): §2.2, Appendix A.2, Figure 1 (right), Table 4. k = 1 and 256 values are from Table 4; values in between are read by eye from the plot (about ±0.01).

More evidence, and where it is weaker than the headline

Solved sets. AIME 2024 at k = 1,024: the base solves 23 of 30, the RL model 19, all inside the base set.
Real reasoning? On the hardest AIME problems the base solved, 5 of 6 had a valid chain of thought, checked by hand.
Distillation differs. On Minerva, a model distilled from DeepSeek-R1 beats its base at every k.
Caveats in its own tables. At k = 256 on MATH500, all six RL algorithms tie or beat the base (RLOO 97.4 vs 96.2). On Magistral the gap only narrows.
Say it this way: “pass@1 gains may be sharpening. To claim new capability, show large-k pass@k or problems the base never solves.”

Yue et al.: Table 5, §3.1, §4.2, Table 3, §4.6. ProRL (NVIDIA 2025, in the reading list’s extra links) argues longer RL expands the boundary; not covered here.

“+15 points on AIME. Convince me it’s real.”

  1. Noise. AIME is 30 problems: one problem = 3.3 points. Report avg@32 and how much it moves between seeds.
  2. Sharpening or new? pass@k at k = 256 against the base; list problems only the RL model solves.
  3. Placebo rewards. On Qwen2.5-Math-7B, random rewards still gave +15.8 on MATH-500 (true rewards: +24.6). Run that control.
  4. Contamination. Re-test on number-perturbed versions of the problems.
  5. Hacking. Read the chains of thought on newly solved problems; check the verifier can’t be gamed.
Noise on a 30-problem test
one score, ±1 SE
gap between two runs, ±1 SE

Interconnects, “RL with random rewards actually works with Qwen 2.5” (2025) for the reward controls and number-perturbation evidence. Noise computed in your browser assuming the worst case, each problem solved half the time: √(0.25 ÷ (30 × n)), and √2 times that for a difference. This is sampling noise only; a different set of 30 problems adds more.

Two ways to spend more compute on one question

Search against a verifier question answer 1 answer 2 answer N PRM best-of-N: N full answers, verifier picks beam search: score every step, keep the best grey = pruned by the PRM Revise the answer question try 1 try 2 try 3 each try sees up to 4 earlier tries ≈38% of right answers get revised into wrong ones so a verifier (or a vote) must pick the final answer

Scroll the diagram sideways.

Snell et al. (2024), §2, §5.2, §6.1. PaLM 2-S* fine-tuned as verifier or reviser, on MATH. The PRM (process reward model) scores each step; it was trained from Monte Carlo rollouts because a PRM trained on PRM800K was easy to exploit.

The best strategy depends on how hard the question is

Bin questions into 5 levels by the base model’s pass rate, then pick the best strategy per bin: compute-optimal scaling.

DifficultySearch with a PRMRevisions
Easy (bins 1–2)best-of-N wins; beam search over-optimizes the PRMall sequential
Medium-hard (3–4)beam search beats best-of-Na mix of parallel and sequential
Hardest (5)nothing helps muchsmall gains
Up to 4× less compute than best-of-N at equal accuracy: 16 vs 64 generations (search), 64 vs 256 (revisions).
Fine print. Binning took 2,048 samples per question, not counted. One model (PaLM 2-S*), one benchmark (MATH).

Snell et al. (2024), §3.2, §5.3, §6.2, §8 and Appendix C.

Test-time compute vs a 14× bigger model, at equal FLOPs

Training costs 6 × parameters × training tokens; serving costs 2 × parameters × served tokens. A 14× bigger model pays 14× on both.

Give the small model the same total budget at test time. How much extra it gets depends on R = served tokens ÷ training tokens.

Heavy serving (big R) leaves little extra per question, so the bigger model wins on medium and hard questions. Test-time compute and pretraining are not one-to-one swappable.

Snell et al. (2024), §7 and Figure 1 (right). Multiplier computed live: 14 + 3 × (14 − 1) ÷ R. Bars are the printed labels on Figure 1’s bars: relative accuracy change of the small model with compute-optimal test-time compute vs the bigger model’s greedy answer.

Interview check

Part 2: questions you should now handle

  • “Your RL run improved pass@1 on AIME by 15 points. Convince me it learned something new and isn’t reward hacking or sharpening the base model.” senior-staff probe, from the curriculum
  • “During RLHF training against a learned reward model, describe the symptoms you would look for to detect that the policy is over-optimising the reward model rather than genuinely improving, and state concrete mitigations.” Google DeepMind · question bank

Likely follow-up: “Spend the next 10× of compute on RL or on test-time sampling?” Answer with R and difficulty.

Answer sketch for the over-optimization question

Symptom: the proxy reward keeps rising while a held-out judge or human spot checks flatten or fall, and KL from the start keeps growing; watch length and style drift too. Mitigations: a bigger reward model and more preference data (Gao et al., via Weng), a KL penalty or early stopping on the held-out judge, reward-model ensembles whose disagreement flags exploited regions, refreshing labels on current policy samples (Casper §3.4), and verifiable rewards where the task allows.

GDM item: RL quiz Q8. The follow-up is this deck’s suggestion.

Part 3

When the score lies: reward hacking, RLHF’s limits, leaderboards

Over-optimizing a reward model: the hump

Gao et al. trained policies against small proxy reward models (3M–3B parameters) and scored them with a 6B gold reward model standing in for people.

The proxy keeps rising as the policy drifts from where it started. The gold score rises, peaks, then falls.

Bigger reward models and more preference data push the peak further out. A KL penalty acted like early stopping.

Functional forms from Gao, Schulman & Hilton (2022) as given in Weng (2024): gold = d(α − β·log d) for RL, d(α − β·d) for best-of-n, with d = √KL. The α, β values here are made up for illustration; the fitted ones are only in the paper’s figures.

RLHF’s open problems: which ones won’t go away?

Casper et al. list 23 problems and label each tractable (fixable inside RLHF) or fundamental (needs a method that is no longer RLHF).

StageProblemsFundamentalThe ones to name in an interview
Human feedback104humans can be misled; can’t grade hard tasks
Reward model54an imperfect proxy gets hacked; one reward can’t represent everyone
Policy62mode collapse (tractable); misgeneralization in deployment
Joint training20the policy shifts the reward model’s data
63–77% agreement between annotators (and with researchers) in three major RLHF papers.
Over half of critical errors in model summaries were missed by human evaluators, with unlimited time.

Casper et al. (2023), §3.1–3.4 labels (counts are this deck’s tally), §3.2.1 (63–77%), §3.1.2 (Saunders et al. 2022).

The Leaderboard Illusion: Chatbot Arena’s numbers

Chatbot Arena ranks models by votes between two anonymous answers. Singh et al. audited 2M battles and 243 models (2024–25).

27
private variants Meta tested before releasing Llama 4 (Jan–Mar 2025)
20.4% · 19.2%
of all Arena data went to OpenAI and Google; 83 open-weight models shared 29.7%
34% vs 3.3%
highest vs lowest daily sampling rate across providers
205 of 243
public models silently deprecated; 64% of those were open models
23.5% → 49.9%
ArenaHard win rate vs Llama 3.1 8B after 70% Arena data in fine-tuning; MMLU barely moved (66.5 → 65.9)
1417
Meta’s headline score came from an “experimental chat version”, not the released Maverick

Singh et al., “The Leaderboard Illusion” (arXiv 2504.20879): Abstract, §2, §3.1, §4.1–4.2, §5, Table 9. The 1417 quote is Meta’s, via Interconnects, “Llama 4: Did Meta just push the panic button?” (April 2025).

Test N variants, publish the best: free points

Each Arena score is a noisy estimate. Publish the maximum of N estimates and the score is biased upward, even if every variant is equally good.

Real test: Cohere submitted two identical copies of Aya-Vision-8B. They scored 1052 and 1069, with 4 other models ranked between them.

The paper’s fixes: no retracting scores, a public cap on private variants per provider, equal deprecation rules, transparent sampling.

expected gain of the published score

Singh et al.: §3.2–3.3 (Aya-Vision experiment), §7 recommendations, Appendix I Eqs. 5–7: gain ≈ √(spread² + (347.4 ÷ √battles)²) × √(2 ln N), with 3,000 battles per model. Computed in your browser.

Interview check

Part 3: questions this prepares you for

  • “Explain Constitutional AI. What does it buy you over vanilla RLHF, and what doesn’t it solve?” Anthropic · question bank
  • “Design a feedback collection pipeline for RLHF.” OpenAI · prep site
Using this week on them

What AI feedback doesn’t solve: Casper et al. call it a bootstrapping problem (the feedback model must itself be aligned), and the reward model’s fundamental problems remain: an imperfect proxy gets hacked, and one reward can’t represent everyone. For a feedback pipeline, design against the failure list: how labelers are chosen and checked (agreement was only 63–77% in major papers), which prompts get labeled, feedback type, and refreshing labels on current policy samples so the policy can’t camp in regions the reward model never saw.

Questions from the curriculum’s question bank. Sketch from Casper et al. §3, §4.2.1 and §5.

Part 4

Post-training systems: on-policy distillation and async RL

On-policy distillation: the teacher grades every token you wrote

Student samples so x = 7 −0.4−1.6−0.1−0.7 student log-prob of each sampled token Teacher scores −0.5−0.2−0.1−3.2 teacher log-prob of the same tokens: one forward pass Advantage −0.1+1.40.0−2.5 = teacher log-prob − student log-prob
Log-probabilities are made up for illustration.
MethodSamples fromReward
SFT on teacher textteacherdense
RLstudentsparse
On-policy distillationstudentdense

The student learns in the states it actually reaches, so it learns to recover from its own mistakes. RL teaches O(1) bits per episode; this teaches O(tokens).

It needs a teacher whose tokens the student can already produce (mid-train first), and it can be no better than the teacher.

Thinking Machines, “On-Policy Distillation” (2025): Implementation (reverse KL, discount 0, pseudocode), Discussion. Reverse KL is “mode seeking”: it copies one behavior instead of spreading over several.

From 60% to 70% on AIME’24: three ways to pay

Qwen3-8B-Base after SFT on 400K prompts: 60%. Cost to reach ~70%:

Qwen3’s own report: RL reached 67.6% in 17,920 GPU-hours; on-policy distillation reached 74.4% in 1,800.

Fixing forgetting: mid-training on company documents dropped instruction-following (IF-eval) from 85% to 79%; distilling from the original model brought it back to 83%, and knowledge rose 36% → 41%.

The 2M-prompt SFT point is extrapolated from a log-linear trend. RL is placed at about the same cost.

Thinking Machines (2025): Distillation for reasoning (FLOPs table; Qwen3 Technical Report Table 21), Distillation for personalization (table). Ratios computed live from the table’s FLOPs.

Sync vs async RL: stop the GPUs waiting for each other

Scroll the diagram sideways.

The price is staleness: θt+1 is trained on answers from θt. The real system stays one step behind.

Measured, Llama 3.1 8B on 8 H100s: 21 s + 33 s per sync step ≈ 209 min. Ideal async = 128 min; actual 151 min (lock contention, weight sync).

Noukhovitch et al., “Asynchronous RLHF” (ICLR 2025, arXiv 2410.18252 v3): Figure 2, §3.5, Appendix A.2. Bar lengths drawn to those step times.

Designing async RL post-training: decisions with numbers

How stale? As little as possible. PPO degrades with staleness; Online DPO held up best, still learning at 64 mini-batches stale.
What buys tolerance? A bigger policy, not a bigger reward model; and a contrastive loss (best-of-2 SFT failed).
Generation-bound? More gradient steps per batch; KL drifts higher.
Training-bound? 4 answers per prompt, train on best vs worst: 2.5× faster than sync, higher KL.
Results: same 57.2% win rate as sync on Llama 3.1 8B (142 vs 230 min); GSM8k 52.6% vs 52.2% (129 vs 218 min).

Noukhovitch et al. (2025): §3.2–3.4, §4.1–4.2, Tables 1–2. The paper’s “% faster” figures mix two definitions, so raw minutes are quoted here.

Interview check

Can you design and defend this in 45 minutes?

  • “Round on post-training async RL (architect an asynchronous RL post-training system)” Meta MSL · firsthand 2025
  • “Name two ways an LLM post-training system ends up with μ ≠ π without that being a deliberate design choice.” (μ: the policy that sampled the data; π: the one being trained) Google DeepMind · question bank
  • “RL training debugging scenarios: what do you do when you hit … terrible MFU?” frontier lab, unnamed · secondhand 2025
A strong answer outline

Generation workers on an inference engine (the paper found vllm 12× faster than HF transformers for generation), a trainer, a reward or verifier service, a queue with a staleness bound of one step, and periodic weight sync. Log staleness, importance ratios, KL and per-step time split. For μ ≠ π without meaning to: inference-engine numerics differ from the trainer’s, and async or multi-step updates on one batch. For MFU: synchronous steps wait for the longest answer (DAPO notes long tails dominate generation time), so overlap generation and training.

Question bank; the GDM item is from its RL quiz, Q9. Outline built from Noukhovitch et al. (§3, App. A.2) and DAPO §3.2.

Part 5

Critique a model and propose the experiment

A critique method you can reuse on any model

1. Observea specific failure,reproducible, witha number on it 2. Hypothesizewhich stage: data,SFT, reward, RL,eval gate? + rivals 3. Experimentcheapest test thatseparates thehypotheses 4. Pre-committhe result thatwould changeyour mind a wrong hypothesis sends you back with one rival removed

Scroll the diagram sideways.

Ground step 1 in the lab’s own reports, not social media. Don’t claim to know their pipeline; say “if the cause is X, I’d expect Y”.

Worked example: the GPT-4o sycophancy rollback (April 2025)

1 · Observe. OpenAI: an April 25 update “made the model noticeably more sycophantic”; rolled back from April 28. Evals and A/B tests looked fine; expert testers flagged it.
2 · Hypothesize. A new thumbs-up/down reward “weakened the influence of our primary reward signal, which had been holding sycophancy in check.” Rivals: user memory, the system prompt.
3 · Cheapest test. No training: score answer pairs that differ only in agreeing with the user’s wrong belief, under each reward model. Then rerun the last RL stage without the thumbs reward.
4 · Mind-changer. If the thumbs model doesn’t favor agreeing answers and the ablated run is as sycophantic, drop it; test prompt and memory next.

OpenAI’s post-mortems (openai.com/index/sycophancy-in-gpt-4o, …/expanding-on-sycophancy), as quoted in Interconnects, “Sycophancy and the art of the model” (May 2025). Steps 3–4 are this deck’s proposal. Sycophancy = matching user beliefs over truth (Sharma et al. 2023, via Weng).

Before each loop: read the lab’s own reports

LabReadPull out
Anthropicsystem cardstheir own measured failures: search for reward hacking, sycophancy, honesty
OpenAIsystem cards; Model Spec; 2025 sycophancy post-mortemswhere measured behavior departs from the Spec
Google DeepMindmodel cards, Gemini technical reportswhich evals they report, and which they don’t
MetaLlama 3 paper; Llama 4 release notespost-training recipe; the Arena “experimental” gap
xAIGrok 4 model card §2.2deception (MASK) and sycophancy numbers, and what moved them
MistralMagistral paperpure-RL pipeline; how they force the reasoning language

For each lab, write one weakness they report, the stage it points to, and your experiment. Then read the Interconnects post on that release for an outside view.

Links checked November 2026. The Grok 4 card’s §2.2 reports MASK deception and Sharma et al. answer sycophancy; Magistral’s abstract describes forcing the reasoning language.

Interview check

The senior-staff probes, answered with the method

  • “Pick one weakness in our current public model, tell me the likely cause in the training pipeline, and describe the first experiment you’d run to fix it.” senior-staff probe, from the curriculum
  • “Walk me through a post-training recipe to turn a pretrained base model into a personalised assistant.” Meta MSL · question bank
  • “After fine-tuning on a customer’s task, target accuracy is up but the model got worse at everything else. What happened and what do you do?” Mistral · question bank
Using this week on the last two

Personalised assistant: mid-train on the user’s or company’s data, then on-policy distill from the original model to recover instruction following (IF-eval 79% → 83% vs 85% before, Thinking Machines). Forgetting: the same recipe; note that even SFT on the model’s own samples drifted in their test, while on-policy distillation stays on-policy.

Questions from the curriculum’s question bank and l7_bar.

Week 9 in one minute

  1. DAPO: four fixes, AIME 30 → 50.
  2. Entropy buys reward; the ceiling shows early. Target high-covariance tokens.
  3. pass@k: RL wins at k = 1, the base often at k = 256. Show large-k evidence.
  4. Test-time compute beats a 14× model on easy questions with light serving only.
  5. Scores lie: reward hacking, RLHF’s fundamental limits, leaderboard selection.
  6. Systems: on-policy distillation 9–30× cheaper than SFT; async RL trades one step of staleness for speed.
  7. Critique: observe → hypothesize → cheapest test → mind-changer.

Your Week 9 plan

ResourceListed hPlan hTake away
DAPO Yu et al.21§3–4: four fixes, Table 1
Entropy mechanism Cui et al.21§2–4: the law, covariance, KL-Cov
Does RL incentivize reasoning? Yue et al.1.51§2–4, Tables 2–5
Test-time compute Snell et al.21§1, §3, §7, Figure 1
Reward Hacking Weng21examples, RLHF hacking, Gao et al.
RLHF open problems Casper et al.31§3 taxonomy, §5 disclosure list
The Leaderboard Illusion1.50.75§1–4 and the recommendations
Interconnects: GRPO tweaks, Llama 4, o3 over-optimization, sycophancy, random rewards61.5how an expert critiques a release
On-Policy Distillation Thinking Machines1.50.75reverse-KL reward, cost table
Asynchronous RLHF Noukhovitch et al.21§3, §5, Appendix A.2

Listed hours total 23.5; the plan column totals 10. The guide gives weeks 8–9 together 15–20 hours, so read the named sections and skim the rest.

Self-test: answer out loud, then open

Why does DAPO raise only the upper clip?

With ε = 0.2 a 0.01 token can only reach 0.012 per update, so rare tokens can’t grow. Raising the lower clip too would push unlikely tokens to 0 and shrink the sampling space.

A prompt is solved 90% of the time, G = 16. How often is its group useless?

0.9¹⁶ + 0.1¹⁶ ≈ 18.5% of the time all 16 are right, so every advantage is zero. DAPO resamples until the batch is full of useful groups.

DAPO’s penalty for an 18,432-token answer?

(16,384 − 18,432) ÷ 4,096 = −0.5, added to the correctness reward. Past 20,480 tokens it is −1.

What does Yue et al.’s AIME 2024 solved-set table show?

At k = 1,024 the base solves 23 of 30, the RL model 19, all inside the base set. RL sharpened, it didn’t extend, in that run.

State the entropy law and its ceiling.

R = −a·eH + b; at H = 0, R = b − a. About 73% of entropy and 76% of the gain go in the first 200 of 2,400 steps.

Serving 22× more tokens than you trained on: test-time compute or a 14× model?

The small model gets only 14 + 3 × 13 ÷ 22 ≈ 15.8× its serving compute. Snell et al. find the bigger model wins on medium and hard questions there.

On-policy distillation’s reward, and its cost?

Per token: teacher log-prob minus student log-prob on the student’s own sample. 70% on AIME’24 in ~150 steps, 9–30× fewer FLOPs than SFT to the same score.

Which loss tolerates stale data best in async RLHF?

Online DPO; bigger policies help, bigger reward models don’t. Llama 3.1 8B: same 57.2% win rate, 142 vs 230 minutes.

That’s the track

Nine weeks in one picture

Week 1
the whole pipeline
Weeks 2–3
architectures, dense to MoE
Week 4
scaling arithmetic
Weeks 5–6
RLHF, DPO, GRPO, R1
Week 7
run a GRPO job
Week 8
frontier recipes at scale
Week 9 and the interview: mechanism → number → failure mode → experiment. Weeks 1–4: how a model is built and what it costs. Weeks 5–8: how it is shaped. Week 9: whether to believe the result.

Next: pick one lab, run the critique method on its latest report, and answer both probes out loud in five minutes each.