Frontier-lab prep · LLM internals · Week 6

Direct alignment and
RL for reasoning

How DPO solves the RLHF objective without an RL loop, what GRPO gives up by dropping PPO’s critic, and how DeepSeek-R1 turned checkable answers into long reasoning.

About 9 hours with the readings · use → or Space to step through · O for contents · T for dark mode

What this week is for

Explain DPO, GRPO and the R1 recipe down to the gradient, then defend a choice between them.

How much it matters: about 2% of collected interview reports and none of the 18 staff-plus ones, but core in research and research-engineer loops.

How labs ask it

  • “Preference-based training (reward-model RLHF vs DPO)” Mistral · firsthand 2026
  • “What is the difference between PPO and GRPO?” frontier lab, unnamed · firsthand 2026
Senior-staff probe: the target

“PPO vs GRPO: what does dropping the critic cost you, what is the length bias in GRPO’s loss, and when would you go back to a value model?”

The map for this week

Part 1 learns offline from fixed preference pairs. Parts 2–3 learn online from answers a program can check. Part 4 asks whether the gains are real and which method fits which job.

Part 1

DPO: the RLHF objective, solved without RL

The DPO derivation in five plain steps

1 Same goal as RLHF score = mean reward − β × KL(policy ‖ ref) β sets how far the policy may move from the SFT model Gibbs: the KL term is 0 only at one policy 2 The best policy has a closed form best(y) = ref(y) × exp(reward(y) ÷ β) ÷ Z(x) Z(x) sums over every possible answer: can’t compute it take logs, solve for the reward 3 Run it backwards reward(y) = β·log[best(y) ÷ ref(y)] + β·log Z(x) so any policy defines an “implicit reward” Bradley–Terry: preference = sigmoid of a reward gap 4 Preferences only see differences P(c beats r) = σ(reward(c) − reward(r)) c and r share prompt x, so β·log Z(x) cancels maximum likelihood on the preference data 5 Fit it like a classifier loss = −log σ(β·[logratio(c) − logratio(r)]) binary cross-entropy on pairs: no reward model, no sampling

logratio(y) = log π(y) − log πref(y), summed over the answer’s tokens. Both come from forward passes you already run.

Step 4 is the trick: the uncomputable Z(x) is the same for both answers, so it drops out of the difference.

Worked: β = 0.1, logratio(c) = +2, logratio(r) = −1 → margin 0.3 → σ = 0.574 → loss 0.554.

Go deeper

Appendix A.1 rewrites the objective as −KL(policy ‖ best) + log Z(x); Z doesn’t depend on the policy, so the optimum is policy = best. With K ranked answers, the Plackett–Luce model gives the same cancellation (A.3).

Rafailov et al. (2023), §4 and Appendix A.1–A.3; RLHF book ch. 8.

What one DPO step does to one pair

Implicit reward = β × logratio. Margin = chosen reward − rejected reward.

Each pair’s update is scaled by σ(−margin): near 1 when the pair is ordered wrongly, near 0 once it is right.

Without this weight (“unlikelihood”), the paper’s summaries degenerated into “when when when…”. β also sets how far the log-ratios must move before a pair stops mattering.

Computed live from the DPO loss. Rafailov et al. (2023), §4 gradient, Appendix A.4, C.3 and Table 3; default β = 0.1 (Appendix B).

DPO in practice: evidence and weak spots

Evidence (paper, models up to 6B)

  • Sentiment: best reward-for-KL trade-off in a 22-run sweep, beating PPO even with the true reward.
  • Summaries: ~61% win rate vs PPO’s 57%; humans preferred DPO 58% of the time.
  • Recipe: β 0.1, learning rate 1e-6, batch 64.

Weak spots

  • Offline: never learns from its own new answers.
  • Displacement: chosen and rejected log-probs both tend to fall.
  • Static KL: β fixes the drift budget.
  • Controlled studies: “a hair behind” online RL.

Cheap trick: cache the reference log-probs once; peak memory drops 50%.

Rafailov et al. (2023), §6.1–6.4, Appendix B; RLHF book ch. 8 (“Numerical Concerns”, “Implementation Details”, “DAAs vs. RL”).

Reward-model RLHF or DPO? Decide by data and loop

Reward model + PPODPO
Models heldpolicy, critic, reward model, referencepolicy + reference (or cached log-probs)
Learns from its own new answersyesno, unless you run online DPO
Reusable judgereward model: also for filtering, best-of-Nnone
Cost per experimentrollouts + two trained modelsone forward/backward per pair
A defensible default: start with DPO on pairs that include your own model’s outputs, to iterate on data fast. Move to online RL when the ceiling matters or the reward is checkable. DeepSeekMath saw online sampling pull ahead of offline later in training.

DeepSeekMath §4.1.1 and Fig. 5; RLHF book ch. 8 (on-policy synthetic pairs in Tülu 3; “data is often far more important than algorithms”).

Interview check

Can you answer these in two minutes?

  • “Preference-based training (reward-model RLHF vs DPO)” Mistral · firsthand 2026
  • “Compare reward-model RLHF and Direct Preference Optimization.” Mistral · question bank
  • “Explain the difference between pre-training, supervised fine-tuning and preference optimisation.” Meta MSL · question bank

Be ready to: derive the loss in five steps, say why Z(x) cancels, explain the σ(−margin) weight, and name two failure modes with a fix each.

From the question bank behind the curriculum (2025–2026 reports and prep sites).

Part 2

GRPO: PPO without the critic

One lens for SFT, DPO, PPO and GRPO

DeepSeekMath writes every method the same way: raise each token’s log-probability × a gradient coefficient. Methods differ in where answers come from and how the coefficient is set.

MethodAnswers come fromCoefficient per token
SFThuman-written1
Rejection-sampling FTthe SFT model1 if correct, else 0
Online RFTthe current policy1 if correct, else 0
DPOpairs from the SFT modelσ(−margin), + chosen / − rejected
PPOthe current policyadvantage from a learned critic
GRPOa group from the current policygroup-normalised reward, plus a KL term

On a 1.3B model: online beat offline late in training; GRPO beat online RFT because it also pushes wrong answers down.

Shao et al. (2024), §5.2.1, Table 10, Fig. 5, Appendix A.1.

GRPO step by step, with the papers’ settings

  1. Sample a group of G answers per question.
  2. Score each: reward model, or a checker.
  3. Advantage = (score − group mean) ÷ group std. Every token of the answer gets it.
  4. PPO-style clipped update. KL to the reference goes in the loss, not the reward.
DeepSeekMath-RL 7BR1-Zero
Answers per question6416
Max length1,02432,768 → 65,536
KL coefficient0.040.001
Learning rate1e-63e-6
ResultMATH 46.8 → 51.7%AIME 15.6 → 77.9%

MATH is a competition-math benchmark; AIME is a US invitational math contest whose answers are numbers. R1-Zero refreshed its reference model every 400 steps.

DeepSeekMath §4.1–4.2, Table 5; DeepSeek-R1 §2.1, §2.3 (arXiv v2, 2026).

A group of answers → advantages → per-token push

rewardlength (tokens)adv.push per token (×10⁻⁴)total

Answers and lengths are made up for illustration; tap a reward to flip it. Policy ratio = 1 (first update), KL ignored, population std. Advantage: DeepSeekMath §4.1.2; aggregation options: RLHF book ch. 6.

PPO vs GRPO: what each one has to run

PPO · one answer per prompt update policy Policytrained Reward modelfrozen Referencefrozen reward − β·KLKL added per token Value modeltrained · GAE Advantageper token GRPO · a group of G answers per prompt update policy Policytrained G = 16answers Reward modelor a rule checker (r − mean) ÷ stdone A per answer Clipped loss+ β·KL in the loss Reference · frozen no value model trained frozen or a program

Scroll the diagram sideways.

PPO trains two policy-sized networks; GRPO trains one and pays with G rollouts per prompt instead.

DeepSeekMath §4.1.1 and Fig. 4 (“the value function … is typically another model of comparable size”); DeepSeek-R1 Supplementary A.3.

Memory arithmetic: what dropping the critic saves

Assume a trained model costs 16 bytes per parameter (bf16 weights 2 + gradients 2 + fp32 master copy 4 + two Adam moments 4 + 4) and a frozen one costs 2.

PPO: 2 trained + 2 frozen = 36 bytes/param. GRPO with a reward model: 20. With a rule checker: 18.

Rollout KV cache and activations come on top. R1’s framework also offloads each model from GPU memory between rollout, scoring and training.

Computed from the stated assumption; policy-sized critic per DeepSeekMath §4.1.1. Offloading: DeepSeek-R1 Supplementary B.1. 80 GB per H100.

What dropping the critic costs you, and buys you

You pay

  • G rollouts per prompt for a baseline: 64 in DeepSeekMath, 16 in R1-Zero.
  • No per-token credit: all tokens of an answer share one advantage.
  • Dead groups: all right or all wrong → advantage 0. Many recipes keep prompts solved 20–80% of the time.
  • Std division up-weights near-unanimous groups.

You get

  • One policy-sized model fewer to train (~44% less memory).
  • No GAE λ to tune: PPO at λ = 0.95 was “considerably worse”; at λ = 1.0 it came close.
  • No critic guessing the final reward of half-written chains that later revise themselves.

DeepSeekMath §4.1–4.2; DeepSeek-R1 Supplementary A.3 and Fig. 4; RLHF book ch. 6 (GRPO) and ch. 7 (difficulty filtering).

The length bias in GRPO’s loss, with numbers

GRPO divides each answer’s token losses by its own length, then averages answers. Each answer gets the same total push, spread thinner over long ones.

Answer (made up)RewardTokensAdvantagePush per token, GRPO÷ all tokens
short, right1300+1+8.3 × 10⁻⁴+2.2 × 10⁻⁴
long, right1900+1+2.8 × 10⁻⁴+2.2 × 10⁻⁴
short, wrong0300−1−8.3 × 10⁻⁴−2.2 × 10⁻⁴
rambling, wrong03,000−1−0.83 × 10⁻⁴−2.2 × 10⁻⁴

Push per token = advantage ÷ (4 × length). The 3,000-token wrong answer is pushed down 10× less per token. Long and wrong is cheap.

Fixes (DAPO’s per-token loss, Dr. GRPO) are Week 9.

Not GRPO-only

A PPO analysis (Fatemi et al.) found a similar pull toward longer wrong answers. R1 notes the opposite force in PPO: a per-token KL in the reward taxes length. The RLHF book’s toy example shows 0.25 vs 0.14 per-token gradients for 4- and 7-token answers.

Loss: DeepSeekMath Eq. 3. Aggregation: RLHF book ch. 6. Raschka on Liu et al. (Dr. GRPO) and Fatemi et al.; DeepSeek-R1 Supp. A.3.

When would you go back to a value model?

  • Rewards per step or turn (tool calls, multi-turn agents) need a per-state baseline.
  • Huge rollouts: 16–64 samples per prompt cost more than a critic.
  • Mostly dead groups (all pass or all fail) waste the samples.
  • GAE is tunable: R1 got PPO close at λ = 1.0; Open-Reasoner-Zero scaled with vanilla PPO.

Two-minute answer

Dropping the critic trades a second trained model for G samples per prompt and coarser credit; per-answer averaging adds a length bias. I’d return to a value model for dense or multi-step rewards, or when sampling costs more than a critic.

RLHF book ch. 6 (“MDP vs. Bandit Framing”); DeepSeek-R1 Supp. A.3; Raschka on Open-Reasoner-Zero.

Interview check

Can you answer these in two minutes?

  • “What is the difference between PPO and GRPO?” frontier lab, unnamed · firsthand 2026
  • “RL for Language Models: Policy Gradients, PPO, GRPO and DPO (9 questions / 45–60 min)” Google DeepMind · question bank
  • “PPO vs GRPO: what does dropping the critic cost you, what is the length bias in GRPO’s loss, and when would you go back to a value model?” senior-staff probe

Be ready to: draw both pipelines, do the memory arithmetic, compute a group’s advantages, and show the 10× per-token gap.

Questions from the curriculum’s question bank; the probe is the track’s senior-staff bar.

Part 3

DeepSeek-R1, stage by stage

Numbers are from the revised paper (arXiv v2, January 2026, with the supplementary material).

R1-Zero: a base model, a checker, nothing else

“…The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think>…</think> and <answer>…</answer> tags…”
The whole training template, minus the question.
  • Base: DeepSeek-V3-Base, a mixture-of-experts model with 671B parameters (37B active per token). No SFT first.
  • Reward: accuracy + format, equal weight. No neural reward model, because those “are susceptible to reward hacking”.
  • Run: GRPO, 10,400 steps (1.6 epochs) on 512 H800 GPUs for about 198 hours.

DeepSeek-R1 Table 1, §2.1–2.2, Supplementary A.1 and B.4.4.

Verifiable rewards are small programs

Accuracy: pull out the final answer and compare it with the reference (R1 compared math expressions with sympy when filtering data). Code: run test cases.

Format: reasoning inside <think> tags, answer after.

Language consistency (R1, stage 1): share of reasoning words in the question’s language.

R1’s RL prompts: 26K math, 17K code, 22K STEM, 15K logic. Proofs left out: hard to check.

Responses are made up for illustration; edit the text to test the checker. Rewards: DeepSeek-R1 §2.2 Eq. 4, §3.2.1 Eq. 7, Supp. B.3 and Table 4.

What pure RL did: accuracy up, answers longer

15.6% → 77.9%
AIME 2024 pass@1 (accuracy averaged over samples)
86.7%
with a majority vote over 16 samples
37.8%
average human contestant
04Ktraining steps8.2K10.4K Max answer length 32,768 tokens65,536 Average answer length (shape only, from Fig. 1b) The word “wait” nearly absent occasional spikes

Nobody rewarded length. Reflective words (“wait”, “verify”, “check”…) rose 5–7× over training. The first arXiv version reported 71.0%.

DeepSeek-R1 §2.1, §2.3, Fig. 1, Fig. 10, Supplementary C.2 and Fig. 9.

The “aha moment”, and how much to believe it

“Wait, wait. Wait. That’s an aha moment I can flag here. Let’s reevaluate this step-by-step…”
an intermediate R1-Zero checkpoint, mid-solution

Hard problems gained most: MATH level 5 rose from about 0.55 to 0.90 accuracy. The cost: hard-to-read reasoning mixing English and Chinese.

The skeptic’s reading

  • Qwen2.5 base models show aha-style reflection with no RL.
  • Self-correction already grows during pretraining.
  • R1’s base saw much math and code, and web pages with OpenAI-model answers.

Defensible claim: RL surfaced and amplified reflection; it may not have invented it.

DeepSeek-R1 Table 2, Supplementary A.1 and C.1; Raschka, “State of RL for LLM Reasoning”, papers [10] and [13].

R1’s four stages, built step by step

StageWhat goes inCheckpoint after it
Cold-start SFTthousands of R1-Zero answers kept if correct and readable, rewritten by V3, checked by peopleDev1: AIME 59.0, IF-Eval 71.7
Reasoning RLGRPO on rule rewards + language-consistency rewardDev2: AIME 74.0
Rejection sampling + SFT~600K correct reasoning + ~200K other samples; retrain V3-Base, 2–3 epochsDev3: AIME 78.1, AlpacaEval 62.1
RL on everythingrules + helpfulness and safety reward models; 1,700 steps, preference reward only in the last 400R1: AIME 79.8, AlpacaEval 87.6

DeepSeek-R1 Fig. 2, §3, §3.2, Table 3, Supplementary B.3–B.4. IF-Eval scores format-instruction following; AlpacaEval 2.0 is a judge-rated chat win rate.

What each stage bought, and what it cost

BenchmarkZeroDev1Dev2Dev3R1
AIME 2024 (math contest)77.959.074.078.179.8
CNMO 2024 (Chinese math olympiad)88.158.073.977.378.8
GPQA Diamond (PhD-level science)75.866.170.771.271.5
LiveCodeBench (contest code)50.057.563.564.665.9
IF-Eval (format instructions)46.671.772.078.183.3
AlpacaEval 2.0 (chat, judged)24.750.155.862.187.6

Cold start cost math, RL won it back, and the last RL stage mostly bought chat quality. R1-Zero still beats R1 on CNMO and GPQA: readability isn’t free.

DeepSeek-R1 Table 3 (pass@1 for math, code and GPQA).

Rejection sampling and distillation: the 800K set

DomainSamplesAvg tokens
Math395,2856,094
Code211,1297,436
General (non-reasoning)177,8121,420
STEM + logic20,519—
Total804,7455,355

Kept only correct samples (rules, or V3 as judge); dropped mixed-language, long-paragraph and code-block reasoning.

Distillation: SFT only, no RL

AIME 2024 pass@1%
GPT-4o (May 2024)9.3
R1-Distill-Qwen-1.5B28.9
R1-Distill-Qwen-7B55.5
R1-Distill-Qwen-32B72.6
R1-Distill-Llama-70B70.0

DeepSeek-R1 Table 5 (STEM 10,124 + logic 10,395), Supplementary B.3.3, B.4.3, F and Table 15.

For small models, distil; pure RL needs a big base

Same 32B base (Qwen2.5)AIME 2024MATH-500GPQALiveCodeBench
RL only, over 10K steps (“Zero”)47.091.655.040.2
SFT on R1’s 800K samples72.694.362.157.2
  • Early R1 tries on a 7B dense and a 16B MoE base “failed to yield meaningful improvements”: answers turned repetitive as they grew. 32B, 230B and 671B bases worked.
  • Small-model RL isn’t useless: a 7B math base trained with pure RL scored 22.3% on AIME 2024, vs 7.9% for its instruct version.
  • RL on top of a distilled 1.5B model helped for about $42, but small-benchmark gains can be seed noise.

DeepSeek-R1 Tables 16–17, Supplementary F.1 and G.1; Raschka, papers [8] and [15].

What R1 tried and dropped

Process reward model

Scores each step. Steps are hard to define and judge; human labels don’t scale; it invites reward hacking. Even PRM800K has ~20% wrong labels.

Monte Carlo tree search

Token search space explodes; capped expansion hits local optima; the value model guiding search is hard to train.

Long preference RL

Helpfulness reward kept rising while Codeforces pass@1 fell. So preference rewards ran only in the last 400 steps.

Pattern: learned judges get exploited. R1 kept judges it could trust: rules, or a model comparing against a known answer.

DeepSeek-R1 Supplementary G.2, B.5 and Fig. 6, §3.2.2; DeepSeekMath §5.2.3 (PRM800K).

Cost and test-time compute, in numbers

Post-trainingH800 GPU-hours
R1-Zero101K
SFT data creation5K
R141K
Total (≈ $294K at $2/hour)147K

Check: 512 GPUs × 198 hours = 101K. Pretraining V3-Base is not included.

Thinks longer when it’s harder: on 366 problems from 2024 contests R1 solved 61.8% with 8,793 thinking tokens on average (under 7,000 easy, over 18,000 hardest). GPT-4o: 24.7% with 711.

Sampling still helps: AIME 2024 pass@1 79.8% → majority vote 86.7% → any of 64 correct 90.0%.

DeepSeek-R1 Table 7, Supplementary B.4.4 and E.4.

Interview check

Can you answer these in two minutes?

  • “RL training debugging scenarios: what do you do when you hit entropy collapse, the model reasoning in another language, or terrible MFU?” frontier lab, unnamed · relayed 2025
  • “Design post-training for a new base model targeting coding and math. Where do you use SFT, DPO, RLVR and distillation, and how do you know each helped?” senior-staff probe

Hooks: language mixing → R1’s consistency reward (small coding cost); what each stage bought; why no PRM or MCTS. Entropy collapse and MFU: Weeks 7 and 9.

Questions from the curriculum’s question bank; the probe is the track’s senior-staff bar.

Part 4

Is the gain real? And which method when?

An outline for the next senior-staff probe; Week 9 goes deep.

“+15 points on AIME”: five checks

Probe: “Your RL run improved pass@1 on AIME by 15 points. Convince me it learned something new and isn’t reward hacking or sharpening the base model.”

  1. Noise: 15 problems per AIME exam, so one problem = 6.7 points. Average 64 samples (R1’s setting) and several seeds.
  2. Contamination: 10-gram filtering; test on a later contest (R1: 75% on AIME 2025).
  3. Sharpening: compare pass@k at large k. DeepSeekMath’s RL raised majority vote, not pass@K.
  4. Hacking: reward vs a held-out metric (R1: reward up, Codeforces down).
  5. Transfer: off-domain gains (CMATH 84.6 → 88.8 from GSM8K/MATH prompts only).

DeepSeek-R1 Table 13, Supp. D.1, E.2, Fig. 6; DeepSeekMath §5.2.2 Fig. 7, Table 5. pass@k vs base models (Yue et al.): Week 9.

Which method for which job

JobMethodWhy
Format, tone, cold startSFTimitate chosen answers; cheap
Fuzzy preferences, fastDPO on on-policy pairsno RL infrastructure
Preferences at the ceilingreward model + online RLlearns from its own answers; cap steps
Math, code, logicRL with verifiable rewardschecker hard to fool; needs a strong base
Small deployable modeldistil a strong reasoner72.6 vs 47.0 on AIME at 32B

Pretraining predicts raw text; SFT imitates chosen answers; preference optimisation learns from comparisons or rewards on the model’s own outputs. R1: SFT where rewards were unreliable, RL where checkable.

DeepSeekMath Appendix A.1; RLHF book ch. 8; DeepSeek-R1 Supp. G.1 (“Iterative pipeline”), Table 16.

Interview check

Can you answer these in two minutes?

  • “Explain the difference between pre-training, supervised fine-tuning and preference optimisation.” Meta MSL · question bank
  • “Your RL run improved pass@1 on AIME by 15 points. Convince me it learned something new and isn’t reward hacking or sharpening the base model.” senior-staff probe

Answer shape for the probe: noise floor first (6.7 points per problem), then contamination, then pass@k, then reward-vs-held-out curves.

Question bank behind the curriculum; the probe is the track’s senior-staff bar.

The week in one minute

  1. DPO solves the RLHF objective in closed form; Z(x) cancels, leaving a pairwise classifier.
  2. Each pair is weighted by σ(−margin): wrong pairs count most; β sets the drift budget.
  3. GRPO swaps PPO’s critic for a group baseline: ~44% less memory, G rollouts per prompt.
  4. Per-answer averaging: a 10× longer wrong answer gets a 10× weaker penalty per token.
  5. R1-Zero: rule rewards took AIME from 15.6% to 77.9%; length and reflection grew unrewarded.
  6. R1: cold start → RL → 800K SFT → RL; small models do better distilled.
  7. Before believing a gain: noise, contamination, pass@k, reward vs held-out.

Your Week 6 plan

#ResourceHoursTake away
1DPO paper §1–6, App. A–B2derivation, gradient weight, β
2DeepSeekMath §4, §5.2, App. A.12GRPO, unified paradigm, Maj@K vs Pass@K
3DeepSeek-R1 main + Supp. A–C, E–G3R1-Zero, stages, distillation
4State of RL for LLM Reasoning Raschka · also Understanding Reasoning LLMs2PPO → GRPO, length bias
5RLHF book: ch. 8, ch. 7, ch. 6 GRPO + loss aggregation Lambert2DPO weak spots, RLVR practice

11 hours listed against a 9-hour budget (book hours are an estimate for these chapters). To fit: skip DeepSeekMath §2–3 and R1’s safety report (Supp. D.3); skim Raschka’s paper list.

Self-test: answer out loud, then open

Why does Z(x) vanish from the DPO loss?

Bradley–Terry only uses the reward gap; both answers share the prompt, so the β·log Z(x) terms cancel.

DPO weight for a pair at margin −2 vs +2?

σ(2) = 0.88 vs σ(−2) = 0.12: the wrongly ordered pair gets about 7.4× the update.

With β = 0.1, what log-ratio gap drops a pair’s weight to 0.1?

ln 9 ÷ 0.1 ≈ 22 nats. With β = 0.5 it is 4.4: higher β, less drift needed.

GRPO group rewards 1, 0, 0, 0: the advantages?

Mean 0.25, std 0.433 → +1.73 for the right answer, −0.58 for each wrong one.

Training memory, 7B: PPO vs GRPO (16 B trained, 2 B frozen)?

PPO 36 × 7B = 252 GB; GRPO with a reward model 20 × 7B = 140 GB: 44% less.

Per-token penalty, 3,000- vs 300-token wrong answers, GRPO loss?

10× smaller for the long one; each answer gets the same total push.

What did R1’s cold start cost and buy?

AIME 77.9 → 59.0, but IF-Eval 46.6 → 71.7; the next RL stage brought AIME back to 74.0.

Why distil a 32B model instead of running RL on it?

Distilled 72.6% vs RL-only 47.0% on AIME 2024, at a fraction of the compute.

Next

Week 7: run a small GRPO job yourself

Reward functions for math, what to log, healthy and broken training curves, and debugging, with TRL’s GRPO trainer, CS336 assignment 5 and nanochat. Plus the DeepMind fundamentals and RL quizzes as practice.

Question to aim for

“RL training debugging scenarios: what do you do when you hit entropy collapse, the model reasoning in another language, or terrible MFU?”

Week 7 →  ·  ← Week 5: RLHF foundations