Frontier-lab prep · LLM internals · Week 7

Run GRPO yourself,
then defend every curve

A small reinforcement-learning run on grade-school math: the reward, the loss written from scratch, what to log, what broken looks like, and how to argue the result is real.

About 9 hours with the readings, plus GPU time · use → or Space to step through · O for contents · T for dark mode

What this week is for

Run a small GRPO job, break it, and write its loss from memory.

Weight: ~2% of interview reports, none of 18 staff-plus ones; core in research-engineer loops.

How labs ask it

  • “Research Engineer tech screen focused on coding RL algorithms (per the recruiter's prep link)” Anthropic · secondhand 2026
  • “RL training debugging scenarios: what do you do when you hit entropy collapse, the model reasoning in another language, or terrible MFU?” frontier lab, unnamed · secondhand 2025
Senior-staff target: “Your RL run improved pass@1 on AIME by 15 points. Convince me it learned something new and isn’t reward hacking or sharpening the base model.”

The map for this week

Parts 1–4 follow one run: a 1B model learning grade-school math with a 0/1 reward. Part 5 is oral-quiz practice.

Part 1

Set up a run you can afford

Three small runs you can copy

CS336 Assignment 5TRL quick startnanochat RL
ModelOLMo-2-0425-1B (base)Qwen2.5-0.5B-Instructits own ~560M chat model (d20, Oct 2025)
DataGSM8KDeepMath-103KGSM8K
Answers per prompt (G)8816
Prompts per step328 on 8 GPUs (defaults)16
Max answer tokens512512256
Hardware, time2 GPUs; “2 B200 hrs” for 4 seeds8 GPUs, ~1 day8×H100, ~1.5 h

Pick CS336: every hyperparameter is written down, it has unit tests, and a 1B base model actually learns (target: ≥25% validation accuracy, averaged over seeds).

Details

GSM8K: grade-school word problems with a single numeric answer after “####”; 7,473 train and 1,319 test problems (Hugging Face openai/gsm8k). nanochat’s RL loop is a simplified GRPO: no KL, no clipping, advantage = reward − group mean. TRL’s default 8 prompts/step comes from per_device_train_batch_size=8 completions × 8 GPUs ÷ 8 per prompt.

CS336 A5 handout §2.2, §4.3; TRL GRPO Trainer docs (quick start, GRPOConfig); nanochat scripts/chat_rl.py and Discussion #1.

One step of the run, on two GPUs

32 promptsGSM8K questions vLLM GPU: sample8 each = 256 answers Checkerreward 0 or 1 each Group advantages(r − mean) ÷ std, per 8 Train GPU: 1 step32 micro-batches, AdamW Sync weightstrain GPU → vLLM

Generation and training live on different GPUs. vLLM, a fast inference engine, samples; the Hugging Face model trains.

One optimizer step per batch of samples makes it on-policy: every sample came from the current weights, so no ratio clipping is needed yet.

Repeat 200 times: 6,400 training prompts. Evaluate on 1,024 held-out problems every 10 steps.

CS336 A5 handout §4.2.2, §4.3 (suggested hyperparameters).

An honest compute estimate

CS336, 1B on GSM8K

2 B200 GPUs per job. Main run labelled “2 B200 hrs” for 4 seeds × 200 steps. All five experiment problems: 26 B200 hrs.

nanochat, ~560M

RL step about 1.5 h on 8×H100: ~$36 at $24/hour. GSM8K went 4.6% → 7.6%. Needs its own SFT checkpoint first.

Bigger references

TRL quick start: 0.5B, 8 GPUs, ~1 day. Dr. GRPO: 7B, 8×A100, 27 hours.

Plan for: 1B model, 2 GPUs with 80 GB or more, 2–4 B200 GPU-hours for four 200-step seeds, plus a sweep. On H100s budget more; no source gives an H100 figure for this setup.
Why “2–4”

The handout heading says “2 B200 hrs”, and the course’s Modal config requests “B200:2”. It doesn’t say whether that is GPU-hours or hours on the 2-GPU job, so the range covers both readings. The 26 = 2 (main) + 4 (learning-rate sweep) + 4 (prompt ablation) + 8 (on-policy variants) + 8 (off-policy variants).

CS336 A5 handout problem headings; cs336_alignment/modal_utils.py; nanochat README and Discussion #1 report card; TRL docs quick start; Liu et al. 2025 (Dr. GRPO) §1.

The reward: a checker, not a model

# TRL calls this with every completion, plus dataset columns as kwargs
import re
ANSWER = re.compile(r"</think> <answer>(.*?)</answer>", re.S)

def norm(s):
    s = s.strip().replace(",", "").replace("$", "").rstrip(".")
    try:
        return str(float(s))      # "72", "72.0", "72." all match
    except ValueError:
        return s

def gsm8k_reward(completions, answer, **kwargs):
    out = []
    for text, sol in zip(completions, answer):
        gold = sol.split("####")[-1]  # GSM8K: "... #### 72"
        m = ANSWER.search(text)
        out.append(1.0 if m and norm(m.group(1)) == norm(gold) else 0.0)
    return out

The prompt ends with Assistant: <think>, so the model reasons, then answers in tags.

Strict on format, like CS336’s grader: no tags, no reward.

No partial credit for format alone: a format bonus is the easiest reward to hack. Unit-tested on the next slide’s cases.

Prompt: CS336 A5 §3.2 (the DeepSeek R1-Zero template). Reward signature: TRL docs, “Using a custom reward function”. Format rule: CS336 drgrpo_grader.r1_zero_reward_fn.

Score model outputs, live

The function from the previous slide, ported to JavaScript. Question: “Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May…” (gold 72, CS336 A5 §2.2). CS336 notes come from reading drgrpo_grader.py.

Reward design: what each choice costs

ChoiceWhoUpsideFailure mode
Accuracy only, 0/1CS336can’t be farmedreward is ~0 at the start; slow first steps
Accuracy + format, equal weightDeepSeek-R1teaches the tags fastformat alone earns half the reward
+1 / −1DAPOwrong answers pushed downsame group signal after normalising
+ language consistencyDeepSeek-R1readable reasoning“slight degradation” in performance
Learned reward modelDeepSeekMathany taskreward hacking; R1 avoided it for reasoning

Language consistency reward = share of words in the target language inside the reasoning.

CS336 A5 §3.3 (“we will not give partial credit”); Guo et al. 2025 (DeepSeek-R1) §2.2, §3.2.1 and Supp. B.5; Yu et al. 2025 (DAPO) §2.3; Shao et al. 2024 (DeepSeekMath) §4.

The same run in TRL

from trl import GRPOConfig, GRPOTrainer

args = GRPOConfig(
    output_dir="olmo1b-gsm8k-grpo",
    num_generations=8,              # G, answers per prompt
    per_device_train_batch_size=8,  # completions per micro-batch
    gradient_accumulation_steps=32, # 8 × 32 = 256 = 32 prompts × 8
    learning_rate=1e-5,             # CS336 value; TRL default 1e-6
    lr_scheduler_type="constant",
    adam_beta2=0.95, weight_decay=0.0, max_grad_norm=1.0,
    max_completion_length=512, temperature=1.0,
    beta=0.0,                       # no KL term (TRL default)
    loss_type="grpo", scale_rewards="group",  # CS336 "standard"
    max_steps=200, use_vllm=True, log_completions=True,
)
trainer = GRPOTrainer(model="allenai/OLMo-2-0425-1B", args=args,
                      reward_funcs=gsm8k_reward, train_dataset=ds)
trainer.train()

One generation batch per optimizer step keeps it on-policy (num_iterations=1, the default).

TRL’s default loss_type is "dapo", not "grpo". Set it on purpose.

ds needs a prompt column (the filled-in template) and an answer column.

Stop string and versions

CS336 stops vLLM at “</answer>”. In TRL, generation_kwargs goes to vLLM’s SamplingParams, so {"stop": ["</answer>"], "include_stop_str_in_output": True} should do the same; check it on your installed versions. Defaults checked against the TRL docs page and trl/trainer/grpo_config.py on GitHub main (16 Nov 2026; latest release v1.18.0).

TRL GRPOConfig reference and source; CS336 A5 §4.3.

Learning rate, KL, group size: what each source used

RunModelGLRKL βClip εMax tokens
DeepSeekMath7B instruct641e-60.041 update per sample1,024
R1-ZeroV3 base163e-60.001not stated32,768→65,536
DAPOQwen2.5-32B base161e-600.2 low, 0.28 high20,480
Dr. GRPOQwen2.5 1.5B, Math-7B81e-600.23,000
CS336 A5OLMo-2 1B base81e-5noneon-policy, none512
TRL defaults—81e-60.00.2512

Scroll the table sideways.

KL: with a rule-based checker, DAPO, Dr. GRPO and TRL’s default drop it; R1-Zero kept 0.001. Keep β > 0 when the reward is a learned model.
Learning rate: start at the closest source, then sweep one value below and one above, as CS336 asks.

R1’s later first RL stage used ε = 10. Shao et al. 2024 §4.2; Guo et al. 2025 §2.1, §3.2.1; Yu et al. 2025 §4.1; Liu et al. 2025 App. G; CS336 A5 §4.3; TRL GRPOConfig.

Interview check

Setup questions you should answer with numbers

  • “RL for Language Models: Policy Gradients, PPO, GRPO and DPO (9 questions / 45–60 min)” Google DeepMind · question bank 2026
  • Why does GRPO particularly suit verifiable rewards, and what happens to a group whose rewards are all equal? community practice (r87, unofficial)
  • “Design post-training for a new base model targeting coding and math. Where do you use SFT, DPO, RLVR and distillation, and how do you know each helped?” curriculum senior-staff probe
Answer sketch

A first RLVR experiment (RL with verifiable rewards): 1B base model, GSM8K, 0/1 checker, G = 8, 32 prompts × 200 steps, about 2–4 B200 GPU-hours for four seeds. GRPO needs the reward many times per prompt, which is cheap for a checker. An all-equal group has zero advantage, so it gives no gradient. KL: 0.04 in DeepSeekMath with a learned reward model, 0.001 in R1-Zero, 0 in DAPO and Dr. GRPO.

Only the first item is an interview report (llm-q47). The second is community practice; the third is from the curriculum’s senior-staff bar (Week 8 goes deeper).

Part 2

Write GRPO’s loss from scratch

The coding-screen version: about 15 lines of PyTorch, checked against the papers and TRL.

From “push up good answers” to GRPO, one change at a time

  1. REINFORCE: raise each answer’s log-probability in proportion to its reward: right 1, wrong 0.
  2. Subtract the group mean: with 4 of 8 right, weights become +0.5 and −0.5. Same direction in expectation.
  3. Divide by the group’s std: a stability trick that changes the objective.
  4. Average over each answer’s tokens: long answers count less per token.
  5. Clip the ratio new ÷ old probability to [0.8, 1.2] when samples are reused.

Dr. GRPO removes steps 3 and 4.

Why the baseline is free

The expected gradient of log-probability under the model’s own samples is zero: the probabilities always sum to 1, so their gradients sum to 0. Any baseline that doesn’t depend on the sampled answer therefore leaves the expected gradient unchanged and only changes its variance. The group mean does depend on the answers, which rescales the expectation by (G − 1)/G.

CS336 A5 §4.1.4–4.1.8 (derivation, eqs. 9–28); Liu et al. 2025 §3.1.

Group advantages: click answers right or wrong

8 answers to one prompt. Tap to flip.

Advantage = (reward − group mean), then optionally divided by something.

Try 1 of 8 right: with the std, the rare correct answer gets +2.47; without it, +0.88.

Try all right or all wrong: every advantage is 0. That prompt teaches nothing this step.

Dividing by the std gives hard and easy prompts bigger updates. Dr. GRPO calls this a difficulty bias.

Std is the sample std (divide by G − 1) plus 1e-4, as in TRL and torch.std. Normalisers from CS336 A5 §5.1–5.3 (MaxRL divides by the group mean); Liu et al. 2025 §3.1.

The clip: stop pushing once you’ve pushed far enough

ρ = new probability ÷ old probability of a token. ε = 0.2.

Good answer, ρ = 1.5: value capped at 1.2 × 2 = 2.4, gradient 0.

Bad answer, ρ = 0.5: value stuck at 0.8 × −2 = −1.6, gradient 0.

Moves the wrong way are never clipped: ρ = 1.5 with A = −2 still gives −3.0 and full gradient.

A = +2 (good answer) 0.81.202 objective 2.4 1.0 A = −2 (bad answer) 0 0.81.202 −1.6 −3.0
Objective = min(ρ·A, clip(ρ, 0.8, 1.2)·A) for one token, ratio ρ on the x-axis. Orange points: zero gradient. Blue: full gradient.

Schulman et al. 2017 (PPO) eq. 7; the four worked values are r87’s RL quiz Q2, recomputed in a unit test.

How you average the tokens changes what the model learns

One right answer (A = +1) and one wrong answer (A = −1) in the batch.

Each token’s push = its advantage × an averaging weight.

Per-answer mean (original GRPO): a long wrong answer is penalised less per token, so wrong answers drift longer.

Per-token mean (DAPO, TRL default) or a constant (Dr. GRPO): every token weighs the same.

Liu et al. 2025 §3.1 (response-level length bias); Yu et al. 2025 §3.3 (token-level loss); TRL loss_type docs. Constant = 2 answers × 512 max tokens.

GRPO’s loss in 15 lines

def grpo_loss(logp, old_logp, rewards, mask, G, eps=0.2, ref_logp=None,
              beta=0.0, scale="std", agg="token", L_max=None):
    # logp, old_logp, ref_logp, mask: [B*G, T]   rewards: [B*G]
    r = rewards.view(-1, G)
    adv = r - r.mean(dim=1, keepdim=True)          # group baseline
    if scale == "std":                              # Dr. GRPO: "none"
        adv = adv / (r.std(dim=1, keepdim=True) + 1e-4)
    adv = adv.reshape(-1, 1)                        # same A on every token
    ratio = torch.exp(logp - old_logp)              # 1.0 on-policy
    clipped = torch.clamp(ratio, 1 - eps, 1 + eps)
    loss = -torch.min(ratio * adv, clipped * adv)
    if beta > 0:                                    # k3 KL(pi || ref)
        d = ref_logp - logp
        loss = loss + beta * (torch.exp(d) - d - 1)
    if agg == "seq":     # DeepSeekMath, CS336 standard
        return ((loss * mask).sum(1) / mask.sum(1)).mean()
    if agg == "token":   # DAPO, TRL default
        return (loss * mask).sum() / mask.sum()
    return (loss * mask).sum() / (mask.shape[0] * L_max)  # Dr. GRPO

1. Advantage per answer, copied to its tokens.

2. PPO clip per token. On-policy, pass logp.detach() as old: ratio 1, gradient still flows.

3. Optional KL to the frozen reference: the always-positive “k3” estimate.

4. Aggregation: the length-bias choice from the last slide.

Matches TRL’s _compute_loss for loss_type grpo, dapo and dr_grpo (checked numerically). Shao et al. 2024 eqs. 3–4; Schulman, “Approximating KL Divergence” (2020).

The tests an interviewer wants you to write

  • On-policy gradient = REINFORCE with baseline. Compare with autograd on a hand-built objective.
  • All-equal group → zero gradient on those rows.
  • Padding gets no gradient. Multiply by the mask, then divide by its count.
  • Clip: ρ = 1.5, A = +2 → zero gradient; A = −2 → gradient flows.
  • Micro-batches: accumulated gradient equals the full-batch gradient.

“My loss is 0.0000 every step”

Not a bug. On-policy with per-answer averaging, the advantages in each group sum to zero, so the loss value is exactly 0. Its gradient isn’t.

CS336: “This expression is not a loss in the traditional sense.” Watch reward, entropy and gradient norm instead.

Tests 1–4 pass in a local PyTorch run of the previous slide’s code. Test 5: CS336 A5 §4.2.4 (rescale each micro-batch loss by its share of the batch).

Interview check

Can you write and defend this in 30 minutes?

  • “Research Engineer tech screen focused on coding RL algorithms (per the recruiter's prep link)” Anthropic · secondhand 2026
  • Implement GRPO’s per-token loss with clipping and a KL term; then make it Dr. GRPO. practice item (this deck)
  • For ε = 0.2, give the clipped objective at (ρ, A) = (1.5, 2), (0.5, 2), (1.5, −2), (0.5, −2), and say which have zero gradient. community practice (r87, unofficial)
  • Name the two biases in GRPO’s loss and the one-line fix for each. community practice (r87, unofficial)
Answers

2.4 (zero gradient), 1.0, −3.0, −1.6 (zero gradient). Biases: per-answer length averaging (fix: token-level or constant normaliser) and per-group std scaling (fix: don’t divide by std, or use the batch std).

The Anthropic item is the only real report for this group (llm-q13); the recruiter’s prep link pointed at coding RL algorithms.

Part 3

Log it, read it, debug it

What to log, and the name TRL gives it

MetricTRL nameWhy
Train reward, per reward functionreward, rewards/{name}/meanis it learning at all
Format reward, logged separatelylog_metric() inside the reward fnformat vs real accuracy
Held-out accuracy, pass@kyour eval looptrain reward can overfit
Token entropyentropycollapse = no exploration
Length, and share truncatedcompletions/mean_length, clipped_ratiolength bias, truncation
Groups with no signalfrac_reward_zero_stddead prompts
Clip fraction, KL, gradient normclip_ratio/*, kl; grad norm: CS336 requires itstaleness, step size
Sample outputslog_completions=Trueread them every 40 steps

TRL docs, “Logged metrics” and reward-function Example 6 (log_metric); CS336 A5 §4.2.4, §4.3 (required metrics, rollouts every 40 steps); nanochat chat_rl.py (pass@1…8).

Healthy vs broken: what the curves look like

main line second line reference. Shapes made up for illustration.

Shapes are made up for illustration; each pattern is described in a source: DAPO §3.1, §3.4, §4.3; Dr. GRPO §3.2; DeepSeek-R1 Supp. B.5, B.6; CS336 A5 §4.3.

Easy prompts go silent: how many groups give no gradient?

groups with no signal
useful prompts of 32

A group is silent when all G answers agree: chance = pG + (1 − p)G.

As training succeeds, p rises and the batch quietly shrinks. Noisier gradients, wasted rollouts.

Fixes: bigger G; filter out solved prompts; DAPO’s dynamic sampling keeps sampling until the batch is full of mixed groups.

Computed in your browser. Yu et al. 2025 (DAPO) §3.2; TRL frac_reward_zero_std; CS336 A5 §5.4 (skip zero-advantage sequences).

Debugging checklist

SymptomFirst checkFix the sources used
Entropy collapse: identical answers, reward stallsentropy, clip_ratio/highraise upper clip to 0.28 (DAPO); entropy bonus or top-20% entropy tokens (TRL)
Length explosionlength of right vs wrong answers; clipped_ratiotoken-level or constant loss; mask truncated answers; soft length penalty (DAPO)
Format hacking: reward up, accuracy flatformat and answer reward apartno format bonus; read 20 “rewarded” outputs
Language mixinglanguage of the reasoninglanguage-consistency reward (R1); SFT cold start
Zero-variance groupsfrac_reward_zero_stddynamic sampling, harder prompts, bigger G
Reward never movesbase model’s pass@8 on train promptsbetter prompt; easier data. If pass@8 is 0, RL has nothing to reinforce

DAPO §3.1–3.4; Dr. GRPO §2.1, §3.1; DeepSeek-R1 §3.2.1, Supp. B.6; TRL entropy_coef, top_entropy_quantile, mask_truncated_completions; CS336 A5 §3.4, §4.3.

Interview check

“Entropy collapse, another language, terrible MFU: what do you do?”

Entropy collapse

Confirm with entropy and identical samples per group. Raise εhigh to 0.28; DAPO’s naive GRPO baseline reached 30 on AIME 2024, clip-higher was one step to 50.

Other language

The checker can’t see language. Add R1’s consistency reward (share of target-language words); expect a slight score cost.

Terrible MFU

MFU = model FLOPs utilisation. Generation, not training, is usually the bottleneck, dominated by the longest answers. Use vLLM, cap length, overlap generation with training.

Staff-level move: say which plot you’d open first and what number would change your mind.

Question: frontier lab, unnamed · secondhand 2025 (llm-q14). DAPO Table 1 and §3.2; DeepSeek-R1 §3.2.1; TRL docs (“Generation is often the main bottleneck”).

Part 4

Defend the result

“Your RL run improved pass@1 on AIME by 15 points. Convince me it learned something new and isn’t reward hacking or sharpening the base model.”

pass@k: sharpening or a new skill?

pass@k = share of problems solved by at least one of k samples.

Sharpening: RL wins at k = 1, base catches up or wins at large k. Yue et al. found this across models and algorithms.

New skill: RL solves problems the base never solves, even in 256 tries.

The estimator

Sample n answers per problem, count c correct; pass@k = 1 − C(n−c, k) / C(n, k), averaged over problems (Chen et al. 2021, used by Yue et al.). The widget uses n = 256 and 40 made-up problems.

Made-up problems, real estimator computed in your browser. Yue et al. 2025 abstract, §2, App. A.2; Shao et al. 2024 §5.2.2 (RL raised Maj@K but not Pass@K).

15 points on AIME is how many problems?

one problem is worth
95% range, one run

AIME 2024 has 30 problems. 15 points is 4.5 problems.

Two noise sources: which samples you drew, and which problems happen to be in the set.

Do: average 32 samples per problem (DAPO’s avg@32), compare base and RL on the same problems, and run 4 seeds with a 1.96 σ/√n band (CS336).

Interval = ±1.96 × √(p(1 − p) / N), treating each problem as one right/wrong outcome; computed live. Sizes: HuggingFaceH4/aime_2024 (30), openai/gsm8k test (1,319), CS336 val (1,024).

The checks to run on your own small run

  1. pass@k, base vs RL, k = 1 to 256, on held-out problems. Crossing curves mean sharpening.
  2. Held-out sets: the GSM8K test split, and a harder set never used as an RL prompt.
  3. Format vs accuracy: count “formatted but wrong” and “right but unparsed”. Compare with the base model’s best prompt, including 3-shot.
  4. Length: right and wrong answers plotted apart; truncation rate.
  5. Entropy: a collapse means less exploration; read it next to pass@256.
  6. Read the reasoning: hand-check reward-1 outputs for lucky guesses and checker bugs.
Why each matters

Dr. GRPO showed templates can destroy a base model’s ability before RL rebuilds it, so part of a gain can be format repair; they also showed GRPO grows wrong answers’ length. Yue et al. hand-checked correct chains of thought: on the hardest GSM8K problems, 24 of 25 base-model solves and 23 of 25 RL solves had at least one valid chain. DeepSeek-R1 saw reward rise while CodeForces pass@1 fell under a learned helpfulness reward.

Yue et al. 2025 §3.1; Liu et al. 2025 §3.2–3.3; Guo et al. 2025 Supp. B.5; CS336 A5 §3.4 (format/answer categories, 3-shot prompt).

Interview check · senior staff

Your two-minute answer to the probe

  1. Concede the base rate: published evidence says RL with checkable rewards mostly sharpens. A pass@1 gain is still worth having.
  2. Show the noise bar: 15 points = 4.5 of 30 problems; avg@32, 4 seeds, paired by problem.
  3. Show pass@k to 256 on AIME and an unseen set. New skill only if RL stays above at large k.
  4. Rule out hacking: format and answer reward apart, hand-read rewarded outputs, check right-vs-wrong length.
  5. Name the problems that went from 0 of 256 to solved, and show their reasoning.

l7_bar probe 6 of the curriculum. Evidence: Yue et al. 2025; Shao et al. 2024 §5.2.2; Liu et al. 2025; Guo et al. 2025 Supp. B.5.

Part 5

Quiz practice: the oral rounds

Questions from an unofficial community repo of Google DeepMind interview notes. Its own README says the problems are reconstructed from second-hand accounts and written from scratch. Every answer below was re-checked against a primary source or recomputed.

RL for LLMs: answer aloud in 2 minutes, then open

Why doesn’t subtracting a baseline bias the policy gradient?

The expected gradient of log π under π is ∇ of the total probability, ∇1 = 0. So b × 0 = 0 for any b that doesn’t depend on the answer. It only changes variance.

Checked: CS336 A5 eqs. 15–17.

GRPO advantages for rewards (1, 0, 1, 0) and (1, 1, 1, 1)?

Mean 0.5, sample std 0.577 → ±0.866. All-equal → all 0. With the population std (divide by G) it would be ±1.0; TRL and torch.std use G − 1.

Recomputed; TRL nanstd uses Bessel’s correction.

Why use k3 = (u − 1) − log u to estimate KL, with u = πref/π?

Unbiased like −log u, but never negative and far lower variance near u = 1: at u = 1.01, −log u = −0.00995 while k3 = 0.0000497.

Checked: Schulman 2020 blog; DeepSeekMath eq. 4 (“guaranteed to be positive”).

In DPO, why does the partition function Z(x) disappear?

Reward = β log(π*/πref) + β log Z(x). Bradley–Terry uses only the reward difference of two answers to the same x, so β log Z(x) cancels.

Checked: Rafailov et al. 2023 eqs. 4–7.

Symptoms of over-optimising a learned reward model?

Proxy reward keeps rising while a held-out measure falls; answers lengthen. DeepSeek-R1 shows reward up while CodeForces pass@1 falls.

Checked: DeepSeek-R1 Supp. B.5, Fig. 6.

Two ways rollouts become off-policy without you choosing it?

Several optimizer steps per batch (CS336 tries 32), and the inference engine computing slightly different probabilities than the trainer. TRL logs that gap.

Checked: CS336 A5 §6; TRL “training-inference mismatch”.

Questions adapted from r87’s RL-for-LLMs quiz (Q1, Q4, Q6, Q7, Q8, Q9). Unofficial; answers verified as noted.

LLM fundamentals: the arithmetic rounds

Variance of q·k with dh unit-variance independent entries, and why scale?

Variance = dh. Large scores saturate softmax, gradients vanish; dividing by √dh brings variance back to 1.

Checked: Vaswani et al. 2017 §3.2.1, footnote 4.

KV cache: 32 layers, 8 KV heads, head size 128, 32,768 tokens, bf16?

2 × 32 × 8 × 128 × 32,768 × 2 bytes = 232 bytes = 4 GiB per sequence.

Recomputed.

Parameters per block, and FLOPs per token for training vs inference?

12 × d² per block (4d² attention, 8d² MLP). About 6N per training token, 2N per generated token.

Checked: Kaplan et al. 2020 §2.1, Table 1.

Probabilities 0.5, 0.2, 0.8, 0.25 on four true tokens: cross-entropy and perplexity?

Product = 0.02 = 1/50. Perplexity = 501/4 ≈ 2.66; cross-entropy = ln 50 ÷ 4 ≈ 0.978 nats.

Recomputed.

Top-p = 0.7 on {a 0.5, b 0.2, c 0.15, d 0.1, e 0.05}?

Keep the smallest top set reaching 0.7: {a, b}.

Recomputed.

8 experts of 2B, 0.5B shared, top-2 routing: total and active?

Total 0.5 + 8 × 2 = 16.5B. Active 0.5 + 2 × 2 = 4.5B per token.

Recomputed.

Adapted from r87’s LLM-fundamentals quiz (Q1, Q3, Q5, Q7, Q11, Q12). Unofficial; answers verified as noted.

Interview check

The rounds these quizzes prepare you for

  • “LLM Fundamentals: Attention, Transformers and the Training Pipeline (12 questions / 45–60 min)” Google DeepMind · question bank 2026
  • “Deep discussion of NLP and transformer internals.” Google DeepMind · firsthand 2026
  • “Walk me through what happens, tensor by tensor, in one forward pass of a decoder-only transformer.” Anthropic · question bank 2026
  • “Why does attention divide by sqrt(d_k)?” frontier lab, unnamed · question bank 2026
  • “ML coding testing familiarity with the Transformer architecture and its basic concepts.” OpenAI · firsthand 2025
Where the community answers need care

The RL quiz calls KL(πθ ‖ πref) the “forward” KL; TRL’s docs call the same direction’s gradient the “reverse-KL gradient”. Name the direction explicitly instead of using either label. CS336 writes the group std with 1/G but tells you to use torch.std, which divides by G − 1; say which you use.

Question groups llm-q46, q12, q24, q15, q10. Week 1 covers the forward pass and √d in depth.

The week in one minute

  1. Setup: 1B base model, GSM8K, G = 8, 32 prompts × 200 steps, 2 GPUs, 2–4 B200 GPU-hours for four seeds.
  2. Reward: a strict 0/1 checker; log format separately; no format bonus.
  3. Loss: group advantage, PPO clip, optional k3 KL, and a deliberate choice of token averaging.
  4. Log: reward, held-out pass@k, entropy, length, dead groups, samples.
  5. Debug: clip-higher for collapse, token-level loss for length, dynamic sampling for dead groups.
  6. Defend: pass@k to 256, noise bars, hand-read outputs.

If you can do all six from memory with one number each, Week 7 is done.

Your Week 7 plan

#ResourceHoursTake away
1CS336 Assignment 5 handout §3–63the run recipe, derivations, variants to ablate
2Write and test the loss from this deck1.5pass the coding-RL screen
3TRL GRPO Trainer docs loss types, config, metrics1.5defaults; what each knob changes
4nanochat chat_rl.py + Discussion #1 RL part1simplest working loop; pass@k logging
5RL quiz + fundamentals quiz unofficial, timed2oral answers with numbers

Total 9 hours plus GPU time. The guide lists far more for these resources (CS336 assignments 15–25 h each, TRL 6, nanochat 12, quizzes 4), so this plan reads only the parts above. Later: open-r1 for a full R1 reproduction.

Self-test: answer out loud, then open

GRPO advantages when 1 of 8 answers is right?

Mean 0.125, sample std 0.354: right +2.47, wrong −0.35. Without the std (Dr. GRPO): +0.875 and −0.125.

Share of silent groups at p = 0.9, G = 8?

0.9⁸ + 0.1⁸ ≈ 43%: only about 18 of 32 prompts carry signal.

Why does per-answer averaging grow wrong answers?

Per-token push is A ÷ length: a 400-token wrong answer is penalised 4× less per token than a 100-token one.

TRL defaults for beta, loss_type, learning rate, num_generations?

0.0, "dapo", 1e-6, 8.

Clipped objective at ρ = 1.5, A = +2, ε = 0.2?

2.4, with zero gradient: the good answer is already pushed past the trust region.

Your on-policy loss reads 0.0000 every step. Bug?

No. With per-answer averaging and ratio 1, each group’s advantages sum to 0. Watch reward, entropy and gradient norm.

15 AIME points: how many problems, and the noise?

4.5 of 30. One run at 50% accuracy has a ±18-point 95% range.

How do you tell sharpening from a new skill?

pass@k to k = 256 on held-out problems. If the base model catches up or wins at large k, RL sharpened (Yue et al.).

Next

Week 8: frontier recipes at scale

What labs actually did: Llama 3, DeepSeek-V3 and V4, Kimi K2’s MuonClip, Olmo 3, Tulu 3, Llama 2’s RLHF and the OPT-175B logbook. Stability at scale and full post-training pipelines.

Staff-level probe to aim for

“Loss spikes at step 40K of a 400B MoE run and one expert’s routing share jumps. What do you check first, and what small-scale run would confirm the cause?”

Week 8 →  ·  ← Week 6