Run GRPO yourself,
then defend every curve
A small reinforcement-learning run on grade-school math: the reward, the loss written from scratch, what to log, what broken looks like, and how to argue the result is real.
About 9 hours with the readings, plus GPU time · use → or Space to step through · O for contents · T for dark mode
What this week is for
Run a small GRPO job, break it, and write its loss from memory.
How labs ask it
- “Research Engineer tech screen focused on coding RL algorithms (per the recruiter's prep link)” Anthropic · secondhand 2026
- “RL training debugging scenarios: what do you do when you hit entropy collapse, the model reasoning in another language, or terrible MFU?” frontier lab, unnamed · secondhand 2025
The map for this week
Parts 1–4 follow one run: a 1B model learning grade-school math with a 0/1 reward. Part 5 is oral-quiz practice.
Set up a run you can afford
Three small runs you can copy
| CS336 Assignment 5 | TRL quick start | nanochat RL | |
|---|---|---|---|
| Model | OLMo-2-0425-1B (base) | Qwen2.5-0.5B-Instruct | its own ~560M chat model (d20, Oct 2025) |
| Data | GSM8K | DeepMath-103K | GSM8K |
| Answers per prompt (G) | 8 | 8 | 16 |
| Prompts per step | 32 | 8 on 8 GPUs (defaults) | 16 |
| Max answer tokens | 512 | 512 | 256 |
| Hardware, time | 2 GPUs; “2 B200 hrs” for 4 seeds | 8 GPUs, ~1 day | 8×H100, ~1.5 h |
Pick CS336: every hyperparameter is written down, it has unit tests, and a 1B base model actually learns (target: ≥25% validation accuracy, averaged over seeds).
Details
GSM8K: grade-school word problems with a single numeric answer after “####”; 7,473 train and 1,319 test problems (Hugging Face openai/gsm8k). nanochat’s RL loop is a simplified GRPO: no KL, no clipping, advantage = reward − group mean. TRL’s default 8 prompts/step comes from per_device_train_batch_size=8 completions × 8 GPUs ÷ 8 per prompt.
CS336 A5 handout §2.2, §4.3; TRL GRPO Trainer docs (quick start, GRPOConfig); nanochat scripts/chat_rl.py and Discussion #1.
One step of the run, on two GPUs
Generation and training live on different GPUs. vLLM, a fast inference engine, samples; the Hugging Face model trains.
One optimizer step per batch of samples makes it on-policy: every sample came from the current weights, so no ratio clipping is needed yet.
Repeat 200 times: 6,400 training prompts. Evaluate on 1,024 held-out problems every 10 steps.
CS336 A5 handout §4.2.2, §4.3 (suggested hyperparameters).
An honest compute estimate
2 B200 GPUs per job. Main run labelled “2 B200 hrs” for 4 seeds × 200 steps. All five experiment problems: 26 B200 hrs.
RL step about 1.5 h on 8×H100: ~$36 at $24/hour. GSM8K went 4.6% → 7.6%. Needs its own SFT checkpoint first.
TRL quick start: 0.5B, 8 GPUs, ~1 day. Dr. GRPO: 7B, 8×A100, 27 hours.
Why “2–4”
The handout heading says “2 B200 hrs”, and the course’s Modal config requests “B200:2”. It doesn’t say whether that is GPU-hours or hours on the 2-GPU job, so the range covers both readings. The 26 = 2 (main) + 4 (learning-rate sweep) + 4 (prompt ablation) + 8 (on-policy variants) + 8 (off-policy variants).
CS336 A5 handout problem headings; cs336_alignment/modal_utils.py; nanochat README and Discussion #1 report card; TRL docs quick start; Liu et al. 2025 (Dr. GRPO) §1.
The reward: a checker, not a model
# TRL calls this with every completion, plus dataset columns as kwargs import re ANSWER = re.compile(r"</think> <answer>(.*?)</answer>", re.S) def norm(s): s = s.strip().replace(",", "").replace("$", "").rstrip(".") try: return str(float(s)) # "72", "72.0", "72." all match except ValueError: return s def gsm8k_reward(completions, answer, **kwargs): out = [] for text, sol in zip(completions, answer): gold = sol.split("####")[-1] # GSM8K: "... #### 72" m = ANSWER.search(text) out.append(1.0 if m and norm(m.group(1)) == norm(gold) else 0.0) return out
The prompt ends with Assistant: <think>, so the model reasons, then answers in tags.
Strict on format, like CS336’s grader: no tags, no reward.
No partial credit for format alone: a format bonus is the easiest reward to hack. Unit-tested on the next slide’s cases.
Prompt: CS336 A5 §3.2 (the DeepSeek R1-Zero template). Reward signature: TRL docs, “Using a custom reward function”. Format rule: CS336 drgrpo_grader.r1_zero_reward_fn.
Score model outputs, live
The function from the previous slide, ported to JavaScript. Question: “Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May…” (gold 72, CS336 A5 §2.2). CS336 notes come from reading drgrpo_grader.py.
Reward design: what each choice costs
| Choice | Who | Upside | Failure mode |
|---|---|---|---|
| Accuracy only, 0/1 | CS336 | can’t be farmed | reward is ~0 at the start; slow first steps |
| Accuracy + format, equal weight | DeepSeek-R1 | teaches the tags fast | format alone earns half the reward |
| +1 / −1 | DAPO | wrong answers pushed down | same group signal after normalising |
| + language consistency | DeepSeek-R1 | readable reasoning | “slight degradation” in performance |
| Learned reward model | DeepSeekMath | any task | reward hacking; R1 avoided it for reasoning |
Language consistency reward = share of words in the target language inside the reasoning.
CS336 A5 §3.3 (“we will not give partial credit”); Guo et al. 2025 (DeepSeek-R1) §2.2, §3.2.1 and Supp. B.5; Yu et al. 2025 (DAPO) §2.3; Shao et al. 2024 (DeepSeekMath) §4.
The same run in TRL
from trl import GRPOConfig, GRPOTrainer
args = GRPOConfig(
output_dir="olmo1b-gsm8k-grpo",
num_generations=8, # G, answers per prompt
per_device_train_batch_size=8, # completions per micro-batch
gradient_accumulation_steps=32, # 8 × 32 = 256 = 32 prompts × 8
learning_rate=1e-5, # CS336 value; TRL default 1e-6
lr_scheduler_type="constant",
adam_beta2=0.95, weight_decay=0.0, max_grad_norm=1.0,
max_completion_length=512, temperature=1.0,
beta=0.0, # no KL term (TRL default)
loss_type="grpo", scale_rewards="group", # CS336 "standard"
max_steps=200, use_vllm=True, log_completions=True,
)
trainer = GRPOTrainer(model="allenai/OLMo-2-0425-1B", args=args,
reward_funcs=gsm8k_reward, train_dataset=ds)
trainer.train()
One generation batch per optimizer step keeps it on-policy (num_iterations=1, the default).
TRL’s default loss_type is "dapo", not "grpo". Set it on purpose.
ds needs a prompt column (the filled-in template) and an answer column.
Stop string and versions
CS336 stops vLLM at “</answer>”. In TRL, generation_kwargs goes to vLLM’s SamplingParams, so {"stop": ["</answer>"], "include_stop_str_in_output": True} should do the same; check it on your installed versions. Defaults checked against the TRL docs page and trl/trainer/grpo_config.py on GitHub main (16 Nov 2026; latest release v1.18.0).
TRL GRPOConfig reference and source; CS336 A5 §4.3.
Learning rate, KL, group size: what each source used
| Run | Model | G | LR | KL β | Clip ε | Max tokens |
|---|---|---|---|---|---|---|
| DeepSeekMath | 7B instruct | 64 | 1e-6 | 0.04 | 1 update per sample | 1,024 |
| R1-Zero | V3 base | 16 | 3e-6 | 0.001 | not stated | 32,768→65,536 |
| DAPO | Qwen2.5-32B base | 16 | 1e-6 | 0 | 0.2 low, 0.28 high | 20,480 |
| Dr. GRPO | Qwen2.5 1.5B, Math-7B | 8 | 1e-6 | 0 | 0.2 | 3,000 |
| CS336 A5 | OLMo-2 1B base | 8 | 1e-5 | none | on-policy, none | 512 |
| TRL defaults | — | 8 | 1e-6 | 0.0 | 0.2 | 512 |
Scroll the table sideways.
R1’s later first RL stage used ε = 10. Shao et al. 2024 §4.2; Guo et al. 2025 §2.1, §3.2.1; Yu et al. 2025 §4.1; Liu et al. 2025 App. G; CS336 A5 §4.3; TRL GRPOConfig.
Setup questions you should answer with numbers
- “RL for Language Models: Policy Gradients, PPO, GRPO and DPO (9 questions / 45–60 min)” Google DeepMind · question bank 2026
- Why does GRPO particularly suit verifiable rewards, and what happens to a group whose rewards are all equal? community practice (r87, unofficial)
- “Design post-training for a new base model targeting coding and math. Where do you use SFT, DPO, RLVR and distillation, and how do you know each helped?” curriculum senior-staff probe
Answer sketch
A first RLVR experiment (RL with verifiable rewards): 1B base model, GSM8K, 0/1 checker, G = 8, 32 prompts × 200 steps, about 2–4 B200 GPU-hours for four seeds. GRPO needs the reward many times per prompt, which is cheap for a checker. An all-equal group has zero advantage, so it gives no gradient. KL: 0.04 in DeepSeekMath with a learned reward model, 0.001 in R1-Zero, 0 in DAPO and Dr. GRPO.
Only the first item is an interview report (llm-q47). The second is community practice; the third is from the curriculum’s senior-staff bar (Week 8 goes deeper).
Write GRPO’s loss from scratch
The coding-screen version: about 15 lines of PyTorch, checked against the papers and TRL.
From “push up good answers” to GRPO, one change at a time
- REINFORCE: raise each answer’s log-probability in proportion to its reward: right 1, wrong 0.
- Subtract the group mean: with 4 of 8 right, weights become +0.5 and −0.5. Same direction in expectation.
- Divide by the group’s std: a stability trick that changes the objective.
- Average over each answer’s tokens: long answers count less per token.
- Clip the ratio new ÷ old probability to [0.8, 1.2] when samples are reused.
Dr. GRPO removes steps 3 and 4.
Why the baseline is free
The expected gradient of log-probability under the model’s own samples is zero: the probabilities always sum to 1, so their gradients sum to 0. Any baseline that doesn’t depend on the sampled answer therefore leaves the expected gradient unchanged and only changes its variance. The group mean does depend on the answers, which rescales the expectation by (G − 1)/G.
CS336 A5 §4.1.4–4.1.8 (derivation, eqs. 9–28); Liu et al. 2025 §3.1.
Group advantages: click answers right or wrong
Advantage = (reward − group mean), then optionally divided by something.
Try 1 of 8 right: with the std, the rare correct answer gets +2.47; without it, +0.88.
Try all right or all wrong: every advantage is 0. That prompt teaches nothing this step.
Dividing by the std gives hard and easy prompts bigger updates. Dr. GRPO calls this a difficulty bias.
Std is the sample std (divide by G − 1) plus 1e-4, as in TRL and torch.std. Normalisers from CS336 A5 §5.1–5.3 (MaxRL divides by the group mean); Liu et al. 2025 §3.1.
The clip: stop pushing once you’ve pushed far enough
ρ = new probability ÷ old probability of a token. ε = 0.2.
Good answer, ρ = 1.5: value capped at 1.2 × 2 = 2.4, gradient 0.
Bad answer, ρ = 0.5: value stuck at 0.8 × −2 = −1.6, gradient 0.
Moves the wrong way are never clipped: ρ = 1.5 with A = −2 still gives −3.0 and full gradient.
Schulman et al. 2017 (PPO) eq. 7; the four worked values are r87’s RL quiz Q2, recomputed in a unit test.
How you average the tokens changes what the model learns
Each token’s push = its advantage × an averaging weight.
Per-answer mean (original GRPO): a long wrong answer is penalised less per token, so wrong answers drift longer.
Per-token mean (DAPO, TRL default) or a constant (Dr. GRPO): every token weighs the same.
Liu et al. 2025 §3.1 (response-level length bias); Yu et al. 2025 §3.3 (token-level loss); TRL loss_type docs. Constant = 2 answers × 512 max tokens.
GRPO’s loss in 15 lines
def grpo_loss(logp, old_logp, rewards, mask, G, eps=0.2, ref_logp=None,
beta=0.0, scale="std", agg="token", L_max=None):
# logp, old_logp, ref_logp, mask: [B*G, T] rewards: [B*G]
r = rewards.view(-1, G)
adv = r - r.mean(dim=1, keepdim=True) # group baseline
if scale == "std": # Dr. GRPO: "none"
adv = adv / (r.std(dim=1, keepdim=True) + 1e-4)
adv = adv.reshape(-1, 1) # same A on every token
ratio = torch.exp(logp - old_logp) # 1.0 on-policy
clipped = torch.clamp(ratio, 1 - eps, 1 + eps)
loss = -torch.min(ratio * adv, clipped * adv)
if beta > 0: # k3 KL(pi || ref)
d = ref_logp - logp
loss = loss + beta * (torch.exp(d) - d - 1)
if agg == "seq": # DeepSeekMath, CS336 standard
return ((loss * mask).sum(1) / mask.sum(1)).mean()
if agg == "token": # DAPO, TRL default
return (loss * mask).sum() / mask.sum()
return (loss * mask).sum() / (mask.shape[0] * L_max) # Dr. GRPO
1. Advantage per answer, copied to its tokens.
2. PPO clip per token. On-policy, pass logp.detach() as old: ratio 1, gradient still flows.
3. Optional KL to the frozen reference: the always-positive “k3” estimate.
4. Aggregation: the length-bias choice from the last slide.
Matches TRL’s _compute_loss for loss_type grpo, dapo and dr_grpo (checked numerically). Shao et al. 2024 eqs. 3–4; Schulman, “Approximating KL Divergence” (2020).
The tests an interviewer wants you to write
- On-policy gradient = REINFORCE with baseline. Compare with autograd on a hand-built objective.
- All-equal group → zero gradient on those rows.
- Padding gets no gradient. Multiply by the mask, then divide by its count.
- Clip: ρ = 1.5, A = +2 → zero gradient; A = −2 → gradient flows.
- Micro-batches: accumulated gradient equals the full-batch gradient.
“My loss is 0.0000 every step”
Not a bug. On-policy with per-answer averaging, the advantages in each group sum to zero, so the loss value is exactly 0. Its gradient isn’t.
CS336: “This expression is not a loss in the traditional sense.” Watch reward, entropy and gradient norm instead.
Tests 1–4 pass in a local PyTorch run of the previous slide’s code. Test 5: CS336 A5 §4.2.4 (rescale each micro-batch loss by its share of the batch).
Can you write and defend this in 30 minutes?
- “Research Engineer tech screen focused on coding RL algorithms (per the recruiter's prep link)” Anthropic · secondhand 2026
- Implement GRPO’s per-token loss with clipping and a KL term; then make it Dr. GRPO. practice item (this deck)
- For ε = 0.2, give the clipped objective at (ρ, A) = (1.5, 2), (0.5, 2), (1.5, −2), (0.5, −2), and say which have zero gradient. community practice (r87, unofficial)
- Name the two biases in GRPO’s loss and the one-line fix for each. community practice (r87, unofficial)
Answers
2.4 (zero gradient), 1.0, −3.0, −1.6 (zero gradient). Biases: per-answer length averaging (fix: token-level or constant normaliser) and per-group std scaling (fix: don’t divide by std, or use the batch std).
The Anthropic item is the only real report for this group (llm-q13); the recruiter’s prep link pointed at coding RL algorithms.
Log it, read it, debug it
What to log, and the name TRL gives it
| Metric | TRL name | Why |
|---|---|---|
| Train reward, per reward function | reward, rewards/{name}/mean | is it learning at all |
| Format reward, logged separately | log_metric() inside the reward fn | format vs real accuracy |
| Held-out accuracy, pass@k | your eval loop | train reward can overfit |
| Token entropy | entropy | collapse = no exploration |
| Length, and share truncated | completions/mean_length, clipped_ratio | length bias, truncation |
| Groups with no signal | frac_reward_zero_std | dead prompts |
| Clip fraction, KL, gradient norm | clip_ratio/*, kl; grad norm: CS336 requires it | staleness, step size |
| Sample outputs | log_completions=True | read them every 40 steps |
TRL docs, “Logged metrics” and reward-function Example 6 (log_metric); CS336 A5 §4.2.4, §4.3 (required metrics, rollouts every 40 steps); nanochat chat_rl.py (pass@1…8).
Healthy vs broken: what the curves look like
Shapes are made up for illustration; each pattern is described in a source: DAPO §3.1, §3.4, §4.3; Dr. GRPO §3.2; DeepSeek-R1 Supp. B.5, B.6; CS336 A5 §4.3.
Easy prompts go silent: how many groups give no gradient?
A group is silent when all G answers agree: chance = pG + (1 − p)G.
As training succeeds, p rises and the batch quietly shrinks. Noisier gradients, wasted rollouts.
Fixes: bigger G; filter out solved prompts; DAPO’s dynamic sampling keeps sampling until the batch is full of mixed groups.
Computed in your browser. Yu et al. 2025 (DAPO) §3.2; TRL frac_reward_zero_std; CS336 A5 §5.4 (skip zero-advantage sequences).
Debugging checklist
| Symptom | First check | Fix the sources used |
|---|---|---|
| Entropy collapse: identical answers, reward stalls | entropy, clip_ratio/high | raise upper clip to 0.28 (DAPO); entropy bonus or top-20% entropy tokens (TRL) |
| Length explosion | length of right vs wrong answers; clipped_ratio | token-level or constant loss; mask truncated answers; soft length penalty (DAPO) |
| Format hacking: reward up, accuracy flat | format and answer reward apart | no format bonus; read 20 “rewarded” outputs |
| Language mixing | language of the reasoning | language-consistency reward (R1); SFT cold start |
| Zero-variance groups | frac_reward_zero_std | dynamic sampling, harder prompts, bigger G |
| Reward never moves | base model’s pass@8 on train prompts | better prompt; easier data. If pass@8 is 0, RL has nothing to reinforce |
DAPO §3.1–3.4; Dr. GRPO §2.1, §3.1; DeepSeek-R1 §3.2.1, Supp. B.6; TRL entropy_coef, top_entropy_quantile, mask_truncated_completions; CS336 A5 §3.4, §4.3.
“Entropy collapse, another language, terrible MFU: what do you do?”
Entropy collapse
Confirm with entropy and identical samples per group. Raise εhigh to 0.28; DAPO’s naive GRPO baseline reached 30 on AIME 2024, clip-higher was one step to 50.
Other language
The checker can’t see language. Add R1’s consistency reward (share of target-language words); expect a slight score cost.
Terrible MFU
MFU = model FLOPs utilisation. Generation, not training, is usually the bottleneck, dominated by the longest answers. Use vLLM, cap length, overlap generation with training.
Staff-level move: say which plot you’d open first and what number would change your mind.
Question: frontier lab, unnamed · secondhand 2025 (llm-q14). DAPO Table 1 and §3.2; DeepSeek-R1 §3.2.1; TRL docs (“Generation is often the main bottleneck”).
Defend the result
“Your RL run improved pass@1 on AIME by 15 points. Convince me it learned something new and isn’t reward hacking or sharpening the base model.”
pass@k: sharpening or a new skill?
pass@k = share of problems solved by at least one of k samples.
Sharpening: RL wins at k = 1, base catches up or wins at large k. Yue et al. found this across models and algorithms.
New skill: RL solves problems the base never solves, even in 256 tries.
The estimator
Sample n answers per problem, count c correct; pass@k = 1 − C(n−c, k) / C(n, k), averaged over problems (Chen et al. 2021, used by Yue et al.). The widget uses n = 256 and 40 made-up problems.
Made-up problems, real estimator computed in your browser. Yue et al. 2025 abstract, §2, App. A.2; Shao et al. 2024 §5.2.2 (RL raised Maj@K but not Pass@K).
15 points on AIME is how many problems?
AIME 2024 has 30 problems. 15 points is 4.5 problems.
Two noise sources: which samples you drew, and which problems happen to be in the set.
Do: average 32 samples per problem (DAPO’s avg@32), compare base and RL on the same problems, and run 4 seeds with a 1.96 σ/√n band (CS336).
Interval = ±1.96 × √(p(1 − p) / N), treating each problem as one right/wrong outcome; computed live. Sizes: HuggingFaceH4/aime_2024 (30), openai/gsm8k test (1,319), CS336 val (1,024).
The checks to run on your own small run
- pass@k, base vs RL, k = 1 to 256, on held-out problems. Crossing curves mean sharpening.
- Held-out sets: the GSM8K test split, and a harder set never used as an RL prompt.
- Format vs accuracy: count “formatted but wrong” and “right but unparsed”. Compare with the base model’s best prompt, including 3-shot.
- Length: right and wrong answers plotted apart; truncation rate.
- Entropy: a collapse means less exploration; read it next to pass@256.
- Read the reasoning: hand-check reward-1 outputs for lucky guesses and checker bugs.
Why each matters
Dr. GRPO showed templates can destroy a base model’s ability before RL rebuilds it, so part of a gain can be format repair; they also showed GRPO grows wrong answers’ length. Yue et al. hand-checked correct chains of thought: on the hardest GSM8K problems, 24 of 25 base-model solves and 23 of 25 RL solves had at least one valid chain. DeepSeek-R1 saw reward rise while CodeForces pass@1 fell under a learned helpfulness reward.
Yue et al. 2025 §3.1; Liu et al. 2025 §3.2–3.3; Guo et al. 2025 Supp. B.5; CS336 A5 §3.4 (format/answer categories, 3-shot prompt).
Your two-minute answer to the probe
- Concede the base rate: published evidence says RL with checkable rewards mostly sharpens. A pass@1 gain is still worth having.
- Show the noise bar: 15 points = 4.5 of 30 problems; avg@32, 4 seeds, paired by problem.
- Show pass@k to 256 on AIME and an unseen set. New skill only if RL stays above at large k.
- Rule out hacking: format and answer reward apart, hand-read rewarded outputs, check right-vs-wrong length.
- Name the problems that went from 0 of 256 to solved, and show their reasoning.
l7_bar probe 6 of the curriculum. Evidence: Yue et al. 2025; Shao et al. 2024 §5.2.2; Liu et al. 2025; Guo et al. 2025 Supp. B.5.
Quiz practice: the oral rounds
Questions from an unofficial community repo of Google DeepMind interview notes. Its own README says the problems are reconstructed from second-hand accounts and written from scratch. Every answer below was re-checked against a primary source or recomputed.
RL for LLMs: answer aloud in 2 minutes, then open
Why doesn’t subtracting a baseline bias the policy gradient?
The expected gradient of log π under π is ∇ of the total probability, ∇1 = 0. So b × 0 = 0 for any b that doesn’t depend on the answer. It only changes variance.
Checked: CS336 A5 eqs. 15–17.
GRPO advantages for rewards (1, 0, 1, 0) and (1, 1, 1, 1)?
Mean 0.5, sample std 0.577 → ±0.866. All-equal → all 0. With the population std (divide by G) it would be ±1.0; TRL and torch.std use G − 1.
Recomputed; TRL nanstd uses Bessel’s correction.
Why use k3 = (u − 1) − log u to estimate KL, with u = πref/π?
Unbiased like −log u, but never negative and far lower variance near u = 1: at u = 1.01, −log u = −0.00995 while k3 = 0.0000497.
Checked: Schulman 2020 blog; DeepSeekMath eq. 4 (“guaranteed to be positive”).
In DPO, why does the partition function Z(x) disappear?
Reward = β log(π*/πref) + β log Z(x). Bradley–Terry uses only the reward difference of two answers to the same x, so β log Z(x) cancels.
Checked: Rafailov et al. 2023 eqs. 4–7.
Symptoms of over-optimising a learned reward model?
Proxy reward keeps rising while a held-out measure falls; answers lengthen. DeepSeek-R1 shows reward up while CodeForces pass@1 falls.
Checked: DeepSeek-R1 Supp. B.5, Fig. 6.
Two ways rollouts become off-policy without you choosing it?
Several optimizer steps per batch (CS336 tries 32), and the inference engine computing slightly different probabilities than the trainer. TRL logs that gap.
Checked: CS336 A5 §6; TRL “training-inference mismatch”.
Questions adapted from r87’s RL-for-LLMs quiz (Q1, Q4, Q6, Q7, Q8, Q9). Unofficial; answers verified as noted.
LLM fundamentals: the arithmetic rounds
Variance of q·k with dh unit-variance independent entries, and why scale?
Variance = dh. Large scores saturate softmax, gradients vanish; dividing by √dh brings variance back to 1.
Checked: Vaswani et al. 2017 §3.2.1, footnote 4.
KV cache: 32 layers, 8 KV heads, head size 128, 32,768 tokens, bf16?
2 × 32 × 8 × 128 × 32,768 × 2 bytes = 232 bytes = 4 GiB per sequence.
Recomputed.
Parameters per block, and FLOPs per token for training vs inference?
12 × d² per block (4d² attention, 8d² MLP). About 6N per training token, 2N per generated token.
Checked: Kaplan et al. 2020 §2.1, Table 1.
Probabilities 0.5, 0.2, 0.8, 0.25 on four true tokens: cross-entropy and perplexity?
Product = 0.02 = 1/50. Perplexity = 501/4 ≈ 2.66; cross-entropy = ln 50 ÷ 4 ≈ 0.978 nats.
Recomputed.
Top-p = 0.7 on {a 0.5, b 0.2, c 0.15, d 0.1, e 0.05}?
Keep the smallest top set reaching 0.7: {a, b}.
Recomputed.
8 experts of 2B, 0.5B shared, top-2 routing: total and active?
Total 0.5 + 8 × 2 = 16.5B. Active 0.5 + 2 × 2 = 4.5B per token.
Recomputed.
Adapted from r87’s LLM-fundamentals quiz (Q1, Q3, Q5, Q7, Q11, Q12). Unofficial; answers verified as noted.
The rounds these quizzes prepare you for
- “LLM Fundamentals: Attention, Transformers and the Training Pipeline (12 questions / 45–60 min)” Google DeepMind · question bank 2026
- “Deep discussion of NLP and transformer internals.” Google DeepMind · firsthand 2026
- “Walk me through what happens, tensor by tensor, in one forward pass of a decoder-only transformer.” Anthropic · question bank 2026
- “Why does attention divide by sqrt(d_k)?” frontier lab, unnamed · question bank 2026
- “ML coding testing familiarity with the Transformer architecture and its basic concepts.” OpenAI · firsthand 2025
Where the community answers need care
The RL quiz calls KL(πθ ‖ πref) the “forward” KL; TRL’s docs call the same direction’s gradient the “reverse-KL gradient”. Name the direction explicitly instead of using either label. CS336 writes the group std with 1/G but tells you to use torch.std, which divides by G − 1; say which you use.
Question groups llm-q46, q12, q24, q15, q10. Week 1 covers the forward pass and √d in depth.
The week in one minute
- Setup: 1B base model, GSM8K, G = 8, 32 prompts × 200 steps, 2 GPUs, 2–4 B200 GPU-hours for four seeds.
- Reward: a strict 0/1 checker; log format separately; no format bonus.
- Loss: group advantage, PPO clip, optional k3 KL, and a deliberate choice of token averaging.
- Log: reward, held-out pass@k, entropy, length, dead groups, samples.
- Debug: clip-higher for collapse, token-level loss for length, dynamic sampling for dead groups.
- Defend: pass@k to 256, noise bars, hand-read outputs.
If you can do all six from memory with one number each, Week 7 is done.
Your Week 7 plan
| # | Resource | Hours | Take away |
|---|---|---|---|
| 1 | CS336 Assignment 5 handout §3–6 | 3 | the run recipe, derivations, variants to ablate |
| 2 | Write and test the loss from this deck | 1.5 | pass the coding-RL screen |
| 3 | TRL GRPO Trainer docs loss types, config, metrics | 1.5 | defaults; what each knob changes |
| 4 | nanochat chat_rl.py + Discussion #1 RL part | 1 | simplest working loop; pass@k logging |
| 5 | RL quiz + fundamentals quiz unofficial, timed | 2 | oral answers with numbers |
Total 9 hours plus GPU time. The guide lists far more for these resources (CS336 assignments 15–25 h each, TRL 6, nanochat 12, quizzes 4), so this plan reads only the parts above. Later: open-r1 for a full R1 reproduction.
Self-test: answer out loud, then open
GRPO advantages when 1 of 8 answers is right?
Mean 0.125, sample std 0.354: right +2.47, wrong −0.35. Without the std (Dr. GRPO): +0.875 and −0.125.
Share of silent groups at p = 0.9, G = 8?
0.9⁸ + 0.1⁸ ≈ 43%: only about 18 of 32 prompts carry signal.
Why does per-answer averaging grow wrong answers?
Per-token push is A ÷ length: a 400-token wrong answer is penalised 4× less per token than a 100-token one.
TRL defaults for beta, loss_type, learning rate, num_generations?
0.0, "dapo", 1e-6, 8.
Clipped objective at ρ = 1.5, A = +2, ε = 0.2?
2.4, with zero gradient: the good answer is already pushed past the trust region.
Your on-policy loss reads 0.0000 every step. Bug?
No. With per-answer averaging and ratio 1, each group’s advantages sum to 0. Watch reward, entropy and gradient norm.
15 AIME points: how many problems, and the noise?
4.5 of 30. One run at 50% accuracy has a ±18-point 95% range.
How do you tell sharpening from a new skill?
pass@k to k = 256 on held-out problems. If the base model catches up or wins at large k, RL sharpened (Yue et al.).
Week 8: frontier recipes at scale
What labs actually did: Llama 3, DeepSeek-V3 and V4, Kimi K2’s MuonClip, Olmo 3, Tulu 3, Llama 2’s RLHF and the OPT-175B logbook. Stability at scale and full post-training pipelines.
“Loss spikes at step 40K of a 400B MoE run and one expert’s routing share jumps. What do you check first, and what small-scale run would confirm the cause?”