Frontier-lab prep · LLM internals · Week 5

RLHF foundations:
from policy gradients to PPO

How a reward becomes a gradient, how human or AI preferences become that reward, and how PPO keeps each update safe.

About 9 hours with the readings · use → or Space to step through · O for contents · T for dark mode

What this week is for

Derive the policy gradient, build a reward model, and explain PPO-based RLHF with numbers.

Senior-staff target (outline; Week 6 finishes)

“PPO vs GRPO: what does dropping the critic cost you…” (the PPO half) and “Design post-training for a new base model… how do you know each helped?”

How much it matters: ~2% of interview reports, none of the 18 staff-plus ones; core in research loops.

How labs ask it

  • “Explain Constitutional AI. What does it buy you over vanilla RLHF, and what doesn’t it solve?” Anthropic · question bank
  • “Design a feedback collection pipeline for RLHF.” OpenAI · prep site 2026
Part 1

Policy gradients from first principles

The week’s map: each part adds one piece of the RLHF loop.

RL words, mapped onto a language model

RL termIn RLHF
Statethe prompt plus the tokens written so far
Actionthe next token (or the whole answer: the “bandit” view)
Policy πthe model’s next-token probabilities
Rewardreward-model score at the end, minus a small per-token KL charge
Returnsum of rewards; discount γ = 1, so nothing decays
Episodeone prompt and one answer; there is no next state
Value V(s)expected return from this prefix (PPO’s critic learns it)

Unlike classic RL, the policy starts as a strong pretrained model. The main risk is drifting too far, not failing to explore.

Lambert, RLHF book ch. 3 (table “Standard RL vs RLHF”) and ch. 6; Spinning Up, part 1.

The policy gradient: push up what worked

Goal: raise the average reward of answers the model samples.

Trick: the gradient of a probability equals the probability times the gradient of its log. So the gradient becomes an average you can sample: reward × ∇ log π(answer).

log π(answer) is a sum of token log-probs. The environment’s own probabilities drop out because they don’t depend on the weights.

logp = get_policy(obs).log_prob(act)
loss = -(logp * weights).mean()
# weights = the episode's return
This “loss” is not a loss. It doesn’t measure performance, and minimizing it on one batch can crater reward. Watch average reward instead.

Spinning Up part 3, “Deriving the simplest policy gradient” and 1_simple_pg.py; REINFORCE: Williams (1992), via RLHF book ch. 6.

Live: REINFORCE with and without a baseline

The model picks one of 4 next tokens. Rewards 1.0, 0.6, 0.3, 0 (made up for illustration) plus an offset you add to all four.
y: probability of the best token; dashed line = the start (¼).

Adding a constant to every reward leaves the expected gradient unchanged…

…but multiplies its noise. With no baseline, every sampled token gets pushed up, and runs lock onto a random token.

Subtracting the other samples’ average reward (leave-one-out, as in RLOO) cancels the offset.

Computed in your browser: softmax policy, 4 samples per step, learning rate 0.5, 12 seeds. Variance is exact at step 0. Spinning Up part 3, “Baselines in policy gradients”.

Every policy-gradient method picks the weight

Total return: every token gets credit for everything.
Reward-to-go: only rewards after the token count. Fewer zero-mean noise terms.
Minus a baseline b(s): same expectation, because Σ π·∇log π = ∇(Σ π) = ∇1 = 0.
Advantage A = Q − V: lowest variance if V is right. PPO learns V; RLOO and GRPO use sibling answers.
RLOO, worked: 4 answers, rewards 1, 0, 1, 0. Leave-one-out baselines ⅓, ⅔, ⅓, ⅔ → advantages +0.67, −0.67, +0.67, −0.67.
At scale: sparse 0/1 rewards are the high-variance case. A learned critic costs a whole model; sibling baselines cost K answers per prompt.

Spinning Up part 3 (reward-to-go, EGLP lemma, baselines); RLHF book ch. 6 (Ψ list from Schulman et al. 2015; RLOO; sparse rewards).

Interview check

Can you answer these in two minutes?

  • “Research Engineer tech screen focused on coding RL algorithms (per the recruiter’s prep link)” Anthropic · secondhand 2026
  • “Derive ∇J(θ) = E[r(x,y) ∇ log π(y|x)] (the log-derivative trick)… State what a well-chosen baseline buys.” Google DeepMind · question bank
  • Why can a policy-gradient loss go down while the policy gets worse? practice, from Spinning Up

Coding drill: Spinning Up’s 1_simple_pg.py is 128 lines. Rebuild it from memory, add a leave-one-out baseline, then switch to per-token log-probs with a completion mask.

Questions from the question bank behind the curriculum (2026). Practice items are labeled as such.

Part 2

Preference data and reward models

Why compare two answers instead of scoring one

  • Judging which of two answers is better is easier than writing a good one.
  • Rankings are standard; ratings are kept as metadata. Anthropic used an 8-point scale, then reduced it to chosen vs rejected.
  • InstructGPT showed labelers 4–9 answers per prompt. K answers give K × (K − 1) ÷ 2 pairs: 9 answers → 36 pairs.
  • Shuffling those correlated pairs separately overfit the reward model in one pass. Fix: all pairs from a prompt form one batch element.
  • “Chosen” means better than the alternatives shown, not correct.

RLHF book ch. 11 (rankings vs ratings, Likert scales); Ouyang et al. 2022 §3.5 and App. C.2.

Live: score gap → probability → loss

Bradley–Terry: P(chosen beats rejected) = σ(score gap), where σ is the logistic function.

Loss = −log of that. Gap 0 → 0.69 (ln 2). Gap 2 → 0.13.

Only the gap matters. Shift both scores and nothing changes, so the reward’s offset is arbitrary. Pipelines normalize it.

The push on the gap is 1 − P. Confident pairs stop teaching.

RLHF book ch. 5, “Training a Bradley-Terry reward model”; Ouyang et al. 2022 eq. 1.

What a reward model is, concretely

prompt +chosen answer LM backbone(the SFT model) state at last tokenlinear head → 1 r = 1.2 prompt +rejected answer r = 0.3 same weights, two passes loss = −log σ(0.9) = 0.34 P(chosen wins) = 0.71
Scores made up for illustration; the arithmetic is real.

Start from the SFT model, drop the unembedding, add a one-output head.

Train one epoch; more overfits fast.

Normalize: shift so labeler demos average 0 (InstructGPT), or to mean 0, std 1 on samples (OpenAI’s 2019 code).

InstructGPT used one 6B reward model for policies up to 175B: 175B RMs trained unstably and cost far more.

Ouyang et al. 2022 §3.5, App. C.2; RLHF book ch. 5; Huang et al. 2023 (reward normalization).

Designing the feedback pipeline (OpenAI asks this)

Screening: labelers needed about 75% agreement with researchers on sensitive-speech flags and rankings, plus a 6/7 score on sensitive demonstrations.

Vendors deliver in weekly batches. A ~$500K contract might run 6 weeks, early batches spent calibrating. Llama 2 collected in 14 stages.

Ouyang et al. 2022 §3.4, Fig. 12 (labeling interface), App. B.1; RLHF book ch. 11 (“Sourcing and contracts”).

Senior-staff angle: where the pipeline fails

Off-policy pairs. Answers from other models train a judge that misreads yours. Collect on your latest checkpoint.
Rater habits become model habits: verbosity, list formatting, sycophancy, prefix bias.
Instruction wording. Asking raters for the “more harmless” answer rewarded evasive refusals. Anthropic later told raters to prefer thoughtful, non-evasive answers.
Near-ties. Labelers agree only 72.6% of the time. Track agreement per category; send hard items to experts.
What I’d build: a fixed held-out prompt set judged by raters who never label training data (InstructGPT’s held-out raters agreed 77.3%).

RLHF book ch. 11 (on-policy data, biases); Bai et al. 2022 (CAI) §4.4; Ouyang et al. 2022 §3.4.

Interview check

Can you answer these in two minutes?

  • “Design a feedback collection pipeline for RLHF.” OpenAI · prep site 2026
  • “Explain the difference between pre-training, supervised fine-tuning and preference optimisation.” Meta MSL · question bank
  • “Compare reward-model RLHF and Direct Preference Optimization.” Mistral · question bank (reward-model half now, DPO in Week 6)

For the pipeline question, say a number at every stage: K answers per prompt, rater agreement, epochs, batch structure, delivery cadence.

From the question bank behind the curriculum (2026 reports and prep sites).

Part 3

PPO for language models

PPO: reuse a batch without stepping too far

Dashed: unclipped r·A. Solid: PPO objective. Dot: your r.

Ratio r = πnew(token) ÷ πold(token). It starts at 1 and lets you take several steps on one batch.

Clipping removes the reward for pushing r past 1 ± ε in the helpful direction. The min keeps every move that makes things worse.

Several epochs with no clip or penalty scored −0.39 (normalized, 7 MuJoCo tasks); clipping at ε = 0.2 scored 0.82, the best setting.

Schulman et al. 2017 §2–3, Table 1. The default point (r = 1.5, |A| = 2) is from the Google DeepMind question-bank item below.

Four models in memory during PPO-RLHF

Prompt Policy πtrained · writes the answer answer tokens y₁ … y_T Referencefrozen SFT copylog π_ref per token Reward modelfrozenone score at the end Value modeltrained criticV(s) per token Reward per tokenscore at end − β·(log π − log π_ref) AdvantagesGAE from r and V PPO updateclipped loss for π · regression for V · repeat
model states
80 GB GPUs, states only

Roles: RLHF book ch. 6; InstructGPT App. C.4 (6B RM and value for all policies). Trained = 16 bytes/parameter (16-bit weights and gradients, 32-bit Adam states; ZeRO §3.1); frozen = 2. Activations and KV cache excluded. GB = 10⁹ bytes.

The reward each token actually gets

Prompt “usually”, answer “, he would” (GPT-2 tokens). Real log-probs; reward-model score 0.4.

The reward model scores only the finished answer; its 0.4 lands on the last token.

Every token pays β × (log π − log π_ref): a one-sample estimate of the KL to the frozen SFT model.

One sample can be negative (the first two tokens here), so the charge becomes a bonus. It is positive only on average.

InstructGPT used β = 0.02. OpenAI’s 2019 code started at 0.15 and adapted β toward a KL target of 6.

Log-probs from Huang et al. 2023 (“N implementation details”), whose worked example flips the sign; we use reward − β·KL as in Ouyang et al. 2022 eq. 2 and RLHF book ch. 15.

Advantages, built backwards with GAE

TD error δt = rt + V(next) − V(now): did this token beat the critic’s forecast?

GAE: At = δt + λ·At+1, from the last token back (γ = 1).

λ = 0: one step, low variance, trusts V. λ = 1: full return minus V, unbiased, noisier. PPO used 0.95.

Value target = At + V(st).

λ:
Rewards from the previous slide at β = 0.15. Values V made up for illustration. Press → to step.

Schulman et al. 2017 eq. 11–12 and Tables 3–5 (λ = 0.95); RLHF book ch. 6, “Generalized Advantage Estimation”.

One PPO iteration, with InstructGPT’s numbers

1. Roll out512 prompts · π_old · temp 1 2. ScoreRM at end · β = 0.02 per token 3. Advantages6B critic + GAE, no discount 4. Update8 minibatches × 64, clip 0.2 θ_old ← θ
256k episodes from about 31k prompts, one pass per batch, plus pretraining gradients mixed in.

One pass per batch means the ratio barely leaves 1, so the clip rarely binds. The KL penalty does most of the regularizing.

Not universal: Tülu 3 ran 4 PPO passes per batch at 8B and 70B, and 1 at 405B for stability. Watch the clip fraction to see whether clipping is active.

Ouyang et al. 2022 App. C.4; RLHF book ch. 6, “Double regularization”.

Implementation details that change results

Reward side

  • Reward model reads prompt + answer; reward taken at the last token.
  • Reward normalized to mean 0, std 1.
  • Answers with no period between tokens 16 and 24 get a fixed −1.
  • Rewards and advantages whitened per minibatch.

Policy side

  • Divide logits by the sampling temperature before log-probs; otherwise KL rises faster than expected.
  • Value head starts at zero; value loss clipped like the policy.
Adam’s ε differs between PyTorch and TensorFlow. PyTorch’s early updates were more aggressive: approximate KL 6× larger, clip fraction 4.4× higher. With gpt2-xl one seed’s KL hit 17.5.
More details from the reproduction

No dropout during policy training. Adaptive KL controller: β starts at 0.15 and moves toward a target KL of 6 over a horizon of 10,000. Reward model trained one epoch (the descriptiveness task had about 5,000 labels) with the learning rate annealed to zero. Different random seed per GPU process. Responses sampled at fixed length, even past an EOS token.

Huang et al. 2023, “The N Implementation Details of RLHF with PPO” (reproducing OpenAI’s 2019 lm-human-preferences code).

Senior-staff angle: “your PPO run diverged — what do you check?”

  1. Step size: KL to π_old, clip fraction, max ratio. The reference run: KL ≈ 0.0004, clip fraction ≈ 0.5%, max ratio 1.23.
  2. Reward vs KL to the reference: score up while KL climbs means reward hacking. Raise β or lower the KL target.
  3. Length: truncated answers push the RM off-distribution. Score only at EOS; penalize overlong answers.
  4. Log-prob plumbing: temperature, padding and position ids, sampler-vs-trainer mismatch (truncated importance sampling caps the correction).
  5. Critic: value loss, and whether advantages are whitened.

Never read the policy loss as a health signal. Plot reward, KL and length together.

Huang et al. 2023 (TF1 Adam column); RLHF book ch. 6 (EOS scoring, truncated importance sampling); Spinning Up part 3.

Senior-staff angle: what the critic buys you

Buys

  • Per-token credit: each token gets its own advantage. Group methods give every token one number.
  • Works with one sample per prompt; group baselines need K.
  • GAE’s λ as a bias–variance knob.

Costs

  • A second trained model: a same-size critic for a 70B policy adds 1.1 TB of states (+80%).
  • Hard to learn from an LM backbone; initialization matters (InstructGPT: from the RM).
  • More knobs and failure modes.
Judgment to defend: keep a critic for long multi-step episodes, intermediate rewards, or expensive rollouts. Drop it for cheap, checkable 0/1 rewards (Week 6, GRPO).

RLHF book ch. 6 (RLOO, GRPO, value functions, MDP vs bandit, VAPO); Ouyang et al. 2022 App. C.2; memory from the four-models widget.

Interview check

Can you answer these in two minutes?

  • “…say which of these cases zero out the gradient and, for those, explain in terms of a trust region what clipping protects against.” Google DeepMind · question bank
  • “What does using a critic cost, concretely?” Google DeepMind · question bank
  • “What is the difference between PPO and GRPO?” frontier lab, unnamed · firsthand 2026
  • “RL training debugging scenarios: what do you do when you hit entropy collapse, the model reasoning in another language, or terrible MFU?” frontier lab, unnamed · secondhand 2025

Google DeepMind “RL for Language Models” quiz (9 questions, 45–60 min), Q2–Q3; others from the curriculum’s question bank.

Part 4

The KL leash and over-optimization

Why the leash: the reward model is a proxy

Push hard enough and the policy finds what the reward model over-scores.

What it looks like: repeated text, switching languages, excess special characters, readable math with wrong answers; boilerplate like “you are valid, valued, and cared for”.

The leash: maximize reward − β·KL(π ‖ π_ref). Bigger β stays closer to SFT; smaller β chases reward.

Measuring KL cheaply. Sample from π and average log π − log π_ref (k1): unbiased, but single samples go negative. With u = π_ref ÷ π, k3 = (u − 1) − log u is also unbiased and never negative.
Alert on: reward-model score rising while held-out human win-rate stalls, KL climbing fast, answers lengthening.

RLHF book ch. 15 (symptoms, KL estimator); Bai et al. 2022 §4.3 (boilerplate); k1/k3 as posed in the Google DeepMind quiz Q6. Alerts are a judgment call.

Other regularizers, and why RL forgets less than SFT

Pretraining mix (InstructGPT “PPO-ptx”): add next-token loss on pretraining text, 8× as many examples as RL episodes, weight 27.8. It shrank, but did not close, regressions on SQuADv2 and DROP.
Other regularizers

A margin term in the reward-model loss (Llama 2) and a length-normalized likelihood term added to DPO. Most regularizers beyond the KL penalty appear in one model generation and are simplified away in the next.

SFT pulls toward a fixed dataset (forward KL). RL learns from its own samples (reverse KL), so updates stay where the model already puts probability.

Navigation task, rule changed at test: RL took accuracy 80.8% → 91.8%; SFT dropped it to 1.3%.

“RL’s Razor”: forgetting tracks KL from the base model on the new task’s data (R² = 0.96).

Ouyang et al. 2022 §3.5, App. C.4; RLHF book ch. 15 (Chu et al. 2025 V-IRL; Chen et al. 2025; Shenfeld et al. 2026).

Interview check

Can you answer these in two minutes?

  • “During RLHF training against a learned reward model, describe the symptoms you would look for to detect that the policy is over-optimising the reward model rather than genuinely improving, and state concrete mitigations.” Google DeepMind · question bank
  • “…show that k₃ ≥ 0 for every sample (not just in expectation), and explain why k₃ is preferred in practice.” Google DeepMind · question bank
  • “After fine-tuning on a customer’s task, target accuracy is up but the model got worse at everything else. What happened and what do you do?” Mistral · question bank

Google DeepMind “RL for Language Models” quiz Q6 and Q8; Mistral from the curriculum’s question bank.

Part 5

AI feedback and Constitutional AI

Constitutional AI in two phases

Phase 1 · supervised: critique and revise
Phase 2 · RL from AI feedback (RLAIF)

Human input on harmlessness: only the constitution and a few examples. Helpfulness labels stayed human: 135,296 comparisons.

Bai et al. 2022, “Constitutional AI”, Fig. 1, §3.1–3.2, §4.1–4.2.

The details that made RLAIF work

  • Soft labels: use the feedback model’s probabilities of (A) and (B) as targets. Much better than 0/1.
  • Chain-of-thought labels come out near 0 or 1. Clamping to 40–60% worked best (20–80% helped less).
  • A random principle per label, from 16, made the preference model more robust than one fixed principle.
  • Critiquing before revising helped small models; at 52B it made little difference.
  • Why it can work: on 438 helpful/honest/harmless comparisons, chain-of-thought judging approached preference models trained on human labels, and the trend suggested models above 52B would match them.

Bai et al. 2022 §2 (Fig. 4), §3.5, §4.1, §4.3.

Senior-staff angle: what CAI buys, and what it doesn’t solve

Buys

  • 16 written principles replace tens of thousands of harmlessness labels.
  • Less harmful at the same helpfulness than human-feedback RLHF (52B, crowdworker Elo).
  • Non-evasive: explains objections instead of refusing.
  • Change the objective by editing a principle and relabeling.

Doesn’t solve

  • Still needs human helpfulness labels and human-chosen principles, picked “in a fairly ad hoc” way.
  • Goodharting: over-trained RL-CAI adds boilerplate; preference scores less calibrated at high values.
  • AI labels are low-noise but high-bias; judges favor their own outputs.
  • Less human testing means unforeseen failures; the method is dual-use.

Bai et al. 2022 §1–1.2, §3.4, §4.3, §6.2; RLHF book ch. 12 (“AI Feedback”, “Building specific LLMs for judgment”).

AI feedback vs human feedback: the trade

Human labelsAI feedback
Cost per label (2026)~$1, sometimes over $10under $0.01
Error profilehigh noise, low biaslow noise, high bias
Changing the objectivenew batches over weeksedit a principle, relabel
Typical failurerater habits: length, flatterysystematic quirks, self-preference
Who leans on itfrontier labs treat it as a moatthe default first attempt

The book’s advice: start with AI feedback, route the hardest items to humans, and expect to add human data as the pipeline scales.

RLHF book ch. 12, “Balancing AI and human feedback data”; ch. 11 (contract timelines).

Interview check

Can you answer these in two minutes?

  • “What is Constitutional AI and how does it differ from RLHF? What is RLAIF?” Anthropic · question bank
  • “Explain Constitutional AI. What does it buy you over vanilla RLHF, and what doesn’t it solve?” Anthropic · question bank
  • “Walk me through a post-training recipe to turn a pretrained base model into a personalised assistant.” Meta MSL · question bank

Name the two phases, the soft labels, and one limit with its evidence (the boilerplate examples).

From the question bank behind the curriculum (2026).

Senior-staff angle: a post-training pipeline, stage by stage

StageOptimizesReal scaleHow you know it helped
SFTimitate demonstrationsInstructGPT 13K prompts; Tülu 3 ~1M, mostly syntheticheld-out instruction following
RM + PPO, or DPOpreferences, under a KL leashInstructGPT 33K RM prompts, 31K RL; Tülu 3 ~1M pairswin-rate vs last checkpoint, held-out raters; regression suite
RLVRcheckable rewards (math, code)Tülu 3 ~10K promptsheld-out pass@1; other evals flat
Distillcopy a stronger or specialist modelR1: ~800K rejection-sampled samplesstudent matches teacher on target evals

Ablate one stage at a time against the previous checkpoint on a fixed suite, and watch the alignment tax. Week 6 deepens DPO and RLVR.

RLHF book ch. 3 (“Canonical training recipes”); Ouyang et al. 2022 Table 6. The “how you know” column is a judgment call.

Week 5 in one minute

  1. Policy gradient: reward × ∇ log π, averaged over samples; a baseline cuts variance, not accuracy.
  2. Preference data: ranked pairs from your current model; raters agree ~73%.
  3. Reward model: LM + scalar head, loss −log σ(gap), one epoch.
  4. PPO: ratio clipped to 1 ± 0.2; critic + GAE give per-token advantages.
  5. Four models: policy and critic trained, reference and RM frozen: ~3.3 TB of states at InstructGPT scale.
  6. Per-token reward: score at the end minus β × (log π − log π_ref).
  7. Constitutional AI: principles replace harmlessness labels, via critique-and-revise SFT, then RLAIF.

Say each line with its number, and Week 5 is done.

Your Week 5 plan

#ResourceHoursTake away
1Spinning Up: Intro to RL, parts 1–3 + VPG and PPO pages Achiam2.5derivation, baselines
2Proximal Policy Optimization Algorithms Schulman et al.1clipping, Table 1, GAE
3The N Implementation Details of RLHF with PPO Huang et al.2knobs that make RLHF work
4RLHF book: ch. 5, 6 (through GAE), 11, 15, 12 (AI feedback, CAI) Lambert4RMs, PPO for LLMs, data, KL
5Constitutional AI Bai et al.1.5two phases, limits
6InstructGPT §3.4–3.5, App. C optional re-read0.5real pipeline numbers

Total 11 h (11.5 with the re-read) vs a ~9 h target. To fit: skip Spinning Up part 2, skim the code in “N details”, read CAI §1, 3–4.

Self-test: answer out loud, then open

Why doesn’t a baseline bias the policy gradient?

Its expected term is b × Σ π·∇log π = b × ∇(Σ π) = b × ∇1 = 0. It changes only the variance.

Chosen scores 1.5, rejected 0.5. P(chosen wins) and loss?

σ(1) = 0.73; loss = −log 0.73 = 0.31. Add 10 to both and nothing changes.

Why train a reward model for only one epoch?

It overfits fast: InstructGPT saw validation loss deteriorate with more epochs, and even one pass overfit when the K-choose-2 pairs were shuffled apart.

PPO, ε = 0.2: objective at (r = 1.5, A = 2) and at (r = 0.5, A = −2)?

min(3, 2.4) = 2.4 and min(−1, −1.6) = −1.6. Both are clipped, so the gradient is zero.

Model-state memory for InstructGPT’s PPO setup?

175B policy × 16 B + 175B reference × 2 B + 6B RM × 2 B + 6B value × 16 B ≈ 3.26 TB, about 41 GPUs of 80 GB before activations.

What do λ = 0 and λ = 1 mean in GAE?

λ = 0: A = δ, one step, low variance, biased by V. λ = 1: return minus V, unbiased, high variance.

In CAI, which labels are human and which are AI?

Helpfulness: 135,296 human comparisons. Harmlessness: 182,831 AI comparisons, one per red-team prompt.

Your PPO run’s clip fraction jumped 4×. First suspect?

Too-large steps: check approximate KL, learning rate and optimizer settings (Adam’s ε differs by framework) before touching the reward.

Next

Week 6: direct alignment and RL for reasoning

Derive DPO from the KL-regularized objective you met here; then GRPO (DeepSeekMath), DeepSeek-R1 in depth, RL from verifiable rewards, and the PPO vs GRPO trade-offs.

Staff-level probe to finish

“PPO vs GRPO: what does dropping the critic cost you, what is the length bias in GRPO’s loss, and when would you go back to a value model?”

← Week 4: scaling and pretrainingWeek 6 →