Frontier recipes at scale:
what labs actually did
How Meta, DeepSeek, Moonshot, Ai2 and the OPT team ran, rescued and post-trained their biggest models, in their own numbers.
About 10 hours with the readings · use → or Space to step through · O for contents · T for dark mode
What this week is for
Answer “how would you run, fix or post-train this?” with what labs reported.
- “Loss spikes mid-run on a large pretraining job. Walk me through your debugging process.” xAI · question bank
- “Design a feedback collection pipeline for RLHF.” OpenAI · prep site
Weight: ~2% of interview reports, none of the 18 staff-plus ones; core in research loops.
Senior-staff targets
The life of a frontier run
Each stage below is told through one or two labs that published real numbers: Llama 3, DeepSeek-V3 and V4, Kimi K2, OPT-175B, Llama 2, Tülu 3 and Olmo 3.
Running at scale
Parallelism, numerics, failures and money, from Llama 3 405B and DeepSeek-V3.
Llama 3 405B: boring on purpose
Size from a scaling law
Budget 3.8 × 10²⁵ FLOPs. The fit said 402B parameters on 16.55T tokens; they trained 405B on 15.6T.
Flat near the optimum: “about right” is enough.
Dense, not MoE
Chosen “to maximize training stability”. Post-training is SFT + rejection sampling + DPO, not PPO.
RL “tends to be less stable and harder to scale”.
Grow the batch
4M tokens, then 8M after 252M tokens, then 16M after 2.87T.
Result: “few loss spikes”, no interventions.
Changes from Llama 2: 8 KV heads (GQA), no attention across documents, 128K vocabulary (3.17 → 3.94 characters per token), RoPE base 500,000.
Rest of the recipe
126 layers, width 16,384, 128 query heads. Context grown 8K → 128K in six stages on about 800B tokens. Last 40M tokens: learning rate annealed to 0 on upweighted high-quality data, then checkpoints averaged.
Llama 3 Herd, §1, §3.2, Table 3, §3.4. Covers “why GQA, RoPE, SwiGLU?” (Meta MSL).
4D parallelism: which GPUs talk to which
TP splits every weight matrix. Talks every layer, so it stays inside one 8-GPU NVLink server.
CP splits a long sequence. Llama 3 all-gathers keys and values, cheap because GQA makes them small.
PP splits the layers into stages.
DP (FSDP) copies the model and splits the batch. Outermost: it hides latency by prefetching, so it can cross pods (1:7 oversubscribed).
Llama 3 Herd, §3.3.1–3.3.2, Fig. 5, Table 4. Widget numbers computed from Table 4.
54 days of a 16K-GPU run: what broke
466 interruptions: 47 planned, 419 unexpected.
That is one unexpected stop every 3.1 hours. Only 3 needed significant manual work; automation handled the rest.
Still above 90% effective training time.
Harder ones: slow “stragglers” that stall thousands of GPUs, a 1–2% daily throughput swing from mid-day heat, and power swings of tens of megawatts when all GPUs pause together.
Verify the table yourself
The counts add to 419, but the printed percentages add to only 94.9%. “Faulty GPU” is 148 of 419 = 35.3%, not the printed 30.1%. By counts, GPU rows are 64% (the text says 58.7%), and GPU + host + host maintenance is 77%, close to the stated “about 78%” hardware. The widget computes from counts.
Llama 3 Herd, Table 5 and §3.3.4.
DeepSeek-V3’s “$5.6M”, as the paper states it
180K GPU-hours per trillion tokens: about 54 days of pretraining on 2,048 H800s.
Excluded, in their words: “prior research and ablation experiments on architectures, algorithms, or data”.
Also not in it: the GPUs themselves (it is a rental price), and R1’s RL runs, costed separately at 147K GPU-hours.
The fair reading: FLOPs explain the gap. 37B active parameters need 11.6× fewer training FLOPs than Llama’s 405B, and the GPU-hours are 11.1× fewer.
DeepSeek-V3 §1, Table 1; DeepSeek-R1 Table 7; Llama 3.1 model card. FLOPs = 6 × active parameters × tokens, computed.
FP8 training in plain terms
FP8 (E4M3) has 4 exponent and 3 fraction bits, max 448. Values are divided by a scale before rounding.
One scale per tensor: one outlier sets it, and small values round to zero.
DeepSeek-V3: one scale per 1×128 activation tile and 128×128 weight block.
Tensor cores keep ~14 bits when summing, ~2% error at 4,096 terms. Fix: add into FP32 every 128 products.
Kept in BF16/FP32: embeddings, output head, MoE router, norms, attention, master weights. Loss stayed within 0.25% of BF16.
DeepSeek-V3 §3.3, Fig. 6–7, App. B. The widget rounds to real E4M3 values in your browser.
DualPipe: hide expert traffic behind compute
Experts spread over nodes made compute and communication take about equal time (1:1).
Each communication now runs while the other chunk computes. 20 of 132 SMs per GPU drive the traffic; each token goes to at most 4 nodes.
Micro-batches enter from both ends of the pipeline, so bubbles shrink too. Cost: two copies of the parameters.
Kimi K2 (1T parameters) rejected DualPipe for that memory cost and overlapped with standard interleaved 1F1B and only 16-way expert parallelism.
DeepSeek-V3 §3.2, Fig. 4–5, Table 2, §3.5.1; Kimi K2 §2.4.2. Slot sizes simplified to equal length.
Balancing experts without an auxiliary loss
8 experts, each token picks 2. Expert 1 is popular (affinities made up for illustration).
Each expert gets a bias added to its score only when choosing the top experts. The output mix still uses the original score.
After every step: overloaded expert → bias − γ, underloaded → + γ. No extra loss term, so no gradient fights the language-modelling loss.
V3: γ = 0.001, plus a tiny per-sequence loss (α = 0.0001). Validation loss at 1B: 2.258 with a per-sequence aux loss vs 2.253 aux-free.
DeepSeek-V3 §2.1.2, §4.2, §4.5.2–4.5.3. Widget runs the bias-update rule on random batches in your browser.
Three big MoEs, three sets of choices
| Choice | DeepSeek-V3 (Dec 2024) | Kimi K2 (Jul 2025) | DeepSeek-V4-Pro (Apr 2026) |
|---|---|---|---|
| Total / active | 671B / 37B | 1.04T / 32B | 1.6T / 49B |
| Routed experts | 256, pick 8 | 384, pick 8 | 384, pick 6 |
| Attention heads | 128 | 64: 128 would cost +83% FLOPs at 128K | new compressed sparse attention |
| Optimizer | AdamW | MuonClip | Muon |
| Low precision | FP8 compute | FP8 only to store activations | FP4 expert weights after training |
| Extra objective | predict 1 extra token; 85–90% accepted as a draft, 1.8× decode speed | none | same as V3 |
| Serving unit | 32 GPUs prefill, 320 decode | — | — |
Scroll the table sideways.
V3’s own limitation: the deployment unit is “relatively large”, a burden for small teams. That is where a 10× total-to-active MoE hurts.
DeepSeek-V3 §3.4, §4.2, §5.4.3, §6; Kimi K2 §2.3–2.4; DeepSeek-V4 card and §4.2.
Can you answer these with numbers?
- “Mixture of experts (and expert parallelism)” Mistral · firsthand 2026
- “Explain a mixture-of-experts transformer and how expert parallelism distributes its work. Trace what happens to a batch of tokens from routing…” Mistral · question bank
- “Discuss trade-offs in MoE routing: top-k vs expert-choice routing, load-balancing vs quality.” Mistral · prep site
- “Our flagship is a sparse MoE with ~10x more total than active parameters. Why is that architecture a good fit for private enterprise deployment, and where does it hurt?” Cohere · question bank
- “Explain the architectural choices in a Llama-class model: why grouped-query attention, RoPE and SwiGLU instead of the vanilla 2017 Transformer?” Meta MSL · question bank
Coverage note
Expert-choice routing is week 3 material; this week gives the balance-vs-quality evidence (aux-loss-free balancing) and the deployment cost of a 10× total-to-active MoE (V3’s 32- and 320-GPU serving units).
Keeping the loss going down
Loss spikes, divergence and the fixes labs reported, from OPT’s 2021 logbook to DeepSeek-V4.
OPT-175B: two months in the logbook
- Oct 18Plan B: fall back to GPT-3’s settings.
- Nov 5Run 11.0. Loss explodes; lower LR, tighter clipping.
- Nov 5–10Grad-norm explosions; loss scale underflows. GeLU → ReLU.
- Nov 11Run 12.00, GPT-3 initialisation. Now restarts are mostly hardware.
- Late Nov40+ restarts in two weeks: lost GPUs, memory errors.
- Dec 1–2Loss scale at its floor. “SGD-like Adam” silently reloads old settings; real SGD stalls.
- Dec 3Back to Adam, lower LR.
- Dec 6Grad-norm spikes, perplexity creeping up: LR cut to 2/3 of GPT-3’s.
- Dec 19Crosses the ~180B-token epoch: uneventful.
- Dec 21Provider deletes the cluster; 16 nodes return broken.
- Jan 6Done: 300B tokens, ~90 restarts.
OPT chronicles: 10%, 27%, 56% and final notes; OPT-175B logbook (2021-12-02, 12-06, 12-19 entries).
What warned them, and what worked
Early warnings
Before each divergence: the FP16 loss scale crashing toward 0, and the last layer’s activation norm spiking.
Restart rule
Go back to a checkpoint where the loss scale was still ≥ 1.0 and norms were falling. Lower the LR. Clip at 0.3 instead of 1.0.
Didn’t work
Switching to SGD (plateaued). Resetting the loss scale helped only sometimes.
Surprise
An accidental Megatron upgrade (one-line JIT change) shifted activation norms and added 2% throughput.
Hardware: 35+ manual restarts, 100+ hosts swapped. One resume took 50 minutes, 30 of them replaying the data loader. Their lesson: budget compute for the experiments that decide how to scale.
OPT paper §2.5; chronicles 27% and 56% notes; logbook 2021-12-05.
Why attention logits explode
Normalised inputs keep ‖x‖ in check, so a logit can only grow if ‖W_q‖ × ‖W_k‖ grows. Both weights multiply, so growth compounds.
Muon gives every direction of an update equal size (full rank); Adam’s updates are dominated by a few. Moonshot’s hypothesis: Muon pushes the top singular values up more often.
Vanilla Muon at 9B active / 53B total: max logits above 1000 quickly, then spikes and occasional divergence.
QK-Clip: after each optimizer step, if a head’s max logit S exceeds τ, shrink W_q and W_k so the logit lands back on τ.
Kimi K2 §2.1, Fig. 2, App. E. Soft-capping lets q·k keep growing underneath; QK-Norm needs full keys, which MLA never builds at inference.
QK-Clip, live: one head, one step
Scaling W_q and W_k each by √γ, with γ = τ ÷ S, scales every logit of that head by γ. Only heads above τ are touched.
For MLA: per-head query and key parts × √γ, per-head rotary query × γ, the shared rotary key untouched.
Kimi K2, τ = 100: 12.7% of heads triggered in the first 70,000 steps; after that none did. 15.5T tokens with no loss spike.
Ablation at 0.5B active / 3B total with an aggressive τ = 30: negligible effect on loss.
Kimi K2 §2.1, Algorithm 1, App. D. DeepSeek-V4 skips QK-Clip: it normalises queries and keys with RMSNorm instead (§2.4).
Five runs, five stability stories
| Run | What went wrong | Fix they reported |
|---|---|---|
| OPT-175B (2021) | Loss-scale collapse, activation-norm spikes | Roll back, lower LR, clip 0.3 |
| Llama 3 405B (2024) | “Few loss spikes” | Small batch early (4M → 16M), FP32 gradient sums |
| DeepSeek-V3 (2024) | No rollbacks. In a test, coarse FP8 on activation gradients diverged (16B) | Fine-grained FP8 scaling |
| Kimi K2 (2025) | Muon logits above 1000 at mid scale | QK-Clip, τ = 100: zero spikes |
| DeepSeek-V4 (2026) | Spikes from MoE outliers, fed by routing; rollbacks didn’t stop repeats | After a spike, route with older weights (~20% slower while on); clamp SwiGLU to ±10 |
Scroll the table sideways.
V4’s own words: the principles behind both fixes “remain insufficiently understood”.
OPT §2.5; Llama 3 §3.3.2, §3.4.1; DeepSeek-V3 §1, App. B.2; Kimi K2 §2.1; DeepSeek-V4 §4.2.3, §6.
The 40K-step spike with one hot expert
- Freeze the evidence. The batch, per-layer grad norms, max logit per head, per-expert load and bias, FP8 scales.
- Split the causes. Replay the batch on an earlier checkpoint (data?). Check logits (optimizer?) and MoE outliers (as V4 found).
- Buy time. Roll back a few hundred steps, skip the batch. V4: rollback alone doesn’t stop repeats.
- Confirm cheaply. Reproduce on a proxy with the same data order (Kimi saw logit blow-up at 9B active, V3 FP8 divergence at 16B), then add the fix.
Determinism helps: V4 built bit-reproducible kernels partly so spikes can be traced to a numerical cause.
A plan built from the reports above: DeepSeek-V4 §3.3, §4.2.3; Kimi K2 §2.1, App. D; DeepSeek-V3 App. B.2.
Three ways labs ask the same question
- “Your pretraining loss suddenly diverges at step 300k of a long run. Diagnose and fix it.” Google DeepMind · question bank
- “Loss spikes mid-run on a large pretraining job. Walk me through your debugging process.” xAI · question bank
- “Your training run's loss is diverging after step 10K. Walk me through your debugging process.” OpenAI · prep site
Name a real case for each branch: hardware (Llama 3’s 419 stops), numerics (OPT’s loss scale, V3’s FP8 test), optimizer (Kimi’s logits), routing (V4’s MoE outliers).
Post-training pipelines, as published
Llama 2 → Llama 3 → Tülu 3 → Olmo 3, DeepSeek and Kimi: what each stage was for.
Llama 2-Chat: a weekly loop with two judges
27,540 human-written SFT examples were enough to start.
1.42M human comparisons, collected from the newest model so the reward model stays on-distribution.
Two RMs because helpfulness and safety pull apart; each won on its own test set, beating GPT-4 as a judge.
RS only on the 70B; smaller models train on its outputs (distillation).
Lesson: V3 trained only on V2’s samples forgot how to rhyme. Fix: keep the best samples from all past rounds.
Llama 2 §3.1–3.2, Table 6. PPO β = 0.01 (7B, 13B) and 0.005 (34B, 70B); safety threshold 0.15 = precision 0.89, recall 0.55.
Ghost Attention: keep a system instruction alive
Problem: early RLHF models forgot “act as Napoleon” after a few turns.
The answers were written while seeing the instruction every turn, so they obey it.
The training copy shows it once, so the model must carry it across turns.
Earlier turns get zero loss: those assistant messages were made under a different context.
Instructions were random mixes of hobbies, languages and public figures. Applied after RLHF V3; held for 20+ turns.
Llama 2 §3.3, Fig. 9–10.
Llama 3: six rounds of reward model → RS → SFT → DPO
Why DPO, not PPO
DPO “required less compute for large-scale models and performed better”, especially on IFEval (instruction following).
Two DPO fixes
Mask chat-format tokens (in both answers, they caused repetition). Add 0.2 × likelihood loss on the chosen answer.
Data does the work
Best of 10–30 samples per prompt. Targets are mostly model-written, so filter: emoji, “I’m sorry”, RM top quartile.
Synthetic code needs a checker: the 405B trained on its own code “is not helpful (and can even degrade)”. With linters and generated tests: ~1M dialogues, 20% self-fixed after feedback.
Two smaller findings
Only 0.1% long synthetic examples in the SFT mix kept 128K-context skills from regressing. Final models average checkpoints from runs with different data and hyperparameters, at each of the RM, SFT and DPO stages.
Llama 3 Herd §4.1–4.3. DPO β = 0.1, LR 10⁻⁵; SFT 8.5K–9K steps.
Tülu 3: an open SFT → DPO → RLVR recipe
RLVR = RL with verifiable rewards: PPO where the reward is 10 if a checker accepts the answer, else 0. About 30K prompts from GSM8K, MATH and checkable instructions.
Findings: value model initialised from a general RM worked best; adding RM scores to the check made it worse; more KL drift usually lowered the average score (over-optimisation).
They kept an “unseen” evaluation split they never looked at during development, to measure how much they overfit their own choices.
Tülu 3 §2, §5, §6. Final 405B RLVR run: 46 hours on 256 GPUs.
Olmo 3: every stage, dataset and checkpoint released
Delta learning: more SFT on Qwen3-32B answers hurt (average 70.3 → 64.5). DPO preferring them over Qwen3-0.6B answers helped (→ 72.9). The gap teaches, not the answer.
RL: GRPO with recent fixes: drop all-same-reward groups, token-level loss, no KL, no std normalisation. Starting from DPO beat starting from SFT.
Cost, honestly framed: about 56 days on 1,024 H100s ($2.75M at $2/hour), of which post-training took about 9 days of sweeps.
Olmo 3 §2, §4.3–4.4, Table 21. Released: data mixes, 9T-token source pool, per-stage checkpoints, code.
DeepSeek: distil a reasoner, then merge specialists
1.5M SFT examples; reasoning data from an internal R1, trimmed for length. GRPO with rule checks, else a reward model that reasons first.
Cold start → RL → 600K + 200K rejection-sampled SFT → RL on everything. Preference reward only in the last 400 of 1,700 steps: longer led to reward hacking.
Specialists (SFT + GRPO) per domain, then on-policy distillation: one student learns from 10+ teachers on its own samples. Replaced the mixed RL stage.
The distillation trade-off, measured: adding R1 data to V2.5 raised MATH-500 from 74.6 to 83.2, and answer length from 769 to 1,510 tokens.
DeepSeek-V3 §5, Table 9; DeepSeek-R1 §3; DeepSeek-V4 §5.1.
Kimi K2: synthetic agents, real sandboxes
Real where it matters: code tasks run in real sandboxes, 10,000+ at once on Kubernetes, scored by tests.
RL without a checker: K2 judges its own answers against rubrics; that critic is retrained on rollouts that do have checks.
Guards: per-task token budgets, a loss on hand-picked data against forgetting, temperature decayed over training.
Kimi K2 §3.1–3.2. Tools: 3,000+ real MCP tools plus 20,000+ synthetic ones.
Post-training pipelines side by side
| Model | SFT data | Preference stage | RL and reward | Teacher |
|---|---|---|---|---|
| Llama 2-Chat ’23 | 27.5K human | 1.4M human pairs → 2 RMs | RS, then PPO | 70B → smaller |
| Llama 3 ’24 | RS + synthetic | DPO, 6 rounds | none (PPO dropped) | 405B → 8B, 70B |
| Tülu 3 ’24 | 939K open + synthetic | DPO, on-policy pairs | PPO + verifiers | — |
| DeepSeek-V3 ’24 | 1.5M, R1-distilled | RM writes reasoning | GRPO, rules + RM | internal R1 |
| DeepSeek-R1 ’25 | cold start, then 800K | RM, last 400 steps | GRPO, rules | R1 → Qwen, Llama |
| Olmo 3 Think ’25 | long-trace SFT | delta-learning DPO | GRPO + fixes | Qwen3 traces |
| Kimi K2 ’25 | synthetic agent runs | self-critique rubrics | RLVR + critic | — |
| DeepSeek-V4 ’26 | per specialist | RL-trained judge | GRPO specialists | 10+ → 1 student |
Scroll the table sideways.
The trend: human pairs → synthetic data with checkers; PPO → DPO and GRPO; one model → specialists merged by distillation.
Summarised from the previous six slides and their sources.
RL infrastructure: generation is the bottleneck
Olmo 3 32B: 20 inference nodes vs 8 training; about 5× more GPU time generating than learning.
Continuous batching refills a slot as soon as an answer ends. Static batches wait for the longest.
Async: train on slightly stale samples while the next ones generate. Tülu trains on the second-latest batch.
Interrupted rollouts: V4 logs every token and resumes; restarting from scratch biases toward short answers. K2 pauses long ones and resumes next round.
Olmo 3 §4.4.3 (defaults are its reported numbers), Table 23; Tülu 3 §6.3; DeepSeek-V4 §5.2.3; Kimi K2 §3.3.
Design a feedback pipeline for RLHF, Meta-style
- Prompts: annotators write them; raise difficulty as the model improves.
- Pairs: two answers from different model variants and temperatures, for diversity.
- Labels: which is better plus by how much (4 levels); separate helpfulness and safety guidelines.
- Edit: annotators improve the winner, giving edited > chosen > rejected (Llama 3).
- Freshness: weekly batches from the newest model; old-model data drifts off-distribution.
- Use and audit: train on clear wins only; hold out 1,000 per batch. Vendors differed “markedly”, so audit samples.
Reward-model accuracy was highest on “significantly better” pairs, where annotators also agree most; near-ties are mostly noise.
Llama 2 §3.1, §3.2.1–3.2.2; Llama 3 §4.2.1. Answers “Design a feedback collection pipeline for RLHF” (OpenAI · prep site).
Post-training for code and math: each stage earns its place
| Stage | Use it for | Evidence it helped | Watch for |
|---|---|---|---|
| SFT | format, reasoning style, cold start | R1’s cold start fixed unreadable reasoning | imitation saturates (Olmo: 70.3 → 64.5) |
| DPO | contrast good vs weak answers | Olmo: → 72.9, and a better start for RL | stop early; length drift |
| RLVR | math, code, checkable instructions | Tülu 3: gains on the targeted tests | over-optimisation as KL grows; mix domains |
| Distillation | small models; merging specialists | V3: MATH-500 74.6 → 83.2 | answers 2× longer |
Scroll the table sideways.
How you know: decontaminate against your tests, keep an unseen split (Tülu), track pass@k as well as pass@1 (Olmo), and plot length and KL every run.
Olmo 3 Table 21, §4.4; Tülu 3 §6–7; DeepSeek-V3 Table 9; DeepSeek-R1 §3.
Post-training questions, answered from the papers
- “Walk me through a post-training recipe to turn a pretrained base model into a personalised assistant.” Meta MSL · question bank
- “Round on post-training async RL (architect an asynchronous RL post-training system)” Meta MSL · firsthand 2025
- “Design a feedback collection pipeline for RLHF.” OpenAI · prep site
- “Compare reward-model RLHF and Direct Preference Optimization.” Mistral · question bank
- “How would you adapt a generative AI model to perform well in a specific domain?” Anthropic · question bank
For RLHF vs DPO, cite a lab that switched: Llama 2 used RS + PPO; Llama 3 dropped PPO for DPO on cost and IFEval.
What would you copy for a new model?
| Situation | Copy | From |
|---|---|---|
| First big run, small team | Dense model, batch ramp, simple RS + DPO post-training | Llama 3 |
| Thousands of GPUs | Automated restarts, collective tracing, spare nodes, fast checkpoint resume | Llama 3, OPT |
| MoE | Bias-based balancing, limit nodes per token, overlap expert traffic | DeepSeek-V3 |
| Muon or long runs | Log max logit per head; QK-Clip or Q/K RMSNorm | Kimi K2, V4 |
| Domain model | Branch pretraining into an expert (1T tokens, 85%+ code); test data by annealing | Llama 3 |
| Reasoning model | SFT → delta DPO → RLVR with open data and infra | Olmo 3, Tülu 3 |
| Agents | Tool synthesis, real sandboxes, rubric critic | Kimi K2 |
Scroll the table sideways.
The annealing test for new data
Take the 50%-trained Llama 3 8B, decay its learning rate to 0 over 40B tokens with 30% of the mix from the new dataset, and compare benchmarks. Cheaper than a scaling-law study per dataset.
Llama 3 §3.1.3, §4.3.1; plus the slides above.
The week in one minute
- Parallelism follows bandwidth: TP inside a server, DP across pods (Llama 3: 8 × 16 × 128 = 16,384 GPUs).
- Failures are routine: 419 unexpected stops in 54 days, still above 90% useful time.
- Cheap runs are sparse runs: V3’s 2.788M GPU-hours match its 11× fewer FLOPs.
- Stability fixes are mechanical: lower LR and roll back (OPT), clip logits (K2), clamp outliers (V4).
- Post-training: data with checkers beats more human labels; DPO is cheap; RLVR needs verifiers; distillation merges specialists.
If you can tell each story with its number, Week 8 is done.
Your Week 8 plan: read these sections
| Resource | Read | Hours | Take away |
|---|---|---|---|
| Llama 3 Herd | §3.1–3.4, §4.1–4.2; skim §4.3 | 2.5 | 4D parallelism, Table 5, DPO choice |
| DeepSeek-V3 | §1, 2.1.2, 2.2, 3.1–3.3, 4.2, 4.5, 5 | 2 | cost table, FP8, DualPipe, balancing |
| Kimi K2 | §2, §3, App. D–E | 1.25 | QK-Clip, agentic data and RL |
| OPT-175B chronicles | 4 notes + paper §2.5 | 1 | a real debugging sequence |
| Llama 2 | §3 | 1 | preference data, 2 RMs, RS + PPO, GAtt |
| Tülu 3 | §2, §5 findings, §6 | 1 | RLVR and its failure modes |
| Olmo 3 | §1–2, §4.3–4.4 | 1 | delta learning, RL infrastructure |
| DeepSeek-V4 | card; report §2.4, 4.2.3, 5.1–5.2 | 0.75 | Muon, spike fixes, on-policy distillation |
Scroll the table sideways.
About 10.5 hours. The full papers list 38; weeks 8–9 share 15–20 hours, so read only these sections.
Self-test: answer out loud, then open
Llama 3’s 16,384-GPU layout, and why that order?
TP 8 × CP 1 × PP 16 × DP 128. Innermost groups talk most, so TP stays in the 8-GPU NVLink server; FSDP is outermost because it can prefetch and hide latency.
How often did Llama 3 405B stop unexpectedly?
419 times in 54 days, about every 3.1 hours; ~78% hardware. Still above 90% effective training time, with 3 significant manual interventions.
What does DeepSeek-V3’s $5.576M include?
2.788M H800 GPU-hours at a $2 rental price: 2,664K pretraining, 119K context extension, 5K post-training. Not research, ablations, hardware or R1’s 147K GPU-hours.
How does aux-loss-free balancing work?
A per-expert bias is added to routing scores only for top-k selection, moved ±γ (0.001) each step by load. No loss term; 1B validation loss 2.253 vs 2.258.
What does QK-Clip do, exactly?
After each step, if a head’s max logit S exceeds τ = 100, multiply W_q and W_k by √(τ/S), so its logits scale by τ/S. Only 12.7% of K2’s heads ever triggered.
What warned OPT-175B before divergence?
The FP16 loss scale crashing toward 0 and last-layer activation norms spiking. They rolled back to a healthy checkpoint, lowered the LR and clipped at 0.3.
Why did Llama 3 use DPO instead of PPO?
Less compute at 405B scale and better results, especially on IFEval. They masked format tokens and added a 0.2 NLL term to keep DPO stable.
Why must interrupted RL rollouts not restart from scratch?
Short answers finish before interruptions more often, so restarting biases training toward short outputs. V4 logs tokens and resumes instead.
Week 9: critique and measurement
DAPO’s fixes, whether RL adds new capability, test-time compute scaling, reward hacking, the Leaderboard Illusion, on-policy distillation and async RLHF. Then build a defended critique of one lab’s model, with a fix you could run.
“Your RL run improved pass@1 on AIME by 15 points. Convince me it learned something new and isn't reward hacking or sharpening the base model.”