Frontier-lab prep · LLM internals · Week 8

Frontier recipes at scale:
what labs actually did

How Meta, DeepSeek, Moonshot, Ai2 and the OPT team ran, rescued and post-trained their biggest models, in their own numbers.

About 10 hours with the readings · use → or Space to step through · O for contents · T for dark mode

What this week is for

Answer “how would you run, fix or post-train this?” with what labs reported.

  • “Loss spikes mid-run on a large pretraining job. Walk me through your debugging process.” xAI · question bank
  • “Design a feedback collection pipeline for RLHF.” OpenAI · prep site

Weight: ~2% of interview reports, none of the 18 staff-plus ones; core in research loops.

Senior-staff targets

“Loss spikes at step 40K of a 400B MoE run and one expert's routing share jumps. What do you check first, and what small-scale run would confirm the cause?” l7 bar: diagnose instabilities
“Design post-training for a new base model targeting coding and math. Where do you use SFT, DPO, RLVR and distillation, and how do you know each helped?” l7 bar: design post-training

The life of a frontier run

Each stage below is told through one or two labs that published real numbers: Llama 3, DeepSeek-V3 and V4, Kimi K2, OPT-175B, Llama 2, Tülu 3 and Olmo 3.

Part 1

Running at scale

Parallelism, numerics, failures and money, from Llama 3 405B and DeepSeek-V3.

Llama 3 405B: boring on purpose

Size from a scaling law

Budget 3.8 × 10²⁵ FLOPs. The fit said 402B parameters on 16.55T tokens; they trained 405B on 15.6T.

Flat near the optimum: “about right” is enough.

Dense, not MoE

Chosen “to maximize training stability”. Post-training is SFT + rejection sampling + DPO, not PPO.

RL “tends to be less stable and harder to scale”.

Grow the batch

4M tokens, then 8M after 252M tokens, then 16M after 2.87T.

Result: “few loss spikes”, no interventions.

Changes from Llama 2: 8 KV heads (GQA), no attention across documents, 128K vocabulary (3.17 → 3.94 characters per token), RoPE base 500,000.

Rest of the recipe

126 layers, width 16,384, 128 query heads. Context grown 8K → 128K in six stages on about 800B tokens. Last 40M tokens: learning rate annealed to 0 on upweighted high-quality data, then checkpoints averaged.

Llama 3 Herd, §1, §3.2, Table 3, §3.4. Covers “why GQA, RoPE, SwiGLU?” (Meta MSL).

4D parallelism: which GPUs talk to which

replica 0 (GPUs 0–7) replica 1 (GPUs 8–15) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 TP CP PP DP GPU number = TP + 2·CP + 4·PP + 8·DP (paper's toy example)

TP splits every weight matrix. Talks every layer, so it stays inside one 8-GPU NVLink server.

CP splits a long sequence. Llama 3 all-gathers keys and values, cheap because GQA makes them small.

PP splits the layers into stages.

DP (FSDP) copies the model and splits the batch. Outermost: it hides latency by prefetching, so it can cross pods (1:7 oversubscribed).

Llama 3 Herd, §3.3.1–3.3.2, Fig. 5, Table 4. Widget numbers computed from Table 4.

54 days of a 16K-GPU run: what broke

466 interruptions: 47 planned, 419 unexpected.

That is one unexpected stop every 3.1 hours. Only 3 needed significant manual work; automation handled the rest.

Still above 90% effective training time.

Harder ones: slow “stragglers” that stall thousands of GPUs, a 1–2% daily throughput swing from mid-day heat, and power swings of tens of megawatts when all GPUs pause together.

Verify the table yourself

The counts add to 419, but the printed percentages add to only 94.9%. “Faulty GPU” is 148 of 419 = 35.3%, not the printed 30.1%. By counts, GPU rows are 64% (the text says 58.7%), and GPU + host + host maintenance is 77%, close to the stated “about 78%” hardware. The widget computes from counts.

Llama 3 Herd, Table 5 and §3.3.4.

DeepSeek-V3’s “$5.6M”, as the paper states it

180K GPU-hours per trillion tokens: about 54 days of pretraining on 2,048 H800s.

Excluded, in their words: “prior research and ablation experiments on architectures, algorithms, or data”.

Also not in it: the GPUs themselves (it is a rental price), and R1’s RL runs, costed separately at 147K GPU-hours.

The fair reading: FLOPs explain the gap. 37B active parameters need 11.6× fewer training FLOPs than Llama’s 405B, and the GPU-hours are 11.1× fewer.

DeepSeek-V3 §1, Table 1; DeepSeek-R1 Table 7; Llama 3.1 model card. FLOPs = 6 × active parameters × tokens, computed.

FP8 training in plain terms

FP8 (E4M3) has 4 exponent and 3 fraction bits, max 448. Values are divided by a scale before rounding.

One scale per tensor: one outlier sets it, and small values round to zero.

DeepSeek-V3: one scale per 1×128 activation tile and 128×128 weight block.

Tensor cores keep ~14 bits when summing, ~2% error at 4,096 terms. Fix: add into FP32 every 128 products.

Kept in BF16/FP32: embeddings, output head, MoE router, norms, attention, master weights. Loss stayed within 0.25% of BF16.

512 activations, one outlier (random numbers, made up for illustration)

DeepSeek-V3 §3.3, Fig. 6–7, App. B. The widget rounds to real E4M3 values in your browser.

DualPipe: hide expert traffic behind compute

One chunk at a time computenetwork attn idle mlp idle idle mlp idle attn disp comb comb disp Forward chunk A paired with backward chunk B computenetwork attn mlp mlp attn comb disp disp comb half the time saved time
Blue = forward chunk A, orange = backward chunk B. “disp” and “comb” send tokens to and from experts on other nodes.

Experts spread over nodes made compute and communication take about equal time (1:1).

Each communication now runs while the other chunk computes. 20 of 132 SMs per GPU drive the traffic; each token goes to at most 4 nodes.

Micro-batches enter from both ends of the pipeline, so bubbles shrink too. Cost: two copies of the parameters.

Kimi K2 (1T parameters) rejected DualPipe for that memory cost and overlapped with standard interleaved 1F1B and only 16-way expert parallelism.

DeepSeek-V3 §3.2, Fig. 4–5, Table 2, §3.5.1; Kimi K2 §2.4.2. Slot sizes simplified to equal length.

Balancing experts without an auxiliary loss

γ:

8 experts, each token picks 2. Expert 1 is popular (affinities made up for illustration).

Each expert gets a bias added to its score only when choosing the top experts. The output mix still uses the original score.

After every step: overloaded expert → bias − γ, underloaded → + γ. No extra loss term, so no gradient fights the language-modelling loss.

V3: γ = 0.001, plus a tiny per-sequence loss (α = 0.0001). Validation loss at 1B: 2.258 with a per-sequence aux loss vs 2.253 aux-free.

DeepSeek-V3 §2.1.2, §4.2, §4.5.2–4.5.3. Widget runs the bias-update rule on random batches in your browser.

Three big MoEs, three sets of choices

ChoiceDeepSeek-V3 (Dec 2024)Kimi K2 (Jul 2025)DeepSeek-V4-Pro (Apr 2026)
Total / active671B / 37B1.04T / 32B1.6T / 49B
Routed experts256, pick 8384, pick 8384, pick 6
Attention heads12864: 128 would cost +83% FLOPs at 128Knew compressed sparse attention
OptimizerAdamWMuonClipMuon
Low precisionFP8 computeFP8 only to store activationsFP4 expert weights after training
Extra objectivepredict 1 extra token; 85–90% accepted as a draft, 1.8× decode speednonesame as V3
Serving unit32 GPUs prefill, 320 decode——

Scroll the table sideways.

V3’s own limitation: the deployment unit is “relatively large”, a burden for small teams. That is where a 10× total-to-active MoE hurts.

DeepSeek-V3 §3.4, §4.2, §5.4.3, §6; Kimi K2 §2.3–2.4; DeepSeek-V4 card and §4.2.

Interview check

Can you answer these with numbers?

  • “Mixture of experts (and expert parallelism)” Mistral · firsthand 2026
  • “Explain a mixture-of-experts transformer and how expert parallelism distributes its work. Trace what happens to a batch of tokens from routing…” Mistral · question bank
  • “Discuss trade-offs in MoE routing: top-k vs expert-choice routing, load-balancing vs quality.” Mistral · prep site
  • “Our flagship is a sparse MoE with ~10x more total than active parameters. Why is that architecture a good fit for private enterprise deployment, and where does it hurt?” Cohere · question bank
  • “Explain the architectural choices in a Llama-class model: why grouped-query attention, RoPE and SwiGLU instead of the vanilla 2017 Transformer?” Meta MSL · question bank
Coverage note

Expert-choice routing is week 3 material; this week gives the balance-vs-quality evidence (aux-loss-free balancing) and the deployment cost of a 10× total-to-active MoE (V3’s 32- and 320-GPU serving units).

Part 2

Keeping the loss going down

Loss spikes, divergence and the fixes labs reported, from OPT’s 2021 logbook to DeepSeek-V4.

OPT-175B: two months in the logbook

  1. Oct 18Plan B: fall back to GPT-3’s settings.
  2. Nov 5Run 11.0. Loss explodes; lower LR, tighter clipping.
  3. Nov 5–10Grad-norm explosions; loss scale underflows. GeLU → ReLU.
  4. Nov 11Run 12.00, GPT-3 initialisation. Now restarts are mostly hardware.
  5. Late Nov40+ restarts in two weeks: lost GPUs, memory errors.
  6. Dec 1–2Loss scale at its floor. “SGD-like Adam” silently reloads old settings; real SGD stalls.
  1. Dec 3Back to Adam, lower LR.
  2. Dec 6Grad-norm spikes, perplexity creeping up: LR cut to 2/3 of GPT-3’s.
  3. Dec 19Crosses the ~180B-token epoch: uneventful.
  4. Dec 21Provider deletes the cluster; 16 nodes return broken.
  5. Jan 6Done: 300B tokens, ~90 restarts.
numericshardwaredecision or milestone

OPT chronicles: 10%, 27%, 56% and final notes; OPT-175B logbook (2021-12-02, 12-06, 12-19 entries).

What warned them, and what worked

Early warnings

Before each divergence: the FP16 loss scale crashing toward 0, and the last layer’s activation norm spiking.

Restart rule

Go back to a checkpoint where the loss scale was still ≥ 1.0 and norms were falling. Lower the LR. Clip at 0.3 instead of 1.0.

Didn’t work

Switching to SGD (plateaued). Resetting the loss scale helped only sometimes.

Surprise

An accidental Megatron upgrade (one-line JIT change) shifted activation norms and added 2% throughput.

Hardware: 35+ manual restarts, 100+ hosts swapped. One resume took 50 minutes, 30 of them replaying the data loader. Their lesson: budget compute for the experiments that decide how to scale.

OPT paper §2.5; chronicles 27% and 56% notes; logbook 2021-12-05.

Why attention logits explode

token i: xᵢ token j: xⱼ RMSNorm caps ‖x‖ × W_q × W_k norm growsnorm grows qᵢkⱼ logit = qᵢ·kⱼ ÷ √dsoftmax → weights logit > 1000 one key takes ~all clip: × √(τ/S)clip: × √(τ/S)

Normalised inputs keep ‖x‖ in check, so a logit can only grow if ‖W_q‖ × ‖W_k‖ grows. Both weights multiply, so growth compounds.

Muon gives every direction of an update equal size (full rank); Adam’s updates are dominated by a few. Moonshot’s hypothesis: Muon pushes the top singular values up more often.

Vanilla Muon at 9B active / 53B total: max logits above 1000 quickly, then spikes and occasional divergence.

QK-Clip: after each optimizer step, if a head’s max logit S exceeds τ, shrink W_q and W_k so the logit lands back on τ.

Kimi K2 §2.1, Fig. 2, App. E. Soft-capping lets q·k keep growing underneath; QK-Norm needs full keys, which MLA never builds at inference.

QK-Clip, live: one head, one step

Scaling W_q and W_k each by √γ, with γ = τ ÷ S, scales every logit of that head by γ. Only heads above τ are touched.

For MLA: per-head query and key parts × √γ, per-head rotary query × γ, the shared rotary key untouched.

Kimi K2, τ = 100: 12.7% of heads triggered in the first 70,000 steps; after that none did. 15.5T tokens with no loss spike.

Ablation at 0.5B active / 3B total with an aggressive τ = 30: negligible effect on loss.

attention before clip
after clip
Six keys; their score pattern is made up for illustration.

Kimi K2 §2.1, Algorithm 1, App. D. DeepSeek-V4 skips QK-Clip: it normalises queries and keys with RMSNorm instead (§2.4).

Five runs, five stability stories

RunWhat went wrongFix they reported
OPT-175B (2021)Loss-scale collapse, activation-norm spikesRoll back, lower LR, clip 0.3
Llama 3 405B (2024)“Few loss spikes”Small batch early (4M → 16M), FP32 gradient sums
DeepSeek-V3 (2024)No rollbacks. In a test, coarse FP8 on activation gradients diverged (16B)Fine-grained FP8 scaling
Kimi K2 (2025)Muon logits above 1000 at mid scaleQK-Clip, τ = 100: zero spikes
DeepSeek-V4 (2026)Spikes from MoE outliers, fed by routing; rollbacks didn’t stop repeatsAfter a spike, route with older weights (~20% slower while on); clamp SwiGLU to ±10

Scroll the table sideways.

V4’s own words: the principles behind both fixes “remain insufficiently understood”.

OPT §2.5; Llama 3 §3.3.2, §3.4.1; DeepSeek-V3 §1, App. B.2; Kimi K2 §2.1; DeepSeek-V4 §4.2.3, §6.

Senior-staff angle

The 40K-step spike with one hot expert

  1. Freeze the evidence. The batch, per-layer grad norms, max logit per head, per-expert load and bias, FP8 scales.
  2. Split the causes. Replay the batch on an earlier checkpoint (data?). Check logits (optimizer?) and MoE outliers (as V4 found).
  3. Buy time. Roll back a few hundred steps, skip the batch. V4: rollback alone doesn’t stop repeats.
  4. Confirm cheaply. Reproduce on a proxy with the same data order (Kimi saw logit blow-up at 9B active, V3 FP8 divergence at 16B), then add the fix.

Determinism helps: V4 built bit-reproducible kernels partly so spikes can be traced to a numerical cause.

A plan built from the reports above: DeepSeek-V4 §3.3, §4.2.3; Kimi K2 §2.1, App. D; DeepSeek-V3 App. B.2.

Interview check

Three ways labs ask the same question

  • “Your pretraining loss suddenly diverges at step 300k of a long run. Diagnose and fix it.” Google DeepMind · question bank
  • “Loss spikes mid-run on a large pretraining job. Walk me through your debugging process.” xAI · question bank
  • “Your training run's loss is diverging after step 10K. Walk me through your debugging process.” OpenAI · prep site

Name a real case for each branch: hardware (Llama 3’s 419 stops), numerics (OPT’s loss scale, V3’s FP8 test), optimizer (Kimi’s logits), routing (V4’s MoE outliers).

Part 3

Post-training pipelines, as published

Llama 2 → Llama 3 → Tülu 3 → Olmo 3, DeepSeek and Kimi: what each stage was for.

Llama 2-Chat: a weekly loop with two judges

Latest chat modelRLHF V1 → V5 Sample 2 answersvaried models, temperatures Annotators pick+ how much better (4 levels)helpful or safety · weekly 2 reward modelshelpfulness RM, safety RMmargin grows with “how much” Rejection samplingbest of K from the 70B,fine-tune on the winners PPO (from V4 on)safety RM if flagged and < 0.15minus β × KL

27,540 human-written SFT examples were enough to start.

1.42M human comparisons, collected from the newest model so the reward model stays on-distribution.

Two RMs because helpfulness and safety pull apart; each won on its own test set, beating GPT-4 as a judge.

RS only on the 70B; smaller models train on its outputs (distillation).

Lesson: V3 trained only on V2’s samples forgot how to rhyme. Fix: keep the best samples from all past rounds.

Llama 2 §3.1–3.2, Table 6. PPO β = 0.01 (7B, 13B) and 0.005 (34B, 70B); safety threshold 0.15 = precision 0.89, recall 0.55.

Ghost Attention: keep a system instruction alive

1. Sample with the instruction on every turn inst + u1 a1 inst + u2 a2 inst + u3 a3 2. Keep it only on turn 1 in the training copy inst + u1 a1 u2 a2 u3 a3 3. Loss only on the last answer inst + u1 a1 · 0 u2 a2 · 0 u3 a3 · loss

Problem: early RLHF models forgot “act as Napoleon” after a few turns.

The answers were written while seeing the instruction every turn, so they obey it.

The training copy shows it once, so the model must carry it across turns.

Earlier turns get zero loss: those assistant messages were made under a different context.

Instructions were random mixes of hobbies, languages and public figures. Applied after RLHF V3; held for 20+ turns.

Llama 2 §3.3, Fig. 9–10.

Llama 3: six rounds of reward model → RS → SFT → DPO

Why DPO, not PPO

DPO “required less compute for large-scale models and performed better”, especially on IFEval (instruction following).

Two DPO fixes

Mask chat-format tokens (in both answers, they caused repetition). Add 0.2 × likelihood loss on the chosen answer.

Data does the work

Best of 10–30 samples per prompt. Targets are mostly model-written, so filter: emoji, “I’m sorry”, RM top quartile.

Synthetic code needs a checker: the 405B trained on its own code “is not helpful (and can even degrade)”. With linters and generated tests: ~1M dialogues, 20% self-fixed after feedback.

Two smaller findings

Only 0.1% long synthetic examples in the SFT mix kept 128K-context skills from regressing. Final models average checkpoints from runs with different data and hyperparameters, at each of the RM, SFT and DPO stages.

Llama 3 Herd §4.1–4.3. DPO β = 0.1, LR 10⁻⁵; SFT 8.5K–9K steps.

Tülu 3: an open SFT → DPO → RLVR recipe

RLVR = RL with verifiable rewards: PPO where the reward is 10 if a checker accepts the answer, else 0. About 30K prompts from GSM8K, MATH and checkable instructions.

Findings: value model initialised from a general RM worked best; adding RM scores to the check made it worse; more KL drift usually lowered the average score (over-optimisation).

They kept an “unseen” evaluation split they never looked at during development, to measure how much they overfit their own choices.

Tülu 3 §2, §5, §6. Final 405B RLVR run: 46 hours on 256 GPUs.

Olmo 3: every stage, dataset and checkpoint released

Delta learning: more SFT on Qwen3-32B answers hurt (average 70.3 → 64.5). DPO preferring them over Qwen3-0.6B answers helped (→ 72.9). The gap teaches, not the answer.

RL: GRPO with recent fixes: drop all-same-reward groups, token-level loss, no KL, no std normalisation. Starting from DPO beat starting from SFT.

Cost, honestly framed: about 56 days on 1,024 H100s ($2.75M at $2/hour), of which post-training took about 9 days of sweeps.

Olmo 3 §2, §4.3–4.4, Table 21. Released: data mixes, 9T-token source pool, per-stage checkpoints, code.

DeepSeek: distil a reasoner, then merge specialists

V3 · Dec 2024

1.5M SFT examples; reasoning data from an internal R1, trimmed for length. GRPO with rule checks, else a reward model that reasons first.

R1 · 2025

Cold start → RL → 600K + 200K rejection-sampled SFT → RL on everything. Preference reward only in the last 400 of 1,700 steps: longer led to reward hacking.

V4 · Apr 2026

Specialists (SFT + GRPO) per domain, then on-policy distillation: one student learns from 10+ teachers on its own samples. Replaced the mixed RL stage.

The distillation trade-off, measured: adding R1 data to V2.5 raised MATH-500 from 74.6 to 83.2, and answer length from 769 to 1,510 tokens.

DeepSeek-V3 §5, Table 9; DeepSeek-R1 §3; DeepSeek-V4 §5.1.

Kimi K2: synthetic agents, real sandboxes

Real where it matters: code tasks run in real sandboxes, 10,000+ at once on Kubernetes, scored by tests.

RL without a checker: K2 judges its own answers against rubrics; that critic is retrained on rollouts that do have checks.

Guards: per-task token budgets, a loss on hand-picked data against forgetting, temperature decayed over training.

Kimi K2 §3.1–3.2. Tools: 3,000+ real MCP tools plus 20,000+ synthetic ones.

Post-training pipelines side by side

ModelSFT dataPreference stageRL and rewardTeacher
Llama 2-Chat ’2327.5K human1.4M human pairs → 2 RMsRS, then PPO70B → smaller
Llama 3 ’24RS + syntheticDPO, 6 roundsnone (PPO dropped)405B → 8B, 70B
Tülu 3 ’24939K open + syntheticDPO, on-policy pairsPPO + verifiers—
DeepSeek-V3 ’241.5M, R1-distilledRM writes reasoningGRPO, rules + RMinternal R1
DeepSeek-R1 ’25cold start, then 800KRM, last 400 stepsGRPO, rulesR1 → Qwen, Llama
Olmo 3 Think ’25long-trace SFTdelta-learning DPOGRPO + fixesQwen3 traces
Kimi K2 ’25synthetic agent runsself-critique rubricsRLVR + critic—
DeepSeek-V4 ’26per specialistRL-trained judgeGRPO specialists10+ → 1 student

Scroll the table sideways.

The trend: human pairs → synthetic data with checkers; PPO → DPO and GRPO; one model → specialists merged by distillation.

Summarised from the previous six slides and their sources.

RL infrastructure: generation is the bottleneck

Olmo 3 32B: 20 inference nodes vs 8 training; about 5× more GPU time generating than learning.

Continuous batching refills a slot as soon as an answer ends. Static batches wait for the longest.

Async: train on slightly stale samples while the next ones generate. Tülu trains on the second-latest batch.

Interrupted rollouts: V4 logs every token and resumes; restarting from scratch biases toward short answers. K2 pauses long ones and resumes next round.

Olmo 3 §4.4.3 (defaults are its reported numbers), Table 23; Tülu 3 §6.3; DeepSeek-V4 §5.2.3; Kimi K2 §3.3.

Senior-staff angle

Design a feedback pipeline for RLHF, Meta-style

  1. Prompts: annotators write them; raise difficulty as the model improves.
  2. Pairs: two answers from different model variants and temperatures, for diversity.
  3. Labels: which is better plus by how much (4 levels); separate helpfulness and safety guidelines.
  1. Edit: annotators improve the winner, giving edited > chosen > rejected (Llama 3).
  2. Freshness: weekly batches from the newest model; old-model data drifts off-distribution.
  3. Use and audit: train on clear wins only; hold out 1,000 per batch. Vendors differed “markedly”, so audit samples.

Reward-model accuracy was highest on “significantly better” pairs, where annotators also agree most; near-ties are mostly noise.

Llama 2 §3.1, §3.2.1–3.2.2; Llama 3 §4.2.1. Answers “Design a feedback collection pipeline for RLHF” (OpenAI · prep site).

Senior-staff angle

Post-training for code and math: each stage earns its place

StageUse it forEvidence it helpedWatch for
SFTformat, reasoning style, cold startR1’s cold start fixed unreadable reasoningimitation saturates (Olmo: 70.3 → 64.5)
DPOcontrast good vs weak answersOlmo: → 72.9, and a better start for RLstop early; length drift
RLVRmath, code, checkable instructionsTülu 3: gains on the targeted testsover-optimisation as KL grows; mix domains
Distillationsmall models; merging specialistsV3: MATH-500 74.6 → 83.2answers 2× longer

Scroll the table sideways.

How you know: decontaminate against your tests, keep an unseen split (Tülu), track pass@k as well as pass@1 (Olmo), and plot length and KL every run.

Olmo 3 Table 21, §4.4; Tülu 3 §6–7; DeepSeek-V3 Table 9; DeepSeek-R1 §3.

Interview check

Post-training questions, answered from the papers

  • “Walk me through a post-training recipe to turn a pretrained base model into a personalised assistant.” Meta MSL · question bank
  • “Round on post-training async RL (architect an asynchronous RL post-training system)” Meta MSL · firsthand 2025
  • “Design a feedback collection pipeline for RLHF.” OpenAI · prep site
  • “Compare reward-model RLHF and Direct Preference Optimization.” Mistral · question bank
  • “How would you adapt a generative AI model to perform well in a specific domain?” Anthropic · question bank

For RLHF vs DPO, cite a lab that switched: Llama 2 used RS + PPO; Llama 3 dropped PPO for DPO on cost and IFEval.

What would you copy for a new model?

SituationCopyFrom
First big run, small teamDense model, batch ramp, simple RS + DPO post-trainingLlama 3
Thousands of GPUsAutomated restarts, collective tracing, spare nodes, fast checkpoint resumeLlama 3, OPT
MoEBias-based balancing, limit nodes per token, overlap expert trafficDeepSeek-V3
Muon or long runsLog max logit per head; QK-Clip or Q/K RMSNormKimi K2, V4
Domain modelBranch pretraining into an expert (1T tokens, 85%+ code); test data by annealingLlama 3
Reasoning modelSFT → delta DPO → RLVR with open data and infraOlmo 3, Tülu 3
AgentsTool synthesis, real sandboxes, rubric criticKimi K2

Scroll the table sideways.

The annealing test for new data

Take the 50%-trained Llama 3 8B, decay its learning rate to 0 over 40B tokens with 30% of the mix from the new dataset, and compare benchmarks. Cheaper than a scaling-law study per dataset.

Llama 3 §3.1.3, §4.3.1; plus the slides above.

The week in one minute

  1. Parallelism follows bandwidth: TP inside a server, DP across pods (Llama 3: 8 × 16 × 128 = 16,384 GPUs).
  2. Failures are routine: 419 unexpected stops in 54 days, still above 90% useful time.
  3. Cheap runs are sparse runs: V3’s 2.788M GPU-hours match its 11× fewer FLOPs.
  4. Stability fixes are mechanical: lower LR and roll back (OPT), clip logits (K2), clamp outliers (V4).
  5. Post-training: data with checkers beats more human labels; DPO is cheap; RLVR needs verifiers; distillation merges specialists.

If you can tell each story with its number, Week 8 is done.

Your Week 8 plan: read these sections

ResourceReadHoursTake away
Llama 3 Herd§3.1–3.4, §4.1–4.2; skim §4.32.54D parallelism, Table 5, DPO choice
DeepSeek-V3§1, 2.1.2, 2.2, 3.1–3.3, 4.2, 4.5, 52cost table, FP8, DualPipe, balancing
Kimi K2§2, §3, App. D–E1.25QK-Clip, agentic data and RL
OPT-175B chronicles4 notes + paper §2.51a real debugging sequence
Llama 2§31preference data, 2 RMs, RS + PPO, GAtt
Tülu 3§2, §5 findings, §61RLVR and its failure modes
Olmo 3§1–2, §4.3–4.41delta learning, RL infrastructure
DeepSeek-V4card; report §2.4, 4.2.3, 5.1–5.20.75Muon, spike fixes, on-policy distillation

Scroll the table sideways.

About 10.5 hours. The full papers list 38; weeks 8–9 share 15–20 hours, so read only these sections.

Self-test: answer out loud, then open

Llama 3’s 16,384-GPU layout, and why that order?

TP 8 × CP 1 × PP 16 × DP 128. Innermost groups talk most, so TP stays in the 8-GPU NVLink server; FSDP is outermost because it can prefetch and hide latency.

How often did Llama 3 405B stop unexpectedly?

419 times in 54 days, about every 3.1 hours; ~78% hardware. Still above 90% effective training time, with 3 significant manual interventions.

What does DeepSeek-V3’s $5.576M include?

2.788M H800 GPU-hours at a $2 rental price: 2,664K pretraining, 119K context extension, 5K post-training. Not research, ablations, hardware or R1’s 147K GPU-hours.

How does aux-loss-free balancing work?

A per-expert bias is added to routing scores only for top-k selection, moved ±γ (0.001) each step by load. No loss term; 1B validation loss 2.253 vs 2.258.

What does QK-Clip do, exactly?

After each step, if a head’s max logit S exceeds τ = 100, multiply W_q and W_k by √(τ/S), so its logits scale by τ/S. Only 12.7% of K2’s heads ever triggered.

What warned OPT-175B before divergence?

The FP16 loss scale crashing toward 0 and last-layer activation norms spiking. They rolled back to a healthy checkpoint, lowered the LR and clipped at 0.3.

Why did Llama 3 use DPO instead of PPO?

Less compute at 405B scale and better results, especially on IFEval. They masked format tokens and added a 0.2 NLL term to keep DPO stable.

Why must interrupted RL rollouts not restart from scratch?

Short answers finish before interruptions more often, so restarting biases training toward short outputs. V4 logs tokens and resumes instead.

Next

Week 9: critique and measurement

DAPO’s fixes, whether RL adds new capability, test-time compute scaling, reward hacking, the Leaderboard Illusion, on-policy distillation and async RLHF. Then build a defended critique of one lab’s model, with a fix you could run.

Staff-level probe to aim for

“Your RL run improved pass@1 on AIME by 15 points. Convince me it learned something new and isn't reward hacking or sharpening the base model.”

Week 9 →  ·  ← Week 7