Frontier-lab prep · LLM internals · Week 4

Scaling arithmetic
and pretraining

How big a model, how many tokens, how many GPU-days, what data, and how to keep the run from blowing up.

10 hours with the readings · use → or Space to step through · O for contents · T for dark mode

What this week is for

Turn a compute budget into a model, a token count and a GPU calendar. Then keep the run alive.

Senior-staff target

“You have 10^25 FLOPs and expect 10^13 inference tokens a month. How big a model, how many tokens, how many GPU-days, and why not Chinchilla-optimal?”

How labs ask it

  • “What do the Chinchilla scaling laws say, and how do they differ from earlier scaling intuitions?” Anthropic · question bank
  • “Your pretraining loss suddenly diverges at step 300k of a long run. Diagnose and fix it.” Google DeepMind · question bank
How much this matters, and the second target

LLM internals is about 2% of collected interview reports and appears in none of the 18 staff-plus reports. It is core in research and research-engineer loops. Second target, worked on the stability slides: a loss spike in a 400B MoE run; what do you check first, and what small run confirms the cause?

Part 1

Compute-optimal scaling: Chinchilla

Same question, two answers

Fixed compute budget: spend it on parameters or on training tokens?

10× more computeparamstokens
Kaplan (2020)× 5.5× 1.8
Chinchilla (2022)× 3.2× 3.2

Chinchilla: grow both equally, so each gets √10 ≈ 3.2×.

Why Kaplan over-sized: one long learning-rate schedule for every run, so short runs looked worse than they were. Most of its models were under 100M parameters.

learning rate training tokens → 130B schedule sized for 130B stop at 40B still high LR here: loss reads too high matched schedule decays
Schematic. Chinchilla found the best loss comes from decaying the LR (10×) over exactly the tokens you train; overshooting by more than 25% hurts.

Hoffmann et al. (2022), §1, §2, App. B and Fig. A1.

One budget, many models: the valley is flat

Each point spends the same FLOPs: a bigger model sees fewer tokens.

Too small: capacity runs out. Too big: too few tokens.

The bottom is broad. At 10^25, a model ¼ the 20:1 size predicts almost the same loss (+0.001). That slack is what over-training spends.

Loss from Chinchilla’s fitted formula (§3.3, App. D.2) with unrounded exponents α = 0.336, β = 0.283 as used by Sardana et al. (2024), App. A. Computed in your browser.

The “20 tokens per parameter” rule

Three fitting methods agree: parameters and tokens each grow as compute0.5 (0.50, 0.49, 0.46).

ParamsCompute (FLOPs)Tokens
1B1.2 × 10²⁰20B
10B1.2 × 10²²205B
67B5.8 × 10²³1.5T
280B9.9 × 10²⁴5.9T
1T1.3 × 10²⁶21.2T

The test

Same compute as Gopher (280B on 300B tokens). Chinchilla: 70B on 1.4T.

MMLU 5-shot: 67.6% vs 60.0%. Better on 51 of 57 subjects.

And 4× cheaper to serve, which Part 2 turns into the main argument.

Hoffmann et al. (2022): Table 2 (exponents), Table 3 (Approach 1 projections), §4, Table 6, Fig. 6.

Where the 20:1 rule is shaky

The fitted loss: 1.69 (text you can never predict) + a size penalty that shrinks with parameters + a data penalty that shrinks with tokens.

  • Method 3 (the formula) puts the optimum at smaller models: about 150B on 11T tokens at 10^25, not 290B on 5.8T.
  • Only two large runs tested it; all runs used less than one pass over the data.
  • The frontier bends at large budgets, so even smaller models may be optimal (App. E).
  • A replication found the reported confidence intervals implausibly narrow.
  • It counts training compute only. Serving is free in this model.
The formula, with numbers

Loss = 1.69 + 406.4 ÷ N0.34 + 410.7 ÷ D0.28. With the unrounded exponents α = 0.336, β = 0.283 it predicts 1.93 for Chinchilla (70B, 1.4T) and 1.98 for Gopher (280B, 300B); the 150B/11T optimum above comes from the same formula.

Hoffmann et al. (2022) §3.4, §5, App. D.2, App. E; replication: Besiroglu et al. (2024), as summarised in Sardana et al. (2024) §7.

Interview check

Can you answer these in two minutes?

  • “What do the Chinchilla scaling laws say, and how do they differ from earlier scaling intuitions?” Anthropic · question bank
  • “Transformer internals and scaling laws, with emphasis on the ‘why’ behind design choices” OpenAI · firsthand 2026
Two-minute skeleton: equal scaling (√10 each per 10× compute) → ~20 tokens/param → Kaplan’s schedule artifact → Chinchilla beat Gopher at equal compute → but it ignores serving, so labs over-train (Part 2).

Questions from the curriculum’s question bank (2026).

Part 2

Over-training on purpose: count the serving bill

Add the serving bill to the objective

Chinchilla: fix compute, minimise loss.

Sardana et al.: fix the quality (loss), minimise 6 × params × training tokens + 2 × params × lifetime inference tokens.

Training costs 6 FLOPs per parameter per token; serving costs 2.

The higher the target quality, the more demand it takes to move the optimum. Assumes demand does not depend on model size.

Target qualityInference tokensCheapest modelSaved
7B-Chinchilla10¹¹6B, 1.18× data—
30B-Chinchilla10¹³13.6B, 2.84× data28%
70B-Chinchilla10¹³41.6B on 7.9T12%

Sardana et al. (2024), Eq. 3, §2, Table 2. The 30B row is reproduced live on the next slide.

Lifetime cost against model size, at fixed quality

Quality of a 20:1 model trained withFLOPs
Count

━ total · ━ training · ┅ inference

Moving left, each model needs more tokens to reach the same loss, so training rises while serving falls.

GPU-hours: training and prompts at 50% MFU, output tokens at the decode MFU you pick, 70 in / 215 out tokens per request (Sardana §6).

Method: Sardana et al. (2024) Eq. 3 and Eq. 6, with Chinchilla’s fitted loss. Computed in your browser.

FLOPs are not dollars, and over-training keeps paying

Count GPU time, not FLOPs

At 2T inference tokens, a 70B Chinchilla model needs only 1.3% more FLOPs than the best model, but costs 36% more.

Why: each output token runs at ~1% MFU versus ~50% in training, a 50× gap the FLOP count hides.

No saturation up to 10,000 tokens/param

47 models, 150M–6B: loss and downstream scores kept improving.

But formulas fitted on ≤100 tokens/param overestimate the gain from extra data at extreme ratios.

ModelTokensTokens per parameter
Chinchilla 70B1.4T20
Llama 3 8B15T~1,900
SmolLM3 3B11T~3,700

Sardana et al. (2024) §4–6; Hoffmann et al. (2022); Smol Training Playbook (“Scaling Laws: How Many Parameters”). Ratios computed.

Part 3

A training run on paper

Where “6 × parameters × tokens” comes from

Forwardalso inference X × W = Y 2 · T·P·M Weight gradfor the update Xᵀ × dY = dW 2 · T·P·M Input gradfor the layer below dY × Wᵀ = dX 2 · T·P·M Sum: 6 × (P·M parameters) × (T tokens)
T tokens pass through one P × M weight matrix. A multiply-add counts as 2 FLOPs.

Every weight matrix follows this pattern, so a training step costs about 6 FLOPs per parameter per token.

Attention scores add work that grows with context: about context ÷ (8 × width) of the matmul FLOPs. Width 8,192 at 8K context: 12.5%.

Recomputing activations in the backward pass (to save memory) pushes it toward 8 × N × D.

Austin et al., “How To Scale Your Model”, Part 4 (Transformer Math): forward/reverse FLOPs, attention ratio T/8D, gradient checkpointing.

Rooflines: why training is compute-bound and decoding is not

weights

Arithmetic intensity = FLOPs per byte read from memory. Below the ridge, the chip waits on memory.

A matmul’s intensity ≈ the number of tokens sharing one weight load.

Training and prompts: thousands of tokens per load. Decoding: one token per user, so you batch ~300 users to stop wasting FLOPs. Attention over the KV cache stays near 1–8 FLOPs/byte.

Matrix: Llama 3 70B MLP, 8,192 × 28,672. Peaks and bandwidths from “How To Scale Your Model” Parts 1, 7 and 12 (H100: 989 TFLOP/s bf16, 3.35 TB/s → ridge ≈ 295).

Cluster size is set by FLOPs and calendar, not memory

Llama 3 70B on paper

6 × 70B × 15T ≈ 6.3 × 10²⁴ FLOPs.

Memory: weights 140 GB + Adam state 560 GB + saved activations ~10.5 TB ≈ 11 TB. That fits on ~117 TPU v5p chips, which would take ~3,400 days.

On 8,960 chips at 40% MFU: about 44 days.

What utilisation to assume

MFU (model FLOPs utilisation) = useful FLOPs ÷ peak FLOPs.

  • Raw H100 bf16 matmuls: 714–758 of 990 TFLOP/s
  • End-to-end training runs: 20–41%
  • DeepSeek-V3 estimate: ~22%
  • SmolLM3 planned: 30%

“How To Scale Your Model” Part 6 (LLaMA 3 training) and Part 4 Q7 (DeepSeek-V3); Smol Training Playbook, “Compute Units and FLOPs” and “Scaling Surprises”.

Compute-budget calculator

GPU cluster

GPU-time = FLOPs ÷ (dense bf16 peak × MFU). Peaks from “How To Scale Your Model” Part 12 table (A100 312, H100/H200 990, B200 2,250 TFLOP/s). Loss from Chinchilla’s fit.

Serving on paper: decoding reads memory

Each decode step must read every weight, plus every user’s KV cache. So:

step time ≥ bytes read ÷ memory bandwidth

FLOPs only matter once a large batch shares each weight load.

Llama 3 70B, bf16, 8 × H100

Weights 141 GB ÷ 26.8 TB/s ≥ 5.3 ms per step.

Batch of 32 users at 8K context: KV cache 32 × 8,192 × 320 KiB = 80 GiB, +3.2 ms.

The math for those 32 tokens: 0.6 ms. Memory-bound by ~15×.

Formula: “How To Scale Your Model” Part 7 (theoretical minimum step time). KV bytes per token from Week 1 (2 × 80 × 8 × 128 × 2). Numbers computed.

Senior-staff probe · 1 of 3

10^25 FLOPs: the Chinchilla answer and the calendar

Assumptions, said out loud

  • Dense model; FLOPs = 6 × N × D
  • H100 at 989 TFLOP/s bf16, 40% MFU
  • 10^13 tokens a month, served for 12 months
  1. 20:1 rule: 6 × N × 20N = 10^25 → N ≈ 290B, D ≈ 5.8T tokens. (The fitted formula says ~150B on 11T; same valley.)
  2. GPU time: 10^25 ÷ (989 × 10¹² × 0.4) = 2.5 × 10¹⁰ GPU-seconds = 7.0M H100-hours ≈ 293K GPU-days.
  3. Calendar: 36 days on 8,192 H100s.
  4. Budget extra: SmolLM3’s ablations and debugging cost 58% of its main run.

Hoffmann et al. (2022); peak from “How To Scale Your Model” Part 1; MFU range from the Smol Training Playbook (161,280 of 276,480 GPU-hours). Arithmetic computed.

Senior-staff probe · 2 of 3

Same 10^25 FLOPs, four model sizes

10^13 tokens a month is 3.9M tokens/s on average. At 290B that costs 58% of the whole training budget every month.

The loss column barely moves from 290B down to 70B; the serving columns move 4×. Losses are the fitted formula’s, less reliable at 340+ tokens/param.

Computed in your browser: D = 10^25 ÷ 6N; serving FLOPs = 2 × N × tokens; H100s at 40% of 989 TFLOP/s for average load; weights at 2 bytes per parameter on 80 GB GPUs.

Senior-staff probe · 3 of 3

The answer you give

“Train about 70B on ~24T tokens: same 10^25 FLOPs, same 36 days. Predicted loss within 0.002 of the 290B, but serving takes ~1,400 H100s instead of ~5,600.”

Why not Chinchilla-optimal: it minimises training compute only. Here a year of serving costs 7× the training run.

What would change my answer

  • Data: 24T good unique tokens is a lot (FineWeb: 15T from 96 crawls).
  • Fit optimism at 340 tokens/param: hedge toward ~100B.
  • Demand 10× lower: the optimum moves up to ~100B (lifetime-cost slide).

Sardana et al. (2024) §4–5; Penedo et al. (2024) §3.7; Smol Training Playbook, “The Unintuitive Nature of Data Mixtures”. Numbers from the previous two slides.

Interview check

Questions this arithmetic answers

  • “Walk me through what happens, tensor by tensor, in one forward pass of a decoder-only transformer.” Add the FLOPs per step. Anthropic · question bank
  • “Walk me through matrix manipulations relevant to LLM architectures.” Anthropic · question bank
  • “Explain self-attention. What is its computational complexity, and what are your options when contexts get long?” Give the cost side: T ÷ 8D, KV reads. OpenAI · question bank

Questions from the curriculum’s question bank (2026). Week 1 covers the tensor shapes; this week adds the FLOPs and bytes.

Part 4

Data: building FineWeb

From 96 Common Crawl snapshots to 15T tokens

36T
tokens after base filtering
20T
after per-snapshot MinHash dedup
15T
FineWeb; the Edu filter keeps 1.3T

Each step raised the ablation score. Text extraction alone (trafilatura on raw HTML instead of Common Crawl’s text files) was a clear win.

Penedo et al. (2024) §3.2–3.7, Fig. 1–9.

How they decided, and the dedup surprise

Every choice was an ablation

1.71B models, 28B tokens per filter test, 350B for dedup. Two seeds each.

Benchmarks kept only if low-noise, rising during training and above random. 70+ models, ~80,000 H100-hours.

Global dedup made things worse

Deduping all 96 crawls against each other left 4T tokens and barely helped.

In an old crawl, the 10% it kept trained worse than the 90% it removed: ads and keyword lists survive because they are unique.

Per-snapshot dedup: 20T tokens, matched RefinedWeb.

The gain comes from removing huge duplicate clusters. Small ablations can’t see dedup: in a 1B-token sample almost every document looks unique.

Penedo et al. (2024) §3.1, §3.4, Fig. 3–5, App. E.2 (Fig. 14).

MinHash: how “75% similar” becomes a hash lookup

Hash each document’s word 5-grams with 112 functions; keep the minimum of each. Two documents agree on one value with probability = their similarity.

Group the 112 values into 14 bands of 8. A match on any whole band flags a duplicate.

More hashes give a sharper cutoff (RefinedWeb used 9,000) at more compute and storage.

Penedo et al. (2024) §3.4 and App. E.1 (match rates 56%, 77%, 92%, 98.8% at 0.70–0.85, reproduced here). Computed in your browser.

FineWeb-Edu: a quality filter trained on LLM labels

Result (1.71B, 350B tokens): MMLU 33% → 37%, ARC 46% → 57%. Reaches the MMLU of Matrix, the next-best open set, with ~10× fewer tokens.
Cost: topics shift toward education and history, away from business, entertainment and travel. Plain FineWeb fits broad web text better. The threshold of 3 balanced knowledge tasks against HellaSwag.

Penedo et al. (2024) §4, Fig. 10–12, 17, 18.

Designing streaming dedup at web scale

  1. Sign: per document, 5-gram shingles → 112 MinHash values → 14 band keys.
  2. Stream: look up each band key in a sharded key-value store. Hit = candidate duplicate; miss = insert. One pass, 14 keys per document.
  3. Cluster: merge candidates transitively (union-find); keep one per cluster.
  4. Scope: per crawl window, not global (FineWeb’s ablation). URL-level and line-level global dedup also lost.
  5. Prove it: ablate at 350B tokens; small samples hide dedup effects.
Extending to a petabyte social-media pipeline with safety filtering

Same skeleton, with stages you ablate one at a time: language ID, URL blocklists, rule filters with thresholds chosen from histograms of good vs bad data (FineWeb’s method for its custom filters), PII anonymisation (FineWeb anonymised emails and IP addresses), and model-based quality or safety classifiers trained on LLM labels. Shuffle at the sequence level downstream: SmolLM3 saw loss spikes when one long low-quality file filled a batch.

Design is this deck’s, built from Penedo et al. (2024) §3.4, §3.6, App. E.1–E.3 and the Smol Training Playbook (“Mystery #3”).

Interview check

Data questions labs ask

  • “Design a deduplication pipeline for a web-scale pretraining corpus. It has to run as a streaming process.” xAI · question bank
  • “Design the data pipeline ingesting X content at petabyte scale for model training with safety filtering.” xAI · prep site
Senior-staff move: name the ablation that proves each stage (model size, tokens, seeds, benchmarks), and the one you can’t run small (dedup).

Questions from the curriculum’s question bank (2026).

Part 5

Keeping the run stable

Two known instabilities, and where the fixes sit

Attention, every layer q, kprojections q·k ÷ √dlogits softmaxweights |q|, |k| grow → logits grow with weight scale² → one-hot attention (entropy collapse) → divergence QK-normnormalise q, k Output, once hiddenlast layer logits yone per token log Zsoftmax sum late in training, no weight decay: logits drift very negative, log Z wanders far from 0 → loss diverges z-loss+10⁻⁴·log²Z Weight decay also prevented it in the larger models tested.

Wortsman et al. (2023) §3.1.1–3.1.2, App. C, Fig. E.1 (growth comes from query/key norms, not their alignment).

Watch attention collapse as weights grow

Logits multiply two projections, so they grow with the square of the weight scale.

In the paper’s runs, every model whose max attention logit passed 10⁴ diverged. Forcing logits to ~10³ in a 10M model already hurt; at 10⁴ it did worse than a zero-layer model (embeddings only).

Toy: one query, 8 random keys, head size 16 (fixed seed). Real QK-norm has a learned gain.

Wortsman et al. (2023) §3.1.1, §3.3, Fig. 9–10. Computed in your browser.

The cheap proxy: sweep the learning rate on small models

final loss 3e-41e-23e-1 learning rate (log) → larger model, no QK-norm small with QK-norm: flat
Schematic, shape only, after the paper’s Fig. 1. Not data.
  • LR sensitivity: average gap to the best loss across 3e-4 to 3e-1. It rises with model size.
  • QK-norm let a 1.2B model train at LR 0.3.
  • Longer warmup and LR-independent weight decay lower sensitivity. Depth raises it faster than width.
  • Predict it: they extrapolated max attention logit and predicted a 4.8B model would diverge at LR 1e-2. It did.
More findings, and the limit of this paper

μParam kept the best LR fixed across widths but neither lowered sensitivity nor removed the need for QK-norm. Gradient RMS shrinks with scale toward AdamW’s ε = 1e-8, shrinking updates; ε = 1e-15 improved a 4.8B run at LR 0.3, ε = 1e-6 made it diverge. Limit: the paper studies slow divergence, not fast loss spikes. Its related-work section links fast spikes to Adam’s second-moment estimate going stale, and reports that lowering β2 removes them.

Wortsman et al. (2023) §2.2, §3.1–3.4, §4, Fig. 1, 5–9, 12.

A loss-spike debugging flow

Loss spike at step S Back on trend withina few hundred steps? yes log it,continue no Datathat exact batch?one long document?shuffling bug? Systemrecent code change?bad node, NaNs?throughput drop? Numericsgrad norm by layermax attention logitlog Z, update size Fixrewind and skip batches · lower LR · tighter clippingadd QK-norm or z-loss Confirm on a small proxyLR sweep with and without the fix: does thedivergence LR move? Replay the batches elsewhere.
  • Recoverable spikes are usually bad data; if recovery is slow, rewind and skip.
  • PaLM: replaying the same batches from another checkpoint didn’t re-spike, so cause = data × weight state.
  • SmolLM3 caught a tensor-parallel seed bug from lagging evals, not the loss.
  • PaLM skipped 200–500 batches; Falcon skipped 1B tokens. Marin added QK-norm mid-run.

Smol Training Playbook, “Fix and Restart vs. Fix on the Fly”, “Mystery #3–4”; Wortsman et al. (2023) for the numerics and the proxy.

Senior-staff probe

“Loss spikes at step 40K of a 400B MoE run and one expert’s routing share jumps.”

Check first

  1. Order: did the expert’s share jump before the loss, or after?
  2. The 40K batches: one narrow domain? Per-device balancing stats mislead on narrow batches; Qwen found global stats better.
  3. Max attention logit, log Z, grad norm by layer.

Cheap confirmation

Small MoE proxy (Kimi ablated K2 on a 3B model, 0.5B active). Raise LR until it breaks. Same expert jump first? Then add one fix (global balancing stats, QK-norm): it should move the breaking LR.

Meanwhile, rewind before 40K and skip the window. Reasoned from the mechanisms; no source here runs this exact case.

Smol Training Playbook, MoE “Load Balancing” and “Setting Up Our Ablation Framework”; Wortsman et al. (2023) §3.3 for proxy prediction.

Interview check

Three labs, one question

  • “Your pretraining loss suddenly diverges at step 300k of a long run. Diagnose and fix it.” Google DeepMind · question bank
  • “Loss spikes mid-run on a large pretraining job. Walk me through your debugging process.” xAI · question bank
  • “Your training run’s loss is diverging after step 10K. Walk me through your debugging process.” OpenAI · prep site
Separate the two: a spike that recovers is usually data; a slow divergence is usually numerics (logit growth, LR). Then give the small-scale experiment.

Questions from the curriculum’s question bank (2026).

The week in one minute

  1. Chinchilla: grow params and tokens equally, ~20 tokens/param; the optimum is a broad valley.
  2. Serving changes it: minimise training + lifetime inference; at 10^13 tokens a 30B-quality model is best as 13.6B.
  3. On paper: 6 × N × D; GPU-days = FLOPs ÷ (peak × MFU); 10^25 ≈ 293K H100-days at 40%.
  4. Rooflines: training is compute-bound; decode is memory-bound until ~300 tokens share a weight load.
  5. Data: ablate every step; dedup per snapshot; LLM-labelled classifiers lift MMLU 33% → 37%.
  6. Stability: QK-norm for logit growth, z-loss for output drift; prove fixes on small LR sweeps.

Your Week 4 plan

#ResourceListed hThis weekTake away
1Chinchilla Hoffmann et al. · also Kaplan21.5equal scaling, 20:1, the schedule artifact
2Beyond Chinchilla-Optimal Sardana et al.1.51training + inference objective; FLOPs ≠ dollars
3How To Scale Your Model Parts 1, 4, 6, 7; skim 5, 883.5rooflines, 6ND, MFU, decode bounds
4Smol Training Playbook ablations, hyperparameters, data, marathon82how real runs are planned and debugged
5FineWeb §3–4, App. E · blog31.2ablation-driven pipeline, dedup scope
6Small-scale proxies for instabilities Wortsman et al.21logit growth, z-loss, LR sweeps

Listed: 24.5 hours; budget: 10. The “this week” column (10.2 h) reads only what this deck uses. Skip Chinchilla’s appendix tables, the book’s TPU chapters and the playbook’s post-training and infrastructure chapters.

Self-test: answer out loud, then open

Compute-optimal size for 10^25 FLOPs by the 20:1 rule?

6 × N × 20N = 10^25 → N ≈ 290B on ≈ 5.8T tokens.

Why did Kaplan recommend bigger models?

One fixed cosine schedule (to 130B tokens) for all runs made shorter runs look worse; most models were under 100M parameters.

H100-days for 10^25 FLOPs at 40% MFU?

10^25 ÷ (989e12 × 0.4) ≈ 2.5e10 GPU-seconds ≈ 293K GPU-days; 36 days on 8,192 GPUs.

Batch size where a bf16 matmul becomes compute-bound on an H100?

About 295 tokens: 989 TFLOP/s ÷ 3.35 TB/s.

Why can FLOP-optimal and cost-optimal differ a lot?

Output tokens run at ~1% MFU vs ~50% in training. At 2T inference tokens: 1.3% extra FLOPs but 36% extra cost for a 70B Chinchilla model.

Why dedup per snapshot instead of globally?

Global dedup left 4T tokens and barely helped; the unique leftovers of old crawls were worse. Per snapshot kept 20T and matched RefinedWeb.

What do QK-norm and z-loss each fix?

QK-norm caps attention logit growth (one-hot attention). z-loss (1e-4 × log² Z) keeps output logits from drifting.

Cheapest way to test a stability fix?

Sweep LR on small models with and without it; a real fix moves the LR where loss blows up (QK-norm let 1.2B train at 0.3).

Next

Week 5: RLHF foundations

Policy gradients from first principles, reward models, PPO and its implementation details, and Constitutional AI.

Carry this forward

RL fine-tuning spends much of its time generating samples. That is decoding, so this week’s rooflines tell you why that phase is memory-bound and how batching helps.

Week 5 →  ·  ← Week 3: MLA and mixture of experts