Scaling arithmetic
and pretraining
How big a model, how many tokens, how many GPU-days, what data, and how to keep the run from blowing up.
10 hours with the readings · use → or Space to step through · O for contents · T for dark mode
What this week is for
Turn a compute budget into a model, a token count and a GPU calendar. Then keep the run alive.
“You have 10^25 FLOPs and expect 10^13 inference tokens a month. How big a model, how many tokens, how many GPU-days, and why not Chinchilla-optimal?”
How labs ask it
- “What do the Chinchilla scaling laws say, and how do they differ from earlier scaling intuitions?” Anthropic · question bank
- “Your pretraining loss suddenly diverges at step 300k of a long run. Diagnose and fix it.” Google DeepMind · question bank
How much this matters, and the second target
LLM internals is about 2% of collected interview reports and appears in none of the 18 staff-plus reports. It is core in research and research-engineer loops. Second target, worked on the stability slides: a loss spike in a 400B MoE run; what do you check first, and what small run confirms the cause?
Compute-optimal scaling: Chinchilla
Same question, two answers
Fixed compute budget: spend it on parameters or on training tokens?
| 10× more compute | params | tokens |
|---|---|---|
| Kaplan (2020) | × 5.5 | × 1.8 |
| Chinchilla (2022) | × 3.2 | × 3.2 |
Chinchilla: grow both equally, so each gets √10 ≈ 3.2×.
Why Kaplan over-sized: one long learning-rate schedule for every run, so short runs looked worse than they were. Most of its models were under 100M parameters.
Hoffmann et al. (2022), §1, §2, App. B and Fig. A1.
One budget, many models: the valley is flat
Each point spends the same FLOPs: a bigger model sees fewer tokens.
Too small: capacity runs out. Too big: too few tokens.
The bottom is broad. At 10^25, a model ¼ the 20:1 size predicts almost the same loss (+0.001). That slack is what over-training spends.
Loss from Chinchilla’s fitted formula (§3.3, App. D.2) with unrounded exponents α = 0.336, β = 0.283 as used by Sardana et al. (2024), App. A. Computed in your browser.
The “20 tokens per parameter” rule
Three fitting methods agree: parameters and tokens each grow as compute0.5 (0.50, 0.49, 0.46).
| Params | Compute (FLOPs) | Tokens |
|---|---|---|
| 1B | 1.2 × 10²⁰ | 20B |
| 10B | 1.2 × 10²² | 205B |
| 67B | 5.8 × 10²³ | 1.5T |
| 280B | 9.9 × 10²⁴ | 5.9T |
| 1T | 1.3 × 10²⁶ | 21.2T |
The test
Same compute as Gopher (280B on 300B tokens). Chinchilla: 70B on 1.4T.
MMLU 5-shot: 67.6% vs 60.0%. Better on 51 of 57 subjects.
And 4× cheaper to serve, which Part 2 turns into the main argument.
Hoffmann et al. (2022): Table 2 (exponents), Table 3 (Approach 1 projections), §4, Table 6, Fig. 6.
Where the 20:1 rule is shaky
The fitted loss: 1.69 (text you can never predict) + a size penalty that shrinks with parameters + a data penalty that shrinks with tokens.
- Method 3 (the formula) puts the optimum at smaller models: about 150B on 11T tokens at 10^25, not 290B on 5.8T.
- Only two large runs tested it; all runs used less than one pass over the data.
- The frontier bends at large budgets, so even smaller models may be optimal (App. E).
- A replication found the reported confidence intervals implausibly narrow.
- It counts training compute only. Serving is free in this model.
The formula, with numbers
Loss = 1.69 + 406.4 ÷ N0.34 + 410.7 ÷ D0.28. With the unrounded exponents α = 0.336, β = 0.283 it predicts 1.93 for Chinchilla (70B, 1.4T) and 1.98 for Gopher (280B, 300B); the 150B/11T optimum above comes from the same formula.
Hoffmann et al. (2022) §3.4, §5, App. D.2, App. E; replication: Besiroglu et al. (2024), as summarised in Sardana et al. (2024) §7.
Can you answer these in two minutes?
- “What do the Chinchilla scaling laws say, and how do they differ from earlier scaling intuitions?” Anthropic · question bank
- “Transformer internals and scaling laws, with emphasis on the ‘why’ behind design choices” OpenAI · firsthand 2026
Questions from the curriculum’s question bank (2026).
Over-training on purpose: count the serving bill
Add the serving bill to the objective
Chinchilla: fix compute, minimise loss.
Sardana et al.: fix the quality (loss), minimise 6 × params × training tokens + 2 × params × lifetime inference tokens.
Training costs 6 FLOPs per parameter per token; serving costs 2.
The higher the target quality, the more demand it takes to move the optimum. Assumes demand does not depend on model size.
| Target quality | Inference tokens | Cheapest model | Saved |
|---|---|---|---|
| 7B-Chinchilla | 10¹¹ | 6B, 1.18× data | — |
| 30B-Chinchilla | 10¹³ | 13.6B, 2.84× data | 28% |
| 70B-Chinchilla | 10¹³ | 41.6B on 7.9T | 12% |
Sardana et al. (2024), Eq. 3, §2, Table 2. The 30B row is reproduced live on the next slide.
Lifetime cost against model size, at fixed quality
━ total · ━ training · ┅ inference
Moving left, each model needs more tokens to reach the same loss, so training rises while serving falls.
GPU-hours: training and prompts at 50% MFU, output tokens at the decode MFU you pick, 70 in / 215 out tokens per request (Sardana §6).
Method: Sardana et al. (2024) Eq. 3 and Eq. 6, with Chinchilla’s fitted loss. Computed in your browser.
FLOPs are not dollars, and over-training keeps paying
Count GPU time, not FLOPs
At 2T inference tokens, a 70B Chinchilla model needs only 1.3% more FLOPs than the best model, but costs 36% more.
Why: each output token runs at ~1% MFU versus ~50% in training, a 50× gap the FLOP count hides.
No saturation up to 10,000 tokens/param
47 models, 150M–6B: loss and downstream scores kept improving.
But formulas fitted on ≤100 tokens/param overestimate the gain from extra data at extreme ratios.
| Model | Tokens | Tokens per parameter |
|---|---|---|
| Chinchilla 70B | 1.4T | 20 |
| Llama 3 8B | 15T | ~1,900 |
| SmolLM3 3B | 11T | ~3,700 |
Sardana et al. (2024) §4–6; Hoffmann et al. (2022); Smol Training Playbook (“Scaling Laws: How Many Parameters”). Ratios computed.
A training run on paper
Where “6 × parameters × tokens” comes from
Every weight matrix follows this pattern, so a training step costs about 6 FLOPs per parameter per token.
Attention scores add work that grows with context: about context ÷ (8 × width) of the matmul FLOPs. Width 8,192 at 8K context: 12.5%.
Recomputing activations in the backward pass (to save memory) pushes it toward 8 × N × D.
Austin et al., “How To Scale Your Model”, Part 4 (Transformer Math): forward/reverse FLOPs, attention ratio T/8D, gradient checkpointing.
Rooflines: why training is compute-bound and decoding is not
Arithmetic intensity = FLOPs per byte read from memory. Below the ridge, the chip waits on memory.
A matmul’s intensity ≈ the number of tokens sharing one weight load.
Training and prompts: thousands of tokens per load. Decoding: one token per user, so you batch ~300 users to stop wasting FLOPs. Attention over the KV cache stays near 1–8 FLOPs/byte.
Matrix: Llama 3 70B MLP, 8,192 × 28,672. Peaks and bandwidths from “How To Scale Your Model” Parts 1, 7 and 12 (H100: 989 TFLOP/s bf16, 3.35 TB/s → ridge ≈ 295).
Cluster size is set by FLOPs and calendar, not memory
Llama 3 70B on paper
6 × 70B × 15T ≈ 6.3 × 10²⁴ FLOPs.
Memory: weights 140 GB + Adam state 560 GB + saved activations ~10.5 TB ≈ 11 TB. That fits on ~117 TPU v5p chips, which would take ~3,400 days.
On 8,960 chips at 40% MFU: about 44 days.
What utilisation to assume
MFU (model FLOPs utilisation) = useful FLOPs ÷ peak FLOPs.
- Raw H100 bf16 matmuls: 714–758 of 990 TFLOP/s
- End-to-end training runs: 20–41%
- DeepSeek-V3 estimate: ~22%
- SmolLM3 planned: 30%
“How To Scale Your Model” Part 6 (LLaMA 3 training) and Part 4 Q7 (DeepSeek-V3); Smol Training Playbook, “Compute Units and FLOPs” and “Scaling Surprises”.
Compute-budget calculator
GPU-time = FLOPs ÷ (dense bf16 peak × MFU). Peaks from “How To Scale Your Model” Part 12 table (A100 312, H100/H200 990, B200 2,250 TFLOP/s). Loss from Chinchilla’s fit.
Serving on paper: decoding reads memory
Each decode step must read every weight, plus every user’s KV cache. So:
step time ≥ bytes read ÷ memory bandwidth
FLOPs only matter once a large batch shares each weight load.
Llama 3 70B, bf16, 8 × H100
Weights 141 GB ÷ 26.8 TB/s ≥ 5.3 ms per step.
Batch of 32 users at 8K context: KV cache 32 × 8,192 × 320 KiB = 80 GiB, +3.2 ms.
The math for those 32 tokens: 0.6 ms. Memory-bound by ~15×.
Formula: “How To Scale Your Model” Part 7 (theoretical minimum step time). KV bytes per token from Week 1 (2 × 80 × 8 × 128 × 2). Numbers computed.
10^25 FLOPs: the Chinchilla answer and the calendar
Assumptions, said out loud
- Dense model; FLOPs = 6 × N × D
- H100 at 989 TFLOP/s bf16, 40% MFU
- 10^13 tokens a month, served for 12 months
- 20:1 rule: 6 × N × 20N = 10^25 → N ≈ 290B, D ≈ 5.8T tokens. (The fitted formula says ~150B on 11T; same valley.)
- GPU time: 10^25 ÷ (989 × 10¹² × 0.4) = 2.5 × 10¹⁰ GPU-seconds = 7.0M H100-hours ≈ 293K GPU-days.
- Calendar: 36 days on 8,192 H100s.
- Budget extra: SmolLM3’s ablations and debugging cost 58% of its main run.
Hoffmann et al. (2022); peak from “How To Scale Your Model” Part 1; MFU range from the Smol Training Playbook (161,280 of 276,480 GPU-hours). Arithmetic computed.
Same 10^25 FLOPs, four model sizes
10^13 tokens a month is 3.9M tokens/s on average. At 290B that costs 58% of the whole training budget every month.
The loss column barely moves from 290B down to 70B; the serving columns move 4×. Losses are the fitted formula’s, less reliable at 340+ tokens/param.
Computed in your browser: D = 10^25 ÷ 6N; serving FLOPs = 2 × N × tokens; H100s at 40% of 989 TFLOP/s for average load; weights at 2 bytes per parameter on 80 GB GPUs.
The answer you give
“Train about 70B on ~24T tokens: same 10^25 FLOPs, same 36 days. Predicted loss within 0.002 of the 290B, but serving takes ~1,400 H100s instead of ~5,600.”
Why not Chinchilla-optimal: it minimises training compute only. Here a year of serving costs 7× the training run.
What would change my answer
- Data: 24T good unique tokens is a lot (FineWeb: 15T from 96 crawls).
- Fit optimism at 340 tokens/param: hedge toward ~100B.
- Demand 10× lower: the optimum moves up to ~100B (lifetime-cost slide).
Sardana et al. (2024) §4–5; Penedo et al. (2024) §3.7; Smol Training Playbook, “The Unintuitive Nature of Data Mixtures”. Numbers from the previous two slides.
Questions this arithmetic answers
- “Walk me through what happens, tensor by tensor, in one forward pass of a decoder-only transformer.” Add the FLOPs per step. Anthropic · question bank
- “Walk me through matrix manipulations relevant to LLM architectures.” Anthropic · question bank
- “Explain self-attention. What is its computational complexity, and what are your options when contexts get long?” Give the cost side: T ÷ 8D, KV reads. OpenAI · question bank
Questions from the curriculum’s question bank (2026). Week 1 covers the tensor shapes; this week adds the FLOPs and bytes.
Data: building FineWeb
From 96 Common Crawl snapshots to 15T tokens
Each step raised the ablation score. Text extraction alone (trafilatura on raw HTML instead of Common Crawl’s text files) was a clear win.
Penedo et al. (2024) §3.2–3.7, Fig. 1–9.
How they decided, and the dedup surprise
Every choice was an ablation
1.71B models, 28B tokens per filter test, 350B for dedup. Two seeds each.
Benchmarks kept only if low-noise, rising during training and above random. 70+ models, ~80,000 H100-hours.
Global dedup made things worse
Deduping all 96 crawls against each other left 4T tokens and barely helped.
In an old crawl, the 10% it kept trained worse than the 90% it removed: ads and keyword lists survive because they are unique.
Per-snapshot dedup: 20T tokens, matched RefinedWeb.
The gain comes from removing huge duplicate clusters. Small ablations can’t see dedup: in a 1B-token sample almost every document looks unique.
Penedo et al. (2024) §3.1, §3.4, Fig. 3–5, App. E.2 (Fig. 14).
MinHash: how “75% similar” becomes a hash lookup
Hash each document’s word 5-grams with 112 functions; keep the minimum of each. Two documents agree on one value with probability = their similarity.
Group the 112 values into 14 bands of 8. A match on any whole band flags a duplicate.
More hashes give a sharper cutoff (RefinedWeb used 9,000) at more compute and storage.
Penedo et al. (2024) §3.4 and App. E.1 (match rates 56%, 77%, 92%, 98.8% at 0.70–0.85, reproduced here). Computed in your browser.
FineWeb-Edu: a quality filter trained on LLM labels
Penedo et al. (2024) §4, Fig. 10–12, 17, 18.
Designing streaming dedup at web scale
- Sign: per document, 5-gram shingles → 112 MinHash values → 14 band keys.
- Stream: look up each band key in a sharded key-value store. Hit = candidate duplicate; miss = insert. One pass, 14 keys per document.
- Cluster: merge candidates transitively (union-find); keep one per cluster.
- Scope: per crawl window, not global (FineWeb’s ablation). URL-level and line-level global dedup also lost.
- Prove it: ablate at 350B tokens; small samples hide dedup effects.
Extending to a petabyte social-media pipeline with safety filtering
Same skeleton, with stages you ablate one at a time: language ID, URL blocklists, rule filters with thresholds chosen from histograms of good vs bad data (FineWeb’s method for its custom filters), PII anonymisation (FineWeb anonymised emails and IP addresses), and model-based quality or safety classifiers trained on LLM labels. Shuffle at the sequence level downstream: SmolLM3 saw loss spikes when one long low-quality file filled a batch.
Design is this deck’s, built from Penedo et al. (2024) §3.4, §3.6, App. E.1–E.3 and the Smol Training Playbook (“Mystery #3”).
Data questions labs ask
- “Design a deduplication pipeline for a web-scale pretraining corpus. It has to run as a streaming process.” xAI · question bank
- “Design the data pipeline ingesting X content at petabyte scale for model training with safety filtering.” xAI · prep site
Questions from the curriculum’s question bank (2026).
Keeping the run stable
Two known instabilities, and where the fixes sit
Wortsman et al. (2023) §3.1.1–3.1.2, App. C, Fig. E.1 (growth comes from query/key norms, not their alignment).
Watch attention collapse as weights grow
Logits multiply two projections, so they grow with the square of the weight scale.
In the paper’s runs, every model whose max attention logit passed 10⁴ diverged. Forcing logits to ~10³ in a 10M model already hurt; at 10⁴ it did worse than a zero-layer model (embeddings only).
Toy: one query, 8 random keys, head size 16 (fixed seed). Real QK-norm has a learned gain.
Wortsman et al. (2023) §3.1.1, §3.3, Fig. 9–10. Computed in your browser.
The cheap proxy: sweep the learning rate on small models
- LR sensitivity: average gap to the best loss across 3e-4 to 3e-1. It rises with model size.
- QK-norm let a 1.2B model train at LR 0.3.
- Longer warmup and LR-independent weight decay lower sensitivity. Depth raises it faster than width.
- Predict it: they extrapolated max attention logit and predicted a 4.8B model would diverge at LR 1e-2. It did.
More findings, and the limit of this paper
μParam kept the best LR fixed across widths but neither lowered sensitivity nor removed the need for QK-norm. Gradient RMS shrinks with scale toward AdamW’s ε = 1e-8, shrinking updates; ε = 1e-15 improved a 4.8B run at LR 0.3, ε = 1e-6 made it diverge. Limit: the paper studies slow divergence, not fast loss spikes. Its related-work section links fast spikes to Adam’s second-moment estimate going stale, and reports that lowering β2 removes them.
Wortsman et al. (2023) §2.2, §3.1–3.4, §4, Fig. 1, 5–9, 12.
A loss-spike debugging flow
- Recoverable spikes are usually bad data; if recovery is slow, rewind and skip.
- PaLM: replaying the same batches from another checkpoint didn’t re-spike, so cause = data × weight state.
- SmolLM3 caught a tensor-parallel seed bug from lagging evals, not the loss.
- PaLM skipped 200–500 batches; Falcon skipped 1B tokens. Marin added QK-norm mid-run.
Smol Training Playbook, “Fix and Restart vs. Fix on the Fly”, “Mystery #3–4”; Wortsman et al. (2023) for the numerics and the proxy.
“Loss spikes at step 40K of a 400B MoE run and one expert’s routing share jumps.”
Check first
- Order: did the expert’s share jump before the loss, or after?
- The 40K batches: one narrow domain? Per-device balancing stats mislead on narrow batches; Qwen found global stats better.
- Max attention logit, log Z, grad norm by layer.
Cheap confirmation
Small MoE proxy (Kimi ablated K2 on a 3B model, 0.5B active). Raise LR until it breaks. Same expert jump first? Then add one fix (global balancing stats, QK-norm): it should move the breaking LR.
Meanwhile, rewind before 40K and skip the window. Reasoned from the mechanisms; no source here runs this exact case.
Smol Training Playbook, MoE “Load Balancing” and “Setting Up Our Ablation Framework”; Wortsman et al. (2023) §3.3 for proxy prediction.
Three labs, one question
- “Your pretraining loss suddenly diverges at step 300k of a long run. Diagnose and fix it.” Google DeepMind · question bank
- “Loss spikes mid-run on a large pretraining job. Walk me through your debugging process.” xAI · question bank
- “Your training run’s loss is diverging after step 10K. Walk me through your debugging process.” OpenAI · prep site
Questions from the curriculum’s question bank (2026).
The week in one minute
- Chinchilla: grow params and tokens equally, ~20 tokens/param; the optimum is a broad valley.
- Serving changes it: minimise training + lifetime inference; at 10^13 tokens a 30B-quality model is best as 13.6B.
- On paper: 6 × N × D; GPU-days = FLOPs ÷ (peak × MFU); 10^25 ≈ 293K H100-days at 40%.
- Rooflines: training is compute-bound; decode is memory-bound until ~300 tokens share a weight load.
- Data: ablate every step; dedup per snapshot; LLM-labelled classifiers lift MMLU 33% → 37%.
- Stability: QK-norm for logit growth, z-loss for output drift; prove fixes on small LR sweeps.
Your Week 4 plan
| # | Resource | Listed h | This week | Take away |
|---|---|---|---|---|
| 1 | Chinchilla Hoffmann et al. · also Kaplan | 2 | 1.5 | equal scaling, 20:1, the schedule artifact |
| 2 | Beyond Chinchilla-Optimal Sardana et al. | 1.5 | 1 | training + inference objective; FLOPs ≠ dollars |
| 3 | How To Scale Your Model Parts 1, 4, 6, 7; skim 5, 8 | 8 | 3.5 | rooflines, 6ND, MFU, decode bounds |
| 4 | Smol Training Playbook ablations, hyperparameters, data, marathon | 8 | 2 | how real runs are planned and debugged |
| 5 | FineWeb §3–4, App. E · blog | 3 | 1.2 | ablation-driven pipeline, dedup scope |
| 6 | Small-scale proxies for instabilities Wortsman et al. | 2 | 1 | logit growth, z-loss, LR sweeps |
Listed: 24.5 hours; budget: 10. The “this week” column (10.2 h) reads only what this deck uses. Skip Chinchilla’s appendix tables, the book’s TPU chapters and the playbook’s post-training and infrastructure chapters.
Self-test: answer out loud, then open
Compute-optimal size for 10^25 FLOPs by the 20:1 rule?
6 × N × 20N = 10^25 → N ≈ 290B on ≈ 5.8T tokens.
Why did Kaplan recommend bigger models?
One fixed cosine schedule (to 130B tokens) for all runs made shorter runs look worse; most models were under 100M parameters.
H100-days for 10^25 FLOPs at 40% MFU?
10^25 ÷ (989e12 × 0.4) ≈ 2.5e10 GPU-seconds ≈ 293K GPU-days; 36 days on 8,192 GPUs.
Batch size where a bf16 matmul becomes compute-bound on an H100?
About 295 tokens: 989 TFLOP/s ÷ 3.35 TB/s.
Why can FLOP-optimal and cost-optimal differ a lot?
Output tokens run at ~1% MFU vs ~50% in training. At 2T inference tokens: 1.3% extra FLOPs but 36% extra cost for a 70B Chinchilla model.
Why dedup per snapshot instead of globally?
Global dedup left 4T tokens and barely helped; the unique leftovers of old crawls were worse. Per snapshot kept 20T and matched RefinedWeb.
What do QK-norm and z-loss each fix?
QK-norm caps attention logit growth (one-hot attention). z-loss (1e-4 × log² Z) keeps output logits from drifting.
Cheapest way to test a stability fix?
Sweep LR on small models with and without it; a real fix moves the LR where loss blows up (QK-norm let 1.2B train at 0.3).
Week 5: RLHF foundations
Policy gradients from first principles, reward models, PPO and its implementation details, and Constitutional AI.
RL fine-tuning spends much of its time generating samples. That is decoding, so this week’s rooflines tell you why that phase is memory-bound and how batching helps.