From GPT-2 to Llama 3:
the modern dense Transformer
RMSNorm, SwiGLU, RoPE and context extension, grouped-query and sliding-window attention: what each buys, what it costs, with numbers.
About 7 hours with the readings · use → or Space to step through · O for contents · T for dark mode
What this week is for
Defend every line of the Llama 3 block with a number.
How labs ask it
- “Explain the architectural choices in a Llama-class model: why grouped-query attention, RoPE and SwiGLU instead of the vanilla 2017 Transformer?” Meta MSL · question bank
- “Normalization (RMSNorm vs LayerNorm in transformer blocks)” Mistral · firsthand 2026
Four changes between a GPT-2 block and a Llama 3 block
Same skeleton: decoder only, norm before each branch, add back to the residual stream, next-token loss.
- Norm: LayerNorm → RMSNorm Part 1
- MLP: GELU → gated SwiGLU Part 1
- Position: learned table → RoPE rotations Part 2
- Attention: 8 K/V heads shared by 32 query heads Part 3
Meta: Llama 3 “does not deviate significantly” from Llama 2; the gains came mostly from data and scale.
GPT-2 paper §2.3 and config; Llama 3 paper §3.2, Table 3, and the Llama 3 8B config.
Inside the block: RMSNorm and SwiGLU
Two small swaps. One saves a little time; the other buys a little quality. Know which is which, and by how much.
LayerNorm vs RMSNorm on a real vector
LayerNorm: subtract the mean, divide by the standard deviation, then apply a learned gain and bias.
RMSNorm: divide by the root mean square (square, average, square-root), then apply a gain. No mean, no bias.
If the mean is zero, the two are identical.
Try the buttons: both ignore a rescale; only LayerNorm ignores a shift.
Vector read from GPT-2’s published weights (wte row 3797 + wpe row 1). Learned gain and bias left out. Zhang & Sennrich (2019) §3–4.
What RMSNorm actually buys
Quality: the same
Transformer on English→German news translation: BLEU 26.6 with LayerNorm, 26.8 with RMSNorm.
BLEU: overlap with human translations; higher is better.
Speed: a little
7–9% faster training steps in the Transformer. The headline 7–64% comes from RNNs, which normalize at every time step.
Robustness
In an RNN translation model with weights initialized off-center (mean 0.2), LayerNorm training became very unstable; RMSNorm stayed more robust.
Zhang & Sennrich (2019), Tables 2–4 and Fig. 4; Touvron et al. (2023), Llama 1 §2.2.
SwiGLU: one branch decides how much of the other passes
Swish (also called SiLU) is z × sigmoid(z): about 0 for negative z, about z for positive z.
Each hidden feature is a value (from V) times a gate (from W). A gate near 0 shuts that feature off for this token.
The product of two linear maps gives the layer multiplicative interactions a plain MLP can’t form in one step. The paper itself offers no theory (next slides).
Shazeer (2020), “GLU Variants Improve Transformer”, Eq. 5–6.
Sizing the gated MLP: the 8/3 × d rule, and who ignores it
A 4d GELU MLP has 2 matrices of d × 4d: 8d² weights.
SwiGLU has 3 matrices of d × h: 3dh. Equal when h = 8d ÷ 3.
Shazeer matched T5-base this way (d = 768: 3,072 → 2,048). Llama 1 used “2/3 · 4d”.
Llama 3 doesn’t: 3.5d at 8B and 70B, 3.25d at 405B. Its MLPs are bigger than a GELU MLP of the same width, by choice.
Hidden sizes from Shazeer (2020) §3.1, Llama 1 §2.2, Llama 3 Table 3, Mistral 7B Table 1. Counts computed in your browser.
The evidence: small, consistent, unexplained
Same parameters and compute for every row. GEGLU and SwiGLU are best: about 0.04 lower than ReLU.
Is that real? At 65K steps the gap is 0.053, while run-to-run spread is 0.005–0.010. Yes.
“We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.”
Senior take: an empirical win at a fixed budget. Say so plainly rather than inventing a theory.
Shazeer (2020), Table 1 and §4.
Can you answer these in two minutes?
- “Normalization (RMSNorm vs LayerNorm in transformer blocks)” Mistral · firsthand 2026
- “Explain SwiGLU and why gated activations replaced ReLU/GELU in modern LLM MLP blocks.” Meta MSL · question bank
- “What are the key components of a Transformer model and why does each matter?” Anthropic · question bank
Good answers carry a number: identical when the mean is zero; 7–9% step time; 8d² vs 3dh; 0.04 nats at matched compute.
From the question bank behind the curriculum (2025–2026 reports and prep sites).
Position: RoPE, and stretching it past the training length
Why rotating queries and keys gives relative position, why it still breaks beyond the trained length, and the three fixes labs use.
Ways to tell attention where tokens are
Attention alone ignores word order, so position must be injected somewhere.
| Method | How | Used in | Past the trained length |
|---|---|---|---|
| Sinusoidal | fixed sine/cosine vector added to the embedding | 2017 Transformer | absolute positions only |
| Learned table | one trained vector per position, added | GPT-2, BERT | no row exists: hard stop at 1,024 |
| Relative bias | a learned (T5) or linear (ALiBi) penalty by distance, added to the score | T5, BLOOM | ALiBi favours nearby tokens |
| RoPE | rotate q and k by angles ∝ position: multiply | Llama, Qwen, Mistral | breaks, but can be rescaled |
| NoPE | nothing; the causal mask leaks order | research models | studied; some hybrids mix it with RoPE |
Long-context survey (Liu et al. 2025) §3.1.1, Table 3, §7.2.3; GPT-2 config (1,024 positions).
RoPE: rotate, then take the dot product
Split q and k into pairs of numbers. Rotate each pair by position × θ, where each pair has its own speed θ.
The dot product depends only on the angle between q and k. Shift both positions equally and nothing changes: only m − n survives.
Nothing is added to the embedding, and values (v) are not rotated.
q and k are made up for illustration; speeds are Llama 3’s (base 500,000, head size 128). Su et al. (2021) §3.2, Eq. 12–16.
64 clock hands turning at different speeds
Fast pairs, like a second hand, separate neighbours but wrap around: positions 100 and 1,000 look alike.
Slow pairs, like an hour hand, barely move: they separate far-apart positions.
Turn length runs from 6.3 tokens to 2.56 million (Llama 3) or 54,410 with RoPE’s original base of 10,000.
In Llama 3’s 8K pretraining, 29 of 64 pairs never finish one turn. Remember this for the next slide.
Why RoPE breaks past its training length
An attention score is a sum of 64 waves, one per pair. Training only pins it down on distances 0 to L.
Fit those waves to points inside L and look outside: values explode. Real LLaMA 7B, trained at 2,048 tokens, hits perplexity above 1,000 at 4,096.
Second view (YaRN): pairs that never turned a full circle in training see angles they never saw.
Random targets in [−1, 1], made up for illustration; set-up of Chen et al. (2023) Fig. 2 (head size 128, base 10,000, L = 2,048), refit live. Perplexity: Chen et al. Table 1. Peng et al. (2023) §3.2.
Position interpolation: squeeze positions, don’t extend them
Chen et al. (2023), Fig. 1, Tables 3–5.
Three ways to stretch the clock hands
PI: every turn length × s. Simple; blurs fast pairs.
NTK-aware: raise the base so the slowest pair stretches by s and fast pairs barely change. Some middle pairs end up slightly out of range.
YaRN: by turns per training window (L ÷ turn length): more than 32, leave alone; fewer than 1, stretch by s; blend in between.
Llama 3.1 ships the same idea with a blend from 1 to 4 turns.
Formulas: Peng et al. (2023) Eq. 9–13 and Appendix A.2 (α = 1, β = 32 for Llama); Llama 3.1 config and Hugging Face _compute_llama3_parameters. Computed in your browser.
YaRN: the stretch above, plus a cooler softmax
Temperature: multiply q and k by 0.1 × ln s + 1 (1.28 at s = 16). Long contexts spread attention thin; this sharpens it. It folds into the rotation table, so it costs nothing at run time.
Training: Llama 2 7B to 64K in 400 steps of 64 × 64K tokens (1.7B tokens, computed), then 200 more for 128K.
Without training, LLaMA 7B stretched 8×: perplexity 3.33 at 16K. PI: above 10.
| 7B extension | A100-hours |
|---|---|
| YaRN, 4K → 64K | 256 |
| PI, 2K → 16K | 640 |
| NTK-aware (Code Llama), ≈100K | 6,400 |
Peng et al. (2023) §3.3, Eq. 15, §4.1, Tables 3, 4, 5 and 9; attention spreading: long-context survey §7.2.2. MMLU: a 57-subject multiple-choice exam.
What Llama 3 actually did for 128K
- RoPE base 500,000 from the start (original RoPE: 10,000), shown effective to 32,768 tokens.
- Pretrain at 8K. Then six stages up to 128K, about 800B tokens.
- Advance only when short-context evals recover and needle-in-a-haystack (find one planted fact) is solved.
- Mask attention across documents: “limited impact” at 8K, “important” for very long sequences.
Llama 3 paper §3.2 and §3.4.2; Llama 3.1 8B config (rope_scaling); YaRN §4.1 for the token count.
Position questions labs ask
- “What are some different ways of encoding positional information?” frontier lab, unnamed · firsthand 2026
- “Explain RoPE and how position interpolation / YaRN extend context beyond the trained length.” Meta MSL · question bank
- “Explain the architectural choices in a Llama-class model: why grouped-query attention, RoPE and SwiGLU instead of the vanilla 2017 Transformer?” Meta MSL · question bank
Strong answer for the second: rotation → relative score; why it breaks (unseen angles); PI vs YaRN with one number each; how you would verify (passkey plus short-context evals).
Attention that fits in memory: MQA, GQA and sliding windows
Generation is limited by bytes read, mostly the KV cache. Two levers: fewer key/value heads, or a shorter look-back.
How many key/value heads do the query heads share?
Every query head still asks its own question. Only keys and values are shared within a group.
Llama 3 8B and Mistral 7B: 32 query heads, 8 K/V heads, so groups of 4 and a cache 4× smaller than MHA.
Cache shrinks by query heads ÷ K/V heads. Attention arithmetic does not: every query head still scores every key.
Ainslie et al. (2023) §2.2, Fig. 2; Llama 3 Table 3; Mistral 7B Table 1.
What sharing costs: the GQA paper
| T5 model | Avg score | Time per sample |
|---|---|---|
| MHA-XXL | 47.2 | 1.51 |
| GQA-8-XXL | 47.1 | 0.28 |
| MQA-XXL | 46.6 | 0.24 |
| MHA-Large (smaller) | 46.0 | 0.37 |
Average over summarization, translation and QA. GQA-8: 5.4× faster than MHA for 0.1 points.
Uptraining: average each group’s K/V heads from a trained MHA model, then train 5% more. Averaging beats keeping one head; both beat random.
MQA failure mode: trained from scratch it had frequent loss spikes and diverged on long-input fine-tuning. GQA was stable.
Sharding: split over 8 GPUs, one MQA head is copied to every GPU; 8 GQA heads give one per GPU.
Caveats the authors list
Encoder-decoder models only, ROUGE as the main metric, and no GQA model trained from scratch for comparison. Uptraining XXL with 5% took about 600 TPUv3 chip-days.
Ainslie et al. (2023), Table 1, Figs. 4–6, §2.2, Appendix A, Limitations.
KV-cache arithmetic from real configs
Bytes per token = 2 (K and V) × layers × K/V heads × head size × 2 bytes (16-bit).
Then × context × sequences served. Compare with what is left of a GPU node after the weights.
Try Llama 3 70B at 128K: then switch to the MHA and MQA versions of the same model.
Configs: Llama 3 / 3.1 (Hugging Face mirrors of Meta’s configs), Mistral 7B, GPT-2. Weights counted from the config at 2 bytes each. Node = 8 GPUs × 80 GB (H100). GiB = 2³⁰ bytes.
Sliding-window attention: reach grows with depth
Each layer can pass information back at most W tokens. After k layers, up to k × W.
Mistral 7B: W = 4,096 and 32 layers, so a theoretical span of 131,072 tokens.
Buys: work per token capped at W, not the whole context. At 16K tokens, 2× faster than vanilla attention.
Costs: no direct look beyond W. A fact 20K tokens back must be relayed layer by layer, and “theoretical” is not effective reach.
Jiang et al. (2023), Mistral 7B, §2 and Fig. 1.
Rolling buffer: the cache stops growing at W
With a fixed window, older keys and values are never read again, so overwrite them.
Mistral 7B at 32K tokens: 4 GiB of cache becomes 512 MiB per sequence, 8× smaller. The paper reports no quality loss.
Long prompts are prefilled in chunks of W, each attending to the cache and to itself.
Jiang et al. (2023) §2, Figs. 2–3. Sizes computed: 2 × 32 × 8 × 128 × 2 bytes = 128 KiB per token.
Can you answer these with numbers?
- “What are Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), and what do they trade away?” Meta MSL · question bank
- “Mistral 7B shipped with grouped-query attention and sliding-window attention. What does each buy you, and what does each cost?” Mistral · question bank
Senior-staff angle: long context, costs and choices
The arithmetic behind “what happens if we make the window bigger?”, the failure modes, and what you would pick.
Where a long context’s cost goes, per generated token
Arithmetic: weights cost 2 × parameters FLOPs. Attention adds 4 × context × width × layers, growing with every token of context.
Bytes read: weights once per step, shared by the batch; each sequence reads its own cache.
GQA shrinks only the cache bar. A sliding window shrinks both attention bars (context capped at W).
Computed from the configs: attention per token = score + mix, 2 × 2 × context × (heads × head size) per layer. Embedding lookup excluded from weight FLOPs.
The probe, GQA half: Llama 3 at 128K and at 1M
| Model | Per token | 128K | 1M |
|---|---|---|---|
| Llama 3 8B | 128 KiB | 16 GiB | 128 GiB |
| Llama 3 70B | 320 KiB | 40 GiB | 320 GiB |
| Llama 3 405B | 504 KiB | 63 GiB | 504 GiB |
16-bit cache, one sequence.
One 1M-token user on 70B: 320 GiB of cache, more than four 80 GB GPUs before weights.
Levers, and what each costs
- 8-bit cache: half the bytes; needs quality checks.
- MQA: 40 GiB at 1M; quality and stability risk.
- Mostly sliding-window layers: capped at W; weaker recall.
- Compress K/V to a small latent: MLA, Week 3.
Shape: number → bottleneck (memory) → options with costs → a pick you’d test.
Failure modes to name
Survey §7.1.1–7.1.2, Table 9 (RULER); Chen et al. Table 5; Peng et al. Table 10, §3.4, Appendix B.5; Ainslie et al. Appendix A.
What would you choose, and why?
| Situation | Pick | Number behind it | Check before shipping |
|---|---|---|---|
| Extend a 4K model to 64K, small budget | YaRN, s = 16 | 400 steps, 1.7B tokens, 256 A100-hours (7B) | passkey; MMLU (YaRN lost 1.3–3) |
| New 70B model, 128K, many users | GQA-8, big RoPE base, staged extension | 320 KiB per token; ~800B tokens | per stage: short evals, needle |
| Cheap long-document reader | sliding window | 512 MiB cache at 32K (Mistral 7B) | long-range recall, not perplexity |
| 1M-token context | GQA alone fails | 320 GiB per user (70B) | needs cache compression: Week 3 |
Interviewers probe the last column: how you would know the choice worked.
Long-context questions, senior framing
- “What is the effect of adjusting an LLM’s context window size?” OpenAI · question bank
- “Explain self-attention. What is its computational complexity, and what are your options when contexts get long?” OpenAI · question bank
- “Name one drawback of transformer models.” frontier lab, unnamed · question bank
- “Transformer internals and scaling laws, with emphasis on the ‘why’ behind design choices” OpenAI · firsthand 2026
For each: give the cost in numbers (FLOPs and bytes), the failure mode, and the option you would pick.
The week in one minute
- RMSNorm: drop the mean and bias; same quality, 7–9% faster steps.
- SwiGLU: gate × value; h = 8d/3 matches parameters; ~0.04 better, unexplained.
- RoPE: rotate q and k so scores depend on m − n; breaks past L on unseen angles.
- Extension: PI squeezes all pairs; YaRN the slow ones; Llama 3 trained 800B tokens.
- GQA: 8 K/V heads for 32–128 query heads; cache 320 KiB per token on 70B.
- Sliding window: cache and work capped at W; reach k × W, but only by relay.
Say each line with its number and failure mode, and Week 2 is done.
Your Week 2 plan
| # | Resource | Listed h | This week | Take away |
|---|---|---|---|---|
| 1 | RMSNorm Zhang, Sennrich | 1 | 0.75 | same BLEU, 7–9% faster |
| 2 | GLU Variants Improve Transformer Shazeer | – | 0.25 | 2/3 rule; unexplained win |
| 3 | RoFormer (RoPE) Su et al. | 1.5 | 1 | §3: rotation → relative |
| 4 | Position Interpolation Chen et al., optional | – | 0.5 | §1–2: why it explodes |
| 5 | YaRN Peng et al. | 1.5 | 1.5 | by-parts + temperature |
| 6 | GQA Ainslie et al. | 1 | 0.75 | uptraining, MQA instability |
| 7 | Mistral 7B Jiang et al. | 1 | 0.5 | §2: window, rolling cache |
| 8 | Long-context survey Liu et al. | 4 | 1.5 | §3.1, 3.2.1, 5.2, 7 |
| 9 | The Llama 3 Herd Meta | 8 | 0.25 | §3.2, 3.4.2 (rest: Week 8) |
Full listed times: 18 hours, mostly the survey and Llama 3. The named parts take about 7 (estimates). Short of time: skip item 4.
Self-test: answer out loud, then open
What does RMSNorm drop, and when does it equal LayerNorm?
The mean subtraction and the bias. Identical when the mean is zero: GPT-2’s “ cat” input has mean 0.0005 against RMS 0.22, and the two outputs differ by at most 0.002.
SwiGLU hidden size matching a 4d MLP at d = 4,096?
8/3 × 4,096 ≈ 10,923, since 3 × d × h = 8d². Llama 3 8B actually uses 14,336 = 3.5d.
Why does RoPE give relative position?
q and k are rotated by angles proportional to their positions; the dot product sees only the angle difference, (m − n) × θ.
LLaMA 7B trained at 2K, run at 4K unchanged: what happens?
Perplexity above 1,000. The score function was never constrained beyond 2K, and slow pairs reach unseen angles.
PI vs YaRN in one line each?
PI divides every position by s, blurring nearby tokens. YaRN leaves fast pairs alone, stretches slow ones by s, blends between, and scales q and k by 0.1 ln s + 1.
KV cache per token, Llama 3 8B, and as full MHA?
2 × 32 × 8 × 128 × 2 bytes = 128 KiB. With 32 K/V heads: 512 KiB.
What does GQA not reduce?
Attention arithmetic: every query head still scores every key. It cuts cache size and bytes read.
Mistral 7B: theoretical span, and cache at 32K?
4,096 × 32 layers = 131,072 tokens. Cache stays at 4,096 tokens: 512 MiB instead of 4 GiB.
Week 3: MLA and mixture of experts
DeepSeek-V2’s multi-head latent attention, Switch Transformer and MoE routing with load balancing, Mixtral, Qwen3 and gpt-oss, and the GPT-2 → Llama 3 → DeepSeek-V3 → Qwen3 diff table.
You have the GQA half: 320 KiB per token on Llama 3 70B, 40 GiB at 128K, 320 GiB at 1M. Week 3 adds MLA’s bytes per token for DeepSeek-V3, and the 1M-context pick.