Frontier-lab prep · LLM internals · Week 2

From GPT-2 to Llama 3:
the modern dense Transformer

RMSNorm, SwiGLU, RoPE and context extension, grouped-query and sliding-window attention: what each buys, what it costs, with numbers.

About 7 hours with the readings · use → or Space to step through · O for contents · T for dark mode

What this week is for

Defend every line of the Llama 3 block with a number.

Weight: ~2% of interview reports, 0 of 18 staff-plus; core in research loops.

How labs ask it

  • “Explain the architectural choices in a Llama-class model: why grouped-query attention, RoPE and SwiGLU instead of the vanilla 2017 Transformer?” Meta MSL · question bank
  • “Normalization (RMSNorm vs LayerNorm in transformer blocks)” Mistral · firsthand 2026
Senior-staff target “Llama 3 uses GQA, DeepSeek-V3 uses MLA. Compute KV-cache bytes per token for each at 128K context and tell me which you’d pick for a 1M-context product and why.” This week: the GQA half. MLA is Week 3.

Four changes between a GPT-2 block and a Llama 3 block

GPT-2 small (2019) Llama 3 8B (2024) Token + positionlearned table, 1,024 rows Token embeddingno position vector added LayerNormRMSNorm Attention12 query heads,12 key/value heads Attention + RoPE32 query heads share 8 K/V;q and k rotated by position ++add to streamadd to stream LayerNormRMSNorm GELU MLP768 → 3,072 → 768 SwiGLU MLP4,096 → 2 × 14,336 → 4,096 ++ × 12 blocks · vocab 50,257× 32 blocks · vocab 128,256

Same skeleton: decoder only, norm before each branch, add back to the residual stream, next-token loss.

  1. Norm: LayerNorm → RMSNorm Part 1
  2. MLP: GELU → gated SwiGLU Part 1
  3. Position: learned table → RoPE rotations Part 2
  4. Attention: 8 K/V heads shared by 32 query heads Part 3

Meta: Llama 3 “does not deviate significantly” from Llama 2; the gains came mostly from data and scale.

GPT-2 paper §2.3 and config; Llama 3 paper §3.2, Table 3, and the Llama 3 8B config.

Part 1

Inside the block: RMSNorm and SwiGLU

Two small swaps. One saves a little time; the other buys a little quality. Know which is which, and by how much.

LayerNorm vs RMSNorm on a real vector

LayerNorm: subtract the mean, divide by the standard deviation, then apply a learned gain and bias.

RMSNorm: divide by the root mean square (square, average, square-root), then apply a gain. No mean, no bias.

If the mean is zero, the two are identical.

Try the buttons: both ignore a rescale; only LayerNorm ignores a shift.

GPT-2’s first norm input for “ cat” at position 1: token row + position row, 768 numbers. First 24 shown.

Vector read from GPT-2’s published weights (wte row 3797 + wpe row 1). Learned gain and bias left out. Zhang & Sennrich (2019) §3–4.

What RMSNorm actually buys

Quality: the same

Transformer on English→German news translation: BLEU 26.6 with LayerNorm, 26.8 with RMSNorm.

BLEU: overlap with human translations; higher is better.

Speed: a little

7–9% faster training steps in the Transformer. The headline 7–64% comes from RNNs, which normalize at every time step.

Robustness

In an RNN translation model with weights initialized off-center (mean 0.2), LayerNorm training became very unstable; RMSNorm stayed more robust.

Interview answer: one statistic instead of two and no bias, so a simpler, cheaper kernel at equal quality. Llama adopted it with pre-norm “to improve the training stability”. Don’t claim it improves loss.

Zhang & Sennrich (2019), Tables 2–4 and Fig. 4; Touvron et al. (2023), Llama 1 §2.2.

SwiGLU: one branch decides how much of the other passes

GPT-2 MLP x × W1 (up) GELU × W2 (down) SwiGLU MLP x × W (gate) Swish near 0 = blocked × V (up) ⊗ multiply,feature by feature × W2 three weight matrices instead of two

Swish (also called SiLU) is z × sigmoid(z): about 0 for negative z, about z for positive z.

Each hidden feature is a value (from V) times a gate (from W). A gate near 0 shuts that feature off for this token.

The product of two linear maps gives the layer multiplicative interactions a plain MLP can’t form in one step. The paper itself offers no theory (next slides).

Shazeer (2020), “GLU Variants Improve Transformer”, Eq. 5–6.

Sizing the gated MLP: the 8/3 × d rule, and who ignores it

A 4d GELU MLP has 2 matrices of d × 4d: 8d² weights.

SwiGLU has 3 matrices of d × h: 3dh. Equal when h = 8d ÷ 3.

Shazeer matched T5-base this way (d = 768: 3,072 → 2,048). Llama 1 used “2/3 · 4d”.

Llama 3 doesn’t: 3.5d at 8B and 70B, 3.25d at 405B. Its MLPs are bigger than a GELU MLP of the same width, by choice.

Hidden sizes from Shazeer (2020) §3.1, Llama 1 §2.2, Llama 3 Table 3, Mistral 7B Table 1. Counts computed in your browser.

The evidence: small, consistent, unexplained

Held-out log-perplexity after 524,288 steps (T5-base, C4 span-filling). Lower is better; bars show improvement over ReLU.

Same parameters and compute for every row. GEGLU and SwiGLU are best: about 0.04 lower than ReLU.

Is that real? At 65K steps the gap is 0.053, while run-to-run spread is 0.005–0.010. Yes.

“We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.”

Senior take: an empirical win at a fixed budget. Say so plainly rather than inventing a theory.

Shazeer (2020), Table 1 and §4.

Interview check

Can you answer these in two minutes?

  • “Normalization (RMSNorm vs LayerNorm in transformer blocks)” Mistral · firsthand 2026
  • “Explain SwiGLU and why gated activations replaced ReLU/GELU in modern LLM MLP blocks.” Meta MSL · question bank
  • “What are the key components of a Transformer model and why does each matter?” Anthropic · question bank

Good answers carry a number: identical when the mean is zero; 7–9% step time; 8d² vs 3dh; 0.04 nats at matched compute.

From the question bank behind the curriculum (2025–2026 reports and prep sites).

Part 2

Position: RoPE, and stretching it past the training length

Why rotating queries and keys gives relative position, why it still breaks beyond the trained length, and the three fixes labs use.

Ways to tell attention where tokens are

Attention alone ignores word order, so position must be injected somewhere.

MethodHowUsed inPast the trained length
Sinusoidalfixed sine/cosine vector added to the embedding2017 Transformerabsolute positions only
Learned tableone trained vector per position, addedGPT-2, BERTno row exists: hard stop at 1,024
Relative biasa learned (T5) or linear (ALiBi) penalty by distance, added to the scoreT5, BLOOMALiBi favours nearby tokens
RoPErotate q and k by angles ∝ position: multiplyLlama, Qwen, Mistralbreaks, but can be rescaled
NoPEnothing; the causal mask leaks orderresearch modelsstudied; some hybrids mix it with RoPE

Long-context survey (Liu et al. 2025) §3.1.1, Table 3, §7.2.3; GPT-2 config (1,024 positions).

RoPE: rotate, then take the dot product

Split q and k into pairs of numbers. Rotate each pair by position × θ, where each pair has its own speed θ.

The dot product depends only on the angle between q and k. Shift both positions equally and nothing changes: only m − n survives.

Nothing is added to the embedding, and values (v) are not rotated.

q and k are made up for illustration; speeds are Llama 3’s (base 500,000, head size 128). Su et al. (2021) §3.2, Eq. 12–16.

64 clock hands turning at different speeds

Hand angle = position × θ for four of Llama 3’s 64 pairs, computed in your browser.

Fast pairs, like a second hand, separate neighbours but wrap around: positions 100 and 1,000 look alike.

Slow pairs, like an hour hand, barely move: they separate far-apart positions.

Turn length runs from 6.3 tokens to 2.56 million (Llama 3) or 54,410 with RoPE’s original base of 10,000.

In Llama 3’s 8K pretraining, 29 of 64 pairs never finish one turn. Remember this for the next slide.

Why RoPE breaks past its training length

An attention score is a sum of 64 waves, one per pair. Training only pins it down on distances 0 to L.

Fit those waves to points inside L and look outside: values explode. Real LLaMA 7B, trained at 2,048 tokens, hits perplexity above 1,000 at 4,096.

Second view (YaRN): pairs that never turned a full circle in training see angles they never saw.

Random targets in [−1, 1], made up for illustration; set-up of Chen et al. (2023) Fig. 2 (head size 128, base 10,000, L = 2,048), refit live. Perplexity: Chen et al. Table 1. Peng et al. (2023) §3.2.

Position interpolation: squeeze positions, don’t extend them

new input, 8,192 tokens 02,0484,0966,1448,192 ÷ 4 02,048 trained range: every angle already seen
Cheap: LLaMA 7B, 2K → 8K. Perplexity 16.1 with no training, 7.1 after 200 steps.
Works: passkey retrieval (find a 5-digit number in filler text) to the full 8,192 after 200 steps. Plain fine-tuning reached 2,560 after 10,000.
Cost: fast pairs get squeezed too, so nearby tokens blur. At 32K, BoolQ drops 76.1 → 64.7.

Chen et al. (2023), Fig. 1, Tables 3–5.

Three ways to stretch the clock hands

PI: every turn length × s. Simple; blurs fast pairs.

NTK-aware: raise the base so the slowest pair stretches by s and fast pairs barely change. Some middle pairs end up slightly out of range.

YaRN: by turns per training window (L ÷ turn length): more than 32, leave alone; fewer than 1, stretch by s; blend in between.

Llama 3.1 ships the same idea with a blend from 1 to 4 turns.

Formulas: Peng et al. (2023) Eq. 9–13 and Appendix A.2 (α = 1, β = 32 for Llama); Llama 3.1 config and Hugging Face _compute_llama3_parameters. Computed in your browser.

YaRN: the stretch above, plus a cooler softmax

Temperature: multiply q and k by 0.1 × ln s + 1 (1.28 at s = 16). Long contexts spread attention thin; this sharpens it. It folds into the rotation table, so it costs nothing at run time.

Training: Llama 2 7B to 64K in 400 steps of 64 × 64K tokens (1.7B tokens, computed), then 200 more for 128K.

Without training, LLaMA 7B stretched 8×: perplexity 3.33 at 16K. PI: above 10.

7B extensionA100-hours
YaRN, 4K → 64K256
PI, 2K → 16K640
NTK-aware (Code Llama), ≈100K6,400
What it costs: Llama 2 7B MMLU 43.8 → 41.7 at 128K. Passkey still 99.4% across 128K.

Peng et al. (2023) §3.3, Eq. 15, §4.1, Tables 3, 4, 5 and 9; attention spreading: long-context survey §7.2.2. MMLU: a 57-subject multiple-choice exam.

What Llama 3 actually did for 128K

  1. RoPE base 500,000 from the start (original RoPE: 10,000), shown effective to 32,768 tokens.
  2. Pretrain at 8K. Then six stages up to 128K, about 800B tokens.
  3. Advance only when short-context evals recover and needle-in-a-haystack (find one planted fact) is solved.
  4. Mask attention across documents: “limited impact” at 8K, “important” for very long sequences.
The released 3.1 config adds a frequency rule (factor 8): 29 pairs untouched, 29 stretched 8×, 6 blended. No temperature.
Senior read: YaRN retrofits with ~2.5B tokens; Llama 3 spent ~800B, 300× more, gated on evals. Budget versus robustness.

Llama 3 paper §3.2 and §3.4.2; Llama 3.1 8B config (rope_scaling); YaRN §4.1 for the token count.

Interview check

Position questions labs ask

  • “What are some different ways of encoding positional information?” frontier lab, unnamed · firsthand 2026
  • “Explain RoPE and how position interpolation / YaRN extend context beyond the trained length.” Meta MSL · question bank
  • “Explain the architectural choices in a Llama-class model: why grouped-query attention, RoPE and SwiGLU instead of the vanilla 2017 Transformer?” Meta MSL · question bank

Strong answer for the second: rotation → relative score; why it breaks (unseen angles); PI vs YaRN with one number each; how you would verify (passkey plus short-context evals).

Part 3

Attention that fits in memory: MQA, GQA and sliding windows

Generation is limited by bytes read, mostly the KV cache. Two levers: fewer key/value heads, or a shorter look-back.

How many key/value heads do the query heads share?

MHA8 K/V headscache 1× GQA2 K/V headscache ¼ MQA1 K/V headcache ⅛

Every query head still asks its own question. Only keys and values are shared within a group.

Llama 3 8B and Mistral 7B: 32 query heads, 8 K/V heads, so groups of 4 and a cache 4× smaller than MHA.

Cache shrinks by query heads ÷ K/V heads. Attention arithmetic does not: every query head still scores every key.

Ainslie et al. (2023) §2.2, Fig. 2; Llama 3 Table 3; Mistral 7B Table 1.

What sharing costs: the GQA paper

T5 modelAvg scoreTime per sample
MHA-XXL47.21.51
GQA-8-XXL47.10.28
MQA-XXL46.60.24
MHA-Large (smaller)46.00.37

Average over summarization, translation and QA. GQA-8: 5.4× faster than MHA for 0.1 points.

Uptraining: average each group’s K/V heads from a trained MHA model, then train 5% more. Averaging beats keeping one head; both beat random.

MQA failure mode: trained from scratch it had frequent loss spikes and diverged on long-input fine-tuning. GQA was stable.

Sharding: split over 8 GPUs, one MQA head is copied to every GPU; 8 GQA heads give one per GPU.

Caveats the authors list

Encoder-decoder models only, ROUGE as the main metric, and no GQA model trained from scratch for comparison. Uptraining XXL with 5% took about 600 TPUv3 chip-days.

Ainslie et al. (2023), Table 1, Figs. 4–6, §2.2, Appendix A, Limitations.

KV-cache arithmetic from real configs

Bytes per token = 2 (K and V) × layers × K/V heads × head size × 2 bytes (16-bit).

Then × context × sequences served. Compare with what is left of a GPU node after the weights.

Try Llama 3 70B at 128K: then switch to the MHA and MQA versions of the same model.

per token
total cache

Configs: Llama 3 / 3.1 (Hugging Face mirrors of Meta’s configs), Mistral 7B, GPT-2. Weights counted from the config at 2 bytes each. Node = 8 GPUs × 80 GB (H100). GiB = 2³⁰ bytes.

Sliding-window attention: reach grows with depth

16 tokens, window W = 3: token i sees positions i − 3 to i. Tinted: the last token’s reach.

Each layer can pass information back at most W tokens. After k layers, up to k × W.

Mistral 7B: W = 4,096 and 32 layers, so a theoretical span of 131,072 tokens.

Buys: work per token capped at W, not the whole context. At 16K tokens, 2× faster than vanilla attention.

Costs: no direct look beyond W. A fact 20K tokens back must be relayed layer by layer, and “theoretical” is not effective reach.

Jiang et al. (2023), Mistral 7B, §2 and Fig. 1.

Rolling buffer: the cache stops growing at W

Press → to generate. Window W = 4: token i is written to slot i mod 4.

With a fixed window, older keys and values are never read again, so overwrite them.

Mistral 7B at 32K tokens: 4 GiB of cache becomes 512 MiB per sequence, 8× smaller. The paper reports no quality loss.

Long prompts are prefilled in chunks of W, each attending to the cache and to itself.

Jiang et al. (2023) §2, Figs. 2–3. Sizes computed: 2 × 32 × 8 × 128 × 2 bytes = 128 KiB per token.

Interview check

Can you answer these with numbers?

  • “What are Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), and what do they trade away?” Meta MSL · question bank
  • “Mistral 7B shipped with grouped-query attention and sliding-window attention. What does each buy you, and what does each cost?” Mistral · question bank
Answer skeleton: GQA cuts cache and bytes read by query heads ÷ K/V heads (4× for Llama 3) for ~0.1 points; MQA saves 8× but loses quality and stability. SWA caps cache and work at W (8× at 32K) but long-range facts must hop through layers.
Part 4

Senior-staff angle: long context, costs and choices

The arithmetic behind “what happens if we make the window bigger?”, the failure modes, and what you would pick.

Where a long context’s cost goes, per generated token

Arithmetic: weights cost 2 × parameters FLOPs. Attention adds 4 × context × width × layers, growing with every token of context.

Bytes read: weights once per step, shared by the batch; each sequence reads its own cache.

GQA shrinks only the cache bar. A sliding window shrinks both attention bars (context capped at W).

Computed from the configs: attention per token = score + mix, 2 × 2 × context × (heads × head size) per layer. Embedding lookup excluded from weight FLOPs.

The probe, GQA half: Llama 3 at 128K and at 1M

ModelPer token128K1M
Llama 3 8B128 KiB16 GiB128 GiB
Llama 3 70B320 KiB40 GiB320 GiB
Llama 3 405B504 KiB63 GiB504 GiB

16-bit cache, one sequence.

One 1M-token user on 70B: 320 GiB of cache, more than four 80 GB GPUs before weights.

Levers, and what each costs

  • 8-bit cache: half the bytes; needs quality checks.
  • MQA: 40 GiB at 1M; quality and stability risk.
  • Mostly sliding-window layers: capped at W; weaker recall.
  • Compress K/V to a small latent: MLA, Week 3.

Shape: number → bottleneck (memory) → options with costs → a pick you’d test.

Failure modes to name

Claimed ≠ effective context. RULER benchmark: Llama 3.1 70B claims 128K, holds up to 64K; 8B to 32K.
Lost in the middle. Accuracy is U-shaped: facts at the start or end are found, the middle is missed.
Short context regresses. BoolQ −11 points (PI, 32K); MMLU −2 (YaRN, 128K).
Perplexity misleads. Code Llama 13B’s perplexity rises past 100K, yet it still finds the passkey at 128K.
MQA instability. Loss spikes from scratch; divergence on long-input fine-tuning.
Stale rotated cache. Dynamic scaling changes s as text grows: cache keys before RoPE.

Survey §7.1.1–7.1.2, Table 9 (RULER); Chen et al. Table 5; Peng et al. Table 10, §3.4, Appendix B.5; Ainslie et al. Appendix A.

What would you choose, and why?

SituationPickNumber behind itCheck before shipping
Extend a 4K model to 64K, small budgetYaRN, s = 16400 steps, 1.7B tokens, 256 A100-hours (7B)passkey; MMLU (YaRN lost 1.3–3)
New 70B model, 128K, many usersGQA-8, big RoPE base, staged extension320 KiB per token; ~800B tokensper stage: short evals, needle
Cheap long-document readersliding window512 MiB cache at 32K (Mistral 7B)long-range recall, not perplexity
1M-token contextGQA alone fails320 GiB per user (70B)needs cache compression: Week 3

Interviewers probe the last column: how you would know the choice worked.

Interview check

Long-context questions, senior framing

  • “What is the effect of adjusting an LLM’s context window size?” OpenAI · question bank
  • “Explain self-attention. What is its computational complexity, and what are your options when contexts get long?” OpenAI · question bank
  • “Name one drawback of transformer models.” frontier lab, unnamed · question bank
  • “Transformer internals and scaling laws, with emphasis on the ‘why’ behind design choices” OpenAI · firsthand 2026

For each: give the cost in numbers (FLOPs and bytes), the failure mode, and the option you would pick.

The week in one minute

  1. RMSNorm: drop the mean and bias; same quality, 7–9% faster steps.
  2. SwiGLU: gate × value; h = 8d/3 matches parameters; ~0.04 better, unexplained.
  3. RoPE: rotate q and k so scores depend on m − n; breaks past L on unseen angles.
  4. Extension: PI squeezes all pairs; YaRN the slow ones; Llama 3 trained 800B tokens.
  5. GQA: 8 K/V heads for 32–128 query heads; cache 320 KiB per token on 70B.
  6. Sliding window: cache and work capped at W; reach k × W, but only by relay.

Say each line with its number and failure mode, and Week 2 is done.

Your Week 2 plan

#ResourceListed hThis weekTake away
1RMSNorm Zhang, Sennrich10.75same BLEU, 7–9% faster
2GLU Variants Improve Transformer Shazeer–0.252/3 rule; unexplained win
3RoFormer (RoPE) Su et al.1.51§3: rotation → relative
4Position Interpolation Chen et al., optional–0.5§1–2: why it explodes
5YaRN Peng et al.1.51.5by-parts + temperature
6GQA Ainslie et al.10.75uptraining, MQA instability
7Mistral 7B Jiang et al.10.5§2: window, rolling cache
8Long-context survey Liu et al.41.5§3.1, 3.2.1, 5.2, 7
9The Llama 3 Herd Meta80.25§3.2, 3.4.2 (rest: Week 8)

Full listed times: 18 hours, mostly the survey and Llama 3. The named parts take about 7 (estimates). Short of time: skip item 4.

Self-test: answer out loud, then open

What does RMSNorm drop, and when does it equal LayerNorm?

The mean subtraction and the bias. Identical when the mean is zero: GPT-2’s “ cat” input has mean 0.0005 against RMS 0.22, and the two outputs differ by at most 0.002.

SwiGLU hidden size matching a 4d MLP at d = 4,096?

8/3 × 4,096 ≈ 10,923, since 3 × d × h = 8d². Llama 3 8B actually uses 14,336 = 3.5d.

Why does RoPE give relative position?

q and k are rotated by angles proportional to their positions; the dot product sees only the angle difference, (m − n) × θ.

LLaMA 7B trained at 2K, run at 4K unchanged: what happens?

Perplexity above 1,000. The score function was never constrained beyond 2K, and slow pairs reach unseen angles.

PI vs YaRN in one line each?

PI divides every position by s, blurring nearby tokens. YaRN leaves fast pairs alone, stretches slow ones by s, blends between, and scales q and k by 0.1 ln s + 1.

KV cache per token, Llama 3 8B, and as full MHA?

2 × 32 × 8 × 128 × 2 bytes = 128 KiB. With 32 K/V heads: 512 KiB.

What does GQA not reduce?

Attention arithmetic: every query head still scores every key. It cuts cache size and bytes read.

Mistral 7B: theoretical span, and cache at 32K?

4,096 × 32 layers = 131,072 tokens. Cache stays at 4,096 tokens: 512 MiB instead of 4 GiB.

Next

Week 3: MLA and mixture of experts

DeepSeek-V2’s multi-head latent attention, Switch Transformer and MoE routing with load balancing, Mixtral, Qwen3 and gpt-oss, and the GPT-2 → Llama 3 → DeepSeek-V3 → Qwen3 diff table.

Finish the probe

You have the GQA half: 320 KiB per token on Llama 3 70B, 40 GiB at 128K, 320 GiB at 1M. Week 3 adds MLA’s bytes per token for DeepSeek-V3, and the 1M-context pick.

Week 3 →  ·  ← Week 1