Frontier-lab prep · LLM internals · Week 3

Smaller caches,
sparser layers

How DeepSeek’s latent attention shrinks the KV cache, how mixture-of-experts adds parameters without adding compute, and how the 2025 open models combine the two.

About 7 hours with the readings (weeks 2–3 share 12–15) · use → or Space to step through · O for contents · T for dark mode

What this week is for

Explain latent attention and mixture of experts with numbers you can work out at a whiteboard.

How much it matters: about 2% of interview reports, none of the 18 staff-plus ones; each question here comes from one report. Core in research loops.

How labs ask it

  • “Explain how a Mixtral-style sparse mixture-of-experts model works. Why does a ~47B-parameter model run at roughly the cost of a ~13B one?” Mistral · question bank
  • “What are Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), and what do they trade away?” Meta MSL · question bank

The target, and the route to it

Senior-staff probe for this week

“Llama 3 uses GQA, DeepSeek-V3 uses MLA. Compute KV-cache bytes per token for each at 128K context and tell me which you’d pick for a 1M-context product and why.”

Part 1 answers the probe. Parts 2–4 cover what Mistral and Cohere ask about experts. Part 5 builds the architecture table.

Part 1

Latent attention: shrink the cache, not the heads

The KV cache is the bill you pay per token

Bytes per token = 2 (K and V) × layers × KV heads × head size × 2 bytes.

Cutting KV heads shrinks the cache, but query heads then share keys and values. In DeepSeek’s own 7B test, quality fell as heads were shared.

Raschka notes the GQA and Llama 2 papers found GQA close to full multi-head attention. Treat the gap as one lab’s result. MLA asks: can we shrink the cache without sharing heads?

Llama 3 70B shapeKV headsper token
Multi-head (MHA)642,560 KiB
Grouped-query (GQA), as shipped8320 KiB
Multi-query (MQA)140 KiB
DeepSeek 7B ablationMHAGQA-8MQA
MMLU (5-shot)45.241.237.9
BBH (3-shot)37.035.633.2

80 layers, head size 128: Llama 3 paper, Table 3. Ablation: DeepSeek-V2, Appendix D.1, Table 8 (7B dense models, 1.33T tokens). Raschka, “The Big LLM Architecture Comparison”, §1.1.

MLA: cache a small latent, rebuild K and V from it

Hidden state h 7,168 numbers per token × W_DKV: shrink Latent c · cached 512 numbers × W_UK × W_UV Keys 128 heads × 128 Values 128 heads × 128 rebuilt when needed, never stored fold W_UK into the query fold W_UV into the output Decode attends to the latent itself all 128 query heads read one shared entry

Each token’s hidden state is squeezed to 512 numbers. Only that latent goes in the cache.

Two up-projections rebuild 128 heads of keys and values from it.

The trick: a query times rebuilt keys equals (query × W_UK) times the latent. So fold W_UK into the query and W_UV into the output, and decode never builds K or V at all.

DeepSeek-V2 §2.1.2 and Appendix C (the “absorb” step); sizes from DeepSeek-V3 §4.2. Queries are also compressed (to 1,536) to save training memory; that does not touch the cache.

Why MLA needs a separate path for position

The problem. RoPE rotates each key by its position. A rotation inside the key would sit between the query and W_UK, so W_UK could no longer be folded in. Every cached key would be rebuilt at every step.

The fix. Keep the latent position-free. Add one small key, 64 numbers, that carries RoPE and is shared by all 128 heads.

Each head adds a content score and a position score. Cache per layer: 512 + 64 = 576 numbers.

Context extension (YaRN, a RoPE rescaling, from 4K to 128K) is applied only to the 64-number key.

Hidden state h Latent c · cached 512, no position Key k_R · cached 64, RoPE(position) one per token, all heads share it Score for head i content (128 wide) + position (64 wide) then ÷ √(128 + 64), softmax, mix values

DeepSeek-V2 §2.1.3 (decoupled RoPE, eqs. 14–19) and §3.1.4 (YaRN on the decoupled key only). V3 uses the same sizes (§4.2).

Cache calculator: one sequence, real configs

Each bar is computed live from the model’s published config.

Tap a model to see its formula.

“Split over 8 GPUs” shows the cache per GPU when attention heads are split 8 ways (tensor parallel). GQA splits by KV head. MLA’s single latent is read by every head, so each GPU needs the whole thing.

Configs: Llama 3 paper Table 3; DeepSeek-V3 §4.2; Qwen3 report Table 2 with head size 128 from its config.json; gpt-oss card (36 layers, 8 KV heads of 64, half the layers a 128-token window); Mixtral Table 1; GPT-2 small. GiB = 2³⁰ bytes.

The probe, part 1

The arithmetic, said out loud

Llama 3 70B · GQA

2 × 80 layers × 8 KV heads × 128 × 2 bytes = 327,680 B = 320 KiB per token.

× 131,072 tokens = 40 GiB per sequence.

DeepSeek-V3 · MLA

(512 + 64) × 61 layers × 2 bytes = 70,272 B ≈ 68.6 KiB per token.

× 131,072 tokens = 8.6 GiB per sequence.

  • 4.7× smaller. Per layer it is 4,096 B vs 1,152 B (3.6×); V3 also has fewer layers.
  • Fairer size match: Llama 3 405B is 504 KiB per token, 7.3× V3.
  • At 1M tokens: 320 GiB (70B) vs 68.6 GiB (V3) per sequence, in BF16.

DeepSeek-V2 Table 1: MLA’s cache equals GQA with 2.25 groups. Same-shape MHA for V3 would be 3,904 KiB per token, 57× more.

The probe, part 2

MLA trades memory reads for arithmetic

Per cached token, per layer, at decode:

GQA (70B): read 4,096 B; 64 heads × 2 × (128 + 128) = 32,768 FLOPs → 8 FLOPs per byte.

MLA (V3): read 1,152 B; 128 heads × 2 × (576 + 512) = 278,528 FLOPs → 242 per byte.

An H800 needs roughly 260–300 FLOPs per byte to keep its math units busy. GQA decode waits on memory; MLA sits at the line. DeepSeek’s kernel write-up calls MLA decode compute-bound.

Attention only, one sequence, one GPU at 990 TFLOPS and 3.35 TB/s (H800 figures in DeepSeek’s FlashMLA deep-dive, Apr 2025), perfect overlap: time = the larger of read time and math time. Blue = reading the cache, orange = arithmetic.

The probe, part 3

What else MLA costs you

Position handling

RoPE lives on a 64-number side key, and YaRN is tuned on it alone. V3 was trained to 128K; 1M needs a new extension you must validate.

An alternative

Kimi Linear drops RoPE from its MLA layers, so they run as pure multi-query attention and need no RoPE retuning.

Kernels

GQA runs on stock FlashAttention kernels. MLA needs its own: a modified FlashAttention-2 in V2, then FlashMLA.

FlashMLA’s Sept 2026 release dropped Hopper GPUs and V3.

Parallelism

8 KV heads over 8 GPUs: 40 KiB per token each for 70B. MLA’s latent is copied to every GPU: 68.6 KiB each.

DeepSeek decodes with little or no tensor parallelism.

DeepSeek-V2 §3.1.3–3.1.4; V3 §3.4.2 (decode attention TP4 + DP80); FlashMLA README and Apr 2025 deep-dive (“we don’t use Tensor Parallel for decoding instances”); Raschka §4 (stock FlashAttention for plain GQA) and §14.4 (Kimi Linear).

The probe, part 4

What I’d pick for a 1M-token product

MLA, plus something that stops each token from reading all 1M others.

  • Memory: 68.6 vs 320 GiB per 1M-token sequence. One Llama 3 70B sequence outweighs its own ~140 GB of weights.
  • Math: V3’s attention at 1M costs 17.8 TFLOP per new token, 240× its 74 GFLOP of active weights. So add sparse or windowed attention, as DeepSeek V3.2 did on top of MLA.
  • Pick GQA if you must serve on stock kernels with tensor parallelism, or long context is rare.
  • Measure: 1M retrieval (needle, multi-hop) and decode tokens/s per GPU at your latency target.

Computed from the configs; 2 FLOPs per active parameter per token. Raschka §16 (V3.2 sparse attention) and §21 (GLM-5 uses MLA and DeepSeek Sparse Attention “to reduce the inference cost when working with long contexts”).

Interview check

Can you answer these in two minutes?

  • “What are Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), and what do they trade away?” Meta MSL · question bank
  • “Transformer internals and scaling laws, with emphasis on the “why” behind design choices” OpenAI · firsthand 2026
  • “Llama 3 uses GQA, DeepSeek-V3 uses MLA. Compute KV-cache bytes per token for each at 128K context and tell me which you’d pick for a 1M-context product and why.” senior-staff probe from the study guide

Strong answers name the trade MLA makes: about 4.7× less cache than Llama 3 70B, paid for in decode arithmetic, custom kernels and a split position path.

Part 2

Mixture of experts: more parameters, same FLOPs per token

An MoE layer: a router picks k experts per token

Token x 4,096 numbers Router: x × W_g one score per expert 1.2−0.32.10.4 −1.00.91.7−0.5 E1 E2 E3 E4 E5 E6 E7 E8 E3 E7 weight 0.60 weight 0.40 0.60 · E3(x) + 0.40 · E7(x) softmax over the two kept scores + x (residual) → next layer

Replace the single feed-forward block with N copies (experts). A tiny linear router scores them per token.

Each token runs only its top k. Mixtral: 8 experts, top 2, at every layer; the two picks can differ for every token and layer.

Six of eight experts do no work for this token. That is where the saving comes from.

Mixtral §2.1: y = Σ softmax(Top2(x · W_g))ᵢ · SwiGLUᵢ(x). Router scores in the figure are made up for illustration.

Total vs active parameters, computed

Why is “8x7B” 47B, not 56B? Attention, norms and embeddings are shared. Only the feed-forward block is copied 8 times.

Active = shared parts + k experts per layer. Compute per token follows active; memory follows total.

Convention, as in the gpt-oss card: the embedding lookup is not counted as active; the output layer is.

total
active per token

Expert and attention sizes from Mixtral Table 1, DeepSeek-V3 §4.2 and config.json, Qwen3 config.json, and the gpt-oss card’s Table 1 (whose per-component counts are used directly).

Where the cost lands: compute, memory, bandwidth

Compute follows active parameters: Mixtral uses 5× fewer active parameters than Llama 2 70B.

Memory follows total: all 47B must sit in GPU memory. The paper says MoE suits batched workloads.

Bandwidth depends on batch size. One token reads 2 of 8 experts; a batch of 8 tokens touches about 90% of them. Big batches pay for every expert’s weights, but share the read.

Share of each layer’s experts whose weights are read this step

Assumes each token picks experts uniformly and independently. Real routing is skewed, and Mixtral found consecutive tokens often reuse experts.

Mixtral §3 (“Size and Efficiency”) and §5 (routing analysis). Share = 1 − (1 − k/N)^batch, computed live.

Senior-staff angle

When sparse helps, and when it hurts

Helps

  • Qwen3 MoE bases match its dense ones with ~1/5 the active parameters.
  • DeepSeek-V2: 172.8K GPU-hours per trillion tokens vs 300.6K for dense 67B.
  • High throughput at large batch.

Hurts

  • Every expert in memory: V3 in FP8 is 671 GB, over one 8 × 80 GB node.
  • All-to-all traffic and load imbalance (Part 4).
  • Spiky, small traffic: weight reads dominate.
  • Fine-tuning overfits: Switch used dropout 0.4 in experts.

Cohere’s ~10× question: you pay memory for 10× the parameters and compute for 1×.

The full answer

Good fit when the customer can hold all weights and traffic batches well: they get a big model’s quality at a small model’s compute per token. It hurts on one small box with bursty single-user traffic (weight reads dominate), when experts span GPUs (all-to-all, imbalance), and when fine-tuning on small private data (overfitting; Switch needed expert dropout).

Qwen3 report §3.3; DeepSeek-V2 §3.2.3; DeepSeek-V3 §6 (“the recommended deployment unit … is relatively large”); Switch Table 4. 80 GB per GPU assumed.

Interview check

Mixture-of-experts questions labs ask

  • “What is a mixture-of-experts architecture and how does it scale capacity without scaling FLOPs?” Mistral · question bank
  • “Explain how a Mixtral-style sparse mixture-of-experts model works. Why does a ~47B-parameter model run at roughly the cost of a ~13B one?” Mistral · question bank
  • “Our flagship is a sparse MoE with ~10x more total than active parameters. Why is that architecture a good fit for private enterprise deployment, and where does it hurt?” Cohere · question bank

Say the arithmetic (46.7B total, about 13B active), then where the memory and bandwidth costs land.

Part 3

Routing and load balancing

Capacity factor: fixed slots, dropped tokens

Compiled kernels want fixed shapes, so each expert gets fixed slots: tokens ÷ experts × capacity factor.

Too many tokens for one expert: the extras skip the layer through the residual (dropped). Empty slots are padding: wasted compute.

Switch did best at factors 1.0–1.25 and, with its balancing loss, dropped under 1%. DeepSeek-V2 dropped tokens in training only; V3 drops none.

Capacity factor:
32 tokens, 4 experts, top-1 routing (made-up router). Numbers are token positions; first come, first served.

Switch Transformers §2.2, eq. 3 and Figure 3; DeepSeek-V2 §2.2.4; DeepSeek-V3 §2.1.2 (“No Token-Dropping”).

Two ways to even the load: a loss, or a bias

Aux loss (Switch, V2): add α × N × Σ fᵢ·Pᵢ, where f is each expert’s share of tokens and P its share of router probability. It equals 1 when balanced. Its gradient reshapes the router, and so the gate weights the model uses.

Bias (V3): add bᵢ to the scores only when choosing the top k. Gate weights still use raw scores. After each step: overloaded bᵢ − γ, underloaded bᵢ + γ.

64 tokens, 8 experts, top-2. Router scores (skewed toward expert 1) and γ = 0.02 are made up for illustration.

Switch §2.2, eqs. 4–6 (α = 0.01). DeepSeek-V3 §2.1.2, eqs. 16–20; γ = 0.001 for the first 14.3T tokens, plus a tiny sequence-wise loss with α = 0.0001. Green dashes: even load (16 picks each).

What the balancing ablations actually show

DeepSeek-V3: the bias beat aux losses on most benchmarks, at 16B and 229B total.

But a batch-wide aux loss matched it. What matters is where balance is enforced: per batch lets experts specialise by domain; per sequence spreads every document over all experts.

Qwen3 uses a global-batch balancing loss “to encourage expert specialization”.

Validation loss1B MoE3B MoE
Sequence-wise aux loss2.2582.085
Bias (aux-loss-free)2.2532.080
Batch-wise aux loss2.2532.080

The cost: small batches or shifted traffic unbalance experts at serving time. V3 adds copies of hot experts (Part 4).

DeepSeek-V3 §4.5.2 (Table 5) and §4.5.3; Qwen3 report §2.

Routing choices and what each costs

ChoiceGainsCosts
Top-1 (Switch)least router work, half the slots of top-2one expert per token
Top-k, k > 1 (Mixtral 2 of 8; V3 8 of 256)more expert combinationsmore traffic per token
Expert choice: experts pick tokensbalance built in, no aux losssome tokens get no expert, others several
Many small experts + a shared one (DeepSeekMoE)finer specialisation; shared expert holds common knowledgemore, smaller matmuls; Qwen3 dropped the shared expert
Device or node limit (V2: 3 devices; V3: 4 nodes)caps communicationV2 found 3+ devices roughly matched unrestricted routing

Switch §2.1; MoE survey §4.1.1 (expert choice; Komatsuzaki et al. kept token choice in the T5 decoder) and §4.2.4; DeepSeek-V2 §2.2.2; V3 §2.1.2.

Failure modes, and the fixes that shipped

Routing collapse. A few experts win early and the rest never train. Fix: balancing (loss or bias).
Router numerics. Switch-Base in bfloat16 diverged. Casting only the router to float32 recovered float32 quality at bfloat16 speed.
Initialization. Scaling init down 10× cut quality spread across seeds from 0.68 to 0.01.
Large router logits. ST-MoE’s z-loss penalises them to cut round-off in the softmax.
Scale. Switch-XXL (395B) was “sometimes unstable”, unsolved in the paper. DeepSeek-V3 reports no irrecoverable loss spikes.
Silent drops. With capacity limits, tokens late in a sequence are dropped more (OpenMoE’s “drop towards the end”).
A related senior-staff probe

“Loss spikes at step 40K of a 400B MoE run and one expert’s routing share jumps. What do you check first?” One way to answer: per-layer expert load and router logit size around the spike; router precision; whether a data shard changed at that step. Then rerun that data slice on a 1–3B MoE with the same router settings. (Week 4 covers instabilities.)

Switch Tables 2–3 and §5.6; MoE survey §4.1.1 (z-loss, OpenMoE) and §7; DeepSeek-V2 §2.2.3; DeepSeek-V3 §1 (no irrecoverable loss spikes).

Interview check

Routing questions

  • “Discuss trade-offs in MoE routing: top-k vs expert-choice routing, load-balancing vs quality.” Mistral · prep site
  • “Mixture of experts (and expert parallelism)” Mistral · firsthand 2026

Name the knob and its price: capacity factor (drops vs padding), top-k (quality vs traffic), loss vs bias (gate interference), sequence vs batch scope (specialisation vs serving balance).

Part 4

Expert parallelism: moving tokens to experts

Expert parallelism: tokens travel, weights stay put

  1. Each GPU runs attention on its own tokens and routes them. Numbers are target experts; colour is the home GPU.
  2. All-to-all dispatch: every token goes to the GPU holding its expert.
  3. Experts run dense matmuls on whatever arrived. GPU 1 got 6 tokens, GPU 3 got 2: everyone waits for GPU 1.
  4. All-to-all combine: outputs return home.
  5. Weighted sum plus residual, then the next layer. Forward and backward: four all-to-alls per MoE layer.

Two experts per GPU, made-up routing. MoE survey §5 and Fig. 8; Switch §5.4; Mixtral §2.1.

Trace one token through a DeepSeek-V3 MoE layer

  1. Score all 256 experts (sigmoid), add the balancing bias.
  2. Limit: pick at most 4 nodes, then the top 8 experts on them.
  3. Dispatch in FP8: one copy per node over InfiniBand (50 GB/s), then NVLink (160 GB/s) to each expert. At most 4 × 7,168 B per token per layer.
  4. Compute: each expert is 7,168 → 2,048 → 7,168. The shared expert runs locally meanwhile.
  5. Combine in BF16 with normalised gate weights, summed per node before the trip back.

Training ran near 1:1 compute to communication, hence DualPipe overlap (week 8).

DeepSeek-V3 §2.1.2 (node-limited routing), §3.2.1–3.2.2, §3.3.3 (FP8 dispatch, BF16 combine), §4.2. Byte count computed: 7,168 FP8 values per copy.

Senior-staff angle

Serving an MoE: DeepSeek-V3’s layout

Prefill: 32 GPUs

Attention: 4-way tensor × 8-way data parallel. Experts: 32-way expert parallel. 32 redundant copies of hot experts, re-chosen about every 10 minutes from live load.

Decode: 320 GPUs

Attention: 4-way tensor × 80-way data parallel. Experts: one per GPU; 64 GPUs hold redundant and shared experts. Batches per expert are usually ≤ 256 tokens, so decode is bound by memory reads.

  • Weights alone: 671B × 1 byte (FP8) = 671 GB, more than 8 × 80 GB. The unit of deployment is a cluster, not a box.
  • Contrast: gpt-oss-120b stores experts in MXFP4 (4.25 bits per parameter) and fits one 80 GB GPU.

DeepSeek-V3 §3.4.1–3.4.2 and §6; gpt-oss model card, “Quantization” (60.8 GiB checkpoint).

Interview check

Expert-parallelism questions

  • “Explain a mixture-of-experts transformer and how expert parallelism distributes its work. Trace what happens to a batch of tokens from routing…” Mistral · question bank
  • “Mixture of experts (and expert parallelism)” Mistral · firsthand 2026
  • “Our flagship is a sparse MoE with ~10x more total than active parameters. Why is that architecture a good fit for private enterprise deployment, and where does it hurt?” Cohere · question bank

Trace it with the five steps: route, dispatch, compute, combine, add. Then name the bottleneck: the most loaded GPU and the slowest link.

Part 5

The 2025 open architectures, side by side

The diff table, part 1: attention and position

GPT-2 smallLlama 3 70BDeepSeek-V3Qwen3-235B-A22Bgpt-oss-120b
AttentionMHA, 12 headsGQA, 64 query / 8 KV headsMLA, 128 heads, cache 512 + 64GQA, 64 / 4, QK-normGQA, 64 / 8; half the layers a 128-token window; sinks
KV cache per token
Position, contextlearned table, 1,024RoPE θ 500K, 128KRoPE on 64-d side key; YaRN to 128KRoPE θ 1M; 32K trained, 128K with YaRNRoPE; YaRN to 131,072
NormLayerNorm, before each branchRMSNorm, beforeRMSNorm, before, plus on latentsRMSNorm, before, plus QK-normRMSNorm, before

Scroll the table sideways.

Cache computed in BF16 from configs (gpt-oss counts dense layers only; window layers add a fixed 4.5 MiB). Llama 3 Table 3; V2 §2.1, §3.1.4; V3 §4.2; Qwen3 §2–3.2; gpt-oss card “Architecture”; Raschka §2 on RMSNorm and QK-norm.

The diff table, part 2: feed-forward and scale

GPT-2 smallLlama 3 70BDeepSeek-V3Qwen3-235B-A22Bgpt-oss-120b
Feed-forwarddense, GELUdense, SwiGLU1 shared + 256 routed, top 8; first 3 layers dense128 experts, top 8, no shared128 experts, top 4
Expert width(3,072)(28,672)2,0481,5362,880
Total / active124M / 124M70B / 70B671B / 37B235B / 22B116.8B / 5.1B
Balancing––bias, plus tiny sequence lossglobal-batch lossnot stated
Layers × width12 × 76880 × 8,19261 × 7,16894 × 4,09636 × 2,880

Scroll the table sideways.

V3 §4.2; Qwen3 §2 and config.json; gpt-oss card Table 1 and “Architecture”, expert width from Raschka §9.1; Llama 3 Table 3. Brackets: dense feed-forward width.

Why each lab chose what it did

DeepSeek: cheap serving at 128K on H800s. V2 vs dense DeepSeek 67B: KV cache −93.3%, training cost −42.5%.
Qwen3: GQA plus QK-norm for stable training. No shared expert: “we did not find significant enough improvement”.
gpt-oss: fits one 80 GB GPU: 4-bit experts, 128-token windows in half the layers, 4 of 128 experts active.
Llama 3: dense with GQA; gains “primarily driven by improvements in data quality and diversity” and scale.

Raschka finds them still “structurally similar” to GPT-2. The differences that matter are attention memory and feed-forward sparsity.

DeepSeek-V2 abstract; Qwen3 §2; Junyang Lin (Qwen) quoted in Raschka §6.2; gpt-oss card; Llama 3 §3.2.

Interview check

Defend every cell

  • “Explain the architectural choices in a Llama-class model: why grouped-query attention, RoPE and SwiGLU instead of the vanilla 2017 Transformer?” Meta MSL · question bank
  • “Transformer internals and scaling laws, with emphasis on the “why” behind design choices” OpenAI · firsthand 2026

For each row, say what it costs in bytes or FLOPs and which lab paid it. Week 2 covers RoPE, RMSNorm and SwiGLU; this week adds the MLA and expert columns.

The week in one minute

  1. MLA caches a 512-number latent plus a 64-number RoPE key: 68.6 KiB per token for V3 vs 320 KiB for Llama 3 70B.
  2. It pays in decode math (~242 vs 8 FLOPs per byte), custom kernels and a split position path.
  3. MoE copies the feed-forward block; compute follows active parameters, memory follows total (Mixtral 46.7B / ~13B).
  4. Routing needs balance: capacity factors drop tokens; aux losses touch the gates; V3’s bias does not.
  5. Expert parallelism moves tokens with two all-to-alls per layer; the busiest GPU sets the pace.
  6. The 2025 open models differ mainly in attention memory and feed-forward sparsity.

Your Week 3 plan

#ResourceListedPlanReadTake away
1DeepSeek-V231.5§2, App. C–Dlatent cache, decoupled RoPE
2DeepSeek-V3 report61§2.1, 3.2.2, 3.4, 4.2, 4.5.2–3configs, bias balancing, serving
3Switch Transformers21§2, 5, App. B–Ccapacity, aux loss, stability
4MoE survey (Cai et al.)41§2, 4.1, 5, 7routing types, all-to-all
5Mixtral of Experts10.75all47B vs 13B, routing locality
6Qwen3 report40.5§1–2, 3.3 summaryQK-norm, no shared expert
7gpt-oss model card20.5pp. 4–6windows, sinks, MXFP4
8Big LLM Architecture Comparison (Raschka) · gallery31§1, 5, 6, 9, 16the diff table

Listed: 25 h, far over this week’s ~7 of the shared 12–15. Read only the parts named (7.25 h); V3’s training story is week 8. Optional: DeepSeekMoE, Shazeer 2017.

Self-test: answer out loud, then open

DeepSeek-V3’s KV cache per token in BF16?

(512 + 64) × 61 layers × 2 bytes = 70,272 B ≈ 68.6 KiB; 8.6 GiB at 128K tokens.

Why can’t MLA put RoPE on the rebuilt keys?

The position rotation would sit between the query and W_UK, so W_UK could not be folded into the query; every cached key would be rebuilt each step. A separate 64-number RoPE key fixes it.

Mixtral 8x7B: total and active parameters, and why not 56B?

About 46.7B total and 13B active. Only the feed-forward is copied 8 times; attention and embeddings are shared.

Capacity per expert: 4,096 tokens, 64 experts, factor 1.25?

4,096 ÷ 64 × 1.25 = 80 slots. Extra tokens are dropped through the residual; empty slots are padding.

Switch aux loss under perfect balance, and why multiply by N?

N × Σ (1/N × 1/N) = 1. Multiplying by N keeps it constant as the expert count changes.

How does V3 balance without an aux loss?

A per-expert bias added only for top-k selection, nudged by γ = 0.001 each step against the load; gate weights use raw scores. Plus a tiny sequence loss (α = 0.0001).

Why is MLA decode compute-bound when GQA’s is memory-bound?

128 heads share one latent: about 242 FLOPs per cached byte vs 8 for Llama 3 70B. An H800 needs roughly 260–300 to be compute-bound.

Two ways MoE hurts serving?

All weights must be resident (V3 FP8: 671 GB, over one 640 GB node), and all-to-all traffic with load imbalance means the busiest GPU sets the pace.

Next

Week 4: scaling arithmetic and pretraining

Chinchilla, why labs over-train for cheaper inference, estimating a run on paper, data curation, and training instabilities with small-scale proxies.

Carry this forward

MoE changes the arithmetic: training FLOPs follow active parameters (6 × active × tokens), memory follows total.

← Week 2  ·  Week 4 →  ·  Study guide