Modern LLM architecture

Why start here. Every other week in this plan assumes this vocabulary. RAG, agents, fine-tuning, RL post-training — all of them refer back to the underlying architecture. An afternoon spent here saves a week of confusion later.

1. The transformer, just enough of it

The transformer is the architecture under every modern frontier model. Every variant you'll meet — GPT, Claude, Gemini, Llama, DeepSeek, Qwen, Mistral — is a decoder-only transformer with a handful of architectural tweaks. The tweaks matter and we'll get to them. But the bones are the same.

Here's the practitioner-level mental model. A modern LLM does this for every token it generates:

1. Embed. Look up a vector for the input token.
2. Attend. Each token computes three projections: a Query (Q), a Key (K), and a Value (V). It then attends to all previous tokens by taking dot products of its Q with their Ks, softmaxing those into weights, and using those weights to combine their Vs. This is called causal self-attention — causal because future tokens are masked out.
3. Feed-forward. Pass the result through a small MLP.
4. Repeat. Stack N of these blocks (often 32–80 layers in modern models).
5. Project. The final hidden vector gets multiplied by an unembedding matrix to produce logits over the vocabulary, which become the next-token probability distribution.

Two facts about this you must internalize, because they drive everything else:

Attention is O(n²) in sequence length. Every token attends to every previous token. Doubling the context quadruples the attention compute. This is why context length is a genuine engineering constraint, not just a number on a spec sheet — and why most "modern arch tweaks" are about reducing this cost.

Inference and training have very different cost profiles. During training, you process whole sequences in parallel. During inference, you generate tokens one at a time, recomputing attention against everything you've generated so far. To avoid recomputing the same Ks and Vs, you cache them — this is the KV cache, and it's the single most important thing to understand about LLM inference economics. The KV cache scales linearly with both sequence length and batch size, and it dominates the memory cost of serving LLMs. We'll revisit this in Week 9.

If any of this feels foggy, watch Karpathy's Let's reproduce GPT-2 video before continuing. Two hours, and you'll never feel lost in this material again.

2. The architectural tweaks that actually matter in 2026

Modern models all share the decoder-only transformer skeleton, but they differ in five mostly-orthogonal ways. Knowing these distinguishes someone who has actually read the papers from someone who has read about the papers.

Rotary Position Embeddings (RoPE)

The original transformer used learned or sinusoidal position embeddings: it added a position vector to each token embedding, and the model figured the rest out. RoPE ("rotary") replaces this. Instead of adding positions, RoPE rotates the Q and K vectors by an angle that depends on the token's position. The dot product between Q at position i and K at position j then naturally encodes the relative distance i − j.

Why the field switched: RoPE generalizes better to sequences longer than what was trained, and it composes cleanly with the modern toolkit for extending context (YaRN, NTK-aware scaling, ABF). Llama, Mistral, Qwen, DeepSeek, and most open-weight models use it. If you read "RoPE base = 10000" or "RoPE base = 500000" in a model card, those are the frequencies controlling how fast the rotation accumulates with position — higher base means slower rotation, which lets the model reach longer contexts before periodicity bites.

Grouped-Query Attention (GQA) and MLA

Standard multi-head attention (MHA) gives each head its own Q, K, and V projections. The KV cache is therefore proportional to n_heads × head_dim × seq_len × batch. For a Llama-70B-class model with long context this gets ugly fast.

Three approaches to shrink this:

• MHA: full quality, full cost. The original.
• MQA (Multi-Query Attention): all heads share a single K and V. Tiny KV cache, but quality drops.
• GQA (Grouped-Query Attention): groups of heads share K and V. The sweet spot. Llama 2 70B was the model that established this as the default; almost everyone uses it now.

In 2024 DeepSeek introduced MLA (Multi-head Latent Attention) in their V2 and V3 models. MLA compresses keys and values into a low-rank latent representation, then projects back up at attention time. The KV cache shrinks even further than GQA at similar quality. It's specific to the DeepSeek line for now, but expect to see it copied.

If you remember one thing about these: KV cache size is the binding constraint on serving LLMs at long context, and these tricks are the levers that move it.

Mixture of Experts (MoE)

In a dense model, every token activates every parameter in the feed-forward layers. In a Mixture of Experts model, the feed-forward layer is replaced with N "expert" FFNs and a small router. The router picks the top-k experts (usually k = 1 or 2) for each token. Total parameters can be huge; active parameters per token are small.

The math: DeepSeek-V3 has 671B total parameters but activates only ~37B per token. Mixtral 8x7B has ~47B total but activates ~13B per token. Llama 4 Maverick uses MoE; GPT-4 is widely believed to. The training is harder (expert load-balancing, routing instability) but inference is dramatically cheaper per token at a given capability level. The tradeoff: you still need enough memory to hold all the experts, even though you only use a fraction at a time. So MoE wins on serving cost but not on memory cost.

If a model card lists "X total / Y active" parameters, it's MoE. Pay attention to both numbers.

Sliding window attention

Mistral 7B popularized this: instead of attending to all previous tokens, attend only to the last W tokens (Mistral used 4096). The model can still propagate information from further back through the layer stack — each layer extends the effective receptive field — but the attention compute and KV cache shrink dramatically.

Less common in pure form on the very largest models now, but the idea lives on in hybrid attention patterns where some layers attend globally and others locally.

Normalization, activation, and other small things

A few smaller tweaks that all open-weight models converged on:

• RMSNorm instead of LayerNorm (cheaper, similar quality).
• SwiGLU activation in the feed-forward layer instead of ReLU/GELU.
• No bias terms in linear projections (saves params, no quality loss).
• Pre-norm (norm before each sublayer, not after) for training stability.

You don't need to argue about these. Just recognize them in code.

3. Tokenization is where weird bugs live

Tokenization is the boring part everyone skips, and it causes more production bugs than any other single thing in this stack.

How it actually works

Almost every modern LLM uses a variant of Byte-Pair Encoding (BPE): start with bytes, greedily merge the most common pairs, repeat until you have a vocabulary of ~32k–256k tokens. The tokenizer is trained on a specific corpus, and its merges reflect that corpus. OpenAI uses a BPE variant called Tiktoken; Llama uses SentencePiece; DeepSeek and Qwen each have their own tokenizers. Critically, tokens are not words. They are statistical fragments.

Concrete examples:

• The string "hello" is one token in most modern tokenizers (~4 bytes per token on English text is typical).
• The string "12345" may be one token, two tokens, or five tokens depending on the tokenizer. This is why models are bizarrely bad at arithmetic on novel numbers.
• Code, especially indentation and uncommon symbols, often tokenizes inefficiently.
• Non-Latin scripts (Chinese, Arabic, Korean) often produce 2–4× more tokens per equivalent text than English. This has direct cost and context-length implications for non-English use.
• The same English text in two different tokenizers can produce token counts that differ by 20% or more.

Why this matters in production

1. Pricing is per-token. A user on a non-English language pays more for the same content. A bug where you accidentally double-encode (UTF-8 in a UTF-16 string, etc.) can silently 4× your costs.
2. Context limits are in tokens, not characters. "128k context" can mean very different things for English vs Japanese.
3. Prompts are tokenization-sensitive. Adding a leading space changes the first token, which changes the model's response. Trailing newlines matter.
4. Anomalous tokens exist. The famous SolidGoldMagikarp and related tokens were artifacts of training-corpus statistics that the model could never properly handle. Every tokenizer has some. Production systems occasionally hit them.

What to actually do

Open a Python REPL and inspect tokenization on real inputs:

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")

cases = [
    "Hello world",
    "12345",
    "1 2 3 4 5",
    "    def foo(x):",
    "你好世界",
    "🎉🚀✨",
]

for c in cases:
    ids = tok.encode(c)
    print(f"{repr(c):40s}  {len(ids):3d} tokens  {ids}")

Run this once and the abstract claim that "tokens aren't words" becomes a thing you've seen with your own eyes. Do the same with the OpenAI Tiktoken library and compare counts. This exercise alone will make you better than most of the people building LLM products today.

4. Sampling: how a probability distribution becomes text

Once the model produces logits over the vocabulary, something has to turn them into a chosen token. That something is the sampler, and the sampler choices have outsized effects on output quality.

The standard knobs:

• Greedy decoding. Take the argmax. Deterministic. Often boring; can also fall into repetition loops.
• Temperature. Scale logits by 1/T before softmax. T = 1.0 is "natural"; lower T sharpens the distribution (more deterministic), higher T flattens it (more random). T = 0 is greedy.
• Top-k. Sample only from the top k tokens by probability. Cuts the long tail.
• Top-p (nucleus). Sample from the smallest set whose cumulative probability ≥ p. Adapts to the distribution: in confident contexts the set is small; in uncertain contexts it's larger.
• Min-p. Reject any token with probability less than p × max_prob. Newer (2023) and increasingly common; often produces better-quality samples than top-p at the same diversity.
• Repetition penalty. Multiplicative penalty on tokens already produced. Brittle but useful for older models that loop.

The sampler interacts with the model. A model that's well-RLHF'd often has very peaked output distributions, and aggressive temperatures can push it off-distribution. A reasoning model's "thinking" tokens benefit from different sampling than its "answer" tokens. Most APIs default to something like temperature=1.0, top_p=1.0 on the assumption you'll set what you need.

A practical decoding cheat sheet

• Code generation. Low temperature (0.2 or 0.0). Determinism matters more than creativity.
• Structured output (JSON, function calls). Use the provider's structured-output mode if available; it constrains decoding to valid output. Fall back to low temperature otherwise.
• General assistant tasks. Temperature 0.7 with top-p 0.9, or temperature 1.0 with min-p 0.05. Both are reasonable defaults.
• Creative writing. Higher temperature (1.0–1.3) with min-p as the cutoff. Don't use top-k alone; it loses too much.
• Need exact reproducibility. Greedy + a fixed seed if your provider exposes one. Note that batched inference can still introduce nondeterminism from floating-point order-of-operations.

5. The 2026 frontier landscape

You don't need to memorize a leaderboard. You do need a working map of who makes what, what each is known for, and roughly how to choose between them. Survey first; details when they matter.

Closed frontier models

• Claude (Anthropic). Currently led by Claude Fable 5 for the most demanding long-horizon reasoning, with Opus 5 for complex agentic and enterprise coding, Sonnet 5 as the fast-frontier tier, and Haiku 4.5 as the cheap tier. Known for: writing quality, instruction following, agentic capability, constitutional AI alignment approach. Extended thinking lets you spend variable inference compute on hard problems. Strong tool use.
• GPT family (OpenAI). The GPT-5.6 family — Sol, Terra and Luna across the frontier, mid and budget tiers; o-series models for reasoning workloads. Known for: best general-purpose API surface, function calling that "just works," good multimodal. Slightly less consistent on long-context retrieval than the others.
• Gemini family (Google). Gemini 2.5 Pro is still the generally-available top tier — the newer 3.1 Pro is preview-only — with 3.6 Flash and 3.5 Flash-Lite for cheap and fast. Known for: native multimodality (especially audio and video), the longest production context windows (>1M tokens), tight Google ecosystem integration via Vertex AI.

The closed frontier moves in unison. By the time you're reading this, version numbers will have ticked. The character of each lab — Anthropic's writing, OpenAI's tool ecosystem, Google's multimodal reach — is more stable than the model versions.

Open-weight models

• Llama (Meta). Llama 4 series with MoE variants. The reference open-weight model — most of the open-source ecosystem (vLLM, Hugging Face, Unsloth, Axolotl) optimizes for it first. Permissive license with a couple of carve-outs.
• DeepSeek. V3 (general purpose) and R1 (reasoning) lineages. Genuinely competitive with closed frontier on many benchmarks at a fraction of the inference cost. MLA architecture gives them a serving-cost edge. If you're cost-sensitive and willing to self-host, DeepSeek is your strongest open option.
• Qwen (Alibaba). Qwen 2.5 / Qwen 3 series. Strong on multilingual (especially Chinese) and code. Excellent small-model variants in the 0.5B–7B range — useful for edge deployment.
• Mistral. Mistral Large, Mixtral (MoE), Codestral. Smaller, efficient, with a French-but-very-capable feel. Strong open licenses. Good for European hosting / data-residency constraints.

How to pick, in practice

If you're building a product, the choice tree usually goes:

1. Frontier capability needed? Claude Opus 5 / GPT-5.6 Sol / Gemini 2.5 Pro. Eat the cost.
2. Good enough, low cost? Sonnet 5 / GPT-5.6 Terra / Gemini 3.6 Flash. Most production traffic ends up here.
3. High volume, tight budget, willing to host? DeepSeek V3 self-hosted, or via an API provider like Together, Fireworks, DeepInfra. Llama 4 and Qwen 3 are the close alternatives.
4. Edge / on-device? Qwen 0.5B–3B, Llama 3.2 1B, Phi-3 mini.

We'll come back to cost optimization in detail in Week 9. For now: get the map, don't get attached to specific model names.

6. Reasoning models and test-time compute

The biggest shift since this plan's inception is the rise of reasoning models — models trained to produce long chains of thought before answering. The watershed was OpenAI's o1 in September 2024; DeepSeek-R1 reproduced the recipe openly in January 2025; Claude's extended thinking arrived shortly after. By 2026, every frontier lab offers a reasoning variant or a thinking-budget control on its general models.

What changed

Pre-reasoning, all your inference compute was spent on the answer. A reasoning model spends compute before the answer, generating tokens that explore the problem, try approaches, and self-correct. Those tokens are usually hidden from the user (Claude shows them; OpenAI summarizes them; DeepSeek shows them in raw form). The visible answer is what you read, but most of the compute went into the thinking.

The result: dramatic gains on hard math, code, and multi-step reasoning. AIME problems that base models couldn't solve at any temperature get solved reliably. Code problems that needed agentic scaffolding get solved end-to-end. The cost: latency goes from seconds to tens of seconds, and per-query token cost rises 5–20×.

How to use them

The decision is per query, not per product. A reasoning model on a "what's the capital of France?" question is wasteful. A regular model on a 15-step debugging problem is wasteful too. Modern APIs let you toggle thinking on/off or set a budget:

• Claude extended thinking. A thinking-budget parameter in tokens; the model uses up to that budget when it deems necessary.
• GPT (o3-style). Reasoning effort (low/medium/high) as a parameter.
• Gemini. Thinking modes per request.

A reasonable pattern in production: use a regular model by default, route queries that the model itself or a classifier deems "hard" to a reasoning model. We'll cover routing economics in Week 9.

Why it matters beyond the obvious

The deeper implication is that training pipelines now look different. You can train a model to reason by giving it a verifiable reward (math has answers, code has tests) and letting RL reinforce whichever chains of thought lead to correct answers. This is RL with verifiable rewards (RLVR), and it's how R1 was trained. We dedicate all of Week 4 to this — it's the most important week of the plan for staying current.

For now: know that reasoning models exist, know the cost/latency profile, and know that you don't need them for everything.

Build this week

The point of this week is to make the abstract concrete. Pick at least two of the following and actually do them.

1. Tokenization tour. Load three different tokenizers (Llama 3, Tiktoken/GPT-4, Qwen 2.5). Encode the same set of prompts — English, code, numbers, Chinese, emoji. Compare token counts. Write up the patterns you find. (~1 hour)

2. Sampling comparison. Load a small open model (Llama 3.2 1B is enough). Generate completions for the same prompt at five different sampling configurations: greedy, temperature 0.7 + top-p 0.9, temperature 1.2 + top-p 0.95, temperature 1.0 + min-p 0.05, and a high-temperature high-top-k. Note the differences in coherence, repetition, and creativity. (~1.5 hours)

3. KV cache size calculation. For a 7B model with 32 layers, hidden dim 4096, and 32 attention heads, calculate the KV cache size in MB at sequence length 8192, batch size 1, in fp16. Now redo the calculation assuming GQA with 8 KV heads. Compare. (~30 minutes)

4. Reasoning model walkthrough. Make the same hard math or coding question to a regular model and a reasoning model. Read both responses carefully. Note what the reasoning model's thinking trace shows you about how it solved (or didn't solve) the problem. (~1 hour)

5. (Stretch) Architecture spotting. Read the DeepSeek-V3 technical report. Identify each architectural choice we covered: tokenizer, RoPE base, MoE structure, MLA, sliding window or full attention. Make a one-page architecture summary. (~2 hours)

Read this

In rough priority order. Don't read everything; read what you need.

• Karpathy, Let's reproduce GPT-2. Two-hour video. The single best refresher on what's actually happening inside a modern LLM. Watch it.
• Jay Alammar, The Illustrated Transformer. Still the best visual explanation. Skip if you're already comfortable.
• Llama 3 paper (sections 3 and 4). The architecture and training details — not the benchmarks. This is the most readable modern flagship paper.
• DeepSeek-V3 technical report. Read for MLA and the MoE specifics. About an hour.
• DeepSeek-R1 technical report. Saves for Week 4, but skim now to know what's coming.
• Lilian Weng, Attention? Attention! and her follow-up posts. The evergreen reference for attention variants.

Interview prompts

Treat these as warm-ups. If any one of them feels shaky, that's a section to revisit.

1. Why is attention O(n²) and what classes of techniques have been used to address that?
2. Walk through what a KV cache is and why it dominates LLM inference memory.
3. What's GQA, and what tradeoff does it make compared to MHA and MQA?
4. Explain MoE in two minutes. Why does inference cost decouple from total parameters?
5. RoPE vs sinusoidal position embeddings — what does RoPE actually do, and what does it buy you?
6. Why did the field move from BERT-style encoders to decoder-only models for language work?
7. Walk through the tradeoffs between top-p, top-k, and min-p sampling.
8. When would you use a reasoning model and when wouldn't you?
9. You're given a 13B-param model and asked to estimate serving cost at 100 QPS, 8k average context. Walk through how you'd estimate KV cache memory.
10. Pick any two frontier models and tell me how they differ — architecturally, in capability, and in pricing.

What "done" looks like

By the end of this week you should be able to:

• Read a model card and identify every architectural choice it lists.
• Explain to a colleague why MoE models report two parameter counts.
• Pick a reasonable sampling configuration for a given task without looking it up.
• Estimate, within an order of magnitude, the KV cache size for a model and configuration.
• Articulate when a reasoning model is worth its cost.

If you can do those, you're ready for Week 2.