The Practitioner's LLM Curriculum ← Week 1 · KV Cache Calculator
Interactive · Week 1 · Section 2

The real cost of long context.

Pick a model, set your workload, and see exactly how much GPU memory the KV cache eats. The same context length costs an order of magnitude more on a 70B model than on a 7B — and an order of magnitude more again with MHA than with GQA. This is the hidden tax that shapes every serving decision.

Model architecture
Workload
Context length 8192
Tokens in the context window. Includes prompt and generated tokens.
Batch size 1
Concurrent requests served. Higher batches = better throughput, more KV cache.
Precision fp16
Bytes per element. Quantized KV cache (fp8/int8) is increasingly common in production.
KV Cache size · current settings
—
If this model used…
Hardware fit
Bars show KV cache as a fraction of total VRAM. Real serving also needs ~2× the model weights — KV cache fights for what's left.

What's happening here

    KV cache size for standard attention = 2 (K,V) × layers × seq_len × n_kv_heads × head_dim × batch × bytes_per_element. For MLA (DeepSeek), the K and V are derived from a shared low-rank latent, so the cache is layers × seq_len × (kv_lora_rank + qk_rope_head_dim) × batch × bytes — no factor of 2. Quantized KV cache uses 1 byte per element for fp8/int8, 0.5 for int4. Model weights and activations are additional memory.
    The Practitioner's LLM Curriculum · Venkatesh Baglodi · all weeks · widgets · about