Inference Optimization

Why this week is positioned where it is. Weeks 5-7 produced working systems. Week 8 produced the eval discipline to iterate them. This week is about the pragmatics of running them at scale. A Q&A system that costs $0.05/query is fine for an internal tool with 100 queries/day. The same system at 100 RPS costs $13,000/day — a $4.7M/year line item. The math forces specific architectural choices, and those choices have stabilized into a roughly canonical playbook.

1. The cost equation

The fundamental equation:

cost_per_request = (input_tokens × input_price + output_tokens × output_price) / 1,000,000
monthly_cost = cost_per_request × requests_per_second × 86,400 × 30

A few representative numbers at 2026 hosted API prices:

Two things in that table are worth pausing on, because both cut against the tidy mental model of "newer and bigger costs more."

Long-context surcharges are now a vendor-specific tax. Gemini 2.5 Pro doubles above 200k input tokens ($2.50 / $15.00). The GPT-5.6 family switches to long-context rates above 272k — Sol goes to $10.00 / $45.00. Anthropic prices a single rate across the whole window. A workload that lives at 300k tokens has a different vendor ranking than the same workload at 30k.

Google's newest Pro model is not generally available. Gemini 3.1 Pro is preview-only, so the production-grade Google flagship is still 2.5 Pro — which is why it sits below the newer 3.6 Flash on price here. Check GA status before you build a cost model on a benchmark chart.

For a typical RAG query (4,000 input tokens, 200 output tokens):

• GPT-5.6 Sol: $0.0260/call → at 10 RPS, $22,464/day = $8.2M/year
• Claude Opus 5: $0.0250/call → at 10 RPS, $21,600/day = $7.9M/year
• Claude Sonnet 5: $0.0150/call → at 10 RPS, $12,960/day = $4.7M/year
• GPT-5.6 Terra: $0.0104/call → at 10 RPS, $8,986/day = $3.3M/year
• Gemini 3.6 Flash: $0.0075/call → at 10 RPS, $6,480/day = $2.4M/year
• Gemini 2.5 Pro: $0.0070/call → at 10 RPS, $6,048/day = $2.2M/year
• Claude Haiku 4.5: $0.0050/call → at 10 RPS, $4,320/day = $1.6M/year
• Gemini 3.5 Flash-Lite: $0.0017/call → at 10 RPS, $1,469/day = $536k/year
• GPT-5.6 Luna: $0.0010/call → at 10 RPS, $899/day = $328k/year

The 25× spread from the most to the least expensive option is the difference between "this is a side project" and "this is a public company." Note also how little separates the three frontier tiers — Sol and Opus 5 land within 4% of each other — while the gap within a single vendor's range is an order of magnitude. Choosing the right tier matters far more than choosing the right vendor. The model-choice decision is still the largest financial decision you will make.

Where the cost actually goes. In RAG, ~95% of input tokens come from retrieved chunks; the user query is rounding error. In agents, input grows with conversation history and tool outputs. In chat, system prompts compound across turns. Most production cost is in input, not output. Caching strategies follow from this.

Capacity vs cost. You can't always trade money for capacity. API rate limits cap your throughput regardless of budget. Self-hosted inference gives you capacity control but adds GPU procurement, ops, and the engineering cost of running a serving stack. Most teams stay on hosted APIs until at least 50 RPS; at that scale, the math starts favoring self-hosting if you have specialized hardware.

2. Caching — the first optimization

If you take only one optimization from this lesson, take prompt caching.

Prompt caching stores the model's KV-cache for a stable prefix of your input. The next request that begins with the same prefix replays the cached state instead of recomputing. The provider charges a small fraction of the normal input price for cached tokens.

The economics:

• Anthropic (2026): cached input tokens at 10% of normal price. Cache write incurs a 25% premium on the first request that creates the cache. Cache TTL: 5 minutes (or 1 hour with extended TTL).
• OpenAI: cached tokens at 50% off automatically (no cache-control headers, no premium write). Cache hit rate is opaque.

The economics of prefix caching:

For a RAG system with 800-token system prompt, 4000-token retrieved chunks, 50-token query:

• Without caching: 4,850 input tokens × $3/M (Sonnet) = $14.55/1k requests
• With system prompt cached at 95% hit rate: (800 × 0.05 × $3 + 800 × 0.95 × $0.30 + 4050 × $3) / 1M × 1000 = $12.30/1k requests

Wait — that's only ~15% savings? The retrieved chunks are 4× larger than the system prompt and they're not cached. The savings from caching only the stable component is moderate.

The real wins come from architectures where the cacheable portion is most of the input:

• Multi-turn chat: the conversation history grows; each turn caches the previous turns. ~70% cost reduction at 20-turn conversations.
• Constant document context: applications that always include the same long document (legal contracts, codebases, policy manuals). 80-95% savings.
• Few-shot prompts with many examples: a long set of in-context examples that doesn't change between calls.

Cache placement matters. Anthropic exposes cache_control breakpoints; OpenAI does it automatically. The rule: cache up to the first byte that varies between requests. Cache too little and you miss savings; cache too much and the cache rarely hits.

Cache TTL matters operationally. A 5-minute TTL means low-traffic hours produce cache misses. Some workloads pin warm prefixes via periodic dummy requests; others use the 1-hour TTL for stability at small price premium.

Output caching is separate and rare in LLM products — the same query rarely produces the exact same output (sampling, randomness). For deterministic queries (greedy decoding, stable prompts), it works; for most generation, no.

Semantic caching — caching based on query meaning rather than exact match. Useful for FAQ-style traffic where many users ask near-identical questions. Risks: drift between the cached answer and current data. Used by some chat products; rare in serious RAG.

3. Model routing

Most production systems have a mix of query difficulties. Hard queries need the strongest model; easy queries don't. Routing is the practice of choosing the model per request.

The simplest router: a small classifier (or just a heuristic) decides "easy" vs "hard" and routes accordingly.

def route(query, conversation_history):
    if is_greeting(query):
        return "haiku"
    if is_simple_factual(query):
        return "haiku"
    if requires_reasoning(query, conversation_history):
        return "sonnet"
    if multi_step_synthesis(query):
        return "opus"
    return "haiku"  # default

The classifier itself can be an LLM call (a tiny one), a fine-tuned model, or just regex/heuristics. The cost of routing has to be small relative to the savings — Haiku-level classifier deciding between Haiku and Opus is a no-brainer.

Tier-based routing typical economics: if 70% of queries can use Haiku (~$0.005/call) and 30% need Opus (~$0.025/call), the average cost is $0.011/call. All-Opus is $0.025. Just over twice as cheap for moderate-quality queries; quality loss negligible if classification is correct.

The routing decision tree:

1. Classify by complexity. Greetings, simple factual lookups → smallest model. Reasoning, synthesis, code → larger.
2. Classify by criticality. First-touch customer support (high-stakes) might warrant a stronger model regardless of complexity.
3. Route by user tier. Free tier on cheap models; paid users on stronger models. Common in production.
4. Fallback on disagreement. Run cheap model first; if confidence is low or output looks malformed, retry on stronger model.

The mixture-of-experts pattern in production: instead of one router, a "rough draft → review" pipeline where the small model produces a draft and the large model reviews/edits. Used in some commercial assistants. Trades latency for cost.

Quality vs cost is genuinely a frontier. You can usually find points where routing saves 50-80% with no measurable quality loss. Going further trades quality for cost. The eval discipline from Week 8 tells you where the line is.

4. Quantization

When you self-host, quantization is the highest-leverage cost optimization. It reduces model weight precision (FP16 → INT8 or INT4), trading a small quality loss for large memory and throughput wins.

The 2026 quantization landscape:

• FP16 / BF16 — the standard "full precision" for inference. Baseline.
• FP8 — Hopper-generation NVIDIA hardware (H100, H200, B100). Same memory footprint as INT8 but typically better quality. The default for new self-hosted deployments on modern GPUs.
• INT8 (W8A8) — 8-bit integer weights and activations. Half the memory of FP16. Typical quality loss: 1-3% on most benchmarks for 7B+ models.
• INT4 (W4A8 or W4A16) — 4-bit weight quantization (often paired with 8-bit activations). 4× memory reduction. GPTQ, AWQ, and exl2 are common formats. Quality loss: 3-8% for 7B+ models, smaller models suffer more.
• Mixed precision — sensitive layers in higher precision, others in lower. The frontier of small-quality-loss quantization in 2026.

The throughput-not-just-memory win. Quantization isn't just about fitting bigger models on a GPU. INT8/INT4 ops use specialized hardware (NVIDIA tensor cores) that run faster than FP16. A quantized 70B model might serve 2-3× more requests per second than an FP16 model on the same GPU.

Quantization quality varies by task. Code generation tolerates quantization well; long-context summarization sometimes worse. Always re-run your eval set after quantizing. The Week 8 discipline matters here.

Calibration data matters for INT8/INT4. The quantization process needs sample data that represents your production distribution. Calibrating on Wikipedia and serving customer support produces worse results than calibrating on customer support transcripts.

When not to quantize: very small models (under 3B), where quantization quality loss is disproportionate; latency-critical inference where the calibration overhead doesn't pay back; hosted APIs (you don't control the precision the provider uses).

5. Speculative decoding

Speculative decoding is the most underused inference optimization with the largest typical speedup. It uses two models — a small "draft" model and the actual "target" model — to generate multiple tokens per forward pass of the target.

The mechanism:

1. Draft model generates K candidate tokens (cheap).
2. Target model evaluates all K positions in one forward pass (parallel attention is fine).
3. For each position, accept the draft token if the target's distribution is "compatible" with it; reject and replace with target's choice on first disagreement.
4. The accepted prefix becomes the generated tokens; the rejection point gets the target's correct token; total tokens this pass = (accepted draft count) + 1.

The speedup depends on the acceptance rate — how often the draft model agrees with the target. Standard formula: expected tokens per pass = (1 - α^(K+1)) / (1 - α) where α is acceptance rate, K is draft length.

For typical workloads:

• α = 0.7, K = 4 → 2.8 tokens/pass → ~2.8× speedup
• α = 0.85, K = 4 → 3.6 tokens/pass → ~3.6× speedup
• α = 0.5, K = 4 → 1.9 tokens/pass → ~1.9× speedup
• α = 0.7, K = 8 → 3.4 tokens/pass (diminishing returns at high K)

Quality is preserved. Because rejections are replaced with the target's correct token (not the draft's wrong one), the output distribution is mathematically identical to greedy decoding from the target. This is "speed without quality loss" in a strict sense.

The catch: latency vs throughput. Speculative decoding accelerates per-request latency for low-batch-size inference. At high batch sizes (continuous batching, see section 7), GPU saturation already provides parallelism, and speculative decoding helps less. Most useful when batch size is small or when you're latency-bound rather than throughput-bound.

Draft model selection. The draft should be smaller than the target (so its forward pass is cheap), but agree with it often (so acceptance rate is high). Common: a fine-tune of the target model that's been distilled, or a smaller model from the same family.

In 2026, speculative decoding is built into most production serving stacks (vLLM, TGI, TensorRT-LLM). Often configurable with a single flag. Take the speedup if you can.

6. KV-cache optimization

The KV cache from Week 1 is the largest memory consumer at inference. It grows linearly with context length and dominates GPU memory at long contexts. Several techniques manage it.

Paged attention (vLLM). Allocates KV cache in fixed-size pages, like virtual memory. Eliminates internal fragmentation that wastes 30-60% of memory in naive implementations. The 2026 default for serious self-hosted deployments.

Prefix caching (RadixAttention, vLLM-prefix). Different requests sharing a common prefix share the same KV cache pages. Identical to prompt caching at the API level, but inside the serving stack. Massive wins for traffic with shared system prompts, repeated context, or templated requests.

Sliding window / streaming. Cap KV cache at the most recent N tokens. Beyond N, earlier context is dropped. Trades capability (no long-context recall) for memory. Used for chat applications where only recent context matters.

Attention sinks. A 2024 finding: keeping the first 4 tokens of context in the KV cache (along with sliding window) preserves model quality much better than naive sliding window. The first tokens carry disproportionate attention weight.

MLA (multi-latent attention) and similar architectures (DeepSeek): factorize the KV cache so it's smaller without losing quality. Architectural rather than runtime optimization.

For most teams using hosted APIs, this is invisible — the provider does it. For self-hosted inference at scale, paged attention via vLLM is the right starting point.

7. Batching and serving stacks

A GPU can serve many requests simultaneously, sharing compute. Batching is what extracts that capacity.

Static batching. Wait for B requests, send them as a batch, return all when finished. The naive approach. Wasteful when requests have different lengths — fast requests wait for slow ones.

Continuous batching (in-flight batching). When any sequence in the batch finishes, replace its slot with a new request without waiting for the others. Implemented in vLLM, TGI, TensorRT-LLM. Lifts GPU utilization from ~30% (static) to ~85% (continuous).

The 2026 serving-stack landscape:

• vLLM — open-source, the most common choice for self-hosted production. Strong continuous batching, paged attention, multi-LoRA, speculative decoding. The default.
• TGI (Text Generation Inference) — Hugging Face's serving stack. Mature, slightly less performant than vLLM in 2026 benchmarks but easier integration with HF ecosystem.
• SGLang — competitive with vLLM in raw throughput, distinguishes on structured-generation features (forced JSON, regex constraints).
• TensorRT-LLM — NVIDIA's. Highest peak performance on NVIDIA hardware; requires more setup. Used at extreme scale.
• llama.cpp — CPU and Apple Silicon inference. Not for cloud production, but the standard for local/edge deployment.

The deployment shape that has stabilized: vLLM behind a load balancer, fronted by an API gateway, with per-customer rate limits and budget controls. Most production serving stacks at companies running their own models look like this in 2026.

Latency vs throughput tradeoff. Larger batches mean higher throughput but higher per-request latency (each request waits for others). The right batch size depends on whether you're latency-bound or throughput-bound. Low-traffic interactive products favor small batches; high-volume async workloads favor large.

TTFT (time-to-first-token). The latency from request start to first generated token. Dominated by prompt processing time and queue wait. Critical for chat UX. Optimization: smaller prompts, paged attention to reduce queue wait, dedicated decoding pools.

TPS (tokens per second). Throughput per request after first token. Dominated by generation step latency. Speculative decoding helps here.

Production latency budget for chat: TTFT under 500ms, TPS over 50 tokens/sec for a useful experience. Gets harder at high traffic; the serving-stack optimizations above are how teams hold the line.

8. The optimization playbook

The order to apply optimizations, ranked by typical impact-per-engineering-day:

1. Measure first. Don't optimize what you haven't measured. Cost dashboards, per-customer/per-feature attribution, latency percentiles.

2. Caching. Identify the stable prefix in your prompt structure. Add prompt caching with appropriate TTLs. Free 10-50% savings depending on workload shape.

3. Model routing. Add a router. Route obvious-easy queries to the cheapest viable model. Maintain eval coverage at every tier. Often 50-80% additional savings with no quality loss.

4. Output optimization. Reduce output tokens where possible. Structured output (JSON) is sometimes shorter than prose. Stop sequences cut off rambling. Output token costs are 4-5× input — reducing 100 output tokens saves more than reducing 100 input tokens.

5. Batch API for non-real-time. 50% off both input and output with 24-hour SLA. Most analytics, summarization, and report-generation workloads can wait — but don't, because no one set up batch.

6. Switch to self-hosting if you're at scale. Past ~50-100 RPS sustained, the math starts favoring self-hosting if you have the engineering capacity. Below that, hosted APIs are cheaper than your eng-time + GPU bills.

7. Quantization (if self-hosting). INT8 or FP8 first. Re-eval after. Then INT4 if you need more memory and quality holds.

8. Speculative decoding (if self-hosting). Single-flag config in modern stacks. 2-3× throughput typically. Take it.

9. Continuous batching, paged attention, prefix caching (if self-hosting). Use vLLM. These are mostly free with the right serving stack.

10. Hardware upgrades. Last resort. New GPUs are expensive; the optimizations above usually deliver more lift per dollar.

The cost-monitoring discipline. Budgets are policies, not aspirations. Set per-customer / per-feature caps. Alert on cost spikes the same way you alert on error spikes. The first month after launch is when surprise costs surface; have monitoring before, not after.

Latency budgets are policies too. TTFT under 500ms for chat. P95 latency under 3s for question-answering. P99 under 5s. Alert on percentile breaches. Tail latency degrades the average user experience invisibly.

Production cost-quality monitoring as Pareto fronts. Don't only track cost or only track quality. Track both, and make tradeoff decisions explicit. "We accepted +2% quality for −30% cost" is a defensible decision; "we just made it cheaper" is not.

Build this week

Pick at least two:

1. Inference cost audit. Pull last month's API bill. Break down: which queries, which models, which customers consume the most. Identify the top 3 cost drivers. Estimate savings from caching + routing on those specifically.

2. Implement prompt caching on your highest-volume endpoint. Measure savings vs baseline. Include cache hit rate as a metric you track ongoing.

3. Build a router. Classify your traffic into 2-3 tiers. Route to appropriate models. Compare quality on your eval set vs the all-frontier baseline. Aim for 50%+ cost reduction at <2% quality loss.

4. Set up per-customer/per-feature cost attribution. The first time someone asks "what did Customer X cost us last month?", you should be able to answer in 30 seconds. If you can't, build the dashboard.

5. Run your eval set on Haiku 4.5 vs Sonnet 5 vs Opus 5. Identify the strata where the small model holds up. That's the routing strategy that emerges from the data.

Read this

• Anthropic prompt caching documentation. Reference for the parameter details and best practices.
• vLLM paper and docs. "Efficient Memory Management for Large Language Model Serving with PagedAttention" (Kwon et al. 2023) plus the project docs.
• The original speculative decoding paper. Leviathan et al., "Fast Inference from Transformers via Speculative Decoding" (2023).
• MLA paper from DeepSeek. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model" (2024). The KV cache architectural innovation.
• Hugging Face's text-generation-inference documentation. Reference for TGI deployment patterns.
• Anyscale, Modal, Replicate engineering blogs. Each has practical writeups on production inference at scale.

Interview prompts

1. Walk through the inference cost equation. What dominates the cost in a typical RAG application?
2. What is prompt caching, and what determines whether it saves you money?
3. Compare quantization techniques: FP8, INT8, INT4. When is each appropriate?
4. Explain speculative decoding. Why does it preserve output quality? When does it not help?
5. What's the difference between static and continuous batching? Why does the latter matter at scale?
6. Walk through model routing. How do you decide which queries to route where?
7. Your team is debating self-hosting vs continuing on a hosted API. What's the deciding question?
8. What's TTFT vs TPS, and which matters when?
9. Your cost spiked 3× last week with no traffic increase. What's your investigation order?
10. What's the cost-quality Pareto frontier, and why is it useful framing?

What "done" looks like

By the end of this week you should be able to:

• Compute the per-request and monthly cost of a given LLM workload.
• Apply prompt caching effectively, including knowing where to put the cache breakpoints.
• Design a model router and validate it on an eval set without quality regression.
• Make the self-hosting vs hosted API decision based on traffic and capacity.
• Know which serving stack to reach for and why (vLLM is the answer for most teams).
• Set up per-customer/per-feature cost monitoring and budgets.
• Recognize when you're in a "latency-bound" vs "throughput-bound" regime and apply the right optimizations.

If you can do those, you're ready for Week 10 — multimodal and code agents — where many of these optimizations matter even more, because multimodal context and long-running code agents are the two most expensive workload classes in 2026.