Production Patterns and Interview Preparation

Why this week is positioned where it is. Weeks 1-11 are technically deep; Week 12 is engineering-cultural. Capable systems still get pulled offline if no one sets SLOs or runs the on-call rotation. The shift from "I can build it" to "I can keep it shipped under load" is the difference between a senior IC and a staff engineer in this space. The interview prep section gives you the muscle for explaining the prior eleven weeks to a panel.

1. Deployment patterns adapted for non-determinism

The deployment patterns from classical web infrastructure — canary, shadow, blue-green — all need adjustment for LLM systems. The reason is non-determinism: the same prompt can produce different outputs across versions, across temperature settings, across time. Classical patterns assume "is the new version functionally equivalent on the same input?" That question doesn't have a clean yes/no answer for LLMs.

Canary deployments. The classical pattern: route 5% of traffic to the new version, monitor error rates and latency, gradually increase if healthy. For LLM systems:

• Quality monitoring is non-trivial. You can't just check error rate. You need a quality signal — typically the eval-in-prod scoring discussed in Week 8. If the new model regresses on your eval set even though no errors are thrown, classical canary won't catch it.
• The 5% sample is small. LLM quality issues often show up on rare query types. 5% of traffic on a low-volume product may be too sparse to detect a 5pp regression.
• Per-segment monitoring. Quality regressions often affect specific user segments more than others. Slice canary metrics by user type, query type, language, time of day.

Shadow deployments. Run the new version in parallel with the old, sending both the same inputs. Compare outputs. The dominant production deployment pattern for LLM updates in 2026.

• The advantage: full traffic on new version, no user impact, direct old-vs-new comparison.
• The challenge: comparing outputs is hard. "Different but both correct" is common. Need a judge or an eval pipeline to grade systematically.
• Cost: 2× inference cost during the shadow window. For high-traffic products this is real money — typically a few days at most.

Blue-green deployments. Two parallel environments; switch traffic instantly between them. Fits LLM systems well because rollback is fast and clean. The wrinkle: stateful caches (KV cache from Week 1, prompt cache from Week 2) don't transfer between blue and green; expect cold-cache performance for the first hour after switch.

Feature-flag-based rollouts. Instead of version-level deployment, gate individual prompt changes, retrieval changes, model swaps behind flags. Roll out per user segment. The 2026 LLM-product norm: weekly tweaks to prompts and retrieval gated behind flags, monthly model swaps with shadow + canary.

The most-skipped step: a defined rollback trigger. Before rolling out, declare: "If quality drops below X on the eval-in-prod set, or error rate exceeds Y, or cost per request exceeds Z, automatically roll back." Without this, rollout decisions get made in the heat of the moment by whoever is paged. With it, the system protects itself.

2. The on-call playbook

What does it look like when an LLM product breaks at 3am?

The four common incident classes:

1. Quality regression. Model output is fluently wrong. No error code thrown. Detected by eval-in-prod, user complaints, or a verifier (Week 11) flagging spikes.
2. Latency spike. P99 latency doubles. Could be model serving, retrieval index, downstream tools, or prompt-cache miss after deploy.
3. Cost explosion. Per-request cost rises sharply. Often: agent that learned to loop, longer context-windows on cached prompts that miss, or a feature flag that routed all traffic to the expensive tier.
4. Reliability layer breakdown. Verifier ensemble disagreement spikes, citation entailment failure rate rises, guardrail false-positive rate jumps.

The playbook structure:

• Detection. What metrics trigger the page? Each metric needs a specific alert threshold.
• Initial assessment (5 min). Is user-facing traffic affected? How many users? How severe?
• Containment. What's the fastest action that reduces blast radius? Usually rollback or fallback to a cheaper-and-known-good path.
• Diagnosis. Why did this happen? Read logs, traces, recent deploys.
• Resolution. Apply the fix.
• Postmortem. What did we learn? What detection or prevention should we add?

Mean time to resolution (MTTR) is the metric. Production teams in 2026 target < 30 minutes for revenue-affecting incidents, < 4 hours for quality-only incidents.

The key intervention list:

• Rollback. Always available; always the fastest containment. Aim for 1-click rollback.
• Fall back to a cheaper-and-known model. When the frontier model misbehaves, route to Sonnet or Haiku as a degraded but stable mode.
• Disable the broken feature. Feature flags let you turn off the agent loop, the new RAG path, the experimental prompt — without rolling back the whole deploy.
• Increase verifier strictness. When quality drops, tightening the reject threshold buys time at the cost of higher false-rejects.
• Throttle. When cost spikes, rate-limit the misbehaving feature.
• Route to human. For high-stakes decisions, switch from auto to human-review until the model is fixed.

What separates a strong on-call from a weak one. Strong: knows the metric thresholds, knows the rollback procedure, has practiced the intervention drills. Weak: discovers each playbook item during the incident. The 2026 norm: chaos days where the team practices intervening on simulated incidents.

3. Cost governance at org scale

Week 9 covered cost optimization at the per-request level. Week 12's concern is cost governance: how teams of 50+ engineers across multiple products keep LLM spend predictable and aligned with business value.

The three layers of cost governance:

Per-product budget. Each product line gets a monthly LLM budget. Engineering leads sign off. Finance reviews monthly. Forecast accuracy improves over quarters. Without budgets, cost grows unbounded — the dominant 2025 failure mode in larger orgs was finance discovering that LLM spend tripled before anyone noticed.

Per-request attribution. Every API call is tagged with: product, feature, user segment, request type. Real-time dashboards show cost-per-feature trends. When the bill spikes, attribution tells you where. Most production stacks in 2026 wrap their LLM client to add this tagging by default.

Per-team chargeback. Internally bill product teams for their LLM consumption. Forces ownership: the team that benefits from a feature pays for it. Teams optimize aggressively when their budget is finite; they don't when it's the platform team's problem.

Cost alerting:

• Per-request anomaly. A request that costs > 10× the median is a bug, not a feature. Alert.
• Daily budget burn rate. If today's spend is 5× yesterday, page the team.
• Per-feature trend break. A feature that's been steady at $2k/day suddenly costs $8k/day.

Cost hedging at scale.

• Reserved capacity. At >$50k/month spend, providers offer reserved capacity at 30-50% discount. Worth negotiating.
• Multi-provider routing. Anthropic + OpenAI + Google all support similar capabilities. Production routers route based on cost, latency, and availability. Limited downside; provider outages don't take you offline.
• Self-hosting open-source for predictable workloads. At >$200k/month on a specific workload, self-hosting Llama or Qwen variants on H100s/GH200s becomes economic. The breakeven moves over time as managed pricing drops.

The political dimension. Cost governance is a finance and engineering negotiation. Engineers want capability; finance wants predictability. The patterns above let both sides win: capability is preserved for high-value features; predictability comes from per-team budgets and attribution.

4. SLOs and SLAs for LLM products

Service Level Objectives (SLOs) — internal commitments to a quality bar. Service Level Agreements (SLAs) — external commitments, often in contracts. LLM products complicate both.

The classical web SLOs that still work:

• Availability. % of requests that don't error. Targets: 99.9% for most products, 99.99% for critical systems.
• Latency. P50/P95/P99 response time. Targets: P99 < 5s for chat, < 1s for completions, varies for agents.
• Error rate. % of requests that throw exceptions. Target < 0.5% for production.

The LLM-specific SLOs that 2026 production teams add:

• Quality SLO. % of responses that pass an eval-in-prod judge. Target: depends on the product. 90% might be acceptable for chat; 99%+ required for medical or legal.
• Hallucination rate. % of responses with verifier-flagged unsupported claims. Target: < 2% for general products, < 0.5% for high-stakes.
• Groundedness rate. % of citations that pass entailment check (Week 11). Target: > 95% for RAG-based products.
• Cost-per-request P99. P99 cost per request stays below threshold. Catches cost-explosion bugs.

The honest difficulty. Quality SLOs depend on the eval set. A 95% quality SLO measured against an old eval set is meaningless if the eval set doesn't cover today's traffic. The discipline: refresh the eval set quarterly (Week 8); SLOs are about the current eval set.

External SLAs are tougher. Contractual commitments with financial penalties. The 2026 norm: LLM vendors commit to availability and latency, not quality. Quality is the customer's problem — explicitly carved out of contracts. If you're selling an LLM-based product to enterprise customers:

• Don't commit to a numeric quality SLA in contracts unless you're confident you can measure and meet it. Most providers don't.
• Commit to availability and latency with the same rigor as classical SaaS.
• Define quality remediation in service terms ("we will investigate quality issues within X hours, deploy fixes within Y days") rather than numeric guarantees.

Error budgets. SRE-style error budgets work for LLM systems. Per quarter: 0.1% downtime allowed (8.5 hours), 5% quality regressions allowed. When the budget is consumed, freeze deploys until incidents are resolved. Forces the discipline of fewer-and-better deploys when problems mount.

5. The system-design interview frame

The dominant senior LLM-engineering interview format in 2026 is system design. The interviewer states a scenario; you propose an architecture and defend the choices.

The 7-pillar mental model. Whatever the scenario, your answer covers:

1. Model selection. Frontier vs cost-tier. Routing strategy. Why this model for this task.
2. Context strategy. RAG, fine-tuning, long-context, or none. The Week 2-6 material applied.
3. Eval discipline. Golden set, judge model, eval-in-prod cadence. Without this, every other choice is unverifiable.
4. Latency budget. Hard targets, timeouts, what's interactive vs batch.
5. Cost target. Per-request budget, monthly cap, what justifies escalation.
6. Reliability layer. Verifiers, tool grounding, self-consistency, guardrails. Week 11 applied.
7. Monitoring and observability. Per-request tracing, drift detection, alert thresholds.

The strongest interview answers structure the response across these seven pillars. The weakest answers focus on one or two (usually model choice and prompting) and leave the rest implicit. Interviewers grade on coverage and reasoning, not novelty.

The common scenarios in 2026:

• "Design a customer-support chat for a SaaS company at 50 RPS."
• "Design a code review agent for a TypeScript monorepo with 200 PRs/week."
• "Design a medical-literature Q&A for clinicians at 5 RPS with citation requirements."
• "Design a multimodal product image search at 1000 RPS."
• "Design a long-document analysis tool for legal teams (input: 100-page contracts)."

Each rewards different pillar emphases. Cost-sensitive products foreground pillars 1, 4, 5. Accuracy-critical products foreground 2, 3, 6. Agent products foreground 1, 5, 6. Read the scenario for the constraints and lean into the matching pillars.

The follow-up questions to expect:

• "What's your eval set look like?"
• "How do you measure success for this?"
• "What happens when the model regresses?"
• "How would you reduce cost by 50%?"
• "How does this scale to 10× traffic?"

Each follow-up exercises a specific week's material. Eval is Week 8. Cost is Week 9. Scale is Week 9 + Week 1 (KV cache). Regression handling is Week 11 + Week 12 (this week).

6. Behavioral and leveling preparation

Beyond system design, expect:

The "describe a project" question. Walk through a system you built. Strong answers: state the problem, the constraints, the architecture choices and why, the tradeoffs, the results, and what you'd do differently. Weak answers describe the architecture without the why.

The "describe a failure" question. Production teams want engineers who can recognize and recover from mistakes. Have one or two concrete examples ready: what went wrong, how you detected it, how you responded, what you learned.

The "explain X to a non-engineer" question. "Explain RAG to a product manager." "Explain why LLMs hallucinate to an executive." Tests communication, not just knowledge.

The leveling questions for staff/principal candidates:

• "Walk me through how you'd help a junior engineer ramp on this domain."
• "How do you decide what to build vs buy?"
• "How do you make a case for a major engineering investment that doesn't ship a feature?"

These exercise the leadership and judgment dimensions that get evaluated above pure technical skill.

Questions to ask the interviewer. A few good ones:

• "What's your team's eval discipline?" — gauges Week 8 maturity.
• "What's the on-call rotation like for this product?" — gauges operational maturity.
• "How do you decide on model upgrades?" — gauges deployment discipline.
• "What's the most painful production incident this team has had recently?" — gauges culture and honesty.

The interviewer's answers tell you whether the team has matured past the prototype stage. Avoid teams that haven't.

7. The reading discipline going forward

The field moves fast. The reading discipline that keeps you current:

Weekly:

• Anthropic, OpenAI, Google research blogs. Major capability releases.
• arXiv ML new section — skim titles, deep-read 1-2.
• HuggingFace papers (the curated daily-papers feed).
• Eugene Yan, Lilian Weng, Jay Alammar blogs. Practitioner depth.

Monthly:

• One frontier-paper deep-read. Pick one important paper, read it slowly, work the math.
• Vendor pricing & capability table refresh. Models change quarterly; your mental model should too.
• One tool you haven't used. New eval frameworks, new deployment patterns.

Quarterly:

• Refresh your eval set. What's changed in production? What new failure modes emerged?
• Revisit one earlier topic. Re-read the foundational paper for something you take for granted.

Annually:

• One major project that pushes you into a new area. Build something publishable or open-source it.
• Career check. Did the year build skill? Are you still operating at the frontier?

The trap. It's easy to spend all your time reading and none building. The 70-30 rule: 70% of your learning time should be hands-on; 30% reading. Reading without building produces fluent commentators, not practitioners.

8. Where the field is going (2026-2027)

Predictions age worse than anything else in this curriculum, so read this section differently from the seven before it. Each claim below is a bet, stated with how confident I am and what would make me wrong. Checking them against what actually happened is worth more than the predictions themselves.

Written August 2026. If you are reading this much later, the scoreboard is the point.

Capability growth slows; engineering matters more. Frontier model improvement per generation is shrinking. The advantage in 2026-2027 accrues to teams that engineer the surrounding stack well — eval discipline, reliability layers, cost optimization, retrieval quality. Confident. Wrong if a generation lands that makes the surrounding stack largely unnecessary: long-context recall that beats a tuned retrieval pipeline, or reliability good enough that verifier ensembles stop paying for themselves.

Agentic systems mature. 2024-2025 agents were demos; 2026 agents are production. By 2027 I expect the dominant new product pattern to be agents in narrow verticals — legal research, code review, support resolution — with the engineering question shifting from "can we build it?" to "can we keep it cost-effective and reliable at scale?" A real bet. Wrong if agent reliability stays at demo level and the products that win keep a human in every loop.

Multimodal becomes default. Vision in interfaces, voice in customer interactions, video in content workflows; text-only products start to feel limited. Cost still constrains scale, so the optimization patterns from Week 10 become standard rather than specialist. Confident on direction, uncertain on pace. Wrong if vision token pricing stays roughly where it is — at current rates a lot of multimodal products do not clear their unit economics.

Local and edge inference grows. Llama-class models at 70-100B parameters with 200k+ context, running on consumer GPUs and high-end laptops. Privacy-sensitive workloads shift there, and hybrid architectures — cheap local model for most traffic, frontier API for the hard cases — become normal. A real bet, and the one I am least sure of. Wrong if hosted inference keeps getting cheaper faster than local hardware gets better, which is roughly what happened through 2024-2025.

Eval becomes a profession. Eval engineering as a named specialty, with dedicated teams in major LLM-product orgs by 2027 and a tooling market to match. Confident. Wrong if eval collapses into the platform layer and never becomes a distinct role — plausible if the model providers ship good-enough eval tooling for free.

Regulation arrives. EU AI Act enforcement is underway; US frameworks are still forming. Compliance becomes a real engineering function in high-stakes domains, and hallucination rates and citation verification (Week 11) become legally relevant in some jurisdictions. Confident on the EU, genuinely unsure on the US. Wrong if enforcement stays nominal and compliance remains a legal-department concern rather than an engineering one.

The skill compounds. Engineers who came up through the 2023-2026 era — who internalized eval, reliability, cost, agent design, multimodal — become the senior practitioners of 2027-2030, because each new technique builds on the same foundations. Confident, and the least falsifiable claim here. Treat it as the reason the rest of this curriculum was worth your time, not as a forecast.

What to bet on personally. Build depth in the engineering layer: eval, reliability, cost, agents. The capability layer moves fast and is mostly the model providers' job. The engineering layer is where production value gets created and where careers compound. This is the one claim I would still make if every prediction above turned out wrong — it follows from the shape of the work, not from a forecast about models.

Build this week

Pick at least two:

1. Run an interview prep cycle. Pick three system-design scenarios. Practice 30-minute mock interviews with a peer. Apply the 7-pillar frame. Refine until your answers are crisp.

2. Write your incident response playbook. For a product you've built or know well, document: alert thresholds, intervention list, rollback procedure, postmortem template. This is interview material and real engineering output.

3. Audit your cost governance. Pull the last 90 days of LLM costs. Categorize by feature. Build the dashboard if it doesn't exist. Identify the top three cost-saving opportunities.

4. Define SLOs for one product. Write availability, latency, quality, hallucination, and cost SLOs. Defend each number. Get a stakeholder to sign off.

5. Run a chaos drill. Simulate an incident — model regression, retrieval index decay, prompt injection wave. Practice the playbook. Time the response.

6. Open-source one of your tools. A small eval framework, a cost calculator, a reliability check. The act of polishing for public consumption produces better engineering.

Read this

• Google SRE Book — Service Level Objectives chapter. The classical foundation; LLM SLOs build on it.
• "Building Effective Agents" (Anthropic, 2024-25). Re-read with the production lens; the patterns hold.
• Hugging Face Optimum and TGI documentation. State-of-the-art self-hosting reference.
• "The Eval Engineer" essays (Hamel Husain, 2024-26). Practitioner-direct on eval as a discipline.
• Eugene Yan, Lilian Weng, Jay Alammar archives. Foundational practitioner references for the whole curriculum.
• Charity Majors on observability. The honeycomb-CTO essays generalize cleanly to LLM systems.
• The major model providers' production guides. Anthropic prompt engineering, OpenAI cookbooks, Google AI docs. Read for the patterns, not the API specifics.

Interview prompts

1. Walk through how you'd design a customer-support agent at 50 RPS, covering all 7 pillars.
2. How do canary, shadow, and blue-green deployments differ for LLM systems vs classical web?
3. What metrics would you alert on for a production RAG system?
4. Describe an incident response playbook for a model regression.
5. How would you set up cost attribution across multiple product teams?
6. What SLOs would you define for a code-review agent? Defend each number.
7. The model regressed silently — quality dropped 8% but no errors thrown. How do you detect and respond?
8. The agent loop is costing 5× projected. Walk through the diagnosis and intervention.
9. How does your eval discipline change between prototype, beta, and GA?
10. Where do you think the field is going in the next 18 months, and how does that change your engineering investments?

What "done" looks like

By the end of this week — and the curriculum — you should be able to:

• Choose deployment patterns appropriate for non-deterministic systems.
• Run an on-call rotation for an LLM product without panic.
• Govern costs at team and org scale with attribution and budgets.
• Define SLOs that survive stakeholder review.
• Walk into a senior LLM-engineering interview and structure answers across the 7 pillars.
• Maintain a reading discipline that keeps your knowledge current.
• See the engineering layer of LLM systems as the durable career investment.

If you can do those, you're a 2026 practitioner who can ship and keep shipped. The eleven prior weeks gave you capability; this week gave you the discipline to deploy capability under load.

Closing

The course is finished. You've covered modern LLM architecture, context engineering, fine-tuning, reinforcement learning, RAG basics, advanced RAG, agents, evaluation, inference optimization, multimodal and code agents, reliability and verification, and production patterns. ~40,000 words of practitioner content. 26 interactive widgets. A path from "I've heard of attention" to "I can architect, build, ship, and operate production LLM systems in 2026."

The field will keep moving. The frameworks here — eval discipline, cost arithmetic, reliability stacks, the 7-pillar interview frame — should hold across the next two years. The specific models will change; the engineering layer is durable.

Build something. Ship it. Watch it break. Fix it. Repeat. That's how the curriculum becomes practice.

Good luck out there.