Evaluation

Why this week is positioned where it is. Weeks 5–7 built systems — RAG, advanced RAG, agents — and each one closed with "measure on your own data." This week is how. Without measurement, every claim of improvement is a vibe, and most "improvements" are silent regressions on cases you didn't think to check. The teams that build LLM products that actually work over time are the teams that built eval first and treated everything else as a function of measurable lift.

1. Why eval is the bottleneck

The most common failure pattern in LLM products: a team builds something, it seems to work on a handful of examples, they ship it, and then they spend the next six months making changes that they think improve things while quality drifts in random directions.

The reason is that LLMs are weakly observable. A change to a prompt, a model swap, a retrieval tweak — none of these have predictable second-order effects. Code reviews don't catch quality regressions. Latency monitoring doesn't catch quality regressions. Eyeballing 5 examples doesn't catch quality regressions. The only thing that catches quality regressions is automated evaluation against a held-out dataset.

The cost of not having eval:

• Silent regression. You changed something, it broke a category of queries you didn't test. You ship. Users notice. You don't.
• Vibes-based development. "I think this is better." Compared to what? Measured how?
• Unbounded debate. Two engineers disagree about whether change X helps. Without eval, the argument is irresolvable.
• No iteration loop. You can't optimize what you can't measure. Improvement becomes random walks.
• No ROI calculation. You don't know if a $50/month embedding upgrade buys you 1% lift or 15%.

The teams that ship LLM products successfully share a common pattern: they built an eval set within the first two weeks, and every subsequent change had to clear that eval set as a gate. Everything else — choice of model, retrieval architecture, prompt engineering — became a function of "did this lift the eval score?"

This is not optional. It's the bottleneck for everything else.

2. Building eval sets

An eval set is a collection of inputs paired with expected outputs (or grading criteria). The dataset is the single most important artifact you build for an LLM product.

The 5-tier dataset model that most production teams converge on:

1. Dev set — what you iterate against. ~50–100 examples. You see this dataset constantly; it's what guides daily decisions.
2. Eval set — held-out, used for major decisions. ~200–500 examples. Sees the eval set rarely (deploy gates, A/B test winners). Distinct from dev to avoid overfitting.
3. Regression set — every bug you fix becomes an example here. Grows over time. Re-runs on every change to prevent reintroducing fixed bugs.
4. Canary set — a small set (~20) you sample queries from in production for ongoing monitoring. Detects drift.
5. Production sample — random sampling of real queries with periodic human review. The source of truth that everything else approximates.

The four origins of eval data:

Hand-curated. You write the queries and answers. Best quality, slowest to build. The first 50 should always be hand-curated — you'll find more bugs sitting down to write 50 examples than from any other activity.

Synthetic / LLM-generated. Use a strong model to generate queries and answers. Fast, scales, but biased toward easy queries. Always sample-review and reject the obvious failures (queries that don't make sense, answers that are wrong).

Drift / production-mined. Sample from your real production queries (with PII removed). Highest signal for what your users actually ask. Requires production traffic.

Hard-negative mining. Find queries your system fails on, then add them. The best way to grow the regression set. Set up a workflow where any reported bug becomes an eval example before being closed.

The dataset is a living artifact. The first version is wrong. The version after a quarter of feedback is much better. Plan for ongoing investment, not a one-time build.

One useful rule: the eval set should contain at least one example of every kind of query your users ask. If you don't know what kinds of queries they ask, that's a research problem to solve before the eval problem.

3. Retrieval metrics

If you have a retrieval system (Weeks 5–6), you need retrieval metrics. The standard set:

Recall@K. What fraction of queries had the relevant chunk in the top-K results? The most important retrieval metric. recall@5 = "did we get the right chunk in the top 5?" If recall@5 is 90%, end-to-end correctness is bounded above by 90%.

Precision@K. What fraction of retrieved chunks were relevant? Less important than recall for RAG (the LLM can ignore irrelevant chunks if the right one is also there) but matters for human-facing search interfaces.

MRR (Mean Reciprocal Rank). Average of 1/rank across queries — where rank is the position of the first relevant chunk. Penalizes the model for putting the right chunk at rank 5 vs rank 1. Useful when ranking quality matters (rerankers).

NDCG (Normalized Discounted Cumulative Gain). Like MRR but accounts for graded relevance (not just binary). The right metric when chunks have multi-level relevance (perfect / partial / tangential / irrelevant).

The one to focus on for most RAG: recall@K. Your retrieval is correct if and only if the right context made it into the prompt. Precision matters less — the LLM is good at ignoring irrelevant chunks. MRR and NDCG matter mostly when you're tuning a reranker.

Per-query vs aggregate. Don't only look at averages. Look at the distribution. A system with 95% recall@5 on most queries but 0% on a 10% subset has a different problem than a system with 80% recall everywhere. Always inspect the worst 10% of queries.

Eval set composition matters. If your eval set is 90% easy queries, your average recall is meaningless. Stratify by query type (factual, paraphrase, multi-hop, ambiguous) and report metrics per stratum.

4. Generation metrics

Retrieval gets you context. Generation turns context into an answer. Generation has its own failure modes and its own metrics.

Correctness. Does the answer match ground truth? The metric you ultimately care about. Hard to grade automatically — usually requires human or LLM-as-judge.

Faithfulness. Does the answer match the retrieved sources? Independent from correctness — you can be faithful but wrong (the source was wrong) or unfaithful but correct (the model knew the right answer but didn't cite it).

Citation accuracy. When the model cites a source, does the cited source actually contain the cited claim? In production this is the difference between "trustworthy" and "looks trustworthy." A frequent failure mode is fake citations — claim is true, source doesn't say it.

Completeness. Does the answer address all parts of the question? Multi-part questions are a common failure mode where the model addresses one part well and skips the rest.

Refusal correctness. When the model refuses, was refusal correct? When it should have refused but didn't, that's a hallucination. When it refused but shouldn't have, that's an over-cautious failure.

The decomposition that matters in production:

• Retrieval recall@K measures: did we get the right context?
• Faithfulness measures: did we use the context correctly?
• Correctness measures: was the final answer right?

If recall@K is 95% but correctness is 60%, the gap (35 points) is the generation problem. If recall@K is 60% and correctness is 60%, retrieval is the bottleneck and generation may be fine.

5. Agent and trajectory metrics

Agents (Week 7) need trajectory-level metrics, not just final-answer metrics. The trace is the artifact, not just the answer.

Goal-completion rate. Did the agent finish the task correctly? Binary or graded. The most important metric.

Trajectory quality. Was the path the agent took reasonable, or did it wander? Often graded by humans. Two agents can both succeed; one took 4 steps, the other took 12 with detours.

Tool-call success rate. What fraction of tool calls succeeded (vs errored)? Tracks tool-design quality. Low rates often indicate tool API issues.

Hallucinated calls. Tool calls that the agent invents (calls to non-existent tools, or with structurally invalid arguments). Should be zero. Non-zero = prompt issue or tool-schema issue.

Step efficiency. How many steps did it take vs. the minimum reasonable? A 4-step task that took 12 steps suggests an agent that's spinning.

Cost / latency per task. Production budget metrics. A slightly higher quality at 5x cost may not be worth it.

The agent eval pattern:

1. Build a set of tasks with known correct outcomes.
2. Run the agent end-to-end on each.
3. Score: success/fail (programmatic where possible), trajectory quality (human or LLM-judge).
4. Aggregate: success rate, average steps, average cost, % hallucinated.
5. Inspect the failures. Categorize by root cause. Add to regression set.

The failure-mode taxonomy you'll see in agent eval:

• Loop / drift — agent calls the same thing repeatedly without progress.
• Hallucinated tools — calls a tool name that doesn't exist.
• Bad arguments — calls a real tool with structurally wrong inputs.
• Premature finish — agent claimed to be done but the task isn't complete.
• Ignored observation — agent makes a decision contradicted by what a tool just returned.
• Over-running — agent takes far more steps than needed.
• Step-limit hit — agent ran out of budget mid-task.

Each failure mode points to a different fix. Loops point to tool design (errors aren't actionable). Hallucinated tools point to prompt / system message. Bad arguments point to schema clarity. Premature finish points to success criteria in the prompt. Each is a distinct intervention, and you only know which one to make if you've inspected the trajectories.

6. LLM-as-judge

Many of the metrics above are hard to compute programmatically. "Correctness" of an open-ended answer requires judgment. "Trajectory quality" requires understanding what the agent did.

LLM-as-judge is the pattern of using a separate LLM call to evaluate the output of your system. Surprisingly effective. Surprisingly noisy.

The pattern:

def llm_judge(query, system_response, criteria):
    prompt = f"""
    You are evaluating an AI assistant's response.

    Query: {query}
    Response: {system_response}

    Grading criteria: {criteria}

    Output JSON: {{"score": 1-5, "reasoning": "..."}}
    """
    return strong_llm.call(prompt)

When LLM-as-judge is reliable:

• Binary, factual checks. "Does this answer mention X?" Agreement with humans typically 90%+.
• Refusal detection. "Did the model refuse?" Almost trivial.
• Format checking. "Is the output valid JSON?" Programmatic is better here, but LLM-judge works.
• Clear-criterion grading. "Is this code syntactically valid?" with code-aware models.

When LLM-as-judge is unreliable:

• Subjective quality. "Is this response delightful?" LLM-judge ≈ random above human inter-rater noise.
• Subtle errors. "Did the model make a logical leap?" Misses subtleties.
• Bias-sensitive. Models prefer certain styles, lengths, formality. Bias your eval if you don't control for this.
• Comparative grading without reference. "Which of two responses is better?" Better than absolute grading but still noisy without anchors.

Calibration is non-optional. Run your LLM judge against ~50 human-graded examples. Compute agreement. If agreement is below human inter-rater agreement, your judge is below-noise. If it's at or near human agreement, you have a useful judge.

Expected agreement levels (rough rules of thumb):

• Inter-human agreement on subjective tasks: 70–85%.
• Inter-human agreement on factual tasks: 90–95%.
• LLM-judge vs humans on factual tasks: 85–95% (close to ceiling).
• LLM-judge vs humans on subjective tasks: 60–75% (notably worse than humans-vs-humans).

Which model to judge with. Use the strongest available — judge accuracy is bottlenecked by model capability. Cost is per-eval-run (not per-production-call), so the strongest model is usually affordable. As of 2026, Claude Opus 5 and GPT-5.6 Sol are the typical choices for high-stakes eval; Sonnet 5 / GPT-5.6 Terra for cost-sensitive runs.

The sandbagging problem. If you use the same model as judge as you use to generate, the judge will systematically prefer its own outputs. Use a different model family for judging.

7. Eval harnesses and tooling

In 2026 the tooling landscape has matured. The notable systems:

Promptfoo. Open-source, file-based eval definitions. Great for prompt comparison and CI integration. Yaml configs, runs locally. The "git-and-CLI" choice.

LangSmith. LangChain's eval and observability platform. Strong for teams already on LangChain. Good UI for trace inspection.

Braintrust. Eval-focused product with strong UX for dataset management and side-by-side comparisons. The "we evaluate seriously" choice.

Inspect AI. Anthropic-built framework for AI evaluations. Strong for structured eval suites with grading rubrics and trajectory inspection. The choice for serious eval programs.

OpenAI Evals. OpenAI's open-source eval framework. Less polished than commercial tools but free and integrates well with their API.

Arize Phoenix, Langfuse. Observability-leaning tools that include eval features. Useful for production monitoring more than dev-time eval.

Custom harness. Many teams build their own thin layer on top of pytest or a notebook. Surprisingly effective for the first 6 months. The fancy tools become more valuable as your eval set grows past ~500 examples and you need dataset management, comparison UI, and sharing.

The decision: start with Promptfoo or a custom harness. Move to Braintrust / LangSmith / Inspect when your eval set passes ~500 examples and dataset management becomes painful.

The 2026 frontier in eval tooling: trace-aware eval. Tools that don't just look at final answers but evaluate trajectories — agent paths, RAG retrieval lists, multi-step reasoning chains. Inspect AI leads here; others are catching up.

8. Eval as a development discipline

The shift from "we have an eval set" to "eval is part of how we develop" is what separates teams that ship reliable LLM products from teams that don't.

Regression testing. Every change runs the eval set. Every drop in score is investigated before the change merges. Standard CI patterns — tests must pass for PRs — apply here, just with a non-binary signal.

Eval-driven development. Before changing anything, ask: how would I know this worked? If you can't answer with a metric on a dataset, build the metric or add the examples first.

The deploy gate. Production releases require eval scores at or above the previous version on the main metrics. New regressions must be either fixed or explicitly accepted.

The investigation loop. When eval drops: 1. Identify which examples regressed. 2. Read the diff in outputs. 3. Identify the root cause. 4. Either fix or add to a "known regression" list.

The dataset growth loop. Every bug report becomes an eval example. Every customer complaint becomes one. Every "I just realized we're doing X wrong" becomes one. The eval set grows continuously.

Don't only evaluate retrieval and generation in isolation. Evaluate end-to-end. Then also evaluate components individually for diagnosis. End-to-end correctness is what users see; component metrics tell you where to focus.

Production monitoring is separate from eval. Eval is offline, deterministic, repeatable. Production monitoring catches drift in real traffic. Both are needed; neither replaces the other.

The honest summary: evaluation is the boring, unglamorous discipline that makes everything else possible. Teams that skip it ship faster initially and slower forever. Teams that invest in it ship slower initially and faster forever. The compounding interest is real.

Build this week

Pick at least two:

1. Build your first eval set. 50 hand-curated examples for whatever LLM product you're working on. Each example: input, expected output (or grading criteria), kind/category. Use them to evaluate your current system. Note what fails.

2. Implement an LLM-as-judge. Pick a metric your eval set needs (faithfulness, correctness, helpfulness). Use a strong model as judge. Calibrate against ~30 human-graded examples. Report agreement.

3. Set up a Promptfoo eval suite. Or any harness of your choice. Get to the point where you can run eval ./ in CI and see scores. Hook to your repo.

4. Build a regression set. Pull 30 historical bugs from your tracker. Convert each into an eval example. Run your current system. Confirm none have regressed (or document which have).

5. Stratified eval. Take an existing eval set. Categorize each example by query type (factual, multi-hop, vague, etc.). Report metrics per stratum. Identify the worst stratum and root-cause it.

Read this

• OpenAI Evals documentation. Reference for the open-source standard.
• "Your AI Product Needs Evals" (Hamel Husain). The single most-cited essay on this in 2025–26. Required reading.
• Inspect AI documentation (Anthropic). The most thorough public framework writeup.
• "Eval is hard" (Eugene Yan's writeups). Skeptical, practitioner-tested.
• Vellum AI's eval guides. Practical, opinionated, well-structured.
• HELM paper and benchmarks (Stanford). The academic framing of holistic eval.

Interview prompts

1. Walk through how you'd set up evaluation for a new RAG product from scratch.
2. What's the difference between faithfulness and correctness, and why does it matter?
3. When is LLM-as-judge appropriate, and when is it not?
4. How do you calibrate an LLM judge against human raters?
5. Your eval scores improved by 5%. How do you know it's a real improvement and not noise?
6. Walk through the metrics you'd track for an agent system.
7. What's a regression set and why does it matter?
8. Your retrieval recall@10 is 92% but recall@3 is 65%. What metric would you optimize, and why?
9. How do you handle eval for tasks where there's no single correct answer?
10. Describe the relationship between offline eval and production monitoring. Why are both needed?

What "done" looks like

By the end of this week you should be able to:

• Build and maintain a multi-tier eval dataset (dev, eval, regression, canary).
• Choose appropriate metrics for retrieval, generation, and trajectory tasks.
• Implement and calibrate an LLM-as-judge against human ratings.
• Diagnose where in a pipeline a quality gap lives by reading the metrics.
• Integrate eval into the development loop so quality regressions are caught at PR time, not in production.
• Convince a skeptical teammate that eval is the highest-leverage investment in an LLM product.

If you can do those, you're ready for Week 9 — inference optimization. The discipline of making the systems you've built run cheaply and quickly enough to be commercially viable.