RAG basics

Why this week is positioned where it is. RAG is the most-deployed pattern in 2026 production GenAI, and the most over-promised. After fine-tuning (Week 3) and RL (Week 4), the natural question is "how do I make the model know my data?" — and the answer is almost always RAG, not training. This week is the foundation; Week 6 covers the advanced techniques (rerankers, query rewriting, hybrid search) that turn baseline RAG into something production-worthy.

1. Why RAG, why now

LLMs are pre-trained on a broad corpus that doesn't include your data. They confabulate confidently when asked about things they don't know. Fine-tuning to inject knowledge is a bad pattern — we covered why in Week 3 (catastrophic forgetting, expensive iteration, training-data leakage). The dominant alternative is retrieval-augmented generation (RAG): at query time, retrieve relevant chunks from your corpus and condition the model's generation on them.

The intuition is straightforward. The model already knows how to read text and answer questions. If you put the right text in front of it, you get a grounded answer. If you put the wrong text in front of it, you get a confidently wrong answer that cites the wrong text. RAG quality is mostly retrieval quality.

What RAG is good for, in practice:

• Question answering over private documentation. Internal wikis, support docs, contract repositories, codebases.
• Live data the model can't have memorized. Today's prices, current inventory, real-time status.
• Reducing hallucination on factual questions with a verifiable corpus.
• Citation-bearing outputs where users need to verify the source.

What RAG is bad for, in practice:

• Tasks that need synthesis across the entire corpus. "What are the themes in this 500-page document?" — retrieval gives you slivers, not the whole picture.
• Reasoning tasks where the answer requires combining many small facts. RAG is shallow; it won't chain inferences.
• Tasks where the user's intent is hard to express as a query. Vague questions retrieve vague chunks.

The 2026 baseline RAG stack is roughly: a chunker → an embedding model → a vector database → a reranker → a generation prompt. Each stage has a default that works passably and a tuning knob that matters. We'll work through each.

2. The retrieve-then-generate loop

Two phases:

Index time (one-time, periodic): 1. Ingest documents from your sources. 2. Split each document into chunks (this is the most underrated step). 3. Embed each chunk into a vector with an embedding model. 4. Store chunks + vectors + metadata in a vector database.

Query time (every request): 1. Embed the user's query into a vector. 2. Search the vector database for the top-K nearest chunks. 3. Optionally rerank (Week 6). 4. Assemble a prompt: system instructions + retrieved chunks + user query. 5. Send to the LLM. Return the generated answer, ideally with citations to the chunks.

This is the architecture. Most teams get the architecture right and the details wrong. The details are everything.

3. Embeddings — what they actually are

An embedding model takes text and produces a fixed-size dense vector. The training objective is some form of contrastive learning: similar texts produce similar vectors, dissimilar texts produce dissimilar vectors. "Similar" is defined by the training data — semantically related queries and documents in most cases.

A 2026 production embedding model produces vectors of 768–3072 dimensions. The vectors aren't interpretable individually; what matters is the geometry of the space. Distance between vectors corresponds to dissimilarity, with cosine similarity as the dominant metric (also dot product in some setups; rarely Euclidean).

The 2026 frontier of embedding models:

• OpenAI text-embedding-3-large — 3072 dim, the default for many teams. Reliable, expensive at scale.
• Voyage-3 — Anthropic-recommended for Claude RAG, strong on technical content. 1024 dim.
• Cohere embed-v4 — multilingual, strong reranker pairing.
• Jina-v3 — strong open-weight option, multilingual.
• BAAI/bge-large-en-v1.5 — open-weight workhorse, runs locally, surprisingly competitive.

The choice matters less than people think for English technical content — the gap between top models is small. The choice matters a lot for multilingual, code, or specialized-domain work.

The "embedding model matters more than vector DB" insight. Teams obsess over which vector DB to use. The DB is mostly commodity — they all do approximate-nearest-neighbor search on dense vectors. The embedding model is what determines whether your nearest neighbors are actually relevant. Spend your evaluation budget on embedding choice; spend your engineering budget on the rest of the stack.

4. Chunking — the most underrated step

Chunking decides what gets retrieved. A bad chunking strategy makes a good embedding model look broken.

Chunk size tradeoffs:

• Too small (under ~200 tokens): chunks lack context. The embedding represents a fragment that could mean many things. Retrieval finds the right chunk but it's not enough to answer the question.
• Too large (over ~1500 tokens): the embedding washes out. A chunk covering three different subtopics produces a vector that's average to all of them, distinctive to none. Retrieval misses the chunk because the query embedding doesn't match the diluted average.

The sweet spot for most prose corpora is 400–800 tokens with 50–100 token overlap. The overlap matters more than people think — it handles the "answer spans the chunk boundary" failure mode at minimal cost.

Strategy options:

• Fixed-size (by token or character count). Simplest, ignores document structure. Cuts mid-sentence regularly. Works passably as a baseline.
• Fixed-size with overlap. Same plus 50–100 tokens of overlap between adjacent chunks. Cheap insurance against boundary cuts.
• Sentence-boundary. Cut on sentence ends, target a token budget per chunk. Better than raw fixed-size for prose. Worse for structured docs (code, tables).
• Paragraph-boundary. Treat paragraphs as natural units. Excellent for well-structured prose. Brittle on docs with inconsistent paragraph length.
• Document-structure-aware. Use headings, sections, code blocks, tables as boundaries. Best for technical docs, requires parsing. The right answer for codebases and Markdown wikis.
• Semantic chunking. Use an embedding model to detect topic shifts and chunk on those boundaries. Trendy, expensive, often only marginally better than structure-aware.

Hierarchical / parent-child chunking. A pattern that's quietly become standard: chunk twice, once small (for retrieval) and once large (for context). Embed and search the small chunks; when one matches, return its larger parent chunk to the LLM. Gives you precise retrieval and sufficient context.

The practical takeaway: chunk smarter, not bigger. Most teams default to 1000-token fixed chunks and never revisit. Spend a day evaluating 3–4 chunking strategies on a held-out eval set. The win is often 5–10 points of retrieval recall.

5. Vector databases — what matters

The vector DB stores chunks with their embeddings and answers nearest-neighbor queries. The interesting algorithm is approximate nearest neighbor (ANN) — exact search is O(n) per query and doesn't scale past a few hundred thousand chunks.

The dominant ANN algorithms:

• HNSW (Hierarchical Navigable Small World). Graph-based, fast queries, slow inserts. The default for most teams. Used by Qdrant, Weaviate, pgvector with hnsw index.
• IVF (Inverted File Index). Cluster-based, fast inserts, slightly slower queries. Better for very large collections.
• ScaNN. Google's algorithm. Fast and accurate; mostly used inside Google services.

The recall-vs-latency tradeoff is the main knob. ANN trades exact correctness for speed; you tune the algorithm's parameters to hit your latency budget at acceptable recall (usually >95%).

The 2026 vector DB landscape:

• pgvector (Postgres extension). The right answer for most teams. You already have Postgres. It scales to ~10M chunks comfortably with HNSW. Avoids a new piece of infrastructure.
• Qdrant. Strong open-source option. Good at filtering on metadata.
• Weaviate. Hybrid search out of the box. Good docs.
• Chroma. Local-first, easy to start with, weaker at scale.
• Pinecone. Hosted, mature, expensive. Use it if you want to outsource the operational burden entirely.
• Turbopuffer / LanceDB. Newer entrants, built for embedding-native workloads, worth watching.

For a team starting out: use pgvector until you have a concrete reason to switch. The scaling pain is far away; the operational simplicity is real today.

Metadata filtering. A vector DB feature that matters enormously in production: filter retrieval by metadata (date range, document type, user permissions, tenant). A "RAG over our docs" query that doesn't filter by which docs the user is allowed to see is a security incident waiting to happen.

6. The retrieval signal — and how it breaks

The output of vector search is a ranked list of chunks with similarity scores. Most failures live here.

Top-K choice. Usually K = 5–10. Smaller K means fewer distractors but higher risk of missing the relevant chunk. Larger K dilutes the prompt and increases cost. There's no universal answer — measure retrieval recall@K against a held-out eval set and pick the smallest K where recall is acceptable.

Score thresholds. A common pattern: retrieve top-K, then drop chunks below a similarity threshold. The threshold catches "no good match" cases where the corpus genuinely doesn't contain the answer. Without it, you retrieve random low-relevance chunks and hallucinate confident answers from them.

Canonical retrieval failures:

1. Lexical mismatch. Query says "auth," docs say "authentication." A weak embedding model misses the connection. Mostly solved by 2026's models, occasionally still bites.
2. Specificity mismatch. Query is broad ("how do I deploy?"), docs are narrow ("how to deploy on AWS Lambda runtime v2"). Top-K returns specific docs that don't answer the broad question. The fix is query rewriting (Week 6).
3. Negation flip. Query asks "what does X not support," docs describe what X does support. Embeddings ignore negation. The match looks great by similarity, the answer is the opposite of what's asked.
4. The between-clusters failure. Query falls in the gap between two topical clusters. Top-K returns mixed chunks from both, none of which answer the question well. Diagnosable by inspecting the retrieved chunks; fixed by either better chunking or a reranker.
5. Stale corpus. Docs were indexed six months ago; the answer changed. The retrieval works, the answer is wrong. Fix is reindexing pipeline, not retrieval.

The single most useful debugging tool: log the retrieved chunks for every query in production. When users complain about wrong answers, the first question is always "what did we retrieve?" — and you can only answer it if you logged.

7. Generation — putting it together

Once you have retrieved chunks, you assemble a prompt. The pattern that works:

[system]
You answer questions using only the provided context. If the context
does not contain enough information to answer, say so clearly.

[context]
<source 1: doc-id, chunk-id>
{chunk 1 text}
</source 1>

<source 2: doc-id, chunk-id>
{chunk 2 text}
</source 2>

...

[user]
{user question}

Things that matter:

• Source delimiters. Wrap each chunk in tags with source metadata. Models attend to the structure and use it for citation.
• The "answer only from context" instruction. Explicit. Models do drift from context without it, especially when the context contradicts their training data.
• Refusal instruction. Tell the model to say "I don't know" when context is insufficient. Without this, the model fabricates plausible-sounding answers from the retrieved chunks plus its priors.
• Citation generation. Ask for inline citations using source IDs. This is free; users (and your evals) need it.

The "model ignored my context" failure. A frequent complaint. The diagnosis is almost always one of three things: 1. The context didn't actually contain the answer (retrieval failure, not generation failure). 2. The context contradicted strong priors and the model chose priors. Fix: tighten the system prompt, sometimes try a different model. 3. The context was buried in the middle of a long prompt and lost-in-the-middle (Week 2) hit you. Fix: keep retrieval results at the top of the context, not the bottom.

In every case the fix is upstream. Debugging RAG by tweaking the generation prompt is almost always the wrong layer. Look at what was retrieved first.

8. Evaluating RAG — the preview

End-to-end evaluation hides everything. A RAG system that's "75% correct" tells you nothing about where to invest. The right framing is two-axis evaluation:

Retrieval evaluation. Build a held-out set of queries with known relevant chunks. Measure: - Recall@K — fraction of queries where at least one relevant chunk is in the top-K. - MRR (mean reciprocal rank) — how high in the ranking the first relevant chunk appears. - NDCG — graded relevance, weights position. Useful when you have multiple relevant chunks per query.

Generation evaluation (assuming retrieval was correct). Measure: - Faithfulness — does the answer say only what's supported by retrieved context? - Answer correctness — given correct context, is the answer right? - Citation accuracy — do the cited sources actually contain the cited claims?

End-to-end evaluation sits on top — does the user-facing system produce correct answers? Useful for tracking, useless for diagnosing.

The two-axis framing is what lets you debug. If retrieval recall@5 is 60% and end-to-end correctness is 55%, you know retrieval is the bottleneck. If retrieval recall@5 is 95% and correctness is 60%, you know the generation step is the problem.

We'll go deep on building evaluation infrastructure in Week 8. For now: even a hand-curated set of 50–100 query/relevant-chunk pairs, scored manually, is more valuable than every fancy automated eval framework — because it tells you the truth about your data.

Build this week

Pick at least two:

1. Build a baseline RAG pipeline. Pick a corpus you have access to (technical docs, papers, transcripts). Use the simplest stack possible: pgvector + OpenAI text-embedding-3-large + GPT-5 or Claude Sonnet. Get end-to-end answers on 20 queries. Note where it fails.

2. Chunking ablation. Take the same corpus, chunk it three different ways (fixed 1000-token, fixed 500-token with 50 overlap, paragraph-boundary). Build the same eval set. Measure retrieval recall@5 for each. Write up which strategy won and why.

3. Embedding model bake-off. Same corpus, same chunking, same eval set. Try two embedding models (e.g., text-embedding-3-large vs voyage-3). Compare recall@5. The gap is often smaller than you'd expect for English technical content — and that's a useful lesson.

4. Build the eval harness. Hand-curate 50 queries from real or expected user questions. For each, manually identify the chunks that contain the answer. This is the single most valuable artifact you'll build. Use it for everything in Weeks 6 and 8.

5. Retrieval failure inspection. Run 100 queries through your baseline RAG. For the 20–30% that fail end-to-end, classify each failure: was it a retrieval miss, a context-too-long failure, or a generation problem? The distribution tells you what to fix first.

Read this

• Anthropic's "Contextual Retrieval" post. Defines the problem with naive chunking and a strong fix. Required reading.
• Vespa, Pinecone, and Weaviate engineering blogs. Best ongoing coverage of retrieval-system internals.
• Patrick Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (the original RAG paper, 2020). Worth reading once to understand where the term came from.
• Voyage AI evaluation reports. Good comparative data on embedding model performance across domains.
• Ben Kehoe's "RAG is more than retrieval" essays. Pragmatic engineering perspective, less hype than most RAG content.
• The pgvector documentation. If you're going to use it (you probably should), read it carefully.

Interview prompts

1. Walk through the architecture of a baseline RAG system. What runs at index time, what runs at query time?
2. Why does chunking matter? Walk through three failure modes and how chunk size choice affects each.
3. Compare cosine similarity, dot product, and Euclidean distance for retrieval. When do they give different results?
4. A user reports the RAG system "doesn't know about a feature in our docs." Walk through your debugging process.
5. What's the difference between retrieval evaluation and end-to-end evaluation, and why does both/either matter?
6. Explain HNSW at a high level. What's it trading off and why is that tradeoff acceptable?
7. You have 10M documents and a tight latency budget. Walk through your indexing strategy.
8. What's the parent-child / hierarchical chunking pattern, and when is it the right choice?
9. Your RAG system has 90% retrieval recall and 60% end-to-end correctness. Where do you focus next, and why?
10. Why would you choose pgvector over a dedicated vector database? When would you not?

What "done" looks like

By the end of this week you should be able to:

• Stand up a RAG system end-to-end on a real corpus.
• Diagnose RAG failures by stage (retrieval vs generation), not just at the end.
• Make principled chunking decisions based on document structure and corpus shape.
• Choose a vector database for a given scale and operational constraint.
• Build a retrieval-only eval set and use it to compare changes.
• Recognize the canonical retrieval failure modes when you see them in logs.

If you can do those, you're ready for Week 6 — advanced RAG, where we add rerankers, query rewriting, and hybrid search to push baseline retrieval recall from "passable" to "production-worthy."