This is the finale. We can already embed a question and pull the nearest chunks from our index (Parts 3 and 4). Now we turn that raw retrieval into a polished, trustworthy answer: how many chunks to fetch, how to combine vector and keyword search, how reranking sharpens precision, and how the model finally generates a grounded reply with citations. Then we'll cover the part everyone skips — how to know if it's actually any good.
Step 1 — Choose how many chunks: top-k
The vector index can return its closest matches ranked by similarity. Top-k is simply how many you keep. It's a balancing act:
| k too small (e.g. 1) | k too large (e.g. 20) |
|---|---|
| If the single chunk misses part of the answer, the model has nothing to fall back on. | You drag in loosely related chunks that distract the model and cost more tokens. |
| Risk: incomplete answers. | Risk: vague answers, higher cost, slower responses. |
A typical starting point is k = 3 to 5. Enough to cover an answer that spans a couple of passages, few enough to stay focused. Like chunk size, you tune it by watching real questions.
Step 2 — Search smarter: hybrid + filters
Pure vector search is excellent at meaning but, as we noted in Part 3, it can be fuzzy on exact tokens — order numbers, product codes, error IDs, proper names. Keyword search is the opposite. So the strongest systems run hybrid search: both at once, then merge the results.
Hybrid search captures meaning and exact tokens — the order number and the policy both surface.
On top of search you add metadata filters — the metadata we attached to every chunk back in Part 2. Filters narrow the candidate pool before ranking, which improves both accuracy and security:
- Freshness: only chunks where
status = current, so you never answer from a superseded policy. - Scope: only
department = HR, orlanguage = en, orregion = EU. - Security: only chunks the signed-in user is permitted to see — the most important filter of all.
Step 3 — Reranking: a smarter second opinion
Vector search is fast but a little rough — it compares the query and each chunk independently. A reranker is a slower, more accurate model that looks at the query and a candidate chunk together and scores how well that chunk actually answers this question. The trick is to use it as a cheap-then-careful, two-stage funnel:
- Retrieve wide, cheaply. Vector/hybrid search grabs a generous candidate set — say the top 50 — fast.
- Rerank narrow, carefully. The reranker re-scores just those 50 and keeps the true best 3–5 to send onward.
You get the speed of approximate search over millions of chunks and the precision of a careful model over a small shortlist. Watch the order change once reranking is applied:
First-pass search ranked the generic "returns overview" top; the reranker recognised the specific "30 days" chunk actually answers the question and promoted it.
Step 4 — Augment: assemble the prompt
Now we build the actual text sent to the language model. The retrieved chunks are pasted in as context, wrapped with clear instructions. A good RAG prompt has three layers:
That instruction layer is doing heavy lifting. Phrases like "answer only from the context" and "if the answer isn't in the context, say you don't know" are what keep the model honest — they tell it to ground itself in your documents rather than improvise from memory.
Step 5 — Generate: a grounded, cited answer
Finally the model reads the assembled prompt and writes the reply. Because the relevant passages are right there and the instructions demand grounding, the answer is accurate and traceable:
Source: Employee Handbook, p.12 (Refunds & Returns)
Grounded in the retrieved chunk, with a citation the user can verify — the opposite of a confident guess.
The citation comes straight from the chunk's metadata (Part 2). This is a defining advantage of RAG over a bare language model: every claim can point back to a source, which is exactly what enterprises need for trust and compliance.
The complete picture
Here's the full end-to-end flow we've built across all five parts — indexing once, then this live path per question:
Embed query
Same model as indexing.
Hybrid search
Vector + keyword + filters.
Rerank
Keep the true top 3–5.
Augment
Build the grounded prompt.
Generate
Cited answer.
How do you know it's any good? Evaluation
You can't improve what you don't measure, and "it looked fine in a demo" isn't measurement. RAG quality splits neatly into two halves — and knowing which half is failing tells you exactly what to fix:
| Half | Question it answers | If it's weak, fix… |
|---|---|---|
| Retrieval quality | Did we fetch the right chunks at all? | Chunking, embedding model, k, hybrid, reranking. |
| Generation quality | Given good chunks, was the answer faithful and complete? | The prompt/instructions, and the generation model. |
The metrics you'll hear most often, in everyday terms:
- Context recall — of the chunks needed to answer, how many did retrieval actually find? Low recall means the answer wasn't even in the context.
- Context precision — of the chunks we retrieved, how many were actually relevant? Low precision means we're feeding the model noise.
- Faithfulness / groundedness — does the answer stick to the provided context, or did the model invent things? This is your hallucination check.
- Answer relevance — does the response actually address what the user asked?
A tuning cheat-sheet
When something's off, this maps symptoms to the dial to turn — and to the part of the series that explains it:
| Symptom | Likely fix | See |
|---|---|---|
| Answer misses context that exists | Increase chunk overlap or k; check chunking boundaries | Parts 2, 5 |
| Answers are vague or unfocused | Smaller chunks; lower k; add reranking | Parts 2, 5 |
| Misses exact codes / names | Add hybrid (keyword) search | Part 5 |
| Surfaces outdated or wrong-scope info | Add metadata filters | Parts 2, 5 |
| Model invents facts | Strengthen "use only the context" instruction | Part 5 |
| Relevance dropped after a change | Did the embedding model change? Re-index everything | Parts 3, 4 |
The takeaway — the whole series in a breath
RAG is a pipeline, not a model. Index once: load → chunk → embed → store. Answer per question: embed the query → search (hybrid + filters) → rerank → augment → generate with citations. Chunking sets the ceiling on quality; embeddings let you search by meaning; the vector index makes that search instant at scale; retrieval, reranking, and a disciplined prompt turn matches into a grounded, traceable answer. Get each box roughly right, measure both retrieval and generation, and tune from real questions — that's how a "chat with your documents" demo becomes something you can trust in production.