This is the opening part of a five-part series on how retrieval-augmented generation (RAG) actually works under the hood — the same technology behind "chat with your documents" and grounded enterprise copilots. In this part we'll build the mental model: the two phases of a RAG system, the eight steps inside them, and a single worked example we'll carry through the whole series. No maths yet — just the shape of the thing.
The one-sentence version
A language model is brilliant at language but it doesn't know your facts — your policies, your product docs, last quarter's numbers. RAG fixes that by finding the relevant text first and handing it to the model before it answers. The model then responds from that supplied material instead of guessing from memory. That's the entire idea. Everything else is plumbing to make the "find the relevant text" step fast and accurate.
Here's the analogy I keep coming back to. Imagine an extremely well-read assistant who has never seen your company's files. If you just ask them a question, they'll improvise. But if you first slide the exact right page across the desk and then ask, they'll give you a precise, grounded answer. RAG is the machinery that, in a fraction of a second, picks the right page out of a library of millions.
RAG happens in two phases
This is the single most important thing to understand, and it's where most beginners get confused. RAG is not one flow — it's two. They happen at completely different times:
- Indexing (offline, done ahead of time). You take all your documents and prepare them for fast searching. This happens once when you set the system up, and again whenever your content changes. The user never sees it.
- Querying (online, done per question). When a user actually asks something, you search the prepared index, grab the best matches, and feed them to the model. This happens live, in milliseconds, every time.
Phase 1 — Indexing: preparing your knowledge
The indexing phase turns a messy pile of documents into a searchable index. It has four steps, and each one is a future part of this series:
Run once up front, then re-run only when your documents change.
Let's walk each step with a concrete example. Say your knowledge base is a single 40-page employee handbook PDF.
- Load — A loader reads the PDF and extracts its raw text. In a Microsoft setup the source might be SharePoint, OneDrive, or a website; the loader's job is just to get clean text out.
- Chunk — You can't store a whole 40-page document as one searchable unit — it covers too many topics at once. So you split it into bite-sized passages, say a few paragraphs each. One chunk might be the refund policy, another the parental-leave policy. (This is deceptively important, which is why Part 2 is entirely about it.)
- Embed — Each chunk is passed through an embedding model that converts its meaning into a long list of numbers — a vector. Chunks about similar topics end up with similar numbers. This is the magic that lets us search by meaning instead of keywords (all of Part 3).
- Store — The vectors, along with the original chunk text, are saved into a vector index (for example Azure AI Search). The index is organised so that, later, we can find the nearest vectors to a query almost instantly (Part 4).
At the end of indexing, your 40-page PDF has become, say, 120 chunks, each paired with a vector and sitting in a searchable store. Nothing has been shown to a user yet. You're simply ready.
Phase 2 — Querying: answering a question
Now a real user types a question. The querying phase reuses the embedding model from indexing, then adds search and generation. Four more steps:
Embed query
Turn the user's question into a vector too — Part 3.
Search
Find the chunks whose vectors are nearest — Part 4 & 5.
Augment
Paste those chunks into the prompt as context — Part 5.
Generate
The model answers using the supplied chunks — Part 5.
Steps 5–8 happen live, in well under a second, for every single question.
Continuing the handbook example, suppose an employee asks: "How many days do I get to return a faulty laptop?"
- Embed the query — The question is run through the same embedding model used during indexing, producing a query vector. Using the same model is essential — it's what puts the question and the chunks in the same "meaning space."
- Search — The vector index compares the query vector against all 120 stored chunk vectors and returns the closest few — the "top-k." Notice the user said "return a faulty laptop" but never the word "refund." Keyword search would miss the refund-policy chunk; vector search finds it because the meaning is close.
- Augment — The top chunks (the refund-policy passage, maybe the IT-equipment passage) are pasted into the prompt, usually with an instruction like "Answer using only the context below, and cite it."
- Generate — The model reads that context and produces a grounded answer: "Faulty equipment can be returned within 30 days…" — drawn from your handbook, not from its imagination.
Why not just give the model everything?
A fair question: if the model is so capable, why not paste the entire 40-page handbook into every prompt and skip all this machinery? Three reasons make retrieval necessary rather than optional:
| Constraint | Why "paste everything" breaks |
|---|---|
| Context window limit | Models can only read so much text at once. Your real knowledge base might be thousands of documents — it simply won't fit. |
| Cost | You pay per unit of text sent. Stuffing entire libraries into every request is wasteful when only one passage is relevant. |
| Accuracy | Burying the one relevant paragraph inside hundreds of irrelevant pages makes answers worse, not better — the model gets distracted. Less, but relevant, beats more. |
Retrieval is what makes RAG scale: instead of sending everything, you send only the handful of passages that matter for this question.
The vocabulary you'll meet in this series
A quick glossary so the later parts feel familiar. Don't memorise these — we'll build each one up properly:
| Term | Plain meaning | Covered in |
|---|---|---|
| Chunk | A small passage of a document — the unit you search over. | Part 2 |
| Embedding / vector | A list of numbers representing the meaning of a piece of text. | Part 3 |
| Embedding model | The model that converts text into that vector. | Part 3 |
| Vector index / store | A database built to find the nearest vectors fast. | Part 4 |
| Similarity / distance | How "close in meaning" two vectors are. | Part 4 |
| Top-k retrieval | Returning the k closest chunks (e.g. the top 5). | Part 5 |
| Reranking | A second, smarter pass to reorder the candidates. | Part 5 |
| Augmentation | Inserting retrieved chunks into the prompt. | Part 5 |
The full picture in one line
If you remember nothing else, remember this sentence: Index your documents once (load → chunk → embed → store); then for every question, embed → search → augment → generate. The first half is preparation; the second half is the live answer. Every remaining part of this series is just a deep, friendly look inside one of those eight boxes.
What's next
In Part 2 we'll open the box that quietly determines whether your whole RAG system is good or useless: chunking. How big should a chunk be? Where do you cut? Why does overlap matter? We'll use real text and visuals to make the trade-offs obvious — because a brilliant embedding model can't rescue badly cut chunks.