Ask an AI assistant about your company's refund policy or last quarter's numbers, and it will often answer confidently — and wrongly. It isn't lying; it simply doesn't have your information, so it fills the gap with a plausible guess. The fix has a name: retrieval-augmented generation, or RAG. It's one of the most important ideas in practical AI, and underneath the acronym is a very intuitive trick: give the model the right material before you ask the question.
Why AI makes things up
A language model learns patterns from a huge amount of text up to a certain point in time. It doesn't have a live connection to your files, your database, or this morning's news. So when you ask about something specific and private — a policy document, an internal report, a niche product — the honest answer would be "I don't know," but the model's instinct is to produce the most likely-sounding answer instead. That's where confident, made-up responses (often called "hallucinations") come from.
Two facts about the model explain it:
- Its knowledge has a cut-off. Anything that happened, or any document written, after training simply isn't in there.
- It can't see your private data. Your contracts, wikis, and spreadsheets were never part of its training.
How RAG works, in four steps
The name sounds technical, but the flow mirrors how a careful person would answer a question they're unsure about: look it up first, then respond. RAG just automates the "look it up" part:
Ask
You pose a question, e.g. "What's our refund window?"
Retrieve
The system finds the most relevant documents from a trusted source.
Augment
Those documents are added to your question as context.
Generate
The model answers using that material — grounded and citable.
Ask → retrieve the right documents → add them as context → generate a grounded answer.
The crucial step is retrieve. Behind the scenes, a search system matches your question to the passages most likely to contain the answer — from a knowledge base, a set of files, or a company wiki — and only those passages get attached to the prompt. The model then has the facts in front of it, so its answer is anchored to real sources rather than to a hazy memory.
You can do RAG by hand — today
Full RAG systems are automated, but the core idea works manually in any chat assistant, right now, with zero setup. You simply play the role of the retriever:
- Find the source. Grab the policy page, the report section, the documentation — whatever actually contains the answer.
- Paste it in. Drop that text into the chat alongside your question.
- Ask against it. "Using only the text above, what is our refund window, and quote the sentence that says so."
That last instruction is the secret ingredient. By telling the model to answer only from the provided text and to quote its source, you both prevent it from wandering off into guesses and give yourself an instant way to verify the answer.
Where RAG shows up in real products
Once you recognise the pattern, you'll see it everywhere. A customer-support bot that answers from your help centre, an assistant that searches your company's documents, a research tool that cites the papers it drew from — all of them are doing RAG. They retrieve trusted content first, then let the model phrase the answer. That's why these tools can stay current and accurate even though the underlying model's training is frozen in time: the freshness comes from the documents, not the model.
Why this matters for trust
Grounding is what turns AI from an entertaining guesser into a dependable assistant. When an answer is tied to a source you can check, you can actually rely on it — and quickly catch the rare case where the retrieval pulled the wrong document. For anything that carries real consequences, "show me where that came from" should be your default, and RAG is what makes that question answerable.
The takeaway
AI invents answers when it lacks the facts; RAG fixes that by retrieving the right material and handing it to the model before it responds. You don't need a special platform to benefit — paste your source into the chat, ask the model to answer only from it, and request a quote. In the final article of the series, we'll zoom out to context itself: how much an AI can hold in mind at once, and how everything you give it quietly shapes the reply.