By now we have a pile of chunks (Part 2), each turned into a vector of numbers (Part 3). This part answers the two questions that turn those vectors into a working search engine: how do we measure that two vectors are "close," and how do we find the closest ones among millions without checking every single one? The answers are cosine similarity and a clever index called HNSW — both of which are far friendlier than they sound.
Measuring "closeness" between vectors
In Part 3 we drew chunks as dots on a meaning-map and said "find the nearest dots." But what does nearest mean for a list of 1,536 numbers? We need an actual measure of distance. There are a few, and you'll meet these names constantly:
| Measure | Intuition | Notes |
|---|---|---|
| Cosine similarity | The angle between two vectors — are they pointing the same way? | The default for text embeddings. Ignores length, focuses on direction (meaning). |
| Dot product | Direction and magnitude combined. | Fast; equivalent to cosine when vectors are normalised (most are). |
| Euclidean distance | Straight-line distance between the two dots. | Intuitive, but for text, angle usually beats raw distance. |
Why cosine, in one picture
Cosine similarity asks: do these two vectors point in the same direction? Two chunks about refunds point almost the same way (small angle → similarity near 1.0). A refund chunk and a leave chunk point in very different directions (big angle → similarity near 0). It deliberately ignores how "long" a vector is, because for meaning we care about direction, not magnitude. The score runs from 1.0 (identical meaning) down toward 0 (unrelated), and the system simply ranks chunks by it.
The system scores every chunk, sorts by score, and keeps the top few. Here the refund and IT-equipment chunks win.
The scaling problem
Scoring four chunks is trivial. But a real knowledge base might hold millions of chunks. Comparing the query against every single one — called a flat or brute-force search — gives perfect results but gets painfully slow as the collection grows. Doing a million 1,536-number comparisons for every keystroke doesn't scale.
So we make a classic engineering trade: give up a tiny bit of accuracy for an enormous speed-up. Instead of checking everything (exact search), we use approximate nearest-neighbor (ANN) search, which finds almost always the right neighbours while looking at only a small fraction of the data.
| Exact (flat / brute-force) | Approximate (ANN) | |
|---|---|---|
| Accuracy | Perfect — always the true nearest | Very high — occasionally misses an edge case |
| Speed at scale | Slow; grows with collection size | Fast; barely grows with collection size |
| When to use | Small collections, or when 100% recall is critical | The default for large, production knowledge bases |
HNSW: how approximate search stays fast
The most popular ANN method is HNSW — Hierarchical Navigable Small World. The name is scary; the idea is delightfully simple, and there's a perfect real-world analogy: travelling across a country.
Suppose you're at a random town and want to reach a specific house. You don't inspect every house in the country. Instead you use layers of transport:
- Top layer — flights. Take a long flight to land in roughly the right region of the country. A few big hops cover huge distances.
- Middle layer — highways. Drive the motorways to get into the right city.
- Bottom layer — local streets. Navigate street by street to the exact house.
HNSW builds exactly this kind of multi-layer map of your vectors. The top layer has a few "long-haul" links that let a search jump across the whole space in a couple of steps; lower layers have progressively shorter, denser links for fine-grained navigation. A query starts at the top, makes a few big jumps to the right neighbourhood, then drops down layer by layer to home in on the true nearest chunks — touching only a tiny handful of vectors along the way.
By starting coarse and refining, HNSW finds the nearest vectors among millions in milliseconds, visiting only a sliver of the data.
What's actually stored in the index
A vector index entry is more than just numbers. For each chunk you typically store three things together, and all three earn their keep:
The vector finds the match; the text is what we hand to the model; the metadata enables filtering and citations (Part 5).
- The vector — what the similarity search actually compares against.
- The original chunk text — because once we've found a chunk, it's the human-readable text (not the numbers) that we feed to the language model.
- The metadata — source, page, date, and permissions from Part 2, used to filter results and to cite sources.
Keeping the index fresh
Your documents change — policies get updated, pages get added, old ones retired. A production index needs a refresh strategy so it doesn't drift out of date:
- Add new or changed documents by chunking and embedding just those, then inserting their vectors.
- Update a changed chunk by re-embedding it and replacing the old vector.
- Delete vectors whose source content was removed, so stale answers can't surface.
Tools like Azure AI Search support indexers that watch a source (such as a SharePoint library or blob storage) and keep the index in sync automatically, including change detection — so you rarely rebuild from scratch.
Where we are now
We can now take a question, embed it, and retrieve the top few most-relevant chunks from millions in milliseconds. That's a complete retrieval engine. What remains is the part the user actually sees: turning those retrieved chunks into a trustworthy answer — and squeezing out the last bit of accuracy with hybrid search and reranking.
What's next
Part 5: Retrieval, Reranking & Generation closes the series. We'll choose how many chunks to fetch (top-k), combine vector search with keyword search (hybrid) and metadata filters, add a reranking pass that reorders candidates with a smarter model, then assemble the final prompt so the model answers only from the retrieved context — with citations. That's the whole pipeline, end to end.