Your RAG System Doesn't Have a Retrieval Problem
When an AI answers badly from your own documents, everyone blames the search. In every system I have debugged, the search was fine. The documents were the problem.
Derpx06
Imagine you hire a very smart assistant who has never seen your company. You want them to answer customer questions, so you hand them a stack of pages torn from your handbook and say: read these, then answer.
If the pages you hand over are the wrong pages, it does not matter how smart the assistant is. And if the pages are the right pages but each one has been torn in half, it matters even less.
That is the whole idea behind RAG — retrieval-augmented generation. You search your documents for the relevant bits, you paste those bits in front of the AI's question, and the AI answers using them. It is the standard way to make a general-purpose model answer questions about your stuff.
And when it goes wrong, almost everybody blames the searching. I want to argue that the searching is usually innocent.
The wrong week
Every one of these systems I have worked on has had the same bad week.
The answers are poor, so somebody swaps the search model. Still poor, so somebody adds a re-ranker — a second pass that reorders the results. Then keyword search alongside the AI search. Then rewriting the question three different ways. Then just fetching more results.
Each change is reasonable on its own. Each one makes the system slower. The answers stay bad.
The reason is almost always the same, and it stings: the search was returning the right pages the whole time. The pages were just not worth returning.
Read what it actually found
When an answer is wrong, the first thing to do is not to change anything. It is to print out exactly what got handed to the AI, and read it yourself as if you were the one who had to answer.
I put this off far too long on AssistFlow AI, a support tool we built that crawls a customer's website and answers questions from it. The answers were vague in a way that felt like an AI problem — that hedging you get from a model that is not sure.
Then I finally printed the pages it had been given for a failing question. The top result was a navigation menu. The second was a cookie banner. The third was the correct paragraph, chopped in half at exactly the sentence containing the answer.
The search did its job. Given the pile it was searching, those genuinely were the closest matches. The pile was the problem, and no amount of re-ranking fixes a pile.
Cutting documents up is a decision, not a setting
Documents are too big to hand over whole, so you cut them into pieces first. These pieces are called chunks, and almost everyone starts with something like cut every thousand characters, overlapping a bit.
That looks like a setting you tune. It is not. It is a claim — that meaning in your documents arrives in thousand-character lumps that survive being cut at an arbitrary point. For most real documents that claim is simply false.
Picture what goes wrong:
A price table cut in half is worse than useless. The header row lands in one chunk and the numbers land in another, so the search confidently returns numbers with nothing to say what they mean.
A step-by-step guide split at step four hands the AI half a procedure, with nothing to signal that the rest exists.
A page headed "Refund policy" whose body says this does not apply to annual plans becomes two chunks. One of them now cheerfully asserts the opposite of the truth.
The fix is not a cleverer cutting tool. It is deciding, for each kind of document, what the smallest complete thought is — and refusing to cut smaller. For a FAQ that is one question with its answer. For an API reference it is one endpoint. For a policy it is one section with its heading still attached. Sometimes the right chunk is a whole short page, and the right setting is don't cut this at all.
Two things that help more than any search upgrade
Both happen when you store the documents, not when you search them.
Put back the context the chunk lost. A paragraph pulled out of a document has been stripped of everything that told you where it lived. So paste the document title and heading trail on top before you store it. Now the stored piece says Refund policy → Annual plans → the text, instead of an anonymous paragraph. Questions about refunds start matching the thing that is actually about refunds.
This single change has rescued more systems, in my experience, than every re-ranker I have ever added.
Store labels, then actually filter on them. AI search has no opinion about whether something is current. If three versions of your pricing page are in there, it will happily hand over the one from two years ago, because being out of date is not a direction in AI-search-space. Product, version, date, language, customer — these are cheap to store and they turn a whole category of "the AI made something up" reports into a simple filter.
Why you cannot tell what is working
Here is what makes the bad week possible. Without a set of test questions, every change feels like it might have helped. You try a re-ranker, you spot-check four questions, two look nicer, you ship it. You have learned nothing and added a delay.
The smallest useful version is thirty questions where you already know which document holds the answer. Thirty. Written by hand in an afternoon, from real customer questions if you have them, from reading your own documents if you do not.
Then measure two separate things:
- Did the right chunk show up at all in what the search returned?
- Given those chunks, was the final answer right?
Splitting those two numbers is the entire point, because it tells you which half of the system to fix. Right chunk found but wrong answer means the search is fine — go look at your prompt or your model. Right chunk not found means stop fiddling with the prompt.
Nearly every team I have watched debug one of these has been busy improving the half that already worked, because they only ever looked at the number at the end.
When it really is the search
Sometimes it is, and there are two honest signals.
Exact codes failing. Part numbers, error codes, function names. AI search works by meaning, and by meaning ERR_4021 and ERR_4012 are practically identical — they are the same kind of thing. This is exactly what old-fashioned keyword search is for, and running both together is the real fix.
Questions that are secretly two questions. How do I move off the free plan, and what happens to my old exports? The system turns your question into a single point and looks for nearby documents — but this question lives in two unrelated places at once, and the answer needs both. Splitting it into two searches is the right tool here, and worth its cost here specifically.
Both of those are diagnoses. You reach for them when a measurement points at them, not as a default.
The order I would work in
- Read twenty failing examples word for word before changing a line.
- Write thirty test questions with known correct sources.
- Fix the chunking so no chunk is half a thought.
- Paste title and headings onto every chunk before storing it.
- Add filters for date, version and customer.
- Now check whether a re-ranker earns its delay.
The search is the most visible part of the system and the least likely to be broken. It should be the last thing you touch. It is, in my experience, the first thing everybody reaches for — me included, for most of a week I would like back.