Search over your own documents (RAG)

A model knows what it read while it was being trained. It has never seen your employee handbook, last quarter’s contracts, or the four thousand support tickets your team filed this year. Ask it about any of those and you get a confident, plausible, invented answer — the thing it is worst at, dressed as the thing it is best at. There are only two honest fixes: put the relevant documents directly in front of the model, or give it a way to go and find them. RAG is the second one.

The mechanics are less mysterious than the acronym. Your documents get chopped into passages — a page, a section, a few paragraphs each. Every passage is run through a model that turns text into a long list of numbers capturing what it means, not which words it used; those lists are called embeddings, and two passages about the same idea end up numerically close even if they share no vocabulary. The lists go into a vector database, which is built to answer one question very fast: what is closest in meaning to this? When you ask something, your question gets turned into numbers the same way, the database returns the handful of nearest passages, and those passages are pasted into the model’s context alongside your question. That is the “retrieval.” The model then writes the answer from what it was handed — the “generation.”

The payoff is real. Answers are grounded in text you can point at, so a good system cites its sources and you can check them. Updating a document updates the answers immediately; there is no retraining involved. And you can put ten thousand documents behind a question without ever needing the model to hold ten thousand documents at once.

The failure mode is equally real, and it is worth understanding before anyone sells you one. A RAG system can only answer from what retrieval happened to find. If the search pulls the wrong three passages, the model will answer fluently and wrongly from them, and nothing about the answer will look different from a correct one. It is also poor at questions that require reading everything — “how many of our contracts include a termination clause?” is an aggregate question, and retrieving the five most relevant contracts does not answer it. Retrieval quality, not model quality, is where these systems usually break.

Now the part that gets skipped in most explanations: you very often do not need this. If your documents fit in a folder on your machine, open the folder in Claude Code and just ask. Claude reads the actual files — searching them, opening what looks relevant, following the trail — which is retrieval, only without a pipeline to build, an index to keep in sync, or infrastructure to pay for. For the common case of “I have sixty documents and some questions,” this is simpler, more accurate, and available this afternoon.

The second escape hatch is a connector. If the material lives in Google Drive, Notion, a ticketing system, or a database, connecting Claude to that tool gives it live search over the real system, and somebody else has already built and maintained the retrieval layer. You get freshness and existing permissions for free, which is a meaningful advantage over a copy of your documents sitting in a vector database slowly going stale.

So when is building RAG genuinely the right call? When the corpus is far too large to hand over — millions of documents, not thousands. When many people need to query it at once, cheaply and fast. When different users must see different subsets. Or when you are building a product, not answering your own questions. Those cases are legitimate and common, but each of them makes RAG an engineering project with an ongoing owner, not a feature you switch on. If someone pitches you “RAG over your data,” the two questions worth asking are how retrieval quality gets measured, and what the system does when it finds nothing relevant — because the answer to that second one is where confident nonsense comes from.