The first time it happens to someone, it’s almost always a citation.
You ask for background on something, and back comes a tidy paragraph with a reference: a paper, a plausible-sounding author, a journal, a year. It’s exactly the sort of thing you’d have spent an afternoon finding. You go to look it up, and it isn’t there. Not moved, not paywalled — it doesn’t exist. Nothing about the answer looked any different from the ten correct things the model told you an hour earlier.
That gap — between how the output looks and whether it’s true — is the whole subject of this guide. The industry calls it hallucination, which is a slightly unfortunate word because it implies malfunction. The behaviour is exactly what the machine is built to do, and once you understand why, you stop being surprised by it and start designing around it.
It isn’t lying, and it isn’t broken
Lying requires knowing the truth and choosing to say something else. Nothing in the system is doing that. There’s no internal ledger of facts being consulted and overridden.
What a language model does — and this is the entire mechanism, honestly — is produce the next most plausible piece of text given everything it has seen. It’s an extraordinarily sophisticated version of that one operation, trained on enough material that “plausible” and “true” overlap most of the time. When you ask what year a famous treaty was signed, the plausible continuation and the correct continuation are the same string, because that fact appeared consistently in the training material.
But the machinery has no way to tell those two cases apart. When you ask for a citation that supports a specific claim, “a paper with a title like that, by an author with a name like that, in a journal like that, around that year” is a superbly plausible continuation. It’s structured exactly like ten thousand real citations. It just doesn’t correspond to anything.
The model isn’t retrieving. It’s composing. Truth is a frequent side effect of composing well, not the goal of the operation.
Why confidence tells you nothing
Here’s the property that makes this genuinely dangerous in a work setting, more than the error rate itself.
In a human colleague, uncertainty leaks. They hedge, they slow down, they say “I think it was around 2019?”, they mention they’re not sure. You’ve spent your whole life reading those signals and you read them unconsciously.
A model’s fluency is completely uncoupled from its accuracy. The invented citation arrives with the same clean formatting, the same measured tone, and the same air of competence as the correct one. There is no tell. When it does hedge, that hedging is itself generated text — a plausible-sounding hedge, not a report from an internal confidence meter.
This is why “I’ll just notice when it’s wrong” fails as a strategy. You won’t. The whole reason smart, careful people get caught is that the failure is invisible from the outside.
The five places it bites hardest
Risk isn’t evenly spread. It clusters, and the clusters are predictable enough to memorize.
1. Specific numbers. Prices, percentages, market sizes, dates, dimensions, version numbers. A number is the highest-risk token in any answer, because “a number of about the right magnitude in about the right place” is trivially plausible and completely unverifiable by looking at it.
2. Citations, sources, and quotes. As above. Also: quotes attributed to real people, page references, clause numbers, case names. The more precisely-formatted the reference, the more convincing the invention.
3. Names and attributions. Who founded what, who said what, which company acquired which. Especially bad for anything niche, regional, or recent — the material was thinner, so the composition has more room to drift.
4. Features and settings that should exist. Ask how to do something in a piece of software and you may get a menu path, a setting name, or an option that is perfectly sensible and entirely fictional. The model has seen a thousand tools with a “Preferences → Export → Scheduled” and it will build you one. This one catches technical and non-technical people equally.
5. Anything about material it wasn’t actually given. If you ask “what does our Q3 report say about churn” without attaching the report, you will get an answer about churn. It will be a beautifully written answer about a report that exists only in that sentence.
Notice the pattern across all five: risk rises exactly when the answer depends on something specific that wasn’t in the room.
Grounding: the fix that actually works
Which points straight at the fix. Don’t ask the model to recall — give it the source and ask it to work from that.
This is called grounding, and the difference in reliability is not subtle. “What are our top three churn drivers?” is an invitation to compose. “Here is the churn report — what are the top three drivers it identifies, and quote the line for each?” is a reading comprehension task, which is something these systems are genuinely excellent at, and which you can check in ten seconds because you asked for the quote.
The practical version of this in daily work:
- Attach or open the actual file instead of describing it. Tools like Claude Code Desktop work on a folder you point them at, which means the real documents are right there — the single biggest accuracy upgrade available to most people is simply using the source rather than the memory.
- Ask for quotes and locations, not just conclusions. “Quote the sentence” converts an unverifiable claim into a two-second check.
- Let it look things up when the tooling supports search or connectors, rather than asking it to remember. A retrieved fact has a link you can click; a recalled one doesn’t.
- Say what to do when it doesn’t know. “If the document doesn’t say, tell me it doesn’t say” genuinely changes the output. It gives the model an acceptable path that isn’t invention.
There’s a deeper version of this discipline — curating everything the model can see before it answers — which we cover in context engineering. Grounding is its most important single move.
The thirty-second habit
Grounding reduces the failure rate. It doesn’t get you to zero, so you also need a checking habit cheap enough that you’ll actually do it every time.
Here it is. Before you send, ship, or act on anything a model produced, run your eye over it and ask one question: which parts of this would be embarrassing if they were invented?
You’re not re-doing the work. You’re triaging. In a typical page of output, the answer is three or four things — a figure, a name, a claim about what some tool does, a date. Those get checked against a source. The rest of the page — the structure, the argument, the phrasing, the summary of material you supplied — carries almost no invention risk and doesn’t need it.
Three rules make that habit hold up:
- Check against a source, not against the model. Asking it to double-check its own answer regenerates; it doesn’t verify. Two plausible compositions in a row are not a corroboration.
- The check has to be cheap or you’ll skip it. This is why “ask for the quote” is such a good move — it front-loads the verification into the original request.
- Never let AI be the last check on anything with consequences. Something going to a client, a regulator, a board, or a customer’s invoice gets a human as the final gate. Always. That’s not distrust of the tool; it’s just where accountability lives.
Why this gets more important as the tools get better
You might expect this to fade as models improve. In one narrow sense it will — the error rate keeps falling. But the risk doesn’t fall with it, and may even rise, for a reason that has nothing to do with the technology.
A tool that’s wrong one time in three keeps you alert. A tool that’s wrong one time in fifty trains you, over a few hundred correct answers, to stop looking. The rare failure then lands in a workflow with no checking left in it. Reliability erodes vigilance faster than it earns it.
And agents raise the stakes again. When a model was producing a paragraph for you to read, one bad fact was one bad fact. When an agent takes a goal and does twelve steps on its own — reading files, pulling numbers, building the deliverable — a fabricated intermediate step can be quietly built on by every step after it. This is the same reason we argue in writing got cheap, understanding didn’t that review, not generation, is now the leveraged skill.
What to take from this
Not “don’t trust AI.” That’s both unhelpful and, at this point, uncompetitive. The useful framing is narrower and much more actionable:
Trust it completely with material you gave it. Trust it not at all with specifics you didn’t.
Summarizing your document, restructuring your notes, drafting from your template, translating what’s in front of it, generating five options for you to choose between — reliable, and the productivity gain is real. Recalling a figure, a source, a name, or a setting from memory — that’s the composition machine running unsupervised, and it needs a source before it leaves your hands.
Get that line clear in your head and hallucination stops being a reason to distrust the whole category. It becomes what it actually is: a known property of a tool, with a known workaround, that costs you thirty seconds a page.