A team’s AI bill jumped month over month. Nobody had changed how they worked — same people, same kinds of requests, same tools. The number just climbed. When they finally traced where the money was going, the answer was almost funny: a typical request was sending about 45,000 words of context to the model, of which maybe 5,000 were actually relevant. The other 40,000 were paid for, transmitted, and ignored — on every single request, all day long.
It’s like ordering one pizza and paying for nine more you never eat, every time you order.
That story is worth understanding, because the reflexes it triggers are almost all wrong, and the one thing that works is unglamorous. If you or your team are paying for AI, this is where the money actually goes.
The 90/10 split nobody expects
Every exchange with an AI has two sides that cost money. There’s the input — everything you send in: your question, plus all the surrounding context (files, history, instructions, background). And there’s the output — what the model writes back.
The surprising part is how lopsided they are. In a typical coding workload, the input is roughly 90% of the cost, and the output only about 10%.
Sit with that for a second, because it inverts the instinct. When a bill is too high, most people look at the answers coming out — they feel expensive, they’re the visible product. But the answers are the cheap side. The expensive side is the pile of context going in, most of which you never see and never think about.
Why “just be concise” saves almost nothing
Once you know the split, the popular fixes reveal themselves as aimed at the wrong target.
Telling the model to write shorter answers works — you can genuinely cut output by three-quarters. But output is only 10% of the bill, so cutting it 75% saves about 8% overall. Real, but small. You optimized the side that was already cheap.
Telling the model to “only use what’s relevant” feels smarter, and it does nothing for cost. Here’s the mechanism, and it’s the crucial one: you’re billed for context the moment you send it, not when the model reads it. By the time the model gets to your instruction saying “ignore the irrelevant parts,” you’ve already paid to transmit all 45,000 words. The instruction can shape the answer, but the money is already spent. You cannot ask for a refund on tokens you’ve already put on the wire.
Changing the model’s settings — its answer length, its creativity dial — only touches the output. Same problem: you’re fiddling with the 10%.
Every one of these treats a symptom. None of them reduces what you send.
The actual fix: send less
The only thing that moves a bill dominated by input is putting less into the input. Not a cleverer prompt, not a bigger budget, not a fancier model — just less context riding along on each request.
The naive way to feed an AI your codebase is to dump whole files at it and let it sort out what matters. It works, and it’s ruinously wasteful, because you pay to transmit the entire file even when three lines of it were relevant. The disciplined way is to put a retrieval layer in front of the model: a fast local lookup that finds the handful of relevant pieces and sends only those, instead of shipping everything and hoping.
Done well, that’s the difference between sending 45,000 words and sending 5,000 — the same answer, a fraction of the cost. This is also the cost face of something covered elsewhere on this site: context engineering is usually pitched as a way to get better answers (a focused context is a clearer signal), and it is — but the same discipline is what keeps the bill down. A tight context is both smarter and cheaper. You rarely have to choose.
Why finding “the relevant bit” is harder than it sounds
There’s a reason naive tools just dump everything: knowing what’s actually relevant is genuinely tricky, and getting it wrong is expensive in a second way.
Two different ways to search a codebase each have a blind spot. One searches by meaning — it understands that “sign in” and “login” are the same idea — but it can confuse two things that sound alike and grab the wrong one. The other searches by exact wording — it nails the precise name you asked for — but it misses anything phrased differently. On its own, each approach misses roughly one relevant result in four. Run both at once and merge them, and the miss rate drops to about one in ten. Two weak searches cover each other’s gaps.
The subtler danger is confidence. If the lookup returns the wrong code and the model treats it as relevant, you don’t get a shrug — you get a confident wrong answer, which is worse than no answer, because it looks trustworthy. So a good retrieval layer doesn’t just find candidates; it scores them and throws away the weak ones before they’re ever sent. Interestingly, the thing that works here is a simple weighted score — a bit of meaning-match, a bit of exact-match, a bit of “how recently was this code touched” — running in a fraction of a millisecond. Asking a second AI to judge relevance was more accurate but far too slow to sit in front of every request. A plain formula beat the cleverer approach.
Be honest about the headline number
You’ll see claims like “94% fewer tokens.” Treat them the way you’d treat any best-case figure.
That number is measured against the worst possible baseline — a tool that dumps entire files every time. Compared against a modern coding tool that’s already somewhat selective, the real-world saving is smaller. The approach also strains on big, messy codebases where files do many unrelated things at once; it shines when each file has one clear job. And the teams who report these numbers often deliberately choose a smaller, faster lookup engine over a larger, more accurate one — trading a little precision for near-instant results, because a cost tool that adds a two-second delay to every request won’t get used.
None of that undoes the core lesson. It just means the honest version is “you can cut a large fraction of wasted context,” not “you’ll always save 94%.”
The model is only part of the bill
Here’s the reframe worth carrying away. When people try to control AI spend, they obsess over model choice — the cheaper tier versus the premium one, this provider versus that one. It matters. But in one measured case, the model accounted for only about 30% of total cost. The other 70% was the context being fed in.
That’s the whole argument in one line. You can agonize over picking the perfect model and still be paying mostly for the stuff you send it. (This is the cost companion to choosing a model: pick a sensible tier, then stop optimizing there and go look at your context.)
What to actually do
You don’t need to build a retrieval engine to benefit from this. The principle scales down to everyday habits:
- Assume the input is where your money goes. Before blaming the model or the answer length, ask what’s being sent on each request.
- Keep the working context tight. Long, sprawling sessions accumulate context you’re paying to re-send every turn. Clearing or compacting a session (see clear vs compact) is a cost move as much as a clarity one.
- Don’t pay to ignore things. If you find yourself telling the AI to disregard half of what you gave it, that’s a signal you’re transmitting — and being billed for — context you could have left out.
- Measure, don’t guess. The teams who fixed this tracked actual tokens sent against what would have been sent, times the price. A real number beats a vibe every time.
The counterintuitive bottom line is the most useful thing here: when an AI bill is too high, the answer is usually not a better model. It’s sending less. Fix the input — that’s where the money goes.