How We Cut Our LLM Bill by 70% Without Users Noticing

Six months after launching an AI writing assistant inside a document tool, our monthly model bill had grown faster than our user count. Finance noticed before engineering did, which is never a good sign.

We spent three weeks on cost work. The bill dropped by roughly 70%, and our quality scores on the evaluation set barely moved. None of the changes were clever. All of them were things we should have done from the start.

Step one: find out where the money goes

We had been logging requests but not attributing cost. The first job was tagging every call with the feature that triggered it and the number of input and output tokens.

The results surprised everyone. The flashy feature, rewriting whole paragraphs, was only 18% of spend. The biggest cost was an autocomplete suggestion that fired on almost every pause in typing. It accounted for more than half the bill, and most suggestions were never accepted.

Step two: the changes that worked

1. Route by difficulty. Autocomplete did not need our largest model. We moved it to a small, fast model and kept the large one for rewrites and summaries. Acceptance rates for suggestions dropped by less than one percentage point.
2. Trim the prompt. Our system prompt had grown to about 1,900 tokens through months of patches. Many instructions contradicted each other. We rewrote it to around 600 tokens and the eval score actually went up slightly.
3. Send less context. We had been sending the whole document for every request. For most features, the current section plus a short outline was enough.
4. Cache repeated work. Document summaries were regenerated every time someone opened the side panel. We cached them and invalidated only when the text changed meaningfully.
5. Debounce the triggers. Autocomplete now waits for a slightly longer pause and skips requests while the user is clearly still typing fast.

Step three: guard against regressions

Every change went through our evaluation set before shipping. Two ideas failed that test. Using the small model for summaries made them noticeably vaguer, and an aggressive context cut broke answers about earlier sections of long documents. We rolled both back.

We also added a weekly cost report per feature, with a simple threshold alert:

feature        calls/day   avg tokens   cost/day
autocomplete   410,000     310          low
rewrite        22,000      2,100        medium
summary        9,500       1,400        low (cached)

What I took away

Cost is a product decision, not just an infrastructure one. The autocomplete feature was expensive because of a design choice about when it fired, not because of the model itself.

The cheapest token is the one you never send.

If your AI bill is growing faster than your usage, do not start by negotiating with your provider. Start by measuring which feature is spending the money and asking whether it needs to.

Cover: Photo by Tyler on Unsplash.