How we cut LLM inference costs by 61% without touching model quality

Six months ago our inference bill was the second-largest line item after salaries. We run an AI pipeline that classifies, extracts and summarizes support tickets for around forty customers, and every ticket touched a model at least twice. The bill grew faster than revenue, which is a bad sign in a business that is supposed to get cheaper with scale.

This is what we changed, the order we changed it in, and what each step was worth. All numbers are from our own billing exports across a comparable eight-week window.

Step 1: Prompt caching (−34%)

Our system prompt was about 3,100 tokens and identical on every call. We were paying full price for it hundreds of thousands of times a day. Moving the stable instructions and the few-shot examples to the front of the request and marking them cacheable cut input token cost on those calls by roughly 90%.

The one thing that bit us: a timestamp inside the system prompt. It changed every second, so nothing ever hit the cache. We only noticed because cache_read_input_tokens was zero in the usage logs. Check that field before you assume caching works.

Step 2: Batching non-urgent work (−15%)

About 40% of our volume was nightly re-classification that nobody looks at until morning. We moved it to the batch API, which runs asynchronously at half price. The trade-off is latency, which for that workload is irrelevant.

Step 3: A routing layer (−12%)

Not every ticket needs the most capable model. We added a small classifier that routes obvious cases (password resets, invoice requests) to a smaller model, and everything ambiguous to the larger one. We evaluated this on a held-out set of 2,000 tickets before shipping; accuracy dropped 0.4 points, which we accepted.

• Routing rule: confidence under 0.85 on the small model escalates.

• Escalation rate settled at 31%.

• We re-check the held-out set monthly because ticket mix drifts.

What did not work

We tried a second vendor purely on price. Latency doubled and the support burden of running two providers was nearly free anyway.

Switching providers on price alone. We ran a two-week trial and the migration cost in engineering time exceeded a year of the savings.

Where we landed

61% lower cost per ticket at slightly higher volume, no measurable quality loss on our eval set. If you take one thing from this: instrument usage first. We spent two weeks guessing before we exported the billing data and the answer was obvious within an hour.