How to Cut 80% of Your AI Token Bill Without Switching Models
AI agents spend 50x more tokens than chat sessions. Five engineering techniques that cut costs 60-90% without replacing your models.
This morning, a developer posted on Hacker News that their AI agent bankrupted their account while trying to scan DN42. The post got 192 points before noon.
It’s a funny story until it happens to your production environment.
TechCrunch reported last week that companies are already “3x over their entire 2026 token budget and it’s only April.” The Linux Foundation launched a dedicated Tokenomics Foundation to bring the same cost discipline to AI tokens that FinOps brought to cloud spend. This is the infrastructure layer’s next big problem.
The instinct is to switch to a cheaper model. That’s often the wrong move — or at least the last move you should make. There are five techniques that cut AI spend 60–90% without changing a single model configuration, and most teams are using none of them.
Why Agents Burn Tokens at a Different Rate Than Chat
When you use Claude or GPT-4 in a chat session, you control the loop. You send a message, you read the reply, and you decide what comes next.
Agents don’t work that way. They run reasoning loops that compound across dozens or hundreds of model calls. Each loop can load tools, retrieve documents, inspect previous outputs, and then call the model again — with all of that accumulated context in the prompt. A task that takes 2,000 tokens in a chat session can require 200,000 tokens when an agent works through the same problem autonomously.
The Lowfat CLI tool demonstrated this clearly: a single kubectl command can dump 10,000+ lines of YAML at an agent. The agent doesn’t need most of it. Filtering the output down to what actually matters saved 91.8% of tokens — without changing anything about the model, the prompt, or the task.
This is the core insight: most token waste is upstream of model inference. It’s in what you send, not which model receives it.
Technique 1: Filter What Goes Into the Context
The largest single opportunity for most teams is output filtering — controlling what tool outputs, API responses, and document retrievals actually make it into the model’s context window.
A kubectl command returns structured YAML. An agent checking pod status needs five fields, not ten thousand lines. A database query returns full row objects when the agent needs one column. A file read loads an entire document when the agent needs a specific function.
The practical approach: build a filtering layer between your tool calls and your model context. Identify your top five tool calls by token volume and write a schema that extracts only what the agent’s reasoning actually requires.
Unblocked’s production benchmarks found that curated context cut tokens by 42% compared to raw retrieval. That’s before any other optimization.
Technique 2: Use Prompt Caching Correctly
Anthropic’s prompt caching feature will reduce your token costs significantly — if you structure your prompts to take advantage of it. The catch: caching only works on the stable prefix of a prompt, and that prefix must be at least 1,024 tokens long to qualify.
The mistake most teams make: they put the system prompt last, or they interleave dynamic content with static content, breaking the cacheable prefix. Restructure your prompts so everything stable — system instructions, tool schemas, background context files — comes first. Dynamic content (the user task, the current loop state) comes at the end.
Properly structured, prompt caching reduces input costs by 50–90% on stable content. Anthropic reports up to 90% cost reduction on stable prefixes for long-running agents with consistent system prompts.
This is a zero-cost change. You’re using the same model, the same prompts. You’re just reordering what’s already there.
Technique 3: Route Tasks to the Right Model
Not every task in an agent workflow needs a frontier model.
Frontier models (GPT-4o, Claude Opus, Gemini Ultra) are priced for tasks that require complex reasoning, ambiguity resolution, or nuanced judgment. But most steps in a typical agent workflow don’t require any of that. Checking whether a value matches a pattern, formatting a structured output, summarizing a known schema, extracting a field from a document — these are tasks a smaller, cheaper model handles correctly 95%+ of the time.
Model routing means classifying the incoming task and sending it to the cheapest model capable of handling it reliably. The numbers are meaningful: routing 80% of routine inference traffic to cost-optimized models while reserving frontier models for complex tasks reduces inference spend by 60–80% with minimal quality impact.
The implementation isn’t complex: a lightweight classifier — even rule-based — that categorizes tasks before dispatching them. Simple extraction goes to the small model. Multi-step reasoning with ambiguous constraints goes to the frontier model. The tricky part is defining the categories and tuning the routing rules, and that’s an engineering problem, not a model problem.
Technique 4: Enforce Hard Token Budgets at the Agent Layer
The agent that bankrupted its operator this morning wasn’t doing anything wrong from its own perspective. It was given a task, it was executing the task, and nothing stopped it from continuing to spend.
Token budgets are circuit breakers. They’re not a quality optimization — they’re a safety mechanism. Without them, you’re relying on task completion being finite, which is a bad assumption for autonomous agents running against production systems.
Claude Code’s architecture is instructive: it enforces hard token limits, automatically compacts conversation history before the context window fills, and runs pre-execution budget checks before starting expensive operations. These controls are what separate a production-ready agent from a prototype that works until it doesn’t.
Practical implementation checklist:
- Per-run hard limits: Set a maximum token budget per agent execution. The agent stops and reports when it hits the limit, not after it already has.
- Per-task soft warnings: Log and alert at 50% and 75% of budget so you see patterns before they become incidents.
- Context window compaction: Summarize earlier turns when context grows past a threshold, rather than letting old tokens accumulate indefinitely.
- Backpressure on tool calls: Limit how many tool calls an agent can make per reasoning step. Unbounded tool-call loops are the fastest path to runaway spend.
Technique 5: Implement Semantic Caching for Repeated Queries
Agents running in production often ask the same or similar questions repeatedly — either across runs (different users, same task) or within a run (checking the same resource multiple times).
Semantic caching intercepts these requests before they reach the model. Instead of sending a new inference call, you check whether a semantically similar query has been answered recently, and return the cached response if the similarity threshold is met.
Published benchmark data shows semantic caching cuts API costs by up to 73% on workloads with high query repetition. That number only applies if your workload has sufficient repetition — a high-throughput multi-tenant agent platform sees this benefit; a bespoke analyst tool running one query per day does not.
Key design decision: set your similarity threshold conservatively. An aggressive threshold will serve stale responses to questions that actually changed. Start at 0.95 cosine similarity and loosen from there based on observed miss rates.
What This Looks Like in Practice
A team running a mid-complexity agent on Claude 3.5 Sonnet can realistically achieve:
| Technique | Expected Reduction |
|---|---|
| Output filtering | 30–50% of input tokens |
| Prompt cache structure | 50–90% on stable prefixes |
| Model routing | 60–80% of total inference cost |
| Token budgets | Eliminates runaway events |
| Semantic caching | 40–73% on repetitive workloads |
These aren’t additive in a simple way — they target different parts of the cost stack. But layering them is the point. Production systems doing this seriously route 70–80% of tokens through either an output filter, a cache, or a smaller model before a frontier model ever sees them.
The TechCrunch piece from last week quoted the Linux Foundation’s Tokenomics project lead: “In April and May, I started hearing from companies: ‘Oh my god, we are 3x over our entire 2026 token budget and it’s only April.’” That’s not a model problem. Those companies chose reasonable models. Their architecture let the tokens accumulate unchecked.
The Model Upgrade Trap
Switching to a cheaper model is tempting because it shows up as a line-item change with an immediate number attached. But it often trades cost for reliability in ways that compound — more retries, more correction loops, more human review, all of which cost tokens too.
Optimize the architecture first. Apply output filtering, prompt cache restructuring, and model routing before you touch the model tier. In most cases you’ll hit your cost targets without any model change. In the cases where you don’t, you’ll have eliminated the noise and can make a cleaner comparison.
If you want a platform that handles model routing, prompt caching, and per-agent token budgets as first-class infrastructure — without sending your data to a third-party cloud — install Auxot or walk through the setup tutorials.