MCP gateway token caching is the practice of intercepting, measuring, and reusing token representations at the gateway layer between AI agents and Model Context Protocol (MCP) servers, so that repeated tool descriptions, schemas, and context payloads are not re-sent (and re-billed) on every model call. Done well, teams report 30-60% reductions in input token spend for agent workloads; done poorly, caching can serve stale tool metadata or break authorization boundaries. This guide walks through what to cache, where, and why, with the trade-offs spelled out.
Why MCP Gateways Became the Caching Layer
Also worth reading: How Do You Calculate AI Agent Costs Without Surprise Token Bills? · How Do Human-in-the-Loop AI Strategies Work in Practice? · What Are the Most Effective Advanced ML Deployment Strategies for 2026?
Since the Model Context Protocol matured through 2025, most enterprise agent deployments converged on a gateway pattern: agents talk to a central control plane, and the gateway fans requests out to registered MCP servers. Amazon's Bedrock AgentCore Gateway, F5's AI Gateway enhancements for cost and access control, and Uber's internal software-factory tooling all reflect the same architectural instinct — put a metering and policy layer between agents and tools. That gateway position is exactly where token caching belongs, because it sees every payload before it hits an LLM provider.
The economics are brutal without caching. A typical agent loop re-sends tool definitions every turn. If an agent has 25 registered tools with average schemas of 800 tokens each, that is 20,000 tokens of static context per call. At 200 agent turns per day per agent and input pricing in the $0.50-$3.00 per million token range, a fleet of 50 agents can burn $1,500-$9,000 per month purely on repeated boilerplate. Providers now offer prompt caching discounts (typically 50-90% off cached input tokens, with cache TTLs from 5 minutes to 1 hour), but those discounts only apply if your payloads are cache-friendly — stable prefixes, consistent ordering, and minimal mid-prefix mutation. The gateway is the only component that can enforce that consistency across many agents.
The Four Caching Layers That Matter
Token caching in an MCP context is not one technique; it is four distinct layers, and confusing them is the most common design mistake. The first layer is provider-side prompt caching, where you mark stable prefixes so the LLM provider's infrastructure reuses KV-compute and charges discounted rates. The second is gateway-side semantic caching, where the gateway hashes incoming prompts and returns previously computed responses for semantically identical requests. The third is tool-schema caching, where the gateway serves cached tool listings and JSON schemas to agents instead of re-fetching and re-serializing them every session. The fourth is context-window deduplication, where the gateway strips or compresses content that already appeared earlier in the conversation.
Each layer attacks a different cost. Provider prompt caching cuts the price of tokens you must send. Semantic caching eliminates the call entirely for repeated queries. Schema caching cuts bandwidth, latency, and gateway egress, and prevents cache-busting when tool servers reorder their listings. Context deduplication shrinks the prompt itself. A mature setup uses all four, but they have very different risk profiles — semantic caching can return wrong answers for queries that differ in a load-bearing detail, so it needs similarity thresholds and per-tool exclusions, while schema caching is nearly risk-free with proper versioning.
Comparison of Caching Approaches
| Feature | Provider Prompt Caching | Gateway Semantic Cache | Tool-Schema Cache | Context Deduplication |
|---|---|---|---|---|
| Typical savings | 50-90% on cached input tokens | 15-40% of total calls skipped | 10-25% of input tokens | 20-50% of context tokens |
| Latency impact | Reduces TTFT 2-5x on hits | Near-zero on hits | Minor reduction | Moderate reduction |
| Staleness risk | Low (TTL-bound) | High without thresholds | Low with versioning | Medium (compression loss) |
| Implementation effort | Low (prefix design) | Medium-High | Low | Medium-High |
| Where it runs | LLM provider | Gateway layer | Gateway layer | Gateway or agent runtime |
| Failure mode | Cache miss on prefix change | Wrong answer on near-duplicate | Outdated tool behavior | Lost detail in summaries |
Practical Implementation Steps
Start by instrumenting before you cache. Every gateway decision should be driven by measured token composition: what percentage of your input tokens are tool definitions, what percentage is conversation history, and what percentage is genuinely new content. Most teams that measure this for the first time find that 40-70% of their input tokens are static or near-static boilerplate. Uber's engineering write-ups on running their software factory at scale emphasize exactly this — you cannot optimize what per-request telemetry does not show.
Second, restructure prompts so static content leads. Provider prompt caches match on prefix, so the layout should be: system prompt, tool definitions, then conversation history, then the new user message. Any dynamic content injected early — a timestamp, a randomized ID, a per-request nonce — invalidates everything after it. Move timestamps to the end of the prompt or remove them. This single reordering often converts a 0% cache hit rate into an 80%+ hit rate within a day.
Third, implement schema caching with versioned invalidation. When the gateway fetches a tool listing from an MCP server, hash the serialized response and store it with a TTL (30-300 seconds is typical for volatile environments; hours for stable internal tools). Serve the cached listing to all agents and refresh lazily. Fourth, add semantic caching only for read-only, deterministic tools, with a minimum similarity threshold of 0.95 and a response TTL tuned per tool — 60 seconds for search-like tools, hours for stable reference lookups. Fifth, apply context deduplication last, because it carries the highest correctness risk: summarize or drop older turns only after verifying with eval sets that task completion does not degrade.
Common Mistakes and How They Bite
The most expensive mistake is putting volatile content at the start of the prompt. Teams inject session IDs, current dates, or retrieved-document lists as the first content after the system prompt, and then wonder why provider cache hit rates sit near zero. Cache-busting prefixes can quietly double an LLM bill. The fix is a prompt-layout lint in CI: flag any request where the first N tokens differ from the previous request in the same session.
The second mistake is semantic caching across users. A cached answer computed for one user's authorized context must never be served to another user with different permissions — this is both a correctness and a security failure, and GitGuardian's coverage of OAuth patterns for MCP authorization highlights how agent authorization is scoped per identity and per tool. Semantic cache keys must include the user or tenant identity, or the cache must be partitioned per principal. The third mistake is caching tool schemas without versioning: when an MCP server changes a tool's parameter shape, agents holding stale cached schemas produce malformed calls that fail confusingly at the server. Hash-based invalidation, not time-based alone, is the answer. Fourth, teams over-tune for cache hits and freeze prompts into rigidity — refusing to improve a system prompt because changing it invalidates caches. A cache invalidation cost of a few dollars is trivially worth a 10% quality improvement; keep the human in the loop on what is allowed to change.
Governance and the Control-Plane Split
An emerging enterprise pattern, described in 2026 writing on AI control-plane architecture, separates governance from execution: the control plane decides what is allowed and what it should cost, while the execution plane (the gateway and MCP servers) does the work. Token caching policies belong in the control plane as declarative rules — which tools are semantically cacheable, what TTLs apply, which tenants get cache sharing — while the gateway enforces them mechanically. This separation matters because cache policy is a security policy. A rule like "cache responses to the customer-lookup tool for 10 minutes" is simultaneously a cost decision and a data-freshness and privacy decision, and it should be reviewable, versioned, and auditable like any other policy.
Budget-aware gating is the other governance win. Because the gateway meters every token, it can enforce per-agent or per-tenant spend caps and degrade gracefully — for example, dropping from full context to summarized context when an agent crosses 80% of its daily budget. This turns caching from a passive optimization into an active cost-control mechanism, which is precisely the framing F5 and AWS use when positioning AI gateways as cost, access, and security controls rather than mere proxies.
When to Act and What It Costs
If you are running more than a handful of production agents, act now: reorder prompts and enable provider-side prompt caching first, since both are low-effort, and measure. That baseline work typically takes one engineer one to two weeks and requires no new infrastructure. Schema caching at the gateway is another one-to-two-week project. Semantic caching is a four-to-eight-week effort once you account for embedding infrastructure, threshold tuning, and exclusion lists, and it only pays back above roughly 10,000-20,000 tool calls per day — below that threshold, the operational overhead usually exceeds the savings.
Cost-wise, the software options span free open-source gateway components to commercial AI gateways priced per-request or as platform subscriptions (commonly $500-$5,000+ per month at enterprise scale). Managed options like Bedrock AgentCore Gateway bundle gateway, caching-friendly routing, and identity handling into usage-based pricing. The honest caveat: caching savings claims above 70% should be treated skeptically. Independent write-ups on token economics suggest that 25-45% total spend reduction is the realistic ceiling for most agent workloads once you exclude one-shot, non-repetitive traffic, and teams chasing higher numbers often pay for it in answer quality.
A Balanced Verdict
MCP gateway token caching is worth doing, but it is not magic and it is not free of risk. Provider prompt caching and schema caching are near-unconditional wins with minimal downside. Context deduplication is a genuine but lossy compression that must be validated against task-completion metrics, not just token counts. Semantic caching is the option most likely to disappoint: it demands careful threshold tuning, per-identity partitioning, and tool exclusions, and for many teams its complexity outweighs a 15-20% call reduction. The sequence that works is measure first, reorder prompts second, cache schemas third, deduplicate carefully fourth, and only then consider semantic caching. Treat the gateway as your control plane for token economics, version everything you cache, and audit cache policy with the same rigor as access policy — because in an agent architecture, they are the same thing viewed from different angles.