Nine researchers, eight of them at Tsinghua University, posted a paper called Cache Engineering for Agents on arXiv on October 20, 2026, and the code behind it sits in a public Apache-2.0 repository created on September 28. The engine is called AKV and it runs on SGLang. The claim on the paper’s page is narrow and numerical: an agent that manages its own key-value cache scores 1.7 points higher on average across four agent benchmarks, and costs 27.7% less, than an agent that summarizes its history the usual way.
What was actually released
AKV splits the problem in two. The agent side holds a cache policy that decides which messages to remove from the cache and when to move the tokens that stay. The system side executes those operations inside an SGLang server that keeps an OpenAI-compatible Chat Completions endpoint, so an existing agent harness talks to it without changes.
The mechanism rests on two operations the paper formalizes. Drop removes selected conversation turns from future attention without editing the text the model was given. Repos re-encodes the position of the retained keys, so a shortened cache stays internally consistent. The trajectory itself is never rewritten, which is the point: a rewritten prefix is a dead prefix, and a dead prefix has to be re-prefilled from scratch.
What ships today
- The repository thu-nics/AKV, Apache-2.0, created on September 28, 2026, with its latest commits dated October 21.
- A Python package that installs the agent policy and the serving system together, plus a server entry point.
- A quick start that serves GPT-OSS-120B with tensor parallelism of two and applies a policy keeping the tool results of the latest completed turn.
The quick start in the repository is a working recipe rather than a sketch: a Rust toolchain, a Python 3.12 environment, a package install, then a launch command pointed at a local model directory. What is missing from it is a version, because none has been tagged.
What the claim rests on
The 27.7% is not a bill the authors paid. Costs in the paper are given in US cents per task, computed from published token rates, Groq’s rates for GPT-OSS-120B, MiniMax’s own rates for MiniMax-M2.7, and the rates of Qwen3-30B-A3B, the base model of MiroThinker-1.7-Mini, applied to the main agent and to its summary calls. Every repositioned token is charged at the cache-hit rate, which the paper says deliberately overestimates the cost of key rotation. The saving is a model of a saving, built on a stated price list.
The baseline also moves depending on which section you read. The abstract compares against conventional threshold-triggered summarization; section 5.2 gives 27.7% against the plain untrimmed run and 21.9% against text engineering. Both come from the authors’ own runs on their own harness, averaged across three models and four benchmarks.
The technical problem the paper names is the one that makes this hard to copy at the application layer:
Identical token sequences can yield different cache states under different cache operation sequences, making prefix matching insufficient for correct cache reuse.
Look at the cells, not the average. On SWE-Verified, MiroThinker-1.7-Mini scores 7.4 under AKV against 7.8 on the plain run, so the headline gain hides one loss, and the paper does not discuss that row. Its cost on the same benchmark does fall, from 33.6 cents to 26.3.
| Model | Plain run: score / cost | Text engineering: score / cost | AKV: score / cost |
|---|---|---|---|
| MiniMax-M2.7 | 52.4 / 5.0 | 52.8 / 5.7 | 53.7 / 4.4 |
| GPT-OSS-120B | 51.0 / 11.2 | 48.3 / 10.2 | 53.1 / 7.3 |
| MiroThinker-1.7-Mini | 44.1 / 20.4 | 37.1 / 18.1 | 45.8 / 14.8 |
Average pass rate across four agent benchmarks and mean cost per task in US cents, as published in Table 1 of the paper. Higher score is better, lower cost is better, and cost is computed from token rates rather than billed.
The serving figures are a separate experiment, not the same runs: 1.18x output throughput and 1.51x faster P95 request latency on eight A800-80GB GPUs, four for prefill and four for decode, in BF16 because the A800 has no native MXFP4 support. The abstract rounds those to 1.2x and 1.5x and qualifies them with up to.
What the paper does not show
- No independent run. Every figure is the authors’ own, on their own harness, on four benchmarks and three open-weight models.
- No memory or storage footprint per session, so the size of the GPU bill that AKV itself creates cannot be worked out from the paper.
- No comparison with the prompt caching that commercial APIs now sell, and no dollar price for serving it yourself.
- No version tag on the repository, whose most recent commits are documentation edits.
On this page
What it changes for teams that run their own agents
If you rent tokens from a hosted API, AKV changes nothing today. Drop and repos have to be executed by the server that holds the cache, and that server is not yours. Your lever stays truncation and summarization, with the bill that comes with them.
If you serve agents on SGLang, the code is runnable and the useful part is not the headline saving. It is the finding that a policy that removes old tool results one at a time beats a threshold. In the paper’s test, trim-based context engineering that retained only the latest 12 rounds of tool responses cost 1.85x the full-context baseline, because the cut invalidates the cached prefix and forces a re-prefill. Removing turns one at a time, before the context hits a ceiling, avoids that cliff. That result is worth more than the percentage, because it applies to anyone with a context policy, whatever engine they run.
Three things to measure before believing any of it for your own workload: cost per resolved task on your trajectories, the share of your bill that is cached input rather than fresh input, and how often your agent’s memory policy throws away something it needs three steps later.
The Neural Minute take
Every agent vendor now sells long-running agents and generous context. This paper is the clearest public evidence we have read that the pruning rule inside those agents is cost per resolved task, not cost per million tokens. Two agents on the same model, with the same published rate, can land a third apart on the invoice depending on when they forget, and neither vendor’s price page shows that.
It also sets a limit on how we quote it. A 27.7% saving measured in modeled cents against the authors’ own baseline is not a price any buyer will see, and a code release without a version tag is research you can reproduce, not a service you can buy. We have not run AKV, we have no independent numbers, and we will not dress its table up as ours. If you self-host SGLang and your agents are long, this is worth a week of measurement. If you pay OpenAI or Anthropic by the token, drop and repos need a server you own, and today that is not your problem to solve.
Sources
The paper: Cache Engineering for Agents, arXiv 2610.28577, submitted on October 20, 2026 and read on October 26, 2026. Every figure above comes from that page and from the paper behind it, including the benchmark table and the serving measurements.
The code: the repository thu-nics/AKV on GitHub, Apache-2.0, created on September 28, 2026, its commit history read on October 26, 2026. The paper links to it directly from its abstract page.
No other source was used for this article, and no coverage of this release was read for the figures.