The submission presents AutoCache, which uses coding agents to search over programs that compress a LM's cached prefix and reconstrct it later. The programs the agent finds mix low-rank approx., quantization, prediction and entropy coding, with settings chosen per model and per workload. Besides the standard KV cache, the method also compresses the recurrent states in hybrid models.
The paper starts from a real deployment issue. Fig 2 shows that cache capacity can decide wether a previously computed prefix is still around to be reused. Fig 5 shows that if you compress just the KV cache, some other parts can be the botleneck. Both come up in practice, and having an agent discover compression programs for them is an appealing direction. The closest baseline to the submission is KVTC (KV Cache Transform Coding, 2026).
On paper the compression numbers are very high. In Sec 4.2 AutoCache acheives 291.7× KV compression for Qwen3.5-9B on RULER, against 16.0× for the best baseline (KVTC). The evaluation covers four models and two workloads, RULER and Multi-Round Coreference Resolution (MRCR). For the comparison in Figure 4 the authors limit AutoCache to KV compression only. Fig 6 has ablations of the search procedure.
My problem witht he submission is that there seems to be evaluation leakage, and I dont think the evaluation supports the main claim that accuracy is preserved on new requests. I tried to find evidence to refute this but it is not clear from the submission. According to Sec 3.3, every candidate program is scored on validation examples, the aggregate scores go into the search history, and later proposals are conditioned on that history. The final program is then picked based on the same scores. These examples have already influenced the search, so reporting accuracy on them can overestimate performance on new requests. None of the reported results come from a seperate test on unseen examples after the full search is done. Freezing a candidate before it is evaluated does not fix this.
What I would want to see is the winning programs frozen and then run on requests that were never touched during search, with every baseline run on exactly the same requests.
In it's current form I recommend rejection. Searching for compressors in an agentic way is a promising approach, but the paper lacks clarity on the experimental design and evaluation protocol and may have serious flaws. I am willing to revisit my score based on further evidence.
A few more comments, and some related work the paper should engage with:
I dont exactly get the gap between the initial low-rank result and the existing baselines in Fig 1. The point labelled “Low-rank” already achieves around 13–14× compression while retaining almost all the quality. What exactly is different about this configuration compared to the low-rank baselines? Even without all the other searches, the submission's method seems to have a noticable improvement using just low-rank, and this is an important thing that deserves more discussion. The authors should describe what is behind this result. Sec 4.3 has some details but these are sparse and rather hand-wavy. Here is almost the entireity of this section, and it seems to suggest that AutoCache selects more granular ranks for low-rank compression, and if so, were the baselines compared at similar granularity or not? It is not clear to me what we can learn from these results.
"For Qwen3.5-9B on RULER, low-rank projection reduces KV dimensionality, prediction exploits re-
dundancy across tokens in the projected representation, and entropy coding further compresses the
quantized prediction residuals. The compressor selectively allocates extra bits to correct tokens
poorly captured by the low-rank representation, while encoding recurrent state through separate
low-rank factors. It also selects ranks and precision at a fine granularity, combining aggressive com-
pression with targeted error correction. "
Additional related work
- KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction, NeurIps 2025
- ManifoldKV: Training-Free KV Cache Compression via Euclidean Outlier Detection, ICML 2026