acceptodds
Under review as a conference paper at ICLR 2027

AutoCache: Agentic Search for LLM Prefix Cache Compression

Abstract

Storing and reusing computed states allows LLM serving systems to avoid redundant prefill computation for shared prefixes, reducing time to first token. As context lengths and the number of reusable prefixes grow, however, limited cache capacity forces evictions and expensive recomputation, making storage efficiency critical. This challenge is further complicated by modern hybrid architectures that maintain recurrent and sliding-window states in addition to KV caches, requiring compression strategies that account for model architecture, workload characteristics, and reconstruction cost. We present AutoCache, an agent-driven framework that automatically discovers compression and reconstruction programs for prefix caches. Given a model, dataset, and accuracy target, AutoCache explores a large combinatorial space of compression techniques to compose effective cache-compression recipes, refine existing methods, discover data and model architecture-specific compression opportunities. On the RULER benchmark with 32K context, AutoCache reach 291× KV-cache compression on Qwen3.5-9B, compared with 16.0× for the strongest evaluated baseline. It does so by composing factorization, quantization, entropy encoding, and protection of high-saliency cache entries in ways that would be difficult to design manually. More importantly, AutoCache can seamlessly extend compression beyond KV caches to recurrent states in hybrid models. By discovering and exploiting a previously unexploited low-rank structure in the recurrent state of linear-attention layers, AutoCache achieves 222× compression of the total cache memory footprint. These results demonstrate the potential of automatically discovering compression algorithms that are tailored to the model architecture and workload, therefore enabling substantially more storage-efficient prefix caching for LLM serving.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
KV-Cache, Hybrid Model, Agentic Search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xVHqOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes AutoCache, an agent harness automatically discovers model- and workload-specific compression algorithms for LLM cache.

Critical strengths
  1. The authors provide a well-designed harness for applying an automated research agent to KV-cache optimization, with an implementation integrated into the SGLang cache manager.
  2. The results are impressive, achieving 222× compression.
Critical weaknesses
  1. My biggest concern is how the KV-cache auto-research agent proposed in this paper differs from a general auto-research agent. What is the technical novelty of the design? For example, how does this approach differ from directly asking an agent to complete the KV-cache compression task? How does it differ from manually combining these optimization methods incrementally? Alternatively, is the proposed method simply a way to facilitate this process rather than requiring manual design for incremental combinations?
Questions for authors

See my comments in weakness.

Reviewer_jik3OpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The submission presents AutoCache, which uses coding agents to search over programs that compress a LM's cached prefix and reconstrct it later. The programs the agent finds mix low-rank approx., quantization, prediction and entropy coding, with settings chosen per model and per workload. Besides the standard KV cache, the method also compresses the recurrent states in hybrid models.

The paper starts from a real deployment issue. Fig 2 shows that cache capacity can decide wether a previously computed prefix is still around to be reused. Fig 5 shows that if you compress just the KV cache, some other parts can be the botleneck. Both come up in practice, and having an agent discover compression programs for them is an appealing direction. The closest baseline to the submission is KVTC (KV Cache Transform Coding, 2026).

On paper the compression numbers are very high. In Sec 4.2 AutoCache acheives 291.7× KV compression for Qwen3.5-9B on RULER, against 16.0× for the best baseline (KVTC). The evaluation covers four models and two workloads, RULER and Multi-Round Coreference Resolution (MRCR). For the comparison in Figure 4 the authors limit AutoCache to KV compression only. Fig 6 has ablations of the search procedure.

My problem witht he submission is that there seems to be evaluation leakage, and I dont think the evaluation supports the main claim that accuracy is preserved on new requests. I tried to find evidence to refute this but it is not clear from the submission. According to Sec 3.3, every candidate program is scored on validation examples, the aggregate scores go into the search history, and later proposals are conditioned on that history. The final program is then picked based on the same scores. These examples have already influenced the search, so reporting accuracy on them can overestimate performance on new requests. None of the reported results come from a seperate test on unseen examples after the full search is done. Freezing a candidate before it is evaluated does not fix this.

What I would want to see is the winning programs frozen and then run on requests that were never touched during search, with every baseline run on exactly the same requests.

In it's current form I recommend rejection. Searching for compressors in an agentic way is a promising approach, but the paper lacks clarity on the experimental design and evaluation protocol and may have serious flaws. I am willing to revisit my score based on further evidence.

A few more comments, and some related work the paper should engage with:

I dont exactly get the gap between the initial low-rank result and the existing baselines in Fig 1. The point labelled “Low-rank” already achieves around 13–14× compression while retaining almost all the quality. What exactly is different about this configuration compared to the low-rank baselines? Even without all the other searches, the submission's method seems to have a noticable improvement using just low-rank, and this is an important thing that deserves more discussion. The authors should describe what is behind this result. Sec 4.3 has some details but these are sparse and rather hand-wavy. Here is almost the entireity of this section, and it seems to suggest that AutoCache selects more granular ranks for low-rank compression, and if so, were the baselines compared at similar granularity or not? It is not clear to me what we can learn from these results.

"For Qwen3.5-9B on RULER, low-rank projection reduces KV dimensionality, prediction exploits re- dundancy across tokens in the projected representation, and entropy coding further compresses the quantized prediction residuals. The compressor selectively allocates extra bits to correct tokens poorly captured by the low-rank representation, while encoding recurrent state through separate low-rank factors. It also selects ranks and precision at a fine granularity, combining aggressive com- pression with targeted error correction. "

Additional related work

  • KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction, NeurIps 2025
  • ManifoldKV: Training-Free KV Cache Compression via Euclidean Outlier Detection, ICML 2026
Critical strengths

See above

Critical weaknesses

See above

Questions for authors

See above

Reviewer_9cVrOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

It is well known in literature, that a single KV cache compression is never the best one for a LLM, task, serving platform combination. It is required to tune the compression scheme to a particular scenario. The proposed Autosafe approach automates this task using a simulated annealing based optimization approach. A suitable evaluation harness is embedded in the optimization module.

The proposed approach involves a simulated annealing based search technique for deciding on the best KV cache compression scheme for a particular LLM on a particular input data and task and for a specific serving environment. The framework implements the evaluation harness, the KV cache simulator, and the search state representation. A suitable heuristic objective function involving the compression ratio and reduction i performance is designed.

Experimentally, the proposed method is compared with state of art approaches like KIVI etc on the RULER benchmark. It is seen that the proposed method compares well with them on the whole sequence as well as the sliding window setting.

Critical strengths

The paper proposes a automatic approach of choosing the best KV cache compression scheme for a given LLM, task, data, server combination. It uses an optimization framework for obtaining the best combination. The implementation involves the evaluation harness as well as search controller. The KV cache search space representation is also interesting and efficient.

The experimental results are extensive and well planned. It shows that the proposed method compares well with state of art techniques. It is often seen that the generated scheme by optimization outperforms common popular compression schemes.

Critical weaknesses
  1. The observation that a single compression scheme is not the best one always is obvious. The proposed work only adds an optimization module. This is a natural extension. The approach is not very attractive in real life scenario unless the optimization scheme is tailor made and does not have low foot print. In some sense there is no novelty here.

  2. The word agentic is not much appropriate for the proposed algorithm. It is not clear why this qualifies as an agentic framework.l

  3. Details of the search controller and optimization schemes and harnesses are missing. For example how does one estimate the parameters used in the objective function without much overhead is not clear from the description.

Questions for authors
  1. There are many sophisticated data structures and techniques for process sequence optimizations that are suitable for similar tasks. They should be explored instead of a simple prefix based representation.

  2. What are the advantages of the particular optimization algorithm used?

  3. What are the computational overhead of each of the component of the AutoSafe optimizer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.