- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets an important problem in KV cache compression or eviction in multi-turn agentic inference. The KV cache grows unboundedly as tool calls, model reasoning and outputs accumulate. To run under fixed memory budget, the engine must continuously evict KV entries before knowing what future turns or queries may ask.
Given the problem, the paper finds that two existing methods are insufficient to solve it. Query-dependent methods prioritize KV entries relevant to current query, potentially discarding entries needed by future turns. Query-agnostic methods are unreliable because they use static KV statistics to predict. The paper observes that, attention statistics of the query token during decoding, which is called Writer, cannot accurately reflect the attention dependence onto past memories, because it mixes attention of recent tokens, syntax checking and attention sinks in order to accurately predict next token. When completed tokens are re-fed (append the same tokens right after the completed tokens, called Readers), their attention concentrate on really relevant information. Based on this, the paper derives the MEMENTO design, a query-agnostic method that centers on re-reading based recall scoring.
In general, the paper gives in-depth analysis about availability and accessibility of KV eviction, identifies why decoding attention statistics fail to serve as retrieval signals and designs a rereading based recall scoring method that can adapt evolving context. Experiment results show that MEMENTO has higher task accuracy or pass rate than existing query agnostic methods on several agentic benchmarks.
Critical strengths
- The paper addresses KV retrieval problem in KV cache eviction. It gives a highly informative and detailed characterization of important challenge of accurately retaining relevant evicted KV entries without exactly knowing future queries. The paper first draws from concepts in cognitive psychology field, and maps them to the availability and accessibility of KV cache entries. It then identifies the root cause why query-dependent and query-agnostic ways are insufficient, especially the failure of static attention statistics of existing query-agnostic ways. Based on that, the paper gives their unique insight that rereading provides more accurate attention statistics onto the memories and can therefore serve the retrieval. These analysis give useful insight for query-agnostic KV eviction design.
- The paper also gives hardware-aware design to support efficient execution of their proposed scoring and computation mechanisms. The interleaved decoding executes rereading alongside the normal decoding to amortize additional computation overhead incurred.
Critical weaknesses
- Although the paper discusses query-dependent ways and consider them less effective compared to query-agnostic ways in multi-turn agentic scenarios, it should includes representative query-dependent methods, such as SnapKV, PyramicKV in the evaluations to justify and convince.
- The decoding speed experiment reports 3.17x over full KV cache, which mainly comes from the KV cache compression itself, rather than the interleaved decoding of MEMENTO. No other baselines are compared in decoding speed experiments. It's unclear how the system performs compared to existing works, and how interleaved decoding contributes to the overall throughput improvement of the system.
Questions for authors
- Could the authors report end-to-end results of representative query-dependent methods, which could make the paper more consistent and comprehensive?
- Could the authors report decoding throughput of other baselines, MEMENTO w/o interleaved decoding to justify the effectiveness of the system optimization? Also, Appendix C.6 reports results on H200 without considering host memory, I'm curious how retrieval would affect the performance under KV cache offloaded settings, i.e. evicted KV cache moved between GPU HBM and host memory.
Summary of Reviewers' Feedback and Our Rebuttals
Dear AC
We thank the AC and all three reviewers for their engagement. During the discussion, we added end-to-end comparisons with query-dependent and online baselines, a controlled study separating scoring from allocation, a revised analysis matching the implemented scoring rule, a comparison with learned eviction (DBTrimKV), and a full systems characterization. JNdo has no further questions and keeps their positive score. rnb8 and cU6h raised narrower follow-up questions, which we answered on 18 Nov 2026; we are awaiting their replies. We provide a summary below for the convenience of the AC.
TL;DR of Our Work
Multi-turn agentic inference needs a bounded KV cache, but what later turns need changes across requests and tool calls. Query-dependent eviction specializes the cache to the current query and can permanently discard evidence a later turn needs, while static query-agnostic statistics are unreliable predictors of future retrieval. We argue that useful memory requires both availability and accessibility.
We propose MEMENTO, a training-free, query-agnostic method that lets the model reread: it re-feeds each completed chunk, and the attention of the reread supplies retention scores. Rereading is interleaved with decoding in the same forward pass to amortize its cost, and a geometric analysis bounds local attention error on the regions that reread queries cover and, under explicit coverage and value-separation conditions, yields a lower expected future-query error than a query-specialized cache at the same budget. On SWE-bench Verified with Muse-Glimmer-30B, MEMENTO reaches 71.5% using about 23% of the cache, vs. 65.0% for the best baseline and 73.5% for the full cache.
Reviewers' Recognition
We are glad to report that:
Reviewers acknowledge the strong empirical results on realistic agent tasks:
Reviewers acknowledge the insight of our analysis:
Reviewers acknowledge our hardware-aware system design: