- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Existing memory compression approaches either retain a subset of cached states or use a learnable compressor that processes documents into latent representations to reduce KV cache accumulation and inference cost. This paper argues that these are not enough for tight memory budgets. It proposes SD-Cache (State-guided and Dynamically updated Cache) and advertises it as a query-agnostic framework used for a frozen LLM. Specifically, the SD-Cache framework learns to process layer-wise states into a fixed-size memory, which can be processed quickly without context-specific optimization. Experiments on RAG-based question answering and long-context benchmarks demonstrate that the proposed framework can reduce the KV cache and computational cost while retaining competitive accuracy.
Critical strengths
- If the experimental results are believable, the proposed SD cache method has a larger compression rate and better performance on the two datasets.
- The presentation of this paper is reasonably acceptable.
Critical weaknesses
- First of all, I want to challenge the motivation of the fixed-size KV working memory. There are many different conditions and situations. For example, we may have different previous messages and different recalled docs. Will we waste context using the fixed working memory? Or sometimes it is not large enough for complex tasks. And of course, will the fixed memory dilute or condense information density, which may cause instability of the system or model performance? I didn't understand the motivation for asking that question about using a fixed-size working memory for consolidation. A growing KV cache and history is not the reason.
- The experiments are not deep enough to prove the effectiveness of the proposed cache method. The page lines in Table 1 and Table 2 are different and have no overlap. I think at least we need to have some baselines tested for each task and evaluation type.
- The experiment's model diversity is not enough; only Llama has been tested. Mistral has been tested a little bit as well. It is necessary to prove the approach is effective on diverse model series. Only one or two may be coincidence and because of the instability of model performance scores in the benchmark.
- Also, I suggest testing on larger models a bit to prove the scaling ability of the approach. I understand there may be computational limits, but at least a 20B-level model should be included in this kind of test.
Questions for authors
- What's the reason for introducing fixed-size KV working memory?
- Why do we specifically mention that the method is targeting query-agnostic? I ask this question for two reasons. Firstly, it is intuitive that the record document should be related to the query, and compression can perform better with the query. And with the query, the cost would not increase a lot.
- The cache design is mainly for RAG situations. Could we expand it to normal agentic scenarios? Currently, we usually don't use the RAG concept specifically. The search and reasoning pipeline has been integrated into agentic scenarios.
- Is the result in Figure 3 reproducible? For example, if we use another model or another layers of representation, could we see the same pattern and get the same conclusion?
8 comments
W2: Overlapping baselines across benchmarks
W2: Table 1 and Table 2 are different and have no overlap
Table 1 (RA-QA) evaluates whether SD-Cache preserves answer quality in practical document-grounded question answering, while Table 2 (RULER) provides controlled tests of information retrieval and reasoning over compressed long contexts. The baselines are taken directly from the official benchmarks for the two settings, so they may not overlap.
To provide comparisons across both benchmarks, we additionally evaluate publicly available KV cache eviction methods including SnapKV, PyramidKV, and KeyDiff on NQ from the RA-QA benchmark (Fig. 4(a)). These methods are also included in our RULER evaluation. SD-Cache outperforms these eviction baselines across all four evaluated KV budgets on NQ. We additionally evaluate PISCO on RULER-4K using Llama-3.1-8B. PISCO achieves average accuracies of 78.21% and 64.47% at 75% and 90% KV cache reduction, respectively, compared with 79.52% and 66.11% for SD-Cache under the corresponding settings.
W3 & W4: Experiment diversity
W3: The experiment's model diversity is not enough
Our evaluation already covers three model families—Qwen, Mistral, and Llama—and two sizes within the Qwen family (Qwen3-4B and Qwen3-8B). RULER results include Llama-3.1-8B, Qwen3-8B, and Mistral-7B (Table 2 and Appendix Tables 4-5), while Section 4.3 evaluates Qwen3-4B. SD-Cache and the corresponding baselines are evaluated on the same backbones for direct comparison.
W4: Suggest testing on at least a 20B-level model
Unfortunately, training SD-Cache with 20B+ backbones exceeds the memory capacity of our available GPU hardware under the current configuration, preventing us from completing a comparison with baseline methods at this scale. Our evaluation covers 4B-8B backbones, with SD-Cache and the corresponding baseline methods evaluated on the same models for direct comparison.
W1 & Q1: Motivation for budget-constrained KV memory
W1 & Q1: Clarify the motivation of the fixed-size KV working memory. Will we waste context using the fixed working memory?
We would like to clarify that SD-Cache is a method for KV cache compression, an active research area [1,2]. It can optionally operate under a fixed, configurable budget for the retained KV cache. Under this deployment policy, compression would be triggered only when the incoming context would cause the full KV cache to exceed the budget, potentially causing out-of-memory (OOM) errors or requiring context truncation. Contexts that fit within the budget would remain uncompressed. Like other KV cache compression methods [3,4], we aim to condense useful information from long contexts and preserve its utility for downstream queries, rather than simply truncate the context. We have replaced “fixed-size KV working memory” with “budget-constrained KV working memory” in the manuscript to clarify this distinction.
We summarise the main motivations for budget-constrained KV working memory [1,2,3,4]:
We acknowledge that tighter memory budgets can reduce downstream accuracy. Our RAG QA results demonstrate substantial KV memory savings while retaining useful downstream performance. At 128× compression, SD-Cache achieves 65.46% average answer-matching accuracy across five datasets, compared with 68.67% at 16× compression—a decrease of 3.21 percentage points (Table 1). Figure 4(a) further characterises this memory-accuracy trade-off across four KV budgets on NQ.
[1] Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P. and Chen, D., 2024. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37, pp.22947-22970.
[2] Xiao, G., Tian, Y., Chen, B., Han, S. and Lewis, M., 2024, May. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (Vol. 2024, pp. 21875-21895).
[3] Eyuboglu, S., Ehrlich, R., Arora, S., Guha, N., Zinsley, D., Liu, E., Rudra, A., Zou, J.Y., Mirhoseini, A. and Re, C., 2026, April. Cartridges: Lightweight and general-purpose long context representations via self-study. In International Conference on Learning Representations (Vol. 2026, pp. 42642-42687).
[4] Yang, X., Gu, H., Xu, B., Li, L., Liu, B., Liu, J., Zhu, Q., Guo, Y. and Han, S., 2026. IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference. arXiv preprint arXiv:2605.25475.