acceptodds
Under review as a conference paper at ICLR 2027

SD-Cache: Learnable Memory Consolidation with State-Guided Dynamic Caching

Abstract

Large language models (LLMs) are increasingly deployed as persistent services that must reason over large volumes of external knowledge. Retrieval-augmented generation (RAG) grounds model responses in relevant documents, but the retrieved content must still be encoded and stored in the key-value (KV) cache. As this content accumulates, the cache grows linearly, increasing memory consumption and inference cost. Existing memory-compression methods address this problem either by retaining a subset of cached states or by learning context compressors that map documents to compact latent representations. Despite these advances, consolidating information under tight memory budgets while preserving its usefulness for future queries remains challenging. We investigate how the model’s native KV states can be selectively consolidated into a fixed-size working memory without access to future queries. To this end, we introduce SD-Cache (State-guided and Dynamically updated Cache), a query-agnostic KV-memory consolidation framework that operates alongside a frozen generator LLM. SD-Cache learns to consolidate layer-wise KV states into fixed-size memory through selective anchor-based pooling and state-guided refinement, processing new contexts in a single forward pass without context-specific optimisation. Experiments on retrieval-augmented question answering and synthetic long-context benchmarks show that SD-Cache maintains competitive downstream performance under constrained memory budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RAG Context Compression, Memory Consolidation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cmPbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Existing memory compression approaches either retain a subset of cached states or use a learnable compressor that processes documents into latent representations to reduce KV cache accumulation and inference cost. This paper argues that these are not enough for tight memory budgets. It proposes SD-Cache (State-guided and Dynamically updated Cache) and advertises it as a query-agnostic framework used for a frozen LLM. Specifically, the SD-Cache framework learns to process layer-wise states into a fixed-size memory, which can be processed quickly without context-specific optimization. Experiments on RAG-based question answering and long-context benchmarks demonstrate that the proposed framework can reduce the KV cache and computational cost while retaining competitive accuracy.

Critical strengths
  • If the experimental results are believable, the proposed SD cache method has a larger compression rate and better performance on the two datasets.
  • The presentation of this paper is reasonably acceptable.
Critical weaknesses
  • First of all, I want to challenge the motivation of the fixed-size KV working memory. There are many different conditions and situations. For example, we may have different previous messages and different recalled docs. Will we waste context using the fixed working memory? Or sometimes it is not large enough for complex tasks. And of course, will the fixed memory dilute or condense information density, which may cause instability of the system or model performance? I didn't understand the motivation for asking that question about using a fixed-size working memory for consolidation. A growing KV cache and history is not the reason.
  • The experiments are not deep enough to prove the effectiveness of the proposed cache method. The page lines in Table 1 and Table 2 are different and have no overlap. I think at least we need to have some baselines tested for each task and evaluation type.
  • The experiment's model diversity is not enough; only Llama has been tested. Mistral has been tested a little bit as well. It is necessary to prove the approach is effective on diverse model series. Only one or two may be coincidence and because of the instability of model performance scores in the benchmark.
  • Also, I suggest testing on larger models a bit to prove the scaling ability of the approach. I understand there may be computational limits, but at least a 20B-level model should be included in this kind of test.
Questions for authors
  • What's the reason for introducing fixed-size KV working memory?
  • Why do we specifically mention that the method is targeting query-agnostic? I ask this question for two reasons. Firstly, it is intuitive that the record document should be related to the query, and compression can perform better with the query. And with the query, the cost would not increase a lot.
  • The cache design is mainly for RAG situations. Could we expand it to normal agentic scenarios? Currently, we usually don't use the RAG concept specifically. The search and reasoning pipeline has been integrated into agentic scenarios.
  • Is the result in Figure 3 reproducible? For example, if we use another model or another layers of representation, could we see the same pattern and get the same conclusion?
8 comments
AuthorsOpenReview ↗

W1 & Q1: Motivation for budget-constrained KV memory

W1 & Q1: Clarify the motivation of the fixed-size KV working memory. Will we waste context using the fixed working memory?


We would like to clarify that SD-Cache is a method for KV cache compression, an active research area [1,2]. It can optionally operate under a fixed, configurable budget for the retained KV cache. Under this deployment policy, compression would be triggered only when the incoming context would cause the full KV cache to exceed the budget, potentially causing out-of-memory (OOM) errors or requiring context truncation. Contexts that fit within the budget would remain uncompressed. Like other KV cache compression methods [3,4], we aim to condense useful information from long contexts and preserve its utility for downstream queries, rather than simply truncate the context. We have replaced “fixed-size KV working memory” with “budget-constrained KV working memory” in the manuscript to clarify this distinction.

We summarise the main motivations for budget-constrained KV working memory [1,2,3,4]:

  • Memory capacity: Support long contexts within a configurable budget for the retained KV cache.
  • Information preservation: Consolidate useful contextual information to mitigate the information loss caused by truncation or eviction.
  • Computational efficiency: Reduce attention computation and memory traffic during decoding while keeping compaction overhead practical.
  • Reuse across queries: Construct a query-agnostic compressed cache once and reuse it across queries over the same corpus, amortising compression costs.

We acknowledge that tighter memory budgets can reduce downstream accuracy. Our RAG QA results demonstrate substantial KV memory savings while retaining useful downstream performance. At 128× compression, SD-Cache achieves 65.46% average answer-matching accuracy across five datasets, compared with 68.67% at 16× compression—a decrease of 3.21 percentage points (Table 1). Figure 4(a) further characterises this memory-accuracy trade-off across four KV budgets on NQ.

[1] Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P. and Chen, D., 2024. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37, pp.22947-22970.

[2] Xiao, G., Tian, Y., Chen, B., Han, S. and Lewis, M., 2024, May. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (Vol. 2024, pp. 21875-21895).

[3] Eyuboglu, S., Ehrlich, R., Arora, S., Guha, N., Zinsley, D., Liu, E., Rudra, A., Zou, J.Y., Mirhoseini, A. and Re, C., 2026, April. Cartridges: Lightweight and general-purpose long context representations via self-study. In International Conference on Learning Representations (Vol. 2026, pp. 42642-42687).

[4] Yang, X., Gu, H., Xu, B., Li, L., Liu, B., Liu, J., Zhu, Q., Guo, Y. and Han, S., 2026. IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference. arXiv preprint arXiv:2605.25475.

AuthorsOpenReview ↗

W2: Overlapping baselines across benchmarks

W2: Table 1 and Table 2 are different and have no overlap


Table 1 (RA-QA) evaluates whether SD-Cache preserves answer quality in practical document-grounded question answering, while Table 2 (RULER) provides controlled tests of information retrieval and reasoning over compressed long contexts. The baselines are taken directly from the official benchmarks for the two settings, so they may not overlap.

To provide comparisons across both benchmarks, we additionally evaluate publicly available KV cache eviction methods including SnapKV, PyramidKV, and KeyDiff on NQ from the RA-QA benchmark (Fig. 4(a)). These methods are also included in our RULER evaluation. SD-Cache outperforms these eviction baselines across all four evaluated KV budgets on NQ. We additionally evaluate PISCO on RULER-4K using Llama-3.1-8B. PISCO achieves average accuracies of 78.21% and 64.47% at 75% and 90% KV cache reduction, respectively, compared with 79.52% and 66.11% for SD-Cache under the corresponding settings.

AuthorsOpenReview ↗

W3 & W4: Experiment diversity

W3: The experiment's model diversity is not enough


Our evaluation already covers three model families—Qwen, Mistral, and Llama—and two sizes within the Qwen family (Qwen3-4B and Qwen3-8B). RULER results include Llama-3.1-8B, Qwen3-8B, and Mistral-7B (Table 2 and Appendix Tables 4-5), while Section 4.3 evaluates Qwen3-4B. SD-Cache and the corresponding baselines are evaluated on the same backbones for direct comparison.


W4: Suggest testing on at least a 20B-level model


Unfortunately, training SD-Cache with 20B+ backbones exceeds the memory capacity of our available GPU hardware under the current configuration, preventing us from completing a comparison with baseline methods at this scale. Our evaluation covers 4B-8B backbones, with SD-Cache and the corresponding baseline methods evaluated on the same models for direct comparison.

Reviewer_YvKBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SD-Cache, a learnable and query-agnostic KV cache consolidation framework designed to maintain a fixed-size working memory for large language models. Unlike conventional KV eviction methods that discard selected token states, SD-Cache aims to consolidate information from the original KV representations while preserving their usefulness for future queries. The proposed framework consists of two main components. First, the Anchor-Pooling Module (APM) initializes memory slots from the original KV states and progressively integrates incoming information through novelty-guided attention pooling and gated updates. Second, the State-Guided Correction (SGC) module constructs an associative state from the original KV stream and uses it to refine the compressed key and value representations. The consolidation network is trained to reproduce the outputs of the full-KV model, while the underlying generator remains frozen. The authors evaluate SD-Cache on retrieval-augmented question answering benchmarks and RULER long-context tasks across multiple LLM backbones. Experimental results show that SD-Cache achieves competitive or superior accuracy compared with existing KV compression methods, particularly under aggressive memory constraints. For example, on RULER-4K with Llama-3.1-8B, SD-Cache achieves 53.78% average accuracy at a 98.44% KV cache reduction ratio, substantially outperforming the strongest reported baseline at that setting.

Critical strengths

Strength 1: Novel and well-motivated KV memory consolidation framework. The paper introduces a learnable KV memory consolidation framework that directly operates on the native KV states of a frozen LLM. Unlike conventional KV eviction methods that discard selected token states, SD-Cache integrates incoming information into a fixed-size memory through two complementary mechanisms: novelty-guided anchor pooling and state-guided correction. The former selectively incorporates new information through attention-based pooling and gated updates, while the latter captures key-value associations from the original KV stream to refine the compressed representations. This design provides an interesting alternative to conventional token-selection-based KV cache compression.

Strength 2: Strong performance under aggressive KV cache compression. The experimental results demonstrate substantial performance advantages under highly constrained KV cache budgets. On the RULER-4K benchmark with Llama-3.1-8B, SD-Cache achieves 53.78% average accuracy at a 98.44% KV cache reduction ratio, outperforming the strongest reported baseline at this setting, KeyDiff, by 37.20 percentage points. Notably, its accuracy decreases by only 1.66 percentage points when increasing the cache reduction ratio from 93.75% to 98.44%. These results provide compelling evidence that learnable KV consolidation can preserve useful information under extremely limited memory budgets.

Strength 3: Meaningful ablation studies and insights into the correction mechanism. The paper provides useful ablation studies to validate the contributions of its major components. On Natural Questions, anchor pooling improves accuracy from 61.72% to 64.44%, while state-guided correction further increases it to 65.74%. Moreover, the analysis of correction targets and directions provides additional insights into the effectiveness of SGC. In particular, the results suggest that the performance improvement primarily comes from adjusting the directions of compressed keys rather than merely rescaling them. These findings strengthen the empirical motivation for the proposed correction mechanism.

Critical weaknesses

Weakness 1: Limited robustness under repeated memory updates. The paper positions SD-Cache as a dynamic memory consolidation framework for persistent LLM services. However, the length-generalization experiments in Figure 6 reveal substantial performance degradation under repeated memory updates. In particular, recurrent rolling achieves only 9.9% and 1.7% accuracy after three and four updates, respectively. Although skip-connected rolling alleviates this degradation, its accuracy remains limited to 26.9% after four updates. These results raise concerns about the robustness of SD-Cache in the persistent-memory scenarios that motivate the proposed framework. Moreover, the current experiments vary context length and update count simultaneously, making it difficult to isolate the effects of repeated consolidation from those of longer input sequences. The authors should provide a controlled evaluation of accuracy versus the number of memory updates, including settings with fixed total context length and varying chunk sizes. Comparisons against single-pass consolidation of the complete context would further clarify whether performance degradation originates from repeated memory updates.

Weakness 2: Insufficient end-to-end efficiency evaluation. Although SD-Cache substantially reduces KV cache storage, its consolidation process introduces additional computation through novelty estimation, Perceiver-based pooling, and state-guided correction. The analytical results in Table 3 show that SD-Cache requires higher total FLOPs than Full KV and several compression baselines in the evaluated settings. For example, on Mistral-7B, SD-Cache requires 13.95 TFLOPs compared with 11.82 TFLOPs for Full KV. The paper primarily relies on analytical FLOPs estimates rather than measured GPU execution latency. However, FLOPs alone cannot capture the runtime overhead of the additional consolidation modules, particularly the recurrent associative-state updates and attention-based pooling operations. To better establish the practical efficiency of SD-Cache, the authors should provide end-to-end latency, prefill and consolidation latency breakdowns, decoding throughput, and peak GPU memory measurements. An amortization analysis across different numbers of queries sharing the same compressed context would also clarify when the additional consolidation overhead is offset by the benefits of KV cache compression.

Weakness 3: Limited evaluation of generalization to unseen query distributions. Although SD-Cache constructs compressed KV memories without accessing future queries, its consolidation mechanism is trained using supervision from specific question-answering distributions. Consequently, the learned compression policy may prioritize information that is useful for the training tasks while discarding information required by substantially different future queries. While the paper evaluates multiple QA datasets and synthetic long-context tasks, the current experiments do not sufficiently isolate generalization to unseen query types or substantially shifted query distributions. This is particularly relevant to the paper's query-agnostic motivation, where the information needs of future users are unknown during memory construction. The authors should consider cross-task generalization experiments in which the consolidation model is trained on a subset of query types and evaluated on held-out task categories, such as transferring from factual retrieval to multi-hop reasoning or frequency-based aggregation. Such evaluations would provide stronger evidence that SD-Cache preserves broadly useful information rather than information primarily aligned with its training distribution.

Weakness 4: Limited evaluation in realistic persistent-memory scenarios. The paper motivates SD-Cache through persistent LLM services that continuously process and consolidate external knowledge. However, the evaluation primarily focuses on retrieval-augmented QA and relatively short synthetic long-context benchmarks, with the main RULER experiments conducted at a context length of 4K tokens. These benchmarks demonstrate compression effectiveness but do not fully characterize the challenges of persistent memory, such as long-term information retention, repeated knowledge updates, and interference between previously stored and newly introduced information. The authors should consider evaluating SD-Cache on longer streaming contexts with many successive memory updates. In particular, measuring retrieval accuracy as a function of the age of stored information would help characterize whether earlier knowledge is progressively forgotten. Additional experiments involving redundant or conflicting information would further clarify the suitability of the proposed framework for persistent-memory applications.

Questions for authors

See weakness

5 comments
AuthorsOpenReview ↗

W1: Robustness under repeated memory updates

W1: Limited robustness under repeated memory updates and insufficient controlled evaluation of accuracy versus updates


We evaluate KILT-NQ questions using Mistral-7B, keeping each example’s context, question, token order, and 64-entry memory budget fixed. SD-Cache initializes its slots from the first 64 tokens and divides the remaining context into 1, 2, or 4 chunks, each updating the same slots. The average total context length is 831 tokens; one update provides the single-pass reference. Accuracy declines with more updates even when total context length and memory capacity remain fixed, indicating sensitivity to the update schedule. We have discussed this limitation and identify improving robustness to repeated updates as future work in Section 5.

Method1 update2 updates4 updates
SD-Cache62.754.247.5
PISCO (recurrent adaptation)61.052.244.1
SnapKV (reselection)42.447.545.8
AuthorsOpenReview ↗

W2: End-to-end efficiency and query reuse

W2: Insufficient end-to-end efficiency evaluation


We evaluate Mistral-7B on one RTX 4090 using 1,024 context tokens and 64 generated tokens per query across three record-lookup contexts with three repetitions each. We report medians of per-context repeat medians for latency and maxima for peak allocated memory.

QQ denotes sequential queries sharing one context. Synchronized E2E timing includes context/query prefill, consolidation, and decoding, excluding retrieval, tokenization, model loading, and detokenization. All methods reuse the context cache except query-aware SnapKV, which rebuilds it per query.

MethodE2E Q=1Q=1 (s)E2E Q=8Q=8 (s)E2E Q=32Q=32 (s)Peak GPU memory (GiB)Context KV (MiB)
Full KV0.9236.75726.77713.765130.125
SD-Cache0.9956.78226.62414.23810.125
PISCO (unmerged LoRA)1.1308.14332.15214.19811.125
SnapKV (query-aware)0.9147.27929.08813.63010.125

SD-Cache / Full KV take 93.0 / 88.4 ms for prefill/preparation and 75.0 / 0 ms for consolidation, with decoding throughput of 77.6 / 76.9 tokens/s. At Q=32Q=32, SD-Cache uses 92.2% less context-KV storage with comparable E2E latency to Full KV, while reducing E2E latency by 17.2% versus PISCO and 8.5% versus query-aware SnapKV. Its peak GPU memory is 3.4% higher than Full KV.

AuthorsOpenReview ↗

W3: Generalization to unseen query distributions

W3: Limited evaluation of generalization to unseen query distributions


Cross-task generalisation. We evaluate SD-Cache on RULER with 4K-token contexts using a consolidation model trained solely on retrieval-augmented QA data. SD-Cache achieves 28.00% accuracy on the Needle-in-a-Haystack subsets, compared with PISCO’s 26.47%, and 80.53% on the frequent-word task, compared with PISCO’s 79.15%.

Multi-hop reasoning. Table 1 already reports results on HotpotQA, where SD-Cache achieves comparable performance to PISCO, scoring only 0.58 and 1.25 percentage points lower at 16×16\times and 128×128\times compression, respectively.

Reviewer_xeRzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SD-Cache is a model-native KV cache consolidation with a fixed-size memory constraints. Unlike RAG which is query dependent and memory compression metods which has increasing memory, SD-cache's consolidation does not require knowing the query inputs and has constant memory size complexity. The core idea is to identify the novelty of new KV cache from the old KV cache and generate the change of KV cache using a SSM like model to get the final KV. Compared with prior KV eviction and RAG methods, SD-cache can achieve better accuracy on retrieval-based QA tasks.

Critical strengths

None

Critical weaknesses
  1. The idea of having fixed-size memory is fundamentally not scalable with the amount of incoming data. It requires the constraint that information does not grow after a certain point, which is not true for many complex reasoning tasks.

  2. The method is overly complex but lack of comparison with a much simpler approach in prior works, with only a small modification. For example, one can have a multi-state SSM with the novelty evaluation similar to B'MOJO at NeurIPS'24

Questions for authors
  1. To achieve fixed memory size and adapt to long context inference, one can simply construct a state-space model with the state larger than a single vector (the state size can be the hyperparameter of the model). Using methods like B'MOJO with a trained lightweight predictor to decide whether the current state is sufficient to infer the next token can also serve as a novelty metric. How does SD-cache compare with such simpler methods? Having multiple attention+FFN layers introduce extra overhead, which is also shown in your latency results. It is important to perform such ablation study and understand whether the method is necessary or over-engineering.

  2. Can you further elaborate Figure 2? It's relatively unclear how the consolidation actually works and what are the intuition behind. Several symbols, such as M, C, and different K, V, are not clearly explained in the figure title. What does Z represents? How does K tranform into Q? What weights did the Attention and FFN used in APM, original model or new parameters?

  3. I can see that Figure 3 is trying to illustrate that the pre-ROPE keys of a model are highly clustered on PCA, which inferences that you only need a few states to represent the whole context. However, there are several problems: a. PCA does not include high-dimensional information. It is possible that vectors that are close on PCA are actually very different when computing the l2 distance between them. b. even when two vectors are close in l2 distance, the actual attention score magnitude difference is also related to the query. In terms of fitting the attention score of full KV with fewer KV, I'm wondering if the evidence given is sufficient.

5 comments
AuthorsOpenReview ↗

W1 - Fixed memory budgets and scalability

W1: The idea of having fixed-size memory is fundamentally not scalable with the amount of incoming data. It requires the constraint that information does not grow after a certain point, which is not true for many complex reasoning tasks.


Our method belongs to the family of KV cache compression approaches [1,2]. KV cache compression is an active research area in efficient LLM inference with significant industry interest, due to direct implications for serving costs, user concurrency, and achievable context length under a given memory budget.

Specifically, we study how to retain useful contextual information within a fixed KV budget before the downstream question is specified. The fixed budget reflects system resource constraints, not an assumption that incoming information stops growing. Our goal is to support longer contexts when storing additional KV states would cause out-of-memory (OOM) errors or require context truncation, while characterising the resulting capacity-quality trade-off. This objective aligns with the active research area of KV cache compression [1].

[1] Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P. and Chen, D., 2024. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37, pp.22947-22970.

[2] Devoto, A., Jeblick, M. and Jégou, S., 2025. Expected attention: Kv cache compression by estimating attention from future queries distribution. arXiv preprint arXiv:2510.00636.

AuthorsOpenReview ↗

W2: Comparison with B’MOJO and related methods

W2: The method is overly complex but lack of comparison with B'MOJO, with only a small modification. For example, one can have a multi-state SSM with the novelty evaluation similar to B'MOJO at NeurIPS'24.


The two methods differ in design purpose: B’MOJO is a hybrid language model architecture with expandable memory, whereas SD-Cache is a KV cache compressor. SD-Cache operates on a pretrained, frozen Transformer and produces compact KV representations directly usable by its original decoder. We compare SD-Cache with the published B’MOJO-F checkpoint on 1,000 RULER QA examples (500 each for SQuAD and HotpotQA). We adjust B’MOJO-F’s attention window to match SD-Cache’s average retained-cache size. At the same average cache footprint of 437.1 MiB, SD-Cache achieves higher average QA accuracy than B’MOJO-F (70.6% vs. 68.2%).

MethodTotal paramsExtra over Qwen3-8BMean cache (MiB)SQuADHotpotQAAverage
B’MOJO-F, default configuration9.26 B1.06 B702.584.461.873.1
Qwen3-8B Full KV8.19 B—419.881.659.070.3
SD-Cache, 3,072 slots8.33 B0.14 B437.179.062.270.6
B’MOJO-F, adjusted window9.26 B1.06 B437.184.052.468.2
B’MOJO-F, SSM only9.26 B1.06 B80.423.222.022.6
SD-Cache, 410 slots8.33 B0.14 B62.735.437.036.2
SD-Cache, 64 slots8.33 B0.14 B14.132.230.231.2

Our original evaluation also includes IndexMem in Tables 2, 4, and 5, which combines learned KV-importance scoring with a fixed-size, online-updated latent memory for evicted tokens. SD-Cache achieves higher accuracy at the shared nonzero KV-cache reduction ratios.

AuthorsOpenReview ↗

Q1: Necessity and overhead of attention–FFN layers

Q1: Necessity and overhead of multiple attention-FFN layers


We adopt the established Perceiver-style attention-FFN architecture for learned pooling. Under the same backbone and KV budget, Section 4.3 reports NQ Match scores of 61.72% (soft-token baseline), 64.44% (APM), and 65.74% (APM+SGC) using Qwen3-4B-Instruct-2507 at 16×16\times compression. The additional APM computation occurs during consolidation and can be amortised across queries reusing the cache. We leave alternative compressor architectures design to future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.