- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies continual KV-cache compaction under a bounded memory budget, where documents arrive sequentially and their source text is discarded after compaction. Building on attention matching and self-study queries, it proposes two update strategies: a modular memory that uses likelihood-based write and read routing, and a monolithic memory that restricts updates to an attention-selected subset of slots with value-change regularization. Experiments use two instruction-tuned models and five datasets, with the main accuracy results on LongHealth and QuALITY. Modular memory avoids the severe failure of fully recursive compaction, although its final gains over no memory are modest and model dependent. Controlled comparisons associate old-memory degradation with repeated recompaction without source text. The monolithic approach performs substantially better with Llama than with Qwen, but its specification and reported results require clarification.
Critical strengths
-
Useful controlled diagnosis of memory degradation. Oracle routing has limited impact on final accuracy, while compacting the original phase once to the same final size yields substantially higher accuracy than repeatedly recompacting its cached representation. This helps distinguish the observed degradation from routing errors and final capacity alone. The paper appropriately acknowledges that repeated recompaction and the absence of source text are not separately identified.
-
A relevant continual-memory evaluation protocol. Discarding source documents and evaluating earlier phases after subsequent updates addresses a meaningful limitation of one-shot context compression. Reporting acquisition, forgetting, answer failures, and model-dependent outcomes makes the study informative even where the proposed methods are unsuccessful.
Critical weaknesses
-
The incremental value of the adaptive modular update is not established against a simple bounded partitioned baseline. No merge occurs in the main five-phase experiments, so these results do not demonstrate the benefit of likelihood-based write gating. The appendix does test merging, but it is worse than keeping separate partitions at shorter horizons and approximately comparable at 24 documents. A baseline that reserves a fixed share of the budget per phase, compacts each phase once, and uses the same read router is missing. KV concat changes both the budget and the read procedure. This matters because rebalancing repeatedly damages older partitions, while final gains over no memory are small and statistically inconclusive for Qwen.
-
The monolithic method is insufficiently specified and its results are inconsistent. The precise attention-based slot score, the advertised usage penalty, and the key/value fitting procedure with frozen slots are not fully defined. Furthermore, for Llama with (B=8192) and (t=1024), Table 2 reports 33.8% and 35.5% on LongHealth and QuALITY, whereas Table 9 reports 35.5% and 38.4%. No explanation identifies different runs or aggregation rules. These discrepancies materially affect the comparison with in-context performance and prevent a reliable assessment of this contribution.
Questions for authors
-
Controlled evidence: For the reported five-phase protocol, how does modular memory compare with a fixed (B/5)-slot cache per phase, compacted once and read using the same likelihood router? This comparison would clarify the benefit of adaptive gating and rebalancing.
-
Results clarification: Why do Tables 2 and 9 report different accuracies for the same monolithic configurations? Please identify the runs, evaluation sets, and aggregation rules, and provide a consistent set of results and confidence intervals.
-
Method clarification: Please specify SELECTSLOTS, including any usage-count penalty, and the exact constrained key/value update: how are retained keys positioned, and how do the unchanged slots contribute to the least-squares objective? How does the method compare with full rewriting under otherwise identical initialization, regularization, and query-source settings?
3 comments
Thank you for the substantial revisions, which address several of my concerns regarding the missing baseline, implementation details, and inconsistent results. However, the new experiments show that fixed partitions perform comparably to gated update, while simpler merging rules are also competitive, leaving the incremental benefit of likelihood-based gating unclear; the monolithic method likewise shows no consistent advantage. Although the revised paper is considerably clearer and the empirical analysis remains valuable, my central concern about the novelty and effectiveness of the proposed update mechanisms persists, so I maintain my original rating.
Thank you for your reply and for the time you spent on the revised paper. Gated update uses likelihood in two places: the router, which chooses the partition to read for each question, and the merge gate, which decides whether a new phase joins an existing partition. Our new experiment shows that the likelihood router raises accuracy by 21 to 23% over a random partition, reaching the accuracy of oracle routing and of a text-retrieval router without storing the documents.
1. Likelihood router
To test the router directly, we let the model answer every question from the same saved final memory of gated update, with a different rule choosing the partition in each run. We keep the prompt and decoding the same in all runs. The table gives final accuracy in % with Qwen3-4B at 16,384 slots:
| Partition read per question | LongHealth | QuALITY |
|---|---|---|
| Likelihood router | 50.8 | 58.4 |
| Oracle (partition of the question's phase) | 51.0 | 53.2 |
| BM25 over the source documents of each phase | 49.3 | 56.4 |
| Random partition | 27.7 | 37.0 |
| Partition of the newest phase | 24.8 | 35.9 |
| No memory | 30.5 | 41.6 |
We find that the router is more accurate than a random partition by 23.1% on LongHealth and by 21.4% on QuALITY. When we give the model a random partition or the partition of the newest phase, its accuracy is lower than the no-memory baseline. Since the router and the oracle select the same partition for nearly all questions, we attribute their difference on QuALITY to the variation between two runs on the same memory.
To test whether the choice needs a likelihood score, we also route each question with BM25 over the source documents of each phase. BM25 is 1.5 to 2.0% less accurate than the likelihood router. Because our setting discards the documents after each phase, a text-retrieval router has no text to search. Storing the documents for retrieval would add memory beyond the fixed budget. The likelihood router reaches at least the accuracy of BM25 from the compacted memory alone.
2. Fixed partitions
Because fixed partitions use the same router as gated update, we conclude from their matching accuracy that rebalancing does not lower accuracy. In both methods, the model answers each question from the partition that the router selects among the partitions of the phases. Rebalancing lets gated update reach this accuracy without knowing the number of phases in advance, which fixed partitions require.
3. Merge gate
We agree with you that merging with the gate is no more accurate than merging with simpler rules. When we cap the number of partitions, gated update reaches similar accuracy (35.6 to 37.9%) whether the gate, the oldest partition, or a random partition receives a merged phase. In our main runs, we set the threshold of the gate above every likelihood gain, which gives every phase its own partition, as Section 4.1 states. Our claims therefore concern the setting with one partition per phase.
4. Monolithic update
We include the monolithic update to test whether a single memory, compacted in the same way, can retain the earlier phases as well as separate partitions. At 16,384 slots, gated update is more accurate than the monolithic update by 9.4 to 18.8% in three of the four settings of Table 2 and level with it in the fourth. We conclude that at 16,384 slots, separate partitions retain the earlier phases better than a single memory with the same compaction. The monolithic update thus shows that limiting the writes within one memory is not enough to retain the earlier phases, in line with the diagnosis of Section 3.
5. Contribution
Our contributions are the continual compaction setting, the diagnosis of why a recompacted memory loses the earlier phases, and gated update. In Section 3, we show that recompaction gives nearly all slots to the newest phase, as do StreamingLLM, SnapKV, and KVzip. We kept gated update simple on purpose, because one partition per phase and the likelihood router are enough to keep the earlier phases.
We believe these results address the concern about the benefit of likelihood-based routing. If they do, we would be grateful if you would consider updating your rating. If a concern remains, could you tell us which one, so that we can address it before the discussion ends or in the final version? Thank you again for your careful review.
Thank you for the questions on the fixed-partition baseline and the monolithic implementation, and for the useful feedback on establishing the gain over the no-memory baseline. We apologize for the late response; we underestimated the time needed for the new experiments. We have also revised the writing throughout the paper for clarity. Section and table numbers refer to the revised PDF.
1. Fixed-partition baseline
We find fixed partitions and gated update equally accurate within the sampling noise. Every phase receives a fifth of the budget, is compacted once and is never recompacted, with the same router. With Llama-3.2-3B on both datasets at both budgets, all four differences are within the sampling noise (Appendix G.3, Table 22):
We attribute the accuracy to the separate partitions and the one-partition read. A second experiment agrees: when a cap forces merging, the likelihood gate, an oldest-partition rule and a random rule reach 35.6, 37.9 and 35.6% (Appendix G.5, Table 24). Rebalancing loses no measurable accuracy and lets the memory run without knowing the number of phases in advance.
2. Rebalancing damage
Thank you for raising this concern; it prompted us to measure the rebalancing cost carefully. We find no measurable rebalancing cost in the revised runs. The submitted version reported a loss that grew with the number of recompactions. In the revised runs, phase 1 compacted once to each partition size is 2.8% more accurate than the rebalanced partition, pooled over the four settings and four sizes, within the sampling noise (Appendix G.3, Table 21). The fixed partitions above, never recompacted, are also no more accurate. The improvement comes from rerunning with one compaction block per document and a value prior (λ=0.01).
3. Gain over the no-memory baseline
Gated update exceeds the no-memory baseline at 16,384 slots and is level with it at 1,024 slots. At 16,384 slots, gated update (called modular update in our initial submission) exceeds the baseline by 8.5 to 20.3% across the four settings, with every interval above 0 (Appendix E). At 1,024 slots the gain is within the sampling noise in all four settings. Sections 5.2 and 7 state this limit. The differences are below:
4. Monolithic update: specification and consistency
The discrepancy between the two tables was an implementation error. We re-implemented the monolithic update to share the same compaction path as gated update and reran it at both budgets under the same pipeline as all other methods. Tables 2 and 13 give one consistent set of results, replacing both tables of the submission. With Qwen3-4B the monolithic update exceeds KV recursive on LongHealth and is lower on QuALITY. At 16,384 slots with Llama-3.2-3B, it reaches 36.5% on LongHealth and 30.7% on QuALITY, level in accuracy with KV recursive. At 1,024 slots with Llama-3.2-3B, it collapses after phases 4 and 5; the collapse correlates with the logical length of the memory exceeding the model's context window (Appendix H).
Section 4.2 and Algorithm 2 describe the current implementation exactly. Each layer scores every slot by the total attention the reference queries place on it, summed over queries and key-value heads. The t highest-scoring slots form the writable set S. Within each key-value head, the t keys with the largest σj among the keys of S and of the new documents fill S, while every slot outside S keeps its key and value. The values of S are fitted so that ASVS matches YT−AFVF, with AF the attention on the frozen slots and VF their values. λ=0.01. The submission's introduction mentioned a usage-count penalty; the method has none, and the revised text no longer mentions it.
With t=B the monolithic update is KV recursive, so Table 2 holds this comparison. At t/B=1/4 the monolithic update exceeds KV recursive on LongHealth but falls below it on QuALITY. Limiting the writable set does not help because the attention-based selection directs each new phase to the slots of the previous phase (Appendix H).
We are happy to address any additional questions.