acceptodds
Under review as a conference paper at ICLR 2027

Mixture of Sub-Memories: Selective Updates for Continual KV-Cache Compaction

Abstract

Large language models can acquire knowledge from long contexts, but this approach becomes increasingly impractical in continual settings, where context accumulates over time: prefill cost grows quadratically with sequence length, and the KV cache grows linearly. Cartridges address this by distilling a long context into a small, fixed-size trainable KV cache. In deployment, however, knowledge is not static: new corpora arrive continually, and a memory with a fixed budget must be able to absorb each one while retaining what it already holds. We introduce two complementary approaches that extend training-free KV compaction updates built on attention matching that limit what each update overwrites: (1) a modular update, which routes each incoming corpus to one of several sub-memories under a likelihood gate and read one sub-memory per question, and (2) a monolithic update, which keeps a single memory but rewrites only a small, attention-selected subset of its slots. On LongHealth and QuALITY with two instruction-tuned models Llama-3.2-3B and Qwen3-4B, the modular update remains above the no-memory baseline at 16,384 slots and is at least as accurate as a text memory of equal size. Its gains on old phases shrinks with every update, and a controlled comparison attributes the loss to repeated recompaction without the source text rather than routing errors or memory size. The monolithic update improves over phases with Llama-3.2-3B and, at 8,192 slots, approaches in-context accuracy on LongHealth and reaches it on QuALITY, but collapses with Qwen3-4B. Our results show that a fixed-size KV memory can survive continual updates when writes are confined, and identify repeated recompaction as the main source of forgetting.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, KV cache compression, memory gating, sparse updates, long-context language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XtW4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continual KV-cache compaction under a bounded memory budget, where documents arrive sequentially and their source text is discarded after compaction. Building on attention matching and self-study queries, it proposes two update strategies: a modular memory that uses likelihood-based write and read routing, and a monolithic memory that restricts updates to an attention-selected subset of slots with value-change regularization. Experiments use two instruction-tuned models and five datasets, with the main accuracy results on LongHealth and QuALITY. Modular memory avoids the severe failure of fully recursive compaction, although its final gains over no memory are modest and model dependent. Controlled comparisons associate old-memory degradation with repeated recompaction without source text. The monolithic approach performs substantially better with Llama than with Qwen, but its specification and reported results require clarification.

Critical strengths
  1. Useful controlled diagnosis of memory degradation. Oracle routing has limited impact on final accuracy, while compacting the original phase once to the same final size yields substantially higher accuracy than repeatedly recompacting its cached representation. This helps distinguish the observed degradation from routing errors and final capacity alone. The paper appropriately acknowledges that repeated recompaction and the absence of source text are not separately identified.

  2. A relevant continual-memory evaluation protocol. Discarding source documents and evaluating earlier phases after subsequent updates addresses a meaningful limitation of one-shot context compression. Reporting acquisition, forgetting, answer failures, and model-dependent outcomes makes the study informative even where the proposed methods are unsuccessful.

Critical weaknesses
  1. The incremental value of the adaptive modular update is not established against a simple bounded partitioned baseline. No merge occurs in the main five-phase experiments, so these results do not demonstrate the benefit of likelihood-based write gating. The appendix does test merging, but it is worse than keeping separate partitions at shorter horizons and approximately comparable at 24 documents. A baseline that reserves a fixed share of the budget per phase, compacts each phase once, and uses the same read router is missing. KV concat changes both the budget and the read procedure. This matters because rebalancing repeatedly damages older partitions, while final gains over no memory are small and statistically inconclusive for Qwen.

  2. The monolithic method is insufficiently specified and its results are inconsistent. The precise attention-based slot score, the advertised usage penalty, and the key/value fitting procedure with frozen slots are not fully defined. Furthermore, for Llama with (B=8192) and (t=1024), Table 2 reports 33.8% and 35.5% on LongHealth and QuALITY, whereas Table 9 reports 35.5% and 38.4%. No explanation identifies different runs or aggregation rules. These discrepancies materially affect the comparison with in-context performance and prevent a reliable assessment of this contribution.

Questions for authors
  1. Controlled evidence: For the reported five-phase protocol, how does modular memory compare with a fixed (B/5)-slot cache per phase, compacted once and read using the same likelihood router? This comparison would clarify the benefit of adaptive gating and rebalancing.

  2. Results clarification: Why do Tables 2 and 9 report different accuracies for the same monolithic configurations? Please identify the runs, evaluation sets, and aggregation rules, and provide a consistent set of results and confidence intervals.

  3. Method clarification: Please specify SELECTSLOTS, including any usage-count penalty, and the exact constrained key/value update: how are retained keys positioned, and how do the unchanged slots contribute to the least-squares objective? How does the method compare with full rewriting under otherwise identical initialization, regularization, and query-source settings?

3 comments

Thank you for the questions on the fixed-partition baseline and the monolithic implementation, and for the useful feedback on establishing the gain over the no-memory baseline. We apologize for the late response; we underestimated the time needed for the new experiments. We have also revised the writing throughout the paper for clarity. Section and table numbers refer to the revised PDF.

1. Fixed-partition baseline

We find fixed partitions and gated update equally accurate within the sampling noise. Every phase receives a fifth of the budget, is compacted once and is never recompacted, with the same router. With Llama-3.2-3B on both datasets at both budgets, all four differences are within the sampling noise (Appendix G.3, Table 22):

SettingFixed partitionsGated updateDifference
LongHealth, 16,384 slots39.038.8+0.3
LongHealth, 1,024 slots23.225.5−2.2
QuALITY, 16,384 slots42.641.3+1.3
QuALITY, 1,024 slots34.934.1+0.8

We attribute the accuracy to the separate partitions and the one-partition read. A second experiment agrees: when a cap forces merging, the likelihood gate, an oldest-partition rule and a random rule reach 35.6, 37.9 and 35.6% (Appendix G.5, Table 24). Rebalancing loses no measurable accuracy and lets the memory run without knowing the number of phases in advance.

2. Rebalancing damage

Thank you for raising this concern; it prompted us to measure the rebalancing cost carefully. We find no measurable rebalancing cost in the revised runs. The submitted version reported a loss that grew with the number of recompactions. In the revised runs, phase 1 compacted once to each partition size is 2.8% more accurate than the rebalanced partition, pooled over the four settings and four sizes, within the sampling noise (Appendix G.3, Table 21). The fixed partitions above, never recompacted, are also no more accurate. The improvement comes from rerunning with one compaction block per document and a value prior (λ=0.01\lambda = 0.01).

3. Gain over the no-memory baseline

Gated update exceeds the no-memory baseline at 16,384 slots and is level with it at 1,024 slots. At 16,384 slots, gated update (called modular update in our initial submission) exceeds the baseline by 8.5 to 20.3% across the four settings, with every interval above 0 (Appendix E). At 1,024 slots the gain is within the sampling noise in all four settings. Sections 5.2 and 7 state this limit. The differences are below:

BudgetLongHealth, QwenLongHealth, LlamaQuALITY, QwenQuALITY, Llama
16,384 slots+20.3+14.5+16.7+8.5
1,024 slots−3.0+1.0−1.2+2.2

4. Monolithic update: specification and consistency

The discrepancy between the two tables was an implementation error. We re-implemented the monolithic update to share the same compaction path as gated update and reran it at both budgets under the same pipeline as all other methods. Tables 2 and 13 give one consistent set of results, replacing both tables of the submission. With Qwen3-4B the monolithic update exceeds KV recursive on LongHealth and is lower on QuALITY. At 16,384 slots with Llama-3.2-3B, it reaches 36.5% on LongHealth and 30.7% on QuALITY, level in accuracy with KV recursive. At 1,024 slots with Llama-3.2-3B, it collapses after phases 4 and 5; the collapse correlates with the logical length of the memory exceeding the model's context window (Appendix H).

Section 4.2 and Algorithm 2 describe the current implementation exactly. Each layer scores every slot by the total attention the reference queries place on it, summed over queries and key-value heads. The tt highest-scoring slots form the writable set SS. Within each key-value head, the tt keys with the largest σj\sigma_j among the keys of SS and of the new documents fill SS, while every slot outside SS keeps its key and value. The values of SS are fitted so that ASVSA_S V_S matches YT−AFVFY_T - A_F V_F, with AFA_F the attention on the frozen slots and VFV_F their values. λ=0.01\lambda = 0.01. The submission's introduction mentioned a usage-count penalty; the method has none, and the revised text no longer mentions it.

With t=Bt = B the monolithic update is KV recursive, so Table 2 holds this comparison. At t/B=1/4t/B = 1/4 the monolithic update exceeds KV recursive on LongHealth but falls below it on QuALITY. Limiting the writable set does not help because the attention-based selection directs each new phase to the slots of the previous phase (Appendix H).

We are happy to address any additional questions.

Thank you for the substantial revisions, which address several of my concerns regarding the missing baseline, implementation details, and inconsistent results. However, the new experiments show that fixed partitions perform comparably to gated update, while simpler merging rules are also competitive, leaving the incremental benefit of likelihood-based gating unclear; the monolithic method likewise shows no consistent advantage. Although the revised paper is considerably clearer and the empirical analysis remains valuable, my central concern about the novelty and effectiveness of the proposed update mechanisms persists, so I maintain my original rating.

Thank you for your reply and for the time you spent on the revised paper. Gated update uses likelihood in two places: the router, which chooses the partition to read for each question, and the merge gate, which decides whether a new phase joins an existing partition. Our new experiment shows that the likelihood router raises accuracy by 21 to 23% over a random partition, reaching the accuracy of oracle routing and of a text-retrieval router without storing the documents.

1. Likelihood router

To test the router directly, we let the model answer every question from the same saved final memory of gated update, with a different rule choosing the partition in each run. We keep the prompt and decoding the same in all runs. The table gives final accuracy in % with Qwen3-4B at 16,384 slots:

Partition read per questionLongHealthQuALITY
Likelihood router50.858.4
Oracle (partition of the question's phase)51.053.2
BM25 over the source documents of each phase49.356.4
Random partition27.737.0
Partition of the newest phase24.835.9
No memory30.541.6

We find that the router is more accurate than a random partition by 23.1% on LongHealth and by 21.4% on QuALITY. When we give the model a random partition or the partition of the newest phase, its accuracy is lower than the no-memory baseline. Since the router and the oracle select the same partition for nearly all questions, we attribute their difference on QuALITY to the variation between two runs on the same memory.

To test whether the choice needs a likelihood score, we also route each question with BM25 over the source documents of each phase. BM25 is 1.5 to 2.0% less accurate than the likelihood router. Because our setting discards the documents after each phase, a text-retrieval router has no text to search. Storing the documents for retrieval would add memory beyond the fixed budget. The likelihood router reaches at least the accuracy of BM25 from the compacted memory alone.

2. Fixed partitions

Because fixed partitions use the same router as gated update, we conclude from their matching accuracy that rebalancing does not lower accuracy. In both methods, the model answers each question from the partition that the router selects among the partitions of the phases. Rebalancing lets gated update reach this accuracy without knowing the number of phases in advance, which fixed partitions require.

3. Merge gate

We agree with you that merging with the gate is no more accurate than merging with simpler rules. When we cap the number of partitions, gated update reaches similar accuracy (35.6 to 37.9%) whether the gate, the oldest partition, or a random partition receives a merged phase. In our main runs, we set the threshold of the gate above every likelihood gain, which gives every phase its own partition, as Section 4.1 states. Our claims therefore concern the setting with one partition per phase.

4. Monolithic update

We include the monolithic update to test whether a single memory, compacted in the same way, can retain the earlier phases as well as separate partitions. At 16,384 slots, gated update is more accurate than the monolithic update by 9.4 to 18.8% in three of the four settings of Table 2 and level with it in the fourth. We conclude that at 16,384 slots, separate partitions retain the earlier phases better than a single memory with the same compaction. The monolithic update thus shows that limiting the writes within one memory is not enough to retain the earlier phases, in line with the diagnosis of Section 3.

5. Contribution

Our contributions are the continual compaction setting, the diagnosis of why a recompacted memory loses the earlier phases, and gated update. In Section 3, we show that recompaction gives nearly all slots to the newest phase, as do StreamingLLM, SnapKV, and KVzip. We kept gated update simple on purpose, because one partition per phase and the likelihood router are enough to keep the earlier phases.

We believe these results address the concern about the benefit of likelihood-based routing. If they do, we would be grateful if you would consider updating your rating. If a concern remains, could you tell us which one, so that we can address it before the discussion ends or in the final version? Thank you again for your careful review.

Reviewer_BZnJOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies continual KV-cache compaction: new documents arrive sequentially, their source text is discarded after compression, and a fixed-size memory must retain information from all previous stages. Building on attention matching, it proposes two update strategies: Modular, which routes documents and questions to separate sub-memories, and Monolithic, which rewrites only a small subset of slots in a shared memory. Modular avoids the severe degradation of full recursive compaction, while Monolithic shows strongly model-dependent results. The paper also investigates information loss caused by repeated recompaction without access to the original text.

Critical strengths
  1. A useful problem setting. The paper goes beyond one-time compression and evaluates whether a bounded memory can absorb new documents while retaining older information.

  2. An informative diagnosis of forgetting. Oracle-routing experiments and comparisons at the same final memory size help distinguish routing errors and capacity limits from losses introduced by repeated recompaction.

  3. Simple, complementary update strategies. Partition-level isolation and slot-level selective writes offer two understandable ways to limit interference without training the backbone. Reporting the failure of Monolithic on Qwen also makes the limitations clear.

Critical weaknesses
  1. The evidence is limited to one compaction method and a narrow evaluation setting. All KV-memory methods use attention matching, so it is unclear whether the gains reflect a general benefit of selective updates or mainly address weaknesses of recursively applied attention matching. Comparisons with other approaches, such as KVzip-based eviction, would help establish this. RULER evaluation would also provide a more direct test of whether the memory preserves specific information across updates.

  2. The comparison between Modular and Monolithic does not lead to a clear practical recommendation. Monolithic performs well with Llama but fails badly with Qwen, apparently already during initialization. The two methods are also evaluated at different memory budgets. Without a matched comparison and an explanation of this failure, it is difficult to tell when either method should be preferred.

Questions for authors
  1. Can you compare against another compaction or eviction approach, such as SnapKV, KVzip, under the same continual-update setting? Do partitioning and selective writes still help beyond attention matching?

  2. Can you evaluate both methods on RULER, especially multi-needle retrieval, with relevant information distributed across different arrival phases? Tracking older needles after each update would help distinguish preservation of specific facts from general answer quality.

  3. Which method should a practitioner choose: Modular or Monolithic? Please compare them at the same memory budget and report accuracy, retention, and update cost. Why does Monolithic fail with Qwen already at phase 1, before continual updates begin?

  4. Since the gate never merges partitions in the main five-phase experiments, how much does adaptive routing contribute? How does Modular compare with a simple baseline that assigns each phase a separate, fixed-size partition and never recompacts older partitions, using the same final total budget?

  5. Could you clarify the Monolithic update implementation? How are new keys assigned to writable slots, and how are frozen slots accounted for when fitting values? The introduction also mentions discounting slots by their write history, but the scoring rule is not clearly specified.

  6. What is the end-to-end cost of writing and reading memory? Modular scores each incoming document and question under every partition, and writes may also recompact older partitions. Does its smaller cache during answer generation translate into lower overall latency?

  7. How does Modular handle questions that require information from multiple partitions? Since it reads only one partition, is there a substantial limitation on cross-document reasoning? How sensitive is routing to explicit patient names or story titles in the question?

2 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

Thank you for the detailed questions on the compaction methods and the comparison scope, and for describing the diagnosis as informative and the setting as useful. We apologize for the late response; we underestimated the time needed for the new experiments. We have also revised the writing throughout the paper for clarity. Section and table numbers refer to the revised PDF.

1. Other compaction and eviction methods

Partitioning helps in both forms we tested. We thank the reviewer for suggesting this comparison, which helped us test the generality of the finding.

We run StreamingLLM, SnapKV and KVzip under the same updates and budget (Section 5.1, Table 2). At 16,384 slots, gated update (called modular update in our initial submission) is at least as accurate as all three in every setting, exceeding KVzip by 23.5% with Qwen3-4B and 5.1% with Llama-3.2-3B averaged over both datasets; the one dataset-model pair within the sampling noise is KVzip with Llama-3.2-3B on LongHealth (Table 14). All three eviction methods concentrate over 99% of their slots in the newest phase (Appendix D.1, Table 9), the same pattern we identify for attention matching.

Compacting every partition with KVzip and keeping the same router, KVzip with one partition per phase exceeds a single KVzip memory by 7.0% with Qwen3-4B and 4.8% with Llama-3.2-3B, averaged over both datasets (Appendix G.4, Table 25). With KVzip as compactor, the router selects the correct partition for 87 to 96% of the questions. Partitioning helps even an eviction method like KVzip, which keeps original values and fits none.

We find the comparison between compactors to be mixed. With Qwen3-4B, attention-matching partitions are 12.8% more accurate on LongHealth and 20.2% on QuALITY. With Llama-3.2-3B at 16,384 slots, attention-matching partitions and KVzip partitions reach comparable accuracy. At 1,024 slots, KVzip partitions are 8.3% more accurate on LongHealth and level in accuracy on QuALITY, though the router then selects the correct partition for only 44% of the questions. At about 200 slots per partition, gated update is close to the no-memory baseline (Sections 5.2 and 7). The contribution of this paper is the diagnosis and the partitioning approach; the accuracy gains require partitions large enough to hold useful content, and extending the method to smaller budgets is an open direction.

The monolithic update refits the values of the writable slots, so it has no direct counterpart under eviction, which keeps original values. Partitioning is the form of selective update that applies to both compaction and eviction, and the KVzip-partition results above show that it helps.

2. Fixed-partition baseline

We find the two equally accurate: all four differences are within the sampling noise (Appendix G.3). The results with Llama-3.2-3B on both datasets at both budgets:

SettingFixed partitionsGated updateDifference
LongHealth, 16,384 slots39.038.8+0.3
LongHealth, 1,024 slots23.225.5−2.2
QuALITY, 16,384 slots42.641.3+1.3
QuALITY, 1,024 slots34.934.1+0.8

We attribute the accuracy to the separate partitions and the one-partition read. Rebalancing lets the memory run without knowing the number of phases in advance, at no measurable cost.

3. Questions across partitions

This is a real limitation. Every question of our benchmarks concerns one document. Reading the two highest-scoring partitions together lowers final accuracy by 5.3 to 13.7% with Qwen3-4B and leaves it unchanged with Llama-3.2-3B (Appendix G.2), consistent with the loss from reading all partitions (Table 20). Section 7 states that questions spanning several phases are outside the one-partition read.

4. Sensitivity to names and titles

Thank you for raising this; the dependence on names was an important gap to quantify. Routing accuracy drops, but final accuracy is unchanged in seven of the eight settings (Appendix G.1; summarized in Section 5.2). The results at 16,384 slots:

LongHealth, QwenLongHealth, LlamaQuALITY, QwenQuALITY, Llama
Routing accuracy, with names98.298.0100.0100.0
Routing accuracy, without83.579.290.785.9
Final accuracy, with names50.036.857.441.4
Final accuracy, without49.237.252.140.8

At 16,384 slots, routing accuracy drops from 98 to 100% down to 79 to 91% without names or titles, so the router still finds the partition from the content of the question for most questions. At 1,024 slots it falls to 47 to 76%. The one exception in final accuracy is QuALITY with Qwen3-4B at 16,384 slots (−5.3%). Oracle routing shows the same loss (−5.1%), so the title helps this model answer from the correct partition, and the loss does not come from routing.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

(Continued from Part 1)

5. End-to-end cost

We find that the smaller cache lowers overall latency. The write is faster than for a single memory in three of the four settings (Appendix C.6), because gated update compacts only the new documents and recompacts partitions smaller than the single memory. With Llama-3.2-3B on LongHealth it is slower: 79 vs 65 minutes total. Total write time at 16,384 slots:

SettingGated updateKV recursiveMonolithic
LongHealth, Qwen3-4B87 min111 min102 min
LongHealth, Llama-3.2-3B79 min65 min52 min
QuALITY, Qwen3-4B89 min131 min123 min
QuALITY, Llama-3.2-3B37 min61 min59 min

At the read, routing takes 0.2 to 0.3 seconds per question on one RTX A5000, after which the answer is decoded from one partition of about 3,300 slots, a fifth of the single memory (Appendix C.6). A five-phase run with Qwen3-4B takes 3.3 to 3.6 hours end to end with gated update, compared with 4.1 to 5.5 hours for KV recursive (Section 7 lists the routing cost as a limitation).

6. Which method to choose

We recommend gated update when the budget allows partitions of several hundred slots or more. Tables 2 and 13 report both methods at both budgets with forgetting; the write cost is in the table above.

At 16,384 slots, gated update is the most accurate memory in every setting: it exceeds KV recursive by 7 to 24% in final accuracy while at least halving forgetting (Table 13). The monolithic update exceeds KV recursive on LongHealth and is lower on QuALITY. At 1,024 slots with Llama-3.2-3B, the monolithic update collapses after phases 4 and 5, with the logical length of the memory exceeding the model's context window (Appendix H). With Qwen3-4B at 1,024 slots, the monolithic update reaches accuracy comparable to KV recursive. At 1,024 slots, gated update is level in accuracy with the no-memory baseline (Appendix E).

7. Monolithic failure with Qwen and implementation

The phase-1 failure of Qwen3-4B in our initial submission was an implementation error. We re-implemented the monolithic update to share the same compaction path as gated update (Section 4.2, Algorithm 2). This resolved the Qwen failure but surfaced a separate failure mode with Llama-3.2-3B at 1,024 slots.

In each layer, every slot of the memory is scored by the total attention that the reference queries place on it, summed over queries and key-value heads. The tt highest-scoring slots form the writable set SS. Within each key-value head, the tt keys with the largest σj\sigma_j, chosen from the keys of SS and of the new documents, fill SS. The values of SS are fitted so that ASVSA_S V_S matches YT−AFVFY_T - A_F V_F, where AFA_F is the attention on the frozen slots and VFV_F their values. With t=Bt = B the monolithic update reduces to KV recursive, so Table 2 holds this comparison. The submission's introduction mentioned a write-history discount; the method has none, and the revised text no longer mentions it.

We trace the collapse of Llama-3.2-3B at 1,024 slots to the logical length of the memory exceeding the context window; fixing the logical length removes the collapse but does not improve accuracy (Appendix H).

8. Needle retrieval across phases

Gated update on Qwen retains needles from phases 1 to 4 at 20 to 33% accuracy, three times the in-context rate on those phases. We appreciate this suggestion and built a five-phase needle stream: each phase contains one document with five unique key-value facts (project name to registration code), for 25 needles total. After all five updates, every method answers one MCQ question per needle seen so far (75 questions total at the final phase). Accuracy at the final phase, grouped by the phase in which the needle was planted:

Method / modelPhase 1Phase 2Phase 3Phase 4Phase 5Overall
In-context Qwen01013109638.7
In-context Llama203020158441.3
Gated update Qwen20303325821.3
Gated update Llama001310129.3

In-context baselines show strong recency bias: 84 to 96% on the newest phase, at most 30% on older ones. Gated update on Qwen retains phases 1 to 4 at three times the in-context rate (average 27% vs 8%), at the cost of a lower score on the newest phase. Overall accuracy is lower because the compacted memory raises the share of replies without a parsable answer (the model produces explanations instead of a single letter); among parseable answers, gated-update Qwen scores 50 to 83% on phases 1 to 4. Gated update on Llama is weaker, consistent with the gap between the two models on LongHealth (Table 2).

We are happy to address any additional questions.

Reviewer_wRzWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies continual KV-cache compaction, where new documents arrive over time, earlier source documents are discarded, and the memory budget remains fixed. It proposes two updates built on attention matching: a modular method that routes documents to sub-memories, and a monolithic method that rewrites only selected slots. Experiments examine knowledge retention across sequential updates with Llama-3.2-3B and Qwen3-4B.

Critical strengths

1.The paper studies a useful setting: updating a bounded KV memory after the original documents are no longer available. Evaluating earlier phases after each update makes the learning–forgetting trade-off explicit.

2.The analysis helps identify limitations of repeated compaction. In particular, the comparison with a cache compacted once from the source text to the same final size provides evidence that the loss is not explained by memory size alone. The paper also reports the substantial differences between the two models.

Critical weaknesses
  1. The main experiments do not establish the benefit of the write-time gate. In the five-phase experiments, every phase opens a new partition and no merging occurs. These results mainly support storing phases separately rather than repeatedly compacting them together. Appendix B.7 tests merging, but does not show a clear advantage: on the 16-document sequence, disabling merges improves accuracy by 9.2 percentage points over the default gate. A comparison with simple partitioning or merging rules in a setting where merging is necessary would help establish the gate’s contribution.

  2. The monolithic method lacks a random-slot selection baseline. It updates a small set of slots selected by attention, but does not test whether this selection is better than randomly choosing the same number of slots. The current results therefore do not separate the benefit of selecting suitable slots from the benefit of simply limiting how much memory changes. A comparison that keeps the update count, regularization, and other settings fixed is needed to assess the selection strategy.

Questions for authors
  1. When the fixed budget requires partitions to be merged, does the likelihood gate outperform simple alternatives, such as merging the oldest partitions or selecting a partition at random?

  2. Does attention-based slot selection outperform random selection when the same number of slots is updated under otherwise identical settings?

3 comments

Thank you for raising the question of whether the gate itself matters and for asking about the monolithic slot selection. Your point about simple partitioning helped us sharpen the claims. We apologize for the late response; we underestimated the time needed for the new experiments. We have also revised the writing throughout the paper for clarity. Section and table numbers refer to the revised PDF.

1. Merge gate vs simple alternatives

All three rules reach the same accuracy. On the 16-document sequence at 1,024 slots with Qwen3-4B and a cap of three partitions, 13 of the 16 documents must be merged. All three rules reach comparable accuracy (35.6 to 37.9%). The gate gives the highest routing accuracy (81.0% against 71.6% and 51.3%), since 11 of its 13 merges join documents of the same dataset. All three memories stay within 6% of the no-memory baseline (32.4%): at about 64 slots per document, the budget limits accuracy more than the choice of receiving partition. Appendix G.5 (Table 24) reports this result.

The results are below (Appendix G.5, Table 24):

RuleSlots per partitionRouting accuracyAccuracyDifference from gate
Largest likelihood gain (gate)740 / 243 / 4181.035.6
Oldest partition938 / 45 / 4171.637.9+2.3
Random partition450 / 329 / 24551.335.60.0

2. Benefit beyond storing phases separately

This is an important point that helped us clarify the contribution. You are right that the main experiments do not establish this. In the revision we open one partition per phase by design (Section 4.1); the accuracy claims concern the partitioned memory and the one-partition read. The merge gate is an extension tested only in Appendix G.5. A new fixed-partition baseline confirms your interpretation: every phase gets a fifth of the budget, is compacted once, is never recompacted and uses the same router. All four differences between fixed partitions and gated update (called modular update in our initial submission) are within the sampling noise (Appendix G.3). The results with Llama-3.2-3B:

SettingFixed partitionsGated updateDifference
LongHealth, 16,384 slots39.038.8+0.3
LongHealth, 1,024 slots23.225.5−2.2
QuALITY, 16,384 slots42.641.3+1.3
QuALITY, 1,024 slots34.934.1+0.8

Rebalancing has a different role: fixed partitions need the number of phases in advance, and gated update reaches the same accuracy without this knowledge.

The 9.2% loss from merging at 16 documents that you cite remains in the paper (Table 23). Merging becomes necessary only beyond 64 documents at 1,024 slots. Section 7 lists the accuracy cost as a limitation.

3. Attention-based vs random slot selection

Both the most-attended and least-attended slot rules produce the same retention and the same collapse point, so the choice of writable slots has no measurable effect in this setting; we have not run a uniform-random rule. We test the least-attended rule in the revision (Appendix H), replacing the most-attended slots with the least-attended slots and keeping all else unchanged. The writes then spread across all old phases, each of which holds at least 10% of the slots after phase 5. The memory still collapses: almost no reply contains a parsable answer, matching the collapse of the standard monolithic update and KV recursive at this budget (1,024 slots, Llama-3.2-3B). The collapse of Llama-3.2-3B at this budget correlates with the logical length of the memory exceeding the model's context window; fixing the logical length removes the collapse (Appendix H). The bottleneck is the logical length, not the slot-selection rule.

We are happy to address any additional questions.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors’ efforts and the clearer presentation of the paper.

However, the new results confirm my main concerns. The merge gate does not outperform simple alternatives in final accuracy, and fixed partitioning achieves comparable performance. Moreover, the additional slot-selection experiments do not establish the benefit of attention-based selection, as the random baseline remains missing.

While I appreciate the authors’ clarification of these limitations, I believe the empirical justification for the proposed designs remains insufficient. Therefore, I maintain my original rating of 2 (Weak Rejection).

Dear Reviewer,

Thank you for your reply and for the further questions. We ran the monolithic update with random slot choice, as you suggested. With Qwen3-4B on LongHealth at 16,384 slots, it reaches 27.2% final accuracy. Its accuracy is 48.8% on questions about the newest phase and 15.0 to 28.7% on questions about the earlier phases. A single memory therefore keeps mainly the newest phase whether the attention scores or a random rule choose the slots to write. Our claim concerns this recency bias in a single recompacted memory. It makes no claim that attention-based selection is better than random selection. We will add this baseline for all settings in our revisions of the paper.

We agree that the merge gate is no more accurate than simpler merge rules, and that fixed partitions reach the accuracy of rebalancing when the number of phases is known. We will state both points in our revisions.

If any concern remains, could you tell us which one, so that we can address it? Thank you again for your careful review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful reviews. We apologize for the late response; we underestimated the time needed for the new experiments. We are encouraged that the reviewers find the setting useful (wRzW, BZnJ) and the analysis informative (BZnJ: "an informative diagnosis of forgetting"; XtW4: "a useful controlled diagnosis of memory degradation"). We have uploaded a revised PDF; section and table numbers below refer to it. Below we address shared concerns, then respond to each reviewer individually.

Since all three reviewers raised the fixed-partition baseline and two raised the monolithic implementation, we address both here. Fixed partitions and gated update are equally accurate: all four differences are within the sampling noise (Appendix G.3, Table 22). Two simple merge rules also match the likelihood gate (Appendix G.5, Table 24). We attribute the accuracy to the separate partitions and the one-partition read; rebalancing lets the memory run without knowing the number of phases in advance, at no measurable cost. The monolithic discrepancy between Tables 2 and 9 of the submission was an implementation error; we re-implemented it from scratch and reran it at both budgets (Tables 2 and 13 of the revised paper).

Here is a summary of our revisions. All modifications are highlighted in blue.

  • Reran all KV memories with one compaction block per document and a value prior (λ=0.01\lambda = 0.01, Appendix C.1). Every table comes from these runs.
  • Added token-eviction baselines: StreamingLLM, SnapKV, KVzip in the same continual setting (Section 5.1, Table 2; Reviewer BZnJ).
  • Added a five-step analysis of why the memory forgets (Section 3, Appendix D), yielding three requirements that gated update meets.
  • Reorganized the paper: analysis (Section 3) precedes the method (Section 4); renamed modular update to gated update throughout.
  • Added Llama-3.1-8B, KVzip partitions, routing without names or titles, forgetting and slots read per answer to Section 5.2 (Reviewers BZnJ, wRzW, XtW4).
  • Added analysis of the monolithic update collapse (Appendix H): per-phase accuracy, logical-length overflow, least-attended slot rule.
  • Revised the writing throughout the paper for clarity, following the presentation feedback from all three reviewers.

New experiments run for this rebuttal (all in the revised appendix):

  1. Fixed-partition baseline (Reviewers wRzW, BZnJ, XtW4): Appendix G.3.
  2. Simple merge rules when merging is forced (Reviewer wRzW): Appendix G.5.
  3. Partitions compacted by KVzip (Reviewer BZnJ): Appendix G.4.
  4. Routing without patient names or story titles, and reads of two partitions (Reviewer BZnJ): Appendices G.1 and G.2.
  5. Write time per phase and routing time (Reviewer BZnJ): Appendix C.6.
  6. Intervals for the gain over the no-memory baseline (Reviewer XtW4): Appendix E.
  7. Monolithic update re-implementation (Reviewers wRzW, XtW4): Tables 2 and 13. Appendix H traces the slot writes, tests an alternative scoring rule, and identifies the collapse of Llama-3.2-3B at 1,024 slots as the logical length exceeding the context window.

We warmly encourage the reviewers to examine the results in the revised manuscript. If there are any additional questions, we are happy to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.