- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MoNe, a modular neural memory for long-context inference. MoNe writes segmented context into fast-weight memories attached to Transformer layers using block-localized associative updates, avoiding backpropagation through the frozen backbone. When answering a query, the model no longer needs to process the original context tokens; instead, each layer reads from the stored memory and incorporates the stored contextual information into its hidden representations. MoNe is meta-trained on 1K–4K contexts and evaluated up to 128K, achieving strong performance on retrieval and QA tasks while reducing computation and memory costs compared with ICL and PERK.
Critical strengths
- MoNe provides a lightweight and modular framework by attaching a dedicated memory module to each Transformer layer while keeping the pretrained backbone frozen. Its block-localized memory writing avoids backpropagation through the entire backbone, substantially reducing test-time adaptation cost.
- On the evaluated long-context tasks, MoNe maintains strong performance while significantly reducing both computational cost and peak memory usage compared with the main baselines.
Critical weaknesses
- The evaluation covers a relatively narrow range of long-context tasks, with no summarization or other tasks requiring broader global information integration. Moreover, although MoNe highlights the reusability of its memory across queries, the current experiments do not directly evaluate shared-context or multi-query scenarios, leaving this claimed advantage insufficiently validated.
- MoNe scales the test-time inner learning rate inversely with the total number of context segments. This segment-count-dependent heuristic is difficult to reconcile with the claimed incremental extensibility of the memory, where the eventual context length may be unknown, and the necessity of this global scaling on top of the learned adaptive updater is insufficiently justified. In addition, the segment-size ablation is limited to at most 1024 tokens, leaving the behavior at larger segment sizes insufficiently explored.
Questions for authors
- The paper claims that MoNe supports incremental memory extension, while the test-time learning rate is scaled inversely with the total number of context segments. In a truly incremental setting where the final number of segments is unknown in advance, what value of the total segment count is used for this scaling? If additional context segments are appended later, are the earlier memory updates rescaled or otherwise adjusted?
- The current evaluation is concentrated on retrieval, aggregation, and short-form QA, leaving it unclear how well MoNe generalizes to other types of long-context tasks such as summarization or global information synthesis. Do the authors have results on broader long-context benchmarks, ideally including settings that also test shared-context or multi-query reuse, such as SCBench?
4 comments
Q1/W2: Incremental Memory Extension, Learning-Rate Scaling, and Segment Size
(a) Learning-rate scaling without knowledge of the final context length.
We agree that the original length-adaptive scaling (1/S) assumes knowledge of the final number of context segments , which is unavailable in incremental settings. However, this scaling is not an inherent requirement of MoNe's memory-writing mechanism.
To examine whether this dependency can be removed, we conducted an additional SQuAD experiment comparing the original length-adaptive scaling with fixed scaling, where the learning-rate scale corresponding to 128K is applied uniformly across all context lengths.
Table 1. Length-adaptive versus fixed learning-rate scaling on SQuAD.
| Scaling strategy | 4K | 8K | 16K | 32K | 48K | 64K | 96K | 128K |
|---|---|---|---|---|---|---|---|---|
| Length-adaptive | 0.62 | 0.65 | 0.60 | 0.62 | 0.59 | 0.57 | 0.59 | 0.47 |
| Fixed | 0.60 | 0.62 | 0.59 | 0.60 | 0.58 | 0.57 | 0.56 | 0.47 |
Fixed scaling achieves comparable performance across all evaluated context lengths, with differences of at most 0.03 and identical performance at 64K and 128K. These results demonstrate that length-adaptive scaling is not essential for MoNe's effectiveness within the evaluated range.
Importantly, fixed scaling directly addresses both aspects of the reviewer's concern: (1) the learning-rate scale does not depend on the eventual number of context segments, and (2) when additional segments arrive, earlier memory updates do not need to be rescaled or otherwise adjusted.
New segments can be incorporated by continuing updates from the existing fast-weight state using the same fixed scale, without reprocessing previous context. This provides a practical length-independent alternative to the original (1/S) scaling for incremental memory extension.
(b) Incremental append-and-query evaluation.
To directly validate incremental memory extension, we conducted an additional SQuAD experiment in which context is appended in successive 32K-token increments, with queries evaluated after each append.
Throughout this experiment, we use the fixed learning-rate scaling described above, without prior knowledge of the final context length or retrospective adjustment of earlier updates. The memory-writing segment size remains 1024 tokens.
We consider two settings based on the location of the answer-bearing paragraph: within the most recently appended 32K-token range, or randomly positioned across the entire accumulated context.
Table 2. Incremental append-and-query performance on SQuAD.
| Gold paragraph location | 32K | 64K | 96K | 128K |
|---|---|---|---|---|
| Newly appended 32K chunk | 0.60 | 0.58 | 0.60 | 0.59 |
| Random across accumulated context | 0.60 | 0.57 | 0.54 | 0.47 |
Columns indicate cumulative context lengths after successive 32K-token additions.
Performance remains stable between 0.58 and 0.60 when the relevant paragraph is within the most recently appended range. When the paragraph is randomly positioned across the accumulated context, performance gradually decreases but remains at 0.47 at 128K.
These results directly demonstrate that MoNe can incrementally extend its memory and answer subsequent queries using fixed learning-rate scaling, without knowing the eventual context length or modifying previously applied updates. This provides empirical validation of the incremental extensibility questioned by the reviewer.
(c) Extended segment-size ablation.
To address the reviewer's concern about segment sizes beyond 1024 tokens, we extended the RULER FWE ablation to include 2048-token segments, examining both performance and computational cost.
Table 3. Segment-size ablation on RULER FWE.
| Segment size | 32K Performance | 32K FLOPs (T) | 64K Performance | 64K FLOPs (T) | 128K Performance | 128K FLOPs (T) |
|---|---|---|---|---|---|---|
| 512 | 0.98 | 36.74 | 0.98 | 72.85 | 0.94 | 145.06 |
| 1024 | 1.00 | 40.40 | 1.00 | 80.12 | 1.00 | 159.55 |
| 2048 | 1.00 | 47.73 | 0.99 | 94.66 | 0.97 | 188.52 |
Increasing the segment size from 1024 to 2048 leads to performance degradation while increasing computational cost, particularly at longer context lengths.
At 128K, performance decreases from 1.00 to 0.97, while total FLOPs increase from 159.55T to 188.52T (approximately 18.2% higher). Conversely, reducing the segment size to 512 lowers computational cost but degrades long-context performance (0.94 at 128K).
These results show that increasing segment size beyond 1024 provides no performance or efficiency benefit at longer context lengths. Instead, 1024 offers a favorable performance–efficiency trade-off among the evaluated configurations, providing empirical justification for our original choice.
Q2/W1: Summarization Tasks and Shared-Context Multi-Query Reuse
(a) Summarization and broader information synthesis.
We agree that the original evaluation focuses primarily on retrieval, aggregation, and short-form QA.
To address the concern about broader information synthesis, we additionally evaluated MoNe on three representative abstractive summarization datasets: CNN/DailyMail, XSum, and TL;DR.
As in our original task-specific evaluation setup, MoNe was separately meta-trained for each task using the original training and validation splits. The datasets contain naturally occurring documents with context lengths extending up to approximately 35K tokens, measured using the Qwen tokenizer.
Table 4. Abstractive summarization performance (ROUGE).
| Method | CNN/DM R1 | CNN/DM RLsum | XSum R1 | XSum RLsum | TL;DR R1 | TL;DR RLsum |
|---|---|---|---|---|---|---|
| ICL-no context | 7.9 | 7.5 | 9.0 | 7.3 | 8.2 | 6.7 |
| ICL-full context | 24.5 | 21.0 | 19.2 | 13.0 | 19.1 | 13.1 |
| MoNe | 36.3 | 33.8 | 32.8 | 25.3 | 35.0 | 26.9 |
MoNe substantially outperforms full-context ICL across all three datasets and both ROUGE metrics.
Specifically, R1/RLsum improves from 24.5/21.0 to 36.3/33.8 on CNN/DailyMail, from 19.2/13.0 to 32.8/25.3 on XSum, and from 19.1/13.1 to 35.0/26.9 on TL;DR.
These results directly address the lack of summarization experiments in the original submission and provide evidence that MoNe's learned memories can support generative information synthesis beyond retrieval, aggregation, and short-form QA.
We emphasize that these results reflect task-specific meta-training rather than zero-shot transfer from retrieval or QA tasks. Furthermore, the datasets are not specifically designed for extreme-length summarization, and length-stratified evaluation would provide further insight into performance on the longest documents.
(b) Shared-context multi-query memory reuse.
To directly evaluate the claimed reusability of MoNe's memory, we conducted an additional SQuAD experiment using 100 contexts, each paired with four distinct questions (400 questions in total).
For MoNe, memory is constructed only once per context and reused unchanged for all four questions, without additional memory updates or reprocessing the original context.
Table 5. Shared-context multi-query performance on SQuAD.
| Method | 4K | 8K | 16K | 32K | 48K | 64K | 96K | 128K |
|---|---|---|---|---|---|---|---|---|
| ICL | 0.59 | 0.52 | 0.56 | 0.49 | 0.20 | 0.14 | 0.09 | 0.05 |
| MoNe | 0.63 | 0.61 | 0.60 | 0.59 | 0.59 | 0.56 | 0.52 | 0.50 |
MoNe consistently outperforms ICL across all evaluated context lengths, with a particularly pronounced advantage at 128K (0.50 versus 0.05).
More importantly, the same constructed memory effectively supports four distinct queries without rebuilding or updating the memory. Unlike ICL, which processes the full context for each query, MoNe reuses the stored fast-weight state, directly validating the shared-context multi-query capability highlighted by the reviewer.
While this is not a full SCBench evaluation, it directly tests the key reuse property motivating the reviewer's suggestion: answering multiple distinct queries against a shared context whose memory representation is constructed only once.
We will incorporate these additional results into the revised manuscript and clarify the corresponding methodology and claims.
Dear Reviewer cW2Q,
As the discussion period approaches its conclusion, we would greatly appreciate your feedback on our rebuttal.
We believe we have directly addressed your principal concerns through additional experiments on incremental memory extension, segment-size efficiency, abstractive summarization, and shared-context multi-query memory reuse.
If you have any further questions or concerns, please do not hesitate to raise them. We are fully committed to addressing them and will respond promptly until the very end of the discussion period.
We would also sincerely appreciate your reconsideration of the initial assessment in light of the new evidence.
Best regards,
Authors of Submission #45685
We thank the reviewer for recognizing MoNe's lightweight, modular design and its computational and memory efficiency. We also appreciate the constructive feedback regarding incremental memory extension, learning-rate scaling, segment-size sensitivity, broader information synthesis, and shared-context multi-query reuse.