acceptodds
Under review as a conference paper at ICLR 2027

MoNe: Modular Neural Memory for Efficient Long Context Inference

Abstract

Long-context reasoning is essential for applications such as personalized assistants, document QA, and agentic systems, yet standard in-context learning (ICL) becomes inefficient and unreliable as context length grows. We introduce (dular ural Memory), a lightweight plugin that equips frozen pretrained Transformers with test-time learned fast-weight memories, without modifying or retraining the backbone. MoNe processes long contexts in segments and writes information into these memories through block-localized gradient updates, without backpropagating through the Transformer backbone. During query inference, the adapted memories provide context-dependent residuals at each Transformer layer without including the original context in the model input, while remaining reusable across queries and incrementally extensible as new context arrives. Across RULER, BABILong, SQuAD, and HotpotQA, MoNe generalizes to 128K-token contexts while substantially outperforming the baselines at long context lengths and reducing both peak GPU memory and total FLOPs by approximately (80%) relative to ICL at 128K.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Efficient inference, Neural memory, Test-time learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cW2QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MoNe, a modular neural memory for long-context inference. MoNe writes segmented context into fast-weight memories attached to Transformer layers using block-localized associative updates, avoiding backpropagation through the frozen backbone. When answering a query, the model no longer needs to process the original context tokens; instead, each layer reads from the stored memory and incorporates the stored contextual information into its hidden representations. MoNe is meta-trained on 1K–4K contexts and evaluated up to 128K, achieving strong performance on retrieval and QA tasks while reducing computation and memory costs compared with ICL and PERK.

Critical strengths
  1. MoNe provides a lightweight and modular framework by attaching a dedicated memory module to each Transformer layer while keeping the pretrained backbone frozen. Its block-localized memory writing avoids backpropagation through the entire backbone, substantially reducing test-time adaptation cost.
  2. On the evaluated long-context tasks, MoNe maintains strong performance while significantly reducing both computational cost and peak memory usage compared with the main baselines.
Critical weaknesses
  1. The evaluation covers a relatively narrow range of long-context tasks, with no summarization or other tasks requiring broader global information integration. Moreover, although MoNe highlights the reusability of its memory across queries, the current experiments do not directly evaluate shared-context or multi-query scenarios, leaving this claimed advantage insufficiently validated.
  2. MoNe scales the test-time inner learning rate inversely with the total number of context segments. This segment-count-dependent heuristic is difficult to reconcile with the claimed incremental extensibility of the memory, where the eventual context length may be unknown, and the necessity of this global scaling on top of the learned adaptive updater is insufficiently justified. In addition, the segment-size ablation is limited to at most 1024 tokens, leaving the behavior at larger segment sizes insufficiently explored.
Questions for authors
  1. The paper claims that MoNe supports incremental memory extension, while the test-time learning rate is scaled inversely with the total number of context segments. In a truly incremental setting where the final number of segments is unknown in advance, what value of the total segment count is used for this scaling? If additional context segments are appended later, are the earlier memory updates rescaled or otherwise adjusted?
  2. The current evaluation is concentrated on retrieval, aggregation, and short-form QA, leaving it unclear how well MoNe generalizes to other types of long-context tasks such as summarization or global information synthesis. Do the authors have results on broader long-context benchmarks, ideally including settings that also test shared-context or multi-query reuse, such as SCBench?
4 comments

We thank the reviewer for recognizing MoNe's lightweight, modular design and its computational and memory efficiency. We also appreciate the constructive feedback regarding incremental memory extension, learning-rate scaling, segment-size sensitivity, broader information synthesis, and shared-context multi-query reuse.

Q1/W2: Incremental Memory Extension, Learning-Rate Scaling, and Segment Size

(a) Learning-rate scaling without knowledge of the final context length.

We agree that the original length-adaptive scaling (1/S) assumes knowledge of the final number of context segments SS, which is unavailable in incremental settings. However, this scaling is not an inherent requirement of MoNe's memory-writing mechanism.

To examine whether this dependency can be removed, we conducted an additional SQuAD experiment comparing the original length-adaptive scaling with fixed scaling, where the learning-rate scale corresponding to 128K is applied uniformly across all context lengths.

Table 1. Length-adaptive versus fixed learning-rate scaling on SQuAD.

Scaling strategy4K8K16K32K48K64K96K128K
Length-adaptive0.620.650.600.620.590.570.590.47
Fixed0.600.620.590.600.580.570.560.47

Fixed scaling achieves comparable performance across all evaluated context lengths, with differences of at most 0.03 and identical performance at 64K and 128K. These results demonstrate that length-adaptive scaling is not essential for MoNe's effectiveness within the evaluated range.

Importantly, fixed scaling directly addresses both aspects of the reviewer's concern: (1) the learning-rate scale does not depend on the eventual number of context segments, and (2) when additional segments arrive, earlier memory updates do not need to be rescaled or otherwise adjusted.

New segments can be incorporated by continuing updates from the existing fast-weight state using the same fixed scale, without reprocessing previous context. This provides a practical length-independent alternative to the original (1/S) scaling for incremental memory extension.


(b) Incremental append-and-query evaluation.

To directly validate incremental memory extension, we conducted an additional SQuAD experiment in which context is appended in successive 32K-token increments, with queries evaluated after each append.

Throughout this experiment, we use the fixed learning-rate scaling described above, without prior knowledge of the final context length or retrospective adjustment of earlier updates. The memory-writing segment size remains 1024 tokens.

We consider two settings based on the location of the answer-bearing paragraph: within the most recently appended 32K-token range, or randomly positioned across the entire accumulated context.

Table 2. Incremental append-and-query performance on SQuAD.

Gold paragraph location32K64K96K128K
Newly appended 32K chunk0.600.580.600.59
Random across accumulated context0.600.570.540.47

Columns indicate cumulative context lengths after successive 32K-token additions.

Performance remains stable between 0.58 and 0.60 when the relevant paragraph is within the most recently appended range. When the paragraph is randomly positioned across the accumulated context, performance gradually decreases but remains at 0.47 at 128K.

These results directly demonstrate that MoNe can incrementally extend its memory and answer subsequent queries using fixed learning-rate scaling, without knowing the eventual context length or modifying previously applied updates. This provides empirical validation of the incremental extensibility questioned by the reviewer.


(c) Extended segment-size ablation.

To address the reviewer's concern about segment sizes beyond 1024 tokens, we extended the RULER FWE ablation to include 2048-token segments, examining both performance and computational cost.

Table 3. Segment-size ablation on RULER FWE.

Segment size32K Performance32K FLOPs (T)64K Performance64K FLOPs (T)128K Performance128K FLOPs (T)
5120.9836.740.9872.850.94145.06
10241.0040.401.0080.121.00159.55
20481.0047.730.9994.660.97188.52

Increasing the segment size from 1024 to 2048 leads to performance degradation while increasing computational cost, particularly at longer context lengths.

At 128K, performance decreases from 1.00 to 0.97, while total FLOPs increase from 159.55T to 188.52T (approximately 18.2% higher). Conversely, reducing the segment size to 512 lowers computational cost but degrades long-context performance (0.94 at 128K).

These results show that increasing segment size beyond 1024 provides no performance or efficiency benefit at longer context lengths. Instead, 1024 offers a favorable performance–efficiency trade-off among the evaluated configurations, providing empirical justification for our original choice.

Q2/W1: Summarization Tasks and Shared-Context Multi-Query Reuse

(a) Summarization and broader information synthesis.

We agree that the original evaluation focuses primarily on retrieval, aggregation, and short-form QA.

To address the concern about broader information synthesis, we additionally evaluated MoNe on three representative abstractive summarization datasets: CNN/DailyMail, XSum, and TL;DR.

As in our original task-specific evaluation setup, MoNe was separately meta-trained for each task using the original training and validation splits. The datasets contain naturally occurring documents with context lengths extending up to approximately 35K tokens, measured using the Qwen tokenizer.

Table 4. Abstractive summarization performance (ROUGE).

MethodCNN/DM R1CNN/DM RLsumXSum R1XSum RLsumTL;DR R1TL;DR RLsum
ICL-no context7.97.59.07.38.26.7
ICL-full context24.521.019.213.019.113.1
MoNe36.333.832.825.335.026.9

MoNe substantially outperforms full-context ICL across all three datasets and both ROUGE metrics.

Specifically, R1/RLsum improves from 24.5/21.0 to 36.3/33.8 on CNN/DailyMail, from 19.2/13.0 to 32.8/25.3 on XSum, and from 19.1/13.1 to 35.0/26.9 on TL;DR.

These results directly address the lack of summarization experiments in the original submission and provide evidence that MoNe's learned memories can support generative information synthesis beyond retrieval, aggregation, and short-form QA.

We emphasize that these results reflect task-specific meta-training rather than zero-shot transfer from retrieval or QA tasks. Furthermore, the datasets are not specifically designed for extreme-length summarization, and length-stratified evaluation would provide further insight into performance on the longest documents.


(b) Shared-context multi-query memory reuse.

To directly evaluate the claimed reusability of MoNe's memory, we conducted an additional SQuAD experiment using 100 contexts, each paired with four distinct questions (400 questions in total).

For MoNe, memory is constructed only once per context and reused unchanged for all four questions, without additional memory updates or reprocessing the original context.

Table 5. Shared-context multi-query performance on SQuAD.

Method4K8K16K32K48K64K96K128K
ICL0.590.520.560.490.200.140.090.05
MoNe0.630.610.600.590.590.560.520.50

MoNe consistently outperforms ICL across all evaluated context lengths, with a particularly pronounced advantage at 128K (0.50 versus 0.05).

More importantly, the same constructed memory effectively supports four distinct queries without rebuilding or updating the memory. Unlike ICL, which processes the full context for each query, MoNe reuses the stored fast-weight state, directly validating the shared-context multi-query capability highlighted by the reviewer.

While this is not a full SCBench evaluation, it directly tests the key reuse property motivating the reviewer's suggestion: answering multiple distinct queries against a shared context whose memory representation is constructed only once.

We will incorporate these additional results into the revised manuscript and clarify the corresponding methodology and claims.

Dear Reviewer cW2Q,

As the discussion period approaches its conclusion, we would greatly appreciate your feedback on our rebuttal.

We believe we have directly addressed your principal concerns through additional experiments on incremental memory extension, segment-size efficiency, abstractive summarization, and shared-context multi-query memory reuse.

If you have any further questions or concerns, please do not hesitate to raise them. We are fully committed to addressing them and will respond promptly until the very end of the discussion period.

We would also sincerely appreciate your reconsideration of the initial assessment in light of the new evidence.

Best regards,

Authors of Submission #45685

Reviewer_vjxUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MoNe augments a frozen Transformer with per-layer fast-weight memories. Context is encoded through block-local associative updates, and subsequent queries use the adapted memories without the original context tokens. Meta-training learns the initialization, feature adaptation, update mechanism, and readout. The paper reports length extrapolation and reductions in memory usage and FLOPs.

Critical strengths
  1. Avoiding backbone-wide backpropagation during test-time writing is a useful design objective, and the separation between local deployment updates and outer-loop meta-training is clearly explained.

  2. The reported extrapolation from at most 4K training contexts to 128K evaluation contexts is promising across retrieval and question-answering tasks.

  3. The cost analysis appropriately includes memory writing as well as inference. Comparisons also include YaRN, DCA, and RAG, providing additional context for the long-context results.

Critical weaknesses
  1. The comparisons do not sufficiently isolate task supervision from the memory mechanism. MoNe is meta-trained separately for each task, whereas ICL and its inference-only extensions do not receive comparable task adaptation. PERK is itself meta-learned, but its outer-loop data, budget, and checkpoint selection in this study are insufficiently specified.

  2. The layer-coverage and segment-size ablations do not directly establish how much improvement comes from writing the test context. A QA comparison against meta-trained initialization without writing, and against an incorrect-context memory, would help separate contextual memory use from task adaptation.

  3. Incremental extension is not fully validated. The test-time learning rate depends on the final number of segments S, whereas the final length may be unknown or change in streaming use. Retaining weights and momentum permits further updates, but does not establish accuracy in that regime. The offline extrapolation results therefore provide incomplete support for the broader deployment claims.

Questions for authors
  1. Was PERK meta-trained using the same task data, length mixture, and a comparable outer-loop budget? Can you clarify its checkpoint selection and provide a task-adapted ICL/PEFT control on one representative QA task?

  2. How does QA performance change when context-writing updates are disabled, or when the query receives a memory constructed from an incorrect context?

  3. How is the 1/S learning-rate rule applied when new segments arrive and the final S is unknown? Is a limited append-and-query or multi-query memory-reuse evaluation available?

3 comments

We thank the reviewer for recognizing MoNe's efficient memory-writing design, promising long-context extrapolation, and comprehensive computational cost analysis. We also appreciate the constructive feedback regarding fair comparisons with task-adapted baselines, the contribution of context-dependent memory writing, and the validation of incremental extension and memory reuse.

Q1/W1: Fair Comparison with PERK and Task-Adapted ICL

(a) PERK training and checkpoint selection.

PERK and MoNe were meta-trained using the same task-specific training data, context-length distribution, and outer-loop training budget. Both use the same frozen Qwen2.5-0.5B-Instruct backbone, with training contexts ranging from 1K to 4K tokens and segment counts uniformly sampled from {1, 2, 3, 4}.

Both follow the same outer-loop optimization settings (Appendix C): AdamW, batch size 16, 500 warmup steps, and cosine learning-rate decay. RULER and BABILong are trained for one epoch, while SQuAD and HotpotQA are trained for up to 20 epochs. Checkpoints are selected based on the highest validation score on the same held-out 4K validation split, without using long-context evaluation results.

PERK retains its method-specific architecture and test-time adaptation procedure, including LoRA rank 256 and four inner-loop update iterations (Appendix D). We will clarify these details in the revised manuscript.


(b) Task-adapted ICL baseline.

To isolate the effect of task-specific supervision, we additionally evaluated ICL+SFT, where the same pretrained backbone is fine-tuned with LoRA using task-specific SQuAD training data.

Table 1. Task-adapted ICL comparison on SQuAD.

MethodInference input32K128K
ICLQuery only0.060.06
ICLFull context + Query0.450.04
ICL+SFTQuery only0.020.02
ICL+SFTFull context + Query0.560.10
MoNeQuery only0.620.47

Task-specific fine-tuning improves full-context ICL from 0.45 to 0.56 at 32K and from 0.04 to 0.10 at 128K. Nevertheless, MoNe achieves 0.62 and 0.47, respectively, without requiring the original context tokens during query inference. Moreover, ICL+SFT achieves only 0.02 when the context is removed.

These results indicate that task-specific supervision alone cannot explain MoNe's improvements, highlighting the contribution of context-dependent memory.

 

Q2/W2: Contribution of Context-Dependent Memory Writing

To distinguish contextual memory utilization from meta-learned task adaptation, we conducted additional SQuAD experiments under three conditions: (i) no memory updates, (ii) updates using an unrelated context, and (iii) updates using the correct context. All MoNe conditions use the same meta-trained model.

Table 2. Effect of context-specific memory updates on SQuAD.

Method32K128K
MoNe without memory updates0.060.06
MoNe with unrelated-context updates0.040.04
MoNe with correct-context updates0.620.47

Without updates, MoNe achieves only 0.06 at both lengths, identical to query-only ICL. Updates using unrelated contexts similarly yield only 0.04, whereas correct-context updates achieve 0.62 at 32K and 0.47 at 128K.

These results directly demonstrate that MoNe's QA performance depends on writing relevant context-specific information into memory. The no-update baseline shows that meta-learned initialization and readout alone are insufficient, while the incorrect-context baseline confirms that performing memory updates without relevant information does not explain the improvements.

We will incorporate these ablations into the revised manuscript.

Q3/W3: Unknown Context Length, Incremental Extension, and Memory Reuse

(a) Fixed learning-rate scaling.

We acknowledge that the original 1/S1/S scaling assumes knowledge of the final segment count. To address this limitation, we compared length-adaptive scaling with a fixed learning-rate scale corresponding to 128K, applied uniformly across all evaluated context lengths.

Table 3. Length-adaptive versus fixed learning-rate scaling on SQuAD.

Scaling strategy4K8K16K32K48K64K96K128K
Length-adaptive0.620.650.600.620.590.570.590.47
Fixed0.600.620.590.600.580.570.560.47

Fixed scaling achieves comparable performance across all lengths, with a maximum difference of only 0.03. This demonstrates that strong performance can be maintained without knowing the final context length.

Importantly, fixed scaling allows the same learning-rate scale to be used as new context arrives, without retrospectively adjusting previous updates.


(b) Incremental append-and-query evaluation.

To directly validate incremental memory extension, we conducted a SQuAD experiment in which context is appended in successive 32K-token chunks, each comprising 32 consecutive 1024-token segments. Queries are issued after each append operation, with accumulated context lengths increasing from 32K to 128K.

Throughout the entire sequence, MoNe uses the same fixed learning-rate scale corresponding to 128K, retaining its fast-weight memory and momentum state without resetting or reprocessing previously written context.

We evaluated two settings: (i) the gold paragraph is located within the newly appended 32K-token chunk, and (ii) the gold paragraph is randomly positioned across the accumulated context.

Table 4. Incremental append-and-query performance on SQuAD.

Gold paragraph location32K64K96K128K
Newly appended 32K chunk0.600.580.600.59
Random across accumulated context0.600.570.540.47

Columns indicate cumulative context lengths after successive 32K-token additions.

When the gold paragraph appears in the newly appended chunk, performance remains stable between 0.58 and 0.60, even after accumulating 128K tokens. When it is randomly positioned across the accumulated context, performance reaches 0.47 at 128K, closely matching the offline fixed-scaling results.

These results directly validate incremental memory extension and intermediate querying under fixed scaling, without reconstructing the memory or requiring advance knowledge of the final context length.


(c) Multi-query memory reuse.

We additionally evaluated whether a single constructed memory can support multiple queries. Using SQuAD, we sampled 100 contexts, each paired with four questions (400 questions total).

MoNe writes each context into memory only once and answers all four questions using the same memory, without additional updates or context reprocessing.

Table 5. Shared-context multi-query performance on SQuAD.

Method4K8K16K32K48K64K96K128K
ICL0.590.520.560.490.200.140.090.05
MoNe0.630.610.600.590.590.560.520.50

MoNe consistently outperforms ICL, achieving 0.50 versus 0.05 at 128K.

More importantly, a single context-specific memory can support multiple distinct queries without repeated context processing, validating the practical capability of memory reuse.

We will incorporate these additional experiments and clarify the corresponding incremental deployment claims in the revised manuscript.

Dear Reviewer vjxU,

As the discussion period approaches its conclusion, we would greatly appreciate your feedback on our rebuttal.

We believe we have directly addressed all three principal concerns through fair comparisons with PERK and task-adapted ICL, controlled validation of context-dependent memory writing, and incremental append-and-query experiments.

If you have any further questions or concerns, please do not hesitate to raise them. We are fully committed to addressing them and will respond promptly until the very end of the discussion period.

We would also sincerely appreciate your reconsideration of the initial assessment in light of the new evidence.

Best regards,

Authors of Submission #45685

Reviewer_icswOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoNe for long-context inference with a frozen pretrained Transformer. Through task-specific meta-training, the method learns to write context into layer-wise fast-weight memories using block-localized updates and incorporates memory readouts into attention representations during query inference. Experiments on RULER, BABILong, SQuAD, and HotpotQA demonstrate the effectiveness of the proposed method

Critical strengths

The paper addresses a practically relevant problem: improving the efficiency of long-context inference with frozen pretrained language models. The framework encodes long contexts into neural memory through local updates, enabling subsequent reasoning without the original context as input. Experiments cover retrieval, aggregation, and multi-fact reasoning. The computational comparison includes both memory writing and query inference, providing empirical support for the reported test-time efficiency.

Critical weaknesses

(1) LaCT already uses a SwiGLU fast-weight network, a dot-product association loss, and normalized weight updates, while PERK encodes context into parameters for question answering without the original text. The main addition is to integrate locally updated neural memories into a frozen pretrained Transformer. Nevertheless, the overall design is reasonable. MoNe is meta-trained separately for each task, whereas the ICL baseline directly uses the frozen pretrained model. The performance gains may therefore reflect both additional task supervision and the proposed memory mechanism. A comparison withICL trained on the same task data would isolate the contribution of the memory mechanism. (2) Existing ablations examine only memory placement across layers and segment size on FWE, primarily illustrating performance–computation trade-offs. They do not isolate the contributions of the learned update rule or context-length-based learning-rate scaling. For example, “differenet” in the caption of Table 1 is a spelling error. The legends and axis labels in Figure 5 are difficult to read, and the statistical meaning of the error bars in Figure 4 is not specified.

Questions for authors

see weakness

7 comments

We thank the reviewer for recognizing MoNe's lightweight, modular design and its computational and memory efficiency. We also appreciate the constructive feedback regarding novelty relative to prior neural-memory approaches, fairness of task-specific comparisons, the contributions of individual memory-writing components, and presentation issues.

Q1/W1: Novelty and Fair Comparison with Task-Supervised ICL

(a) Clarifying the contribution relative to prior work.

We agree that several individual components of MoNe, including the SwiGLU fast-weight memory and associative learning objective, build directly upon LaCT. We do not claim these components as novel in isolation.

Rather, our contribution lies in combining three properties that are not jointly supported by the closest prior methods:

  1. Compatibility with frozen pretrained Transformers: Unlike Titans and LaCT, MoNe can be integrated into an existing pretrained Transformer without retraining its backbone or requiring memory modules to be present during backbone pretraining.
  2. Efficient test-time memory writing: Unlike PERK, MoNe uses block-localized updates that do not require backpropagation through the Transformer backbone during context encoding.
  3. Context-free query inference: Unlike methods such as δ-mem that continue processing textual context through the backbone, MoNe encodes context into reusable fast-weight memories, allowing subsequent queries to be answered without the original context tokens.

We acknowledge that this is primarily an architectural and computational contribution rather than an entirely new associative learning algorithm. Nevertheless, enabling these properties simultaneously is meaningful, particularly for applications requiring efficient processing of long contexts with existing pretrained models.

Importantly, the efficiency gains are observed even within the backbone's native context window. At 32K, MoNe reduces peak GPU memory by 43% and total FLOPs by 39% relative to ICL. At 128K, these reductions increase to 80% and 82%, respectively, accounting for both memory writing and query inference (Table 4 of the original submission).


(b) Addressing the fairness of task-specific supervision.

We agree that comparing task-specifically meta-trained MoNe against a frozen ICL baseline does not fully isolate the contribution of memory learning from that of additional task supervision.

To address this concern, we conducted additional experiments with ICL+SFT, where the same pretrained backbone is fine-tuned with LoRA using task-specific SQuAD training data.

Table 1. Task-adapted ICL comparison on SQuAD.

MethodInference input32K128K
ICLQuery only0.060.06
ICLFull context + Query0.450.04
ICL+SFTQuery only0.020.02
ICL+SFTFull context + Query0.560.10
MoNeQuery only0.620.47

Task-specific fine-tuning improves full-context ICL from 0.45 to 0.56 at 32K and from 0.04 to 0.10 at 128K. Nevertheless, MoNe achieves 0.62 and 0.47, respectively, without requiring the original context tokens during query inference. Moreover, ICL+SFT achieves only 0.02 when the context is removed.

These results indicate that task-specific supervision alone cannot explain MoNe's improvements, highlighting the contribution of context-dependent memory.

Q2/W2: Ablations of the Memory-Writing Mechanism

We agree that the original ablations on layer selection and segment size primarily characterize performance–efficiency trade-offs, rather than isolating the contributions of individual memory-writing components.

We therefore conducted three additional analyses on SQuAD.

(a) Contribution of the learned update components.

To evaluate the importance of the token-wise learning rates and segment-level momentum, we progressively incorporate these components while retaining the remaining meta-learning framework.

Table 2. Ablation of learned update components on SQuAD.

Update mechanism32K128K
Without token-wise LR or momentum0.290.16
+ Token-wise learning rates0.550.41
+ Segment-level momentum (Full MoNe)0.620.47

Token-wise learning rates provide the largest improvement, increasing performance from 0.29 to 0.55 at 32K and from 0.16 to 0.41 at 128K. Segment-level momentum provides an additional improvement of 0.07 and 0.06, respectively.

These results indicate that both components contribute to MoNe's effectiveness, with token-wise adaptation being particularly important.

We acknowledge that these experiments isolate components of the learned update mechanism rather than proving the necessity of meta-learning as a whole.


(b) Importance of context-specific memory writing.

We additionally examine whether MoNe's performance depends on encoding the correct context, rather than simply introducing additional memory parameters.

Table 3. Effect of context-specific memory updates on SQuAD.

Method32K128K
MoNe without memory updates0.060.06
MoNe with unrelated-context updates0.040.04
MoNe with correct-context updates0.620.47

Without memory updates, performance drops to the query-only ICL baseline of 0.06. Updating memory using unrelated context similarly produces near-zero performance.

These controls confirm that MoNe's predictions depend on information encoded from the relevant context, rather than solely on the additional learned parameters.

While this property is also expected of other context-encoding approaches, the experiment provides useful validation of MoNe's memory-writing mechanism.


(c) Necessity of context-length-based learning-rate scaling.

To evaluate the necessity of length-adaptive scaling, we additionally compare the proposed strategy against fixed scaling, where the learning-rate scale corresponding to 128K is applied uniformly across all context lengths.

Table 4. Length-adaptive versus fixed learning-rate scaling on SQuAD.

Scaling strategy4K8K16K32K48K64K96K128K
Length-adaptive0.620.650.600.620.590.570.590.47
Fixed0.600.620.590.600.580.570.560.47

Across context lengths from 4K to 128K, the two strategies achieve comparable performance, with differences of at most 0.03. At 128K, both achieve 0.47.

Both strategies achieve comparable performance across all evaluated context lengths, with differences of at most 0.03. These findings indicate that length-adaptive scaling is not essential to MoNe's effectiveness, and that fixed scaling provides a viable alternative across the evaluated context lengths.


(d) Presentation Improvements

We thank the reviewer for identifying the presentation issues.

We will correct the typo "differenet" in Table 1, improve the readability of the legends and axis labels in Figure 5, and explicitly clarify that the error bars in Figure 4 represent 95% Wilson score confidence intervals computed over 200 evaluation samples.

We will also incorporate the additional experiments and corresponding methodological details into the revised manuscript.

Dear Reviewer icsw,

As the discussion period approaches its conclusion, we would greatly appreciate your feedback on our rebuttal.

We believe we have directly addressed your main concerns through a task-adapted ICL comparison, controlled memory-writing ablations, and clarification of MoNe's contributions.

If you have any further questions or concerns, please do not hesitate to raise them. We are fully committed to addressing them and will respond promptly until the very end of the discussion period.

We would also sincerely appreciate your reconsideration of the initial assessment in light of the new evidence.

Best regards,

Authors of Submission #45685

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.