acceptodds
Under review as a conference paper at ICLR 2027

Beyond Overwriting: Selective Memory Integration for Long-Context Reasoning

Abstract

Efficiently processing long contexts remains a key challenge for large language models (LLMs). The memorize-while-reading paradigm addresses this challenge by processing documents chunk by chunk and dynamically updating a textual memory, with subsequent work introducing retrieval to recover lost evidence and gating mechanisms for controlling when to update memory and stop reading. Despite these advances, two challenges remain: (i) repeated overwrite-based updates lack explicit protection for previously stored information, potentially causing information loss that is difficult to recover through retrieval; and (ii) optimization based on rule-based rewards provides only indirect guidance for learning when to update memory and stop reading, which may lead to unstable optimization. To address these challenges, we propose GraftMem, which separates evidence acquisition from incremental memory integration through acquisition–integration (AcIn) updates. Combined with selective memory updating, this design effectively mitigates information loss from repeated overwriting. GraftMem also incorporates an early-exit mechanism to improve efficiency. To strengthen training, we combine multi-level reward shaping with decision supervision: alongside trajectory-level and state-level rewards, we directly supervise memory-update and early-exit decisions. Together, these signals alleviate supervision sparsity and support more stable policy optimization. Extensive experiments demonstrate that GraftMem achieves competitive accuracy on long-context reasoning tasks while substantially accelerating inference, achieving a favorable accuracy–efficiency trade-off.

Primary area:
reinforcement learning
Keywords:
LLM Agent, Reinforcement Learning, Long-Context LLM

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_3uF8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GraftMem to mitigate evidence loss from repeated memory overwriting in long-context reasoning. In this framework, the model processes document chunks sequentially and appends newly extracted evidence to existing memory instead of regenerating the entire memory. Additionally, multi-level GRPO rewards and direct cross-entropy decision supervision are used to train the agent across long-context QA benchmarks.

Critical strengths
  • The append-based memory update design is practical. Replacing full-memory overwriting with incremental evidence appending (Mt−1⊕EtM_{t-1} \oplus E_t, Eq. 5) is simple, prevents previously stored evidence from being lost during regeneration, and reduces per-step decoding length.

  • The internal ablation verifies the append mechanism. The controlled ablation in Table 4 confirms that replacing overwrite-based updates with append-based updates improves QA accuracy at longer context lengths (e.g., 224K and 448K).

Critical weaknesses
  • Problematic academic attribution and experimental comparison. First, the paper claims to introduce the update gate and early-exit mechanism, which were actually proposed by GRU-Mem [1]. The authors are merely following GRU-Mem [1] rather than introducing these gates as their own contribution. Moreover, the entire paper closely follows the experimental setup and figure/table styles of GRU-Mem [1], yet the authors only briefly mention this work without comparing against or discussing it in the experiments. This raises serious concerns regarding proper credit attribution and academic integrity.

  • Overly strong supervision assumption and risk of hallucination. Eqs. 9 and 15 supervise the update and exit gates using static chunk-position labels (Ut⋆=1[Ft≠∅]U_t^\star = \mathbf{1}[F_t \neq \varnothing] and Vt⋆=1[t≥t⋆]V_t^\star = \mathbf{1}[t \ge t^\star]). Even if the model failed to extract an earlier supporting fact and the current memory is incomplete, this supervised training still forces the model to output end at step t⋆t^\star and generate a final answer from insufficient evidence, which may easily induce hallucination.

  • Lack of organic connection between the two proposed components. The two main modifications proposed in this paper—replacing overwriting with append-based updates (AcIn) and adding token-level decision supervision—appear to be independent of each other without an intrinsic connection, reading like a stitched-together combination of two separate tricks.

[1] When to Memorize and When to Stop: Gated Recurrent Memory for Long-Context Reasoning (GRU-Mem).

Questions for authors

Please refer to the weaknesses part. My main concern is that this paper is too similar to GRU-Mem, including all the experiments and visualizations, without comparison, necessary discussion, and contribution seperation. If the author can address my issue. I will raise my scores.

Reviewer_h8NJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GraftMem combines selective evidence appending with direct supervision of memory-update and stopping decisions for long-context QA. Experiments with Qwen2.5-3B/7B on three QA benchmarks report competitive accuracy and faster inference compared with MemAgent and ReMemR1.

Critical strengths
  1. The paper clearly explains the problems of memory overwriting and indirect supervision of update and stopping decisions.
  2. Ablations, stress tests, and failure analyses provide useful validation, including the accuracy–efficiency trade-off of early exit.
Critical weaknesses
  1. The incremental accuracy–efficiency benefits over closely related methods remain insufficiently established, particularly regarding baseline comparisons and overall training and inference costs.
  2. The evaluation provides limited evidence of broader generalization, robustness under sustained memory pressure, and the statistical reliability of the reported gains and training-stability claims.
Questions for authors
  1. Closest baselines and contribution attribution. Why is GRU-Mem, which already incorporates update and exit gates, absent from the evaluation? Could you provide a comparison under matched backbones, training data, supervision, and memory/chunk budgets, or explain the obstacles?

  2. Memory pressure, consolidation, and evidence retention. Appendix C.3 tests overflow using repeated supporting material. How does performance change when the number of distinct necessary facts increases and repeated consolidation becomes necessary? Please report consolidation frequency and evidence retention before and after compression. How is the budget enforced if the consolidated memory remains overlength, and how does the method handle conflicting, corrected, or erroneous earlier information?

  3. Decision supervision and annotation requirements. The Update/Next targets depend on supporting-fact annotations and evidence positions rather than the actual contents of the generated memory. How do these targets handle redundant evidence and failed acquisition of earlier facts? How often does an “exact exit” occur with insufficient memory or an incorrect answer?

  4. Model and task generalization. Evaluation covers two sizes within Qwen2.5 and three Wikipedia-based QA benchmarks with similar long-context construction. What evidence supports generalization to another model family or to QA requiring dense evidence integration or reasoning over naturally long documents? Are the component-ablation gains consistent beyond 3B HotpotQA?

  5. End-to-end inference efficiency and resource usage. For Figure 3, please specify the model scale, hardware, batching/concurrency, decoding settings, averaging procedure, and timing boundaries. Does timing include input processing, all memory updates, retrieval where applicable, consolidation, and final-answer generation?

  6. Overall training cost. Table 2 reports average per-step time, but were all methods trained for the same number of steps under comparable configurations? Please report total wall-clock time and GPU-hours, or time to reach comparable validation quality, and clarify what each timed step includes. Are annotation processing, target construction, and auxiliary CE computation included? What additional cost does decision supervision introduce beyond the existing reward calculations?

  7. Statistical reliability and data protocol. With only 128 examples per benchmark, how robust are the reported improvements? Could you provide paired confidence intervals, results on a larger evaluation sample, and variability across independent training runs, particularly for the training-stability claim? Please also specify the training/validation/evaluation splits, the validation procedure for hyperparameter selection, and whether questions and document arrangements are held consistent across methods at each context length.

Reviewer_o41sOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose GraftMem, a memory agent approach for allowing models to handle longer contexts. GraftMem using multiple training rewards, in particular an early-exit reward in which the agent stops acting and forces the model to produce a final answer before processing all documents in which a context is split, as well as gate and memory rewards which measure if the model chooses to update the memory for the correct documents and if the correct information in each is retained. Altogether, these are trained in a typical RL setup, producing a model that is able to identify correct facts to include in the memory while also exiting when the model is confident that all relevant documents have been processed. The method attains better performance than competitors on HotPotQA, 2WikiMultiHopQA and SQuAD when trained on HotPotQA, with ablations showing the ability to handle different scenarios where the relevant facts can be separated in the overall context.

Critical strengths

The paper is clearly presented, with the motivation of the work clearly stated.

The stated training method is also straightforward to understand. Given the setting and the explanation of how the model is trained relative to the compared baselines, the gains are sensible and fairly expected on the datasets.

Critical weaknesses

The main concern about the work is the setting in which it exists.

First, the setup works on the assumption that there is data which is clearly annotated regarding the relevance of different sections/contexts, such that both the exist reward as well as the memory reward can clearly be trained, as each of them directly depend on the ground-truth supporting facts. This means that the tested settings (HotPotQA, 2WikiMultiHopQA and SQuAD) are rather directly suited to this setting due to the existence of these exact details. The compared baselines MemAgent and ReMemR1 meanwhile do not rely on such an assumption; while ReMemR1 uses an information-gain style loss, this is based on the final ground-truth answer as opposed to intermediate contexts and thus imposes less requirements than the proposed GraftMem approach. Thus there is a major question of robustness beyond the tested settings that the authors fail to directly address. For example, settings such as long-context generation where the relevance of facts to the final answer can be more ambiguous than atomic facts is not tested.

Overall, the approach appears to be something specifically designed for single or multi-hop QA settings where supporting documents are easy to distinguish as opposed to a general long-context approach. Without these, it's hard to see how this approach could be applied given the necessity of each reward component.

Additionally, one major question that the paper doesn't explicitly answer is whether or not the entire gains come from the decision to update the memory or not based on the gating, as if so the method can be considered more simply an information filtering mechanism. This would also bring about further questions about whether or not the learned approach is most useful, as the supervision is again based on a known signal about the existence of relevant information. Instead, Comparing against other approaches for this, for example a comparison of embedded context and query/question (as has been explored in IR research) or querying the relevance to another model, could reveal many of these details and confirm whether or not it is truly the entire approach that is necessary, as the individual components themselves are not novel.

Questions for authors
  • In the Exit Reward (L205-211), the values appear to be chosen arbitrarily or through some type of specific tuning, which also does not take into account the actual number of chunks more than or less than the appearance of the last relevant context. This appears confusing, as the exit should be based on the number of relevant facts that exist within the overall number of contexts processed, rather than a fixed value.
  • This approach appears styled for a RLVR type setting, where explicit binary rewards are expected. Would this scale to other reward scales such as those based on preference scores that extend beyond simply true-or-false? Can this be explicitly tested/verified?
  • In Table 1, is the distinction of in and out-of-distribution only based on what was used for training? This might be misleading, as it is possible that the original/base model already contains a great amount of factual details or knowledge that would be sufficient in answering each task.
  • For Figure 3, the results feel somewhat incomplete as there is no details on how often the model early exits and where the early exit occurs in practice. Furthermore, with the result being that without early exiting achieves a more than 2x speedup relative to the other baselines, it is important to also show how often the memory is actually written to, as this would be relevant to understanding other settings. For example, if all documents have relevant information or there is for some reason a need to write to memory at nearly every step, what would happen to the speed of the approach. Thus having a 2x speedup appears somewhat implausible in this type of setting unless one controller implementation is significantly slower than another. The explanation in L342-350 is lacking on this front and the code does not include any implementations. If the comparison was further made with different setups or implementations this should be documented and the comparison should be more qualified to reflect this fact. Otherwise the authors should implement the direct strategy of these other methods in their own code and make a more fair comparison.
  • In Section 3.3.1, the ablation can be modified to be more convincing. Assuming that the model has learned a controller that can more accurately identify relevant facts (done during the training process), the fact that items are distantly separated should not matter since the controller already has learned to discriminate relevant facts fairly well. Rather there are 2 approaches that would make this more convincing. First, either modify the intermediate documents to be more adversarial, i.e. harder to distinguish from the actual facts, to see how robust the controller truly is. The other is to modify training, such that facts are trained on contexts where the facts are distantly separated, as a way of showing the robustness of the training approach as well. The current results are fairly redundant, as the compared approaches always conduct a memory read-write for every document thus without filtering will naturally accumulate noise on the separating documents.
  • The previous point leads to another question about extrapolation, which is whether the model actually is learning to extrapolate or simply has learned a reasonable filtering approach to information. Given the short nature of HotPotQA facts, the training approach would naturally provide the controller with a lot of signal to learn how to filter due to the abundance of noise or irrelevant facts in each sequence. Thus I would contend that an actual way to measure robustness would be to use very short training contexts, or ones that have very few irrelevant facts relative to the relevant ones, and then testing on longer contexts, as a way of confirming that gains do not solely come from the controller filtering details.
  • The maximum document length utilized is under one million tokens, which falls within the native long-context window of current foundational models; for instance, models such as DeepSeek-V4 already support a context length of one million tokens. Given that base models are increasingly capable of handling such lengths directly, both the necessity and the fundamental innovation of the proposed method are brought into question. Though this isn't a question of the research direction itself, this point could be clarified as a point of the actual impact that can be made to the direction itself, as this is a question of practicality that can be discussed.
  • Another relevant comparison would be to act on fixed-size memories. Given that the document can be very long and the model can still make decisions to store information from irrelevant documents, a feasible robustness check would be to fix the size of the memory and verify if the model by the end is able to identify storing only the relevant documents. This could serve as a better justification of why the gating module has advantages over letting the memory always decide when to read or write.
Reviewer_8MpJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper improves memory-augmented long-context reading by selectively appending new evidence instead of overwriting memory, and by adding direct supervision on when to update and when to stop.

Critical strengths
  1. From the method side. The problem in this paper is clear, and the method corresponds relatively directly to the problem. Because: the old methods repeatedly overwrite memory, which may lose existing evidence; this paper changes to selective appending, and directly supervises “when to remember and when to stop”.

  2. Overall, the experiments provide support for the effectiveness of the method. Compared with MemAgent and ReMemR1, this paper achieves higher accuracy in most settings.

Critical weaknesses
  1. I think the method in this paper is effective, but the novelty of this work cannot be said to be insufficient, yet it still raises some concerns. Because selective updating and early stopping already have related work; the clearer novelty of this paper is concentrated on append-style evidence integration, and on adding direct supervision to the control decisions.

  2. In the experiments, a very critical comparison is missing — the comparison with GRU-Mem; the competitiveness of the method has not been sufficiently verified. GRU-Mem already has update gates and exit gates. This paper discusses it, but does not directly compare with it.

  3. A very important question and confusion is — is the dataset used by the authors not sufficient enough, that is, does the experiment lack datasets that consider complex situations, or does it need to expand beyond the 128 questions? Can the authors add tests for this situation or dataset — to see how much accuracy and advantage can still be retained after the notes are full and repeated compression happens?

Others:
InfMem-4B in the appendix has higher accuracy on 2Wiki at every length, but there is no corresponding time-cost comparison.

Questions for authors

Please refer to the above comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.