acceptodds
Under review as a conference paper at ICLR 2027

MemTrain: Self-Supervised Context Memory Training

Abstract

Memory is an indispensable capability for long-horizon LLM agents, enabling them to preserve and utilize information accumulated across extended interactions. Existing memory-agent approaches typically rely on task-specific end-to-end training. However, collecting high-quality annotated problems for memory-intensive scenarios is costly, and the resulting training data often lack sufficient diversity to cover general memory behaviors. In this work, we propose MemTrain, a self-supervised memory training framework for generally enhancing the context-memory capability of LLM agents. MemTrain introduces two coupled proxy tasks over unlabeled Wikipedia corpora: (1) an end-to-end masked reconstruction objective, which requires the model to recover masked entities after multiple rounds of memory updates, thereby encouraging memory maintenance from the final outcome perspective; and (2) an intermediate memory recall objective, which requires the model to reconstruct masked historical information using intermediate memory states, encouraging faithful compression and memory completeness throughout the interaction process. The two objectives are jointly optimized using GRPO. Extensive experiments on long-text QA and search-based QA benchmarks demonstrate that MemTrain significantly improves downstream memory-intensive reasoning performance across different context lengths and models, achieving gains of up to 17.67 points over direct task-specific post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent memory, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JnEjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MemTrain is a self-supervised training framework for LLM agents that maintain fixed-size textual context memory. It combines end-to-end masked-entity reconstruction with intermediate recall of historical entities on Wikipedia-derived documents. Recall success contributes to the reward for memory-writing trajectories, and both tasks are optimized with GRPO. The resulting checkpoint initializes subsequent MemAgent or MEM1 training. Experiments report improved average performance on long-document and search-based QA across two backbones.

Critical strengths
  1. The framework derives verifiable training targets from raw text, reducing dependence on task-specific QA annotations. Coupling historical recall with trajectory rewards provides a concrete way to train memory generation and utilization jointly.

  2. MemTrain initialization improves average performance across two backbones in both long-document and search-based QA, with gains of 5.17–17.67 and 8.50–10.58 points, respectively. The End-to-End and Decoupled ablations further support the usefulness of the training recipe.

Critical weaknesses
  1. IMR encourages recalling randomly sampled historical entities, but the alignment of this proxy objective with practical memory utility is unclear. Under a fixed memory budget, retaining low-value details may compete with preserving information relevant to future tasks. Although random entity recall encourages broad coverage, the paper does not examine whether it promotes effective selective retention rather than indiscriminate memorization.

  2. IMR provides both an intermediate memory state and a masked historical chunk, so the model may infer the missing entity from the remaining context or pretrained knowledge without relying on memory. Although downstream ablations demonstrate the benefit of IMR training, the paper does not directly evaluate whether reconstruction depends on information preserved in intermediate states. Without memory-removal controls or systematic measurements of historical-fact retention, the claimed improvement in memory preservation remains insufficiently validated.

Questions for authors
  1. What assumptions about future tasks motivate random entity recall? Can the authors analyze whether this objective promotes useful selective retention under a fixed memory budget, potentially by comparing random and relevance-aware entity sampling?

  2. Can the authors report IMR accuracy with and without memory, together with the proportion of sampled historical facts correctly retained in intermediate memory states?

7 comments

Thank you for the helpful comments. We have conducted additional analyses and incorporated them into the manuscript.

[W1.1 & Q1.1] What assumptions about future tasks motivate random entity recall?

Our goal is to use unlabeled text to teach models to write compact memories and recover information from them under a fixed memory budget. Random recall is motivated by the fact that different downstream tasks may need different parts of the same history, and some facts become relevant only in light of later evidence. In our case study, for example, a later chunk identifies Adriana Trigiani as the director of Big Stone Gap, making clear why her previously mentioned location matters.

The E2E objective rewards reconstruction of a single masked entity. IMR adds a historical recall target sampled after the memory has been written. Successful recall trains the model to recover historical information and also contributes to the reward for the memory-writing trajectory. Random sampling provides a range of recall targets without requiring downstream relevance labels. Task-specific post-training then adapts what the model prioritizes within the memory budget.

[W1.2 & Q1.2] Evaluation on Factual Retention

To examine whether IMR promotes effective retention of task-relevant information rather than crowding out the fixed memory budget with irrelevant details, we directly evaluate the memory states on HotpotQA (Qwen3-4B-Instruct-2507) across three dimensions (Section 5.3):

  • Final Fact Retention Rate (FR): The proportion of supporting facts retained in the final memory state.
  • Fact Capture Rate (FC): The proportion of supporting facts immediately integrated into memory upon their appearance.
  • Fact Retention Duration (FD): The relative duration for which supporting facts persist in memory once captured.

Specifically, we employ Qwen3.8-27B as an automated judge to inspect each memory state mtm_t (t∈{1,…,T}t \in \{1, \dots, T\}) and identify which annotated supporting facts from HotpotQA are present. For a given query xx, let S(x)S(x) denote its ground-truth supporting fact set, and let P(mt,a)∈{0,1}P(m_t, a) \in \{0, 1\} be a binary indicator denoting whether memory state mtm_t contains fact a∈S(x)a \in S(x). Furthermore, let ta∈{1,…,T}t_a \in \{1, \dots, T\} denote the turn in which fact aa appears in the input context, and let τa=min⁡{t∈{1,…,T}∣P(mt,a)=1}\tau_a = \min \{ t \in \{1, \dots, T\} \mid P(m_t, a) = 1 \} denote the first turn at which fact aa is recorded in memory. The three metrics for query xx are formally defined as:

FR(x)=1∣S(x)∣∑a∈S(x)P(mT,a),FC(x)=1∣S(x)∣∑a∈S(x)P(mta,a)\text{FR}(x) = \frac{1}{\vert S(x) \vert} \sum_{a \in S(x)} P(m_T, a), \quad \text{FC}(x) = \frac{1}{\vert S(x) \vert} \sum_{a \in S(x)} P(m_{t_a}, a)

FD(x)=1∣Scap(x)∣∑a∈Scap(x)∑t=τa+1TP(mt,a)T−τa\text{FD}(x) = \frac{1}{\vert S_{\text{cap}}(x) \vert} \sum_{a \in S_{\text{cap}}(x)} \frac{\sum_{t=\tau_a+1}^{T} P(m_t, a)}{T - \tau_a}

where Scap(x)={a∈S(x)∣∑t=1TP(mt,a)>0 and τa<T}S_{\text{cap}}(x) = \{ a \in S(x) \mid \sum_{t=1}^{T} P(m_t, a) > 0 \text{ and } \tau_a < T \} represents the subset of supporting facts captured by memory prior to the final turn (with FD(x)=0\text{FD}(x) = 0 if Scap(x)=∅S_{\text{cap}}(x) = \emptyset).

Final Fact Retention Rate (FR, %)

Method7k14k28k56k112k224k448k896kAvg
MemAgent92.8693.1892.8691.8888.3174.5173.0575.0085.21
MemTrain (E2E) + MemAgent87.9988.6484.4277.9279.8774.3574.3568.8379.55
MemTrain + MemAgent95.7894.4896.1093.5192.5389.8084.4287.3491.75

Fact Capture Rate (FC, %)

Method7k14k28k56k112k224k448k896kAvg
MemAgent91.7091.5592.3188.8587.1277.7870.5270.6883.81
MemTrain (E2E) + MemAgent90.1491.0890.8489.3587.7388.5788.4385.7188.98
MemTrain + MemAgent98.1994.7897.3496.2493.7093.4989.3388.8994.00

Fact Retention Duration (FD, %)

Method7k14k28k56k112k224k448k896kAvg
MemAgent94.6995.2893.6494.2595.5291.4992.4692.7593.76
MemTrain (E2E) + MemAgent93.0492.9290.2987.1586.2783.5383.1281.4887.23
MemTrain + MemAgent94.8996.5997.3596.7795.2692.7291.6093.3194.81

Under the same memory budget, adding IMR to the E2E objective improves average fact capture from 88.98% to 94.00% and retention after capture from 87.23% to 94.81%. Final retention of gold supporting facts increases from 79.55% to 91.75%. Compared with directly trained MemAgent, the full model captures more gold supporting facts when they first appear (94.00% vs. 83.81%) and maintains comparable retention after capture (94.81% vs. 93.76%). Overall, the random recall objective improves the coverage of task-relevant evidence in the final memory under a fixed budget.

[W2 & Q2] IMR accuracy with and without memory.

To verify whether IMR reconstruction relies on the intermediate memory state, we sample 800 end-to-end trajectories using MemTrain-Qwen3-4B-Instruct-2507 on unseen data constructed in the same way as the training set, from which we collect 6,400 IMR reconstruction pairs evaluated with and without memory (Appendix C.3).

SettingIMR Accuracy (%)
w/o Memory13.89
w/ Memory18.70

The results show that historical entity reconstruction is a very hard task (E2E accuracy is about 30% for reference). We admit that the model can infer some masked entities from the chunk itself, but over 86% of the entities still cannot be inferred without memory. With memory, the IMR accuracy improves to 18.70%, showing that the memory really matters in the IMR task. Among these sampled memories and entities, about 2.4% of the target entities are literally contained in the memory. This is reasonable because it is impossible to store all entities from a 40k-token document in a 1k-token memory. However, our purpose is not to put as many entities as possible in the memory, but to store compressed and restorable information for later reasoning. From this perspective, IMR indeed takes effect.

Reviewer_33z5OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
1(−2)
Read review
Paper summary

This paper proposes MemTrain, a self-supervised training framework to improve the memory capability of LLM agents. The method is based on the existing fixed-size context memory setting, where the LLM keeps updating a limited-size textual memory while reading new information.

MemTrain adds a training stage before the existing task-specific training methods such as MemAgent and MEM1. It uses Wikipedia passages to construct two tasks: end-to-end masked entity reconstruction and intermediate memory recall. The first asks the model to recover a masked entity after reading multiple chunks and updating its memory. The second asks the model to recover another historical entity using an intermediate memory state. Both are trained jointly with GRPO.

The authors evaluate the method on long-context QA and search-based multi-hop QA. The results show that models trained with MemTrain before MemAgent or MEM1 achieve higher QA accuracy than models directly trained with MemAgent or MEM1.

Critical strengths

1. Using self-supervised data for memory-related training is an interesting direction.

The paper tries to avoid relying on expensive annotated QA data and instead constructs training tasks from Wikipedia. I think this is a potentially useful direction, especially if the goal is to learn some general memory behavior before training on specific downstream tasks.

2. The empirical improvements are quite significant.

The reported improvements on both long-context QA and search-based QA are substantial. The method also shows improvements across different context lengths and backbone models. Although I have concerns about what exactly causes these improvements, the empirical results themselves are interesting.

Critical weaknesses

1. I have great difficulty understanding what exactly is the research problem being proposed, and why the proposed method should solve this problem.

My main concern starts from the Introduction. The paper argues that existing memory agents rely on task-specific training and that final-answer rewards cannot guarantee memory completeness. But I do not quite understand how these observations directly motivate the proposed training objectives.

For example, why should a memory agent preserve more historical information? In a fixed-size memory setting, isn't the purpose of memory management exactly to decide which information should be kept and which information can be discarded? If some historical information is not useful for the current task, forgetting it may actually be the correct behavior.

The paper seems to assume that preserving more historical entities means better memory management, but I do not see this assumption being sufficiently justified.

This also makes the two proposed training tasks confusing to me. I understand technically how E2E masked reconstruction and IMR are constructed, but I am not convinced why these particular tasks correspond to the important problems in memory management. Why is randomly recovering a historical entity a good proxy for learning general memory management? What specific failure of existing memory agents is this task supposed to address?

My impression is that the paper first designs two self-supervised training tasks and then uses the general importance of memory to motivate them. However, I would expect the paper to first clearly identify a concrete problem in existing memory agents, explain why this problem matters, and then show how the proposed training objectives are designed to address it.

Currently, I find this connection missing. The problem formulation feels somewhat arbitrary, and it is difficult for me to understand why the proposed approach is a necessary or well-motivated solution rather than simply another auxiliary training objective that happens to improve performance.

2. I am not convinced that the experiments actually demonstrate improved memory management capability.

This is another fundamental concern for me.

As I understand it, MemTrain is not a new memory module or a new memory management architecture. It is an additional training procedure applied to the same underlying LLM, followed by the existing MemAgent or MEM1 training.

Then, what the experiments essentially show is that adding this training stage improves the final QA accuracy.

But how do we know the improvement actually comes from better memory management?

The training tasks are both based on recovering masked entities from Wikipedia texts, and the evaluation is mainly based on final QA accuracy. I do not see a direct evaluation of the memory itself. For example, does the trained model actually preserve more task-relevant information? Does it lose less important information during memory updates? Is it better at selecting what should be kept under the same memory budget?

These seem to be the actual questions if the paper claims to improve memory management.

Also, since the whole LLM is being trained, the performance improvement could come from better entity prediction, improved reasoning, additional factual knowledge, or simply the benefit of extra training. I do not think the current experiments clearly distinguish these possibilities.

The paper provides a case study where MemTrain helps retain a useful fact that MemAgent forgets, which is relevant, but one example is not enough to establish the mechanism behind the overall improvements.

So I think there is a gap between what the experiments demonstrate and what the paper claims. The experiments show that this additional training procedure improves downstream QA performance, but I am not sure they demonstrate that the model has learned a generally better memory management capability.

This distinction is important to my assessment because the claimed contribution of the paper is specifically about memory training, not simply improving QA performance through additional self-supervised RL.

Questions for authors
  1. What exactly is the memory management problem that MemTrain is trying to solve?
  2. Why is random historical entity reconstruction an appropriate training objective for memory management?
  3. How can the authors establish that MemTrain improves memory management itself, rather than simply improving the underlying LLM through additional training?
  4. Is there a direct quantitative evaluation of the memory states generated by the models?
3 comments

We sincerely thank the reviewer for the constructive comments. We address each point below and update the manuscript accordingly.

[W1 & Q1] What is the research problem of MemTrain?

Our research problem is whether LLMs can learn transferable memory-related capabilities from unlabeled corpora, serving as a general initialization before downstream agentic memory post-training.

MemTrain is motivated by capturing the foundational capability shared across memory agents: writing a compact and restorable memory while fully utilizing the fixed memory budget. This is particularly beneficial in multi-hop tasks where information relevance is delayed (e.g., Case Study, where Chunk 1 mentions Adriana Trigiani's location before Chunk 2 reveals she directed "Big Stone Gap"), as standard models often prematurely discard useful early clues. After MemTrain, these delayed clues are much more likely to be preserved because the model learns to pack more information into memory and actively utilize the remaining budget to retain potentially useful context.

Because MemTrain is not designed for a single downstream task, we do not constrain it to memorize only narrow, task-specific information. Downstream post-training (such as MemAgent or MEM1) subsequently adapts these general capabilities to task-specific selection.

[Q2] Why is random historical entity reconstruction an appropriate training objective?

Entity reconstruction shares the core process of downstream long-document QA: the model must identify relevant clues, update its memory across turns, and reason over the memory for subsequent updates and final prediction.

Specifically, end-to-end entity reconstruction serves as the primary objective, requiring the model to aggregate clues related to the masked target entity across the document. Through this process, the model learns both how to write a memory and how to read from it. IMR serves as a secondary objective designed to encourage the model to preserve additional meaningful details on top of collecting target-related information. Intuitively, we do not want to waste the fixed memory budget: as long as identified task-relevant information is not compromised, retaining more historical details is beneficial because they may become useful in later hops. This also pushes the model to compress representations more compactly—effectively expanding the usable memory capacity—so that more information can be stored and useful clues are more easily retained across turns.

[W2&Q3] How can we disentangle the improvements attributable to memory from those arising from gains in other capabilities?

We conduct a memory-swapping ablation on Qwen3-4B-Instruct-2507 across HotpotQA by decoupling the memory update steps from the final answer prediction (added in Section 5.3): (1) MemAgent generates the final answer based on the final memory produced by MemTrain+MemAgent, and (2) MemTrain+MemAgent generates the final answer based on the final memory produced by MemAgent. Below is the accuracy difference (Δ\Delta) compared with each model using its own memory:

Memory GeneratorAnswer Model7k14k28k56k112k224k448k896kAvg Δ\Delta
MemTrain+MemAgentMemAgent+0.78+4.69+6.25+4.69+3.12+4.69+4.69+6.25+4.40
MemAgentMemTrain+MemAgent-0.78-2.34-3.12-3.91-0.78-7.81-4.69-3.91-3.42

MemAgent benefits substantially (+4.40 on average) from using MemTrain+MemAgent's final memory, whereas MemTrain+MemAgent suffers a clear performance drop (-3.42 on average) when given MemAgent's memory. With the answer model held fixed, replacing MemAgent’s final memory with that of MemTrain + MemAgent improves accuracy by 3.42–4.40 percentage points on average. This shows that the generated memory accounts for a substantial part of the improvement on long-context HotpotQA.

[W2&Q4] Is there a direct quantitative evaluation of the memory states generated by the models?

We evaluate the quality of memory states on HotpotQA (Qwen3-4B-Instruct-2507) across three dimensions (added in Section 5.3):

  • Final Fact Retention Rate (FR): The proportion of supporting facts retained in the final memory state.
  • Fact Capture Rate (FC): The proportion of supporting facts immediately integrated into memory upon their appearance.
  • Fact Retention Duration (FD): The relative duration for which supporting facts persist in memory once captured.

Specifically, we employ Qwen3.8-27B as an automated judge to inspect each memory state mtm_t (t∈{1,…,T}t \in \{1, \dots, T\}) and identify which annotated supporting facts from HotpotQA are present. For a given query xx, let S(x)S(x) denote its ground-truth supporting fact set, and let P(mt,a)∈{0,1}P(m_t, a) \in \{0, 1\} be a binary indicator denoting whether memory state mtm_t contains fact a∈S(x)a \in S(x). Furthermore, let ta∈{1,…,T}t_a \in \{1, \dots, T\} denote the turn in which fact aa appears in the input context, and let τa=min⁡{t∈{1,…,T}∣P(mt,a)=1}\tau_a = \min \{ t \in \{1, \dots, T\} \mid P(m_t, a) = 1 \} denote the first turn at which fact aa is recorded in memory. The three metrics for query xx are formally defined as:

FR(x)=1∣S(x)∣∑a∈S(x)P(mT,a),FC(x)=1∣S(x)∣∑a∈S(x)P(mta,a)\text{FR}(x) = \frac{1}{\vert S(x) \vert} \sum_{a \in S(x)} P(m_T, a), \quad \text{FC}(x) = \frac{1}{\vert S(x) \vert} \sum_{a \in S(x)} P(m_{t_a}, a)

FD(x)=1∣Scap(x)∣∑a∈Scap(x)∑t=τa+1TP(mt,a)T−τa\text{FD}(x) = \frac{1}{\vert S_{\text{cap}}(x) \vert} \sum_{a \in S_{\text{cap}}(x)} \frac{\sum_{t=\tau_a+1}^{T} P(m_t, a)}{T - \tau_a}

where Scap(x)={a∈S(x)∣∑t=1TP(mt,a)>0 and τa<T}S_{\text{cap}}(x) = \{ a \in S(x) \mid \sum_{t=1}^{T} P(m_t, a) > 0 \text{ and } \tau_a < T \} represents the subset of supporting facts captured by memory prior to the final turn (with FD(x)=0\text{FD}(x) = 0 if Scap(x)=∅S_{\text{cap}}(x) = \emptyset).

Final Fact Retention Rate (FR, %)

Method7k14k28k56k112k224k448k896kAvg
MemAgent92.8693.1892.8691.8888.3174.5173.0575.0085.21
MemTrain (E2E) + MemAgent87.9988.6484.4277.9279.8774.3574.3568.8379.55
MemTrain + MemAgent95.7894.4896.1093.5192.5389.8084.4287.3491.75

Fact Capture Rate (FC, %)

Method7k14k28k56k112k224k448k896kAvg
MemAgent91.7091.5592.3188.8587.1277.7870.5270.6883.81
MemTrain (E2E) + MemAgent90.1491.0890.8489.3587.7388.5788.4385.7188.98
MemTrain + MemAgent98.1994.7897.3496.2493.7093.4989.3388.8994.00

Fact Retention Duration (FD, %)

Method7k14k28k56k112k224k448k896kAvg
MemAgent94.6995.2893.6494.2595.5291.4992.4692.7593.76
MemTrain (E2E) + MemAgent93.0492.9290.2987.1586.2783.5383.1281.4887.23
MemTrain + MemAgent94.8996.5997.3596.7795.2692.7291.6093.3194.81

The ablation shows how E2E reconstruction and IMR affect fact capture and retention.

  • E2E-only initialization improves fact capture, with lower post-capture retention. Compared with directly trained MemAgent, MemAgent initialized with E2E-only MemTrain captures more supporting facts at 896k tokens (85.71% vs. 70.68%) and on average (88.98% vs. 83.81%). However, post-capture retention drops from 93.76% to 87.23%, and final fact retention drops from 85.21% to 79.55%.
  • Adding IMR improves both capture and retention. Successful historical recall contributes to the reward for the memory-writing trajectory. Compared with the E2E-only variant, the full model increases average fact capture from 88.98% to 94.00%, post-capture retention from 87.23% to 94.81%, and final fact retention from 79.55% to 91.75%.

Dear Reviewer 33z5,

We sincerely appreciate your valuable comments and constructive feedback, which have greatly helped us improve our manuscript.

As the discussion period is coming to an end soon, we would like to engage in further discussion with you if time permits. Could you please let us know whether our response and the newly added experiments have adequately addressed your concerns, or if you have any remaining questions we can clarify before the deadline?

Thank you very much for your time and review!

Reviewer_pTYpOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

MemTrain introduces a self-supervised training stage for context-memory agents using unlabeled Wikipedia text. It combines end-to-end masked entity reconstruction with intermediate memory recall and couples their rewards through GRPO. The resulting checkpoint is subsequently trained with MemAgent for long-document QA or MEM1 for search-based QA. Experiments on two model backbones show improvements over directly applying these downstream training methods.

Critical strengths

The paper addresses a practical problem: training agents to preserve useful information across repeated memory updates without requiring manually annotated examples for the memory-training stage. The training pipeline is clearly motivated, and the complementary objectives are intuitive. Rewarding intermediate recall provides an explicit incentive to retain historical details beyond those immediately necessary for the final prediction. Evaluation covers both document reading and search-based interaction, providing encouraging evidence of transfer between these settings. The ablations support the usefulness of intermediate recall and reward coupling. The additional downstream-training comparison also helps assess whether longer task training alone explains the gains.

Critical weaknesses

The baseline coverage is limited to MemAgent and MEM1. These comparisons establish that MemTrain benefits two existing frameworks, but leave its competitiveness against other representative memory and RAG approaches unclear. Because the contribution is a training procedure, comparisons with alternative training objectives under comparable computational budgets would also be valuable.

More importantly, the evidence for improved information preservation under a fixed memory budget remains indirect. The context-length evaluation is relevant, but the paper does not clearly isolate increasing distractor content from evidence placement and repeated memory compression. A controlled experiment keeping questions and supporting evidence fixed while increasing irrelevant context would better substantiate the proposed mechanism. Varying memory capacity would further characterize the accuracy–memory trade-off. The existing QA results and qualitative example do not directly measure factual retention or compression faithfulness.

Some reporting could also be strengthened. The paper specifies the 30,000-document memory-training corpus but does not clearly report downstream training and evaluation sample counts. Although three seeds are mentioned for long-document experiments, the main results lack variability estimates. Claims of improvement at every context length should also acknowledge the Qwen3-4B exception at 448k tokens.

Questions for authors

Does MemTrain retain its advantage when only distractor volume or the distance between evidence and the final question increases?

How sensitive are the gains to memory capacity, and can the authors report factual retention alongside final QA accuracy?

How does it compare with additional strong memory/RAG baselines and alternative training procedures under matched budgets?

2 comments

Thank you for the constructive feedback and insightful suggestions. We have conducted additional experiments to address your questions and incorporated these new results into the revised manuscript.

[Q1] Does MemTrain retain its advantage when only distractor volume or the distance between evidence and the final question increases?

To isolate the effects of distractor volume and evidence placement, we keep the questions and supporting evidence fixed while varying document length from 7k to 896k across three position settings: Head, Middle, and Tail, where the supporting evidence is placed within the first 10%, middle 10%, and last 10% of the document, respectively. We evaluate Qwen3-4B-Instruct-2507 on HotpotQA:

PositionMethod7k14k28k56k112k224k448k896kAvg
HeadMemAgent78.1280.4774.2275.0075.7871.8869.5368.7574.22
HeadMemTrain + MemAgent80.4780.4778.9179.6975.7873.4472.6672.6676.76
MiddleMemAgent77.3473.4472.6671.8871.0962.5060.1658.5968.46
MiddleMemTrain + MemAgent82.0378.1272.6672.6671.8868.7567.9767.1972.66
TailMemAgent73.4474.2272.6667.1960.1651.5653.1246.8862.40
TailMemTrain + MemAgent78.1278.1274.2272.6667.9766.4164.8464.0670.80

The results reveal two key patterns:

  1. Evidence appearing in later turns (Tail) is more challenging than in early turns (Head): By the time the model reaches the tail of a long document, its memory state has already accumulated substantial distractor information from preceding turns, which interferes with capturing new evidence and answering at the final step.
  2. Consistent and enlarging gains across all positions: MemTrain + MemAgent consistently outperforms MemAgent across all three placement settings (+2.54% on Head, +4.20% on Middle, and +8.40% on Tail on average, reaching +17.18% on Tail at 896k) and degrades much less as distractor volume grows.

[Q2.1] How sensitive are the gains to memory capacity?

We evaluate MemAgent and MemTrain + MemAgent (Qwen3-4B-Instruct-2507) under maximum memory budgets of 512, 1024, and 2048 tokens on long-document QA (averaged across context lengths):

Method512 Tokens1024 Tokens2048 Tokens
MemAgent56.4565.1467.19
MemTrain + MemAgent70.0270.3170.60

MemAgent is sensitive to memory capacity, dropping by 8.69% when the budget is reduced from its 1024-token training setting to 512 tokens. In contrast, MemTrain + MemAgent remains remarkably stable across budgets. Notably, MemTrain + MemAgent with a 512-token budget (70.02%) outperforms MemAgent even with a 2048-token budget (67.19%), confirming that MemTrain teaches the model to write compact representations and utilize limited memory space much more efficiently.

[Q2.2] Can the authors report factual retention alongside final QA accuracy?

We directly evaluate factual retention in Section 5.3 of the revised manuscript using Final Fact Retention Rate (FR), which measures the proportion of ground-truth supporting facts preserved in the final memory state mTm_T.

Specifically, we employ Qwen3.8-27B as an automated judge to inspect the final memory state mTm_T and identify which annotated supporting facts from HotpotQA are present. For a given query xx with ground-truth supporting fact set S(x)S(x), let P(mT,a)∈{0,1}P(m_T, a) \in \{0, 1\} denote whether the final memory state mTm_T contains fact a∈S(x)a \in S(x). We define: FR(x)=1∣S(x)∣∑a∈S(x)P(mT,a).\text{FR}(x) = \frac{1}{\vert S(x) \vert} \sum_{a \in S(x)} P(m_T, a).

Results on HotpotQA (Qwen3-4B-Instruct-2507) are shown below (more detailed intermediate retention analyses across turns, including Fact Capture Rate and Fact Retention Duration, are also presented in Section 5.3 and our response to Reviewer [33z5] [Q4]):

Method7k14k28k56k112k224k448k896kAvg
MemAgent92.8693.1892.8691.8888.3174.5173.0575.0085.21
MemTrain + MemAgent95.7894.4896.1093.5192.5389.8084.4287.3491.75

MemTrain + MemAgent consistently improves final factual retention across all context lengths (+6.54% on average, and +12.34% at 896k), directly verifying improved information preservation under a fixed memory budget.

[Q3] Comparison with additional memory/RAG baselines and alternative training procedures

  1. Additional RL Memory Framework (ReMemR1): We evaluate MemTrain combined with ReMemR1 on long-document QA using Qwen3-4B-Instruct-2507 (without hyperparameter tuning due to the rebuttal timeframe):
Method7k14k28k56k112k224k448k896kAvg
ReMemR172.1469.5369.0165.1064.5862.2462.5059.3865.56
MemTrain + ReMemR173.4471.8871.0969.5368.7563.2862.5061.7267.77

MemTrain consistently improves ReMemR1 (+2.21% on average), demonstrating that its benefits generalize beyond MemAgent and MEM1.

  1. RAG and External Memory Baselines (IRCoT and A-MEM): On search-based QA (Qwen3-4B-Instruct-2507), we additionally compare against IRCoT (which interleaves chain-of-thought reasoning with iterative retrieval) and A-MEM (an agentic external memory system that dynamically organizes and retrieves structured memory notes). All methods are evaluated under the exact same context length limit, search turn budget, and retrieval top-kk:
MethodTriviaQABamboogleHotpotQANQPopQA2WikiMuSiQueAvg
IRCoT26.1112.1012.0211.2021.139.654.0113.75
A-MEM43.3529.1720.2519.0220.9813.015.7021.64
MEM144.2923.3918.8021.9723.6212.805.6321.50
MemTrain + MEM155.6334.6827.8532.2437.9125.8410.4332.08

While A-MEM performs comparably to MEM1, MemTrain + MEM1 outperforms both IRCoT and A-MEM by a wide margin, showing that pre-training transferable context memory capabilities yields substantial gains over standard RAG and external memory pipelines.

Clarifications on Reporting

We thank the reviewer for pointing out these helpful reporting details. For downstream dataset sizes, the long-document QA setting uses 32,768 training samples and 128 test questions per context length, while the search-based QA setting uses 84,807 training samples and 25,854 test samples across all evaluation benchmarks. Across random seeds, the standard deviations of our long-document QA results are generally around 1%–2%, and we also acknowledge the minor exception on Qwen3-4B at 448k tokens. Once the remaining ReMemR1 evaluations across all backbones are completed, we will reorganize the experimental layout in the final revision to include these dataset statistics, report the exact per-setting standard deviations, and revise the wording in Section 4 to reflect the 448k case.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviews and Rebuttal

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and Reviewers pTYp, 33z5, and JnEj for the time and care you have devoted to our paper. Below, we summarize the strengths noted in the reviews, the main concerns raised, and how we addressed them.

Recognized Strengths

  • Annotation-free, self-supervised memory training (pTYp, 33z5, JnEj). All three reviewers saw value in training general context-memory capabilities on unlabeled Wikipedia text without task-specific QA annotations. pTYp found the pipeline clearly motivated and its complementary objectives intuitive, while JnEj highlighted coupling historical recall with trajectory rewards to jointly train memory writing and utilization.
  • Substantial gains across tasks and backbones (pTYp, 33z5, JnEj). Reviewers noted strong improvements on long-document QA (5.17–17.67 points) and search-based QA (8.50–10.58 points) across two backbones. pTYp highlighted the encouraging cross-setting transfer, and both pTYp and JnEj noted that the ablations supported our design.

Concerns and How We Addressed Them

Lack of direct evidence that better memory management accounts for the QA gains (33z5, JnEj, pTYp)

  • We conducted a memory-swapping ablation on HotpotQA. Holding the answering model fixed, switching from MemAgent’s memory to MemTrain + MemAgent’s memory improved accuracy by 4.40 points (MemAgent answering) and 3.42 points (MemTrain + MemAgent answering), showing that the generated memory accounts for a substantial part of the gain.
  • We directly evaluated memory-state quality against HotpotQA’s annotated supporting facts. Compared with MemAgent, MemTrain + MemAgent captured more supporting facts upon appearance (94.00% vs. 83.81%) and retained more in the final memory (91.75% vs. 85.21%). Adding IMR to E2E-only training also improved retention after capture and raised final retention from 79.55% to 91.75%.
  • We tested IMR’s dependence on intermediate memory on held-out examples: removing the intermediate memory dropped recall accuracy from 18.70% to 13.89%, confirming reliance on preserved memory.

Unclear rationale for random historical entity recall and risk of crowding out important facts under a fixed budget (33z5, JnEj)

  • We clarified that MemTrain uses unlabeled text to teach compact memory writing and recovery before downstream post-training. In multi-hop tasks, early clues often gain relevance only after later evidence appears; sampling historical recall targets encourages preserving such clues, while downstream post-training adapts task-specific prioritization.
  • We clarified the priority hierarchy under a fixed budget: E2E is the primary objective securing identifiable task-relevant clues, whereas IMR is secondary (λ=0.5\lambda=0.5) and increases information density in the remaining space. We acknowledged that random masking does not explicitly model fact importance and leave this for future work.

Long-context evaluation conflates distractor volume with evidence placement and lacks memory-capacity analysis (pTYp)

  • We ran a controlled HotpotQA experiment holding questions and supporting facts fixed while varying distractor volume and evidence placement. MemTrain + MemAgent outperformed MemAgent by 2.54, 4.20, and 8.40 points on average for head, middle, and tail placements, with the largest gains after reading most distractors.
  • We evaluated memory-budget sensitivity across 512, 1,024, and 2,048 tokens: MemTrain + MemAgent remained stable (70.02% to 70.60%), and even with a 512-token budget (70.02%) it outperformed MemAgent using 2,048 tokens (67.19%).

Limited comparisons with representative memory/RAG approaches and alternative training objectives (pTYp)

  • We added ReMemR1 on long-document QA (complementing our objective and extended-training ablations), where MemTrain initialization improved ReMemR1’s average accuracy from 65.56% to 67.77%, showing transfer to a third downstream framework.
  • On search-based QA under matched context and retrieval limits, MemTrain + MEM1 (32.08%) outperformed both IRCoT (13.75%) and A-MEM (21.64%).

Incomplete reporting of dataset sizes, seed variability, and an overlooked exception at 448k (pTYp)

  • We reported downstream training and evaluation sample counts, clarified that seed standard deviations on long-document QA are generally 1–2 points, and acknowledged the Qwen3-4B exception at 448k. Exact per-setting standard deviations, full ReMemR1 results, and revised wording for this exception will be included in the final revision.

Thank you again to the AC and all reviewers for your effort and constructive feedback. We remain available and welcome any further discussion before the deadline.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.