acceptodds
Under review as a conference paper at ICLR 2027

All-in-One Memory: Process-Rewarded Reinforcement Learning for Memory Agents

Abstract

Large language models increasingly rely on external memory to answer queries grounded in long-horizon histories such as multi-session dialogues and lengthy documents. However, existing memory methods often optimize retrieval and memory construction separately, without a shared reward space for evaluating search, writing, and stopping decisions. To address this, we propose **A**ll-**i**n-**O**ne **Mem**ory (**AiO-Mem**), which organizes these operations as transitions over a shared evidence state and directly evaluates the semantic quality of each action within a unified reward space. AiO-Mem compiles heterogeneous histories into source-traceable multi-view events, providing a shared substrate for retrieval and memory writing. A unified memory policy actor then rolls out complete interaction trajectories over this shared evidence state. A separate trajectory-conditioned process reward model is trained through evidence-grounded supervised fine-tuning and trajectory-level DPO to assign process scores to the collected trajectories. These scores are aggregated into an action-balanced trajectory reward and used by **P**rocess-**A**ligned **DAPO** (**PA-DAPO**) to optimize the memory policy. Across long-context memory benchmarks, AiO-Mem consistently improves answer accuracy over both static memory systems and RL-trained memory agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM memory, RLAIF, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DU1JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AiO-Mem learns how to gather evidence from a long history before answering a question. It first compiles the history into source-traceable events with several retrieval representations. A memory agent then repeatedly searches this event bank, adds useful information to working notes, skips unhelpful results, or stops. A separate reward model evaluates these operations using the full interaction trajectory. Its step scores are combined into a trajectory reward, which trains the agent through PA-DAPO. Experiments on LoCoMo, MemoryAgentBench, and PersonaMem report gains over static and RL-trained memory systems.

Critical strengths
  1. The memory operations are explicit and grounded in source evidence. The agent maintains a question, working notes, and retrieved candidates, and acts through search, add, skip, and stop operations. Event locators connect the resulting notes to the original history. This makes the workflow inspectable and provides a concrete basis for evaluating whether an operation contributes useful evidence.

  2. The evaluation covers several forms of memory use. The paper tests conversational memory, broader memory-agent tasks, and dynamic user profiling rather than relying on a single benchmark. Ablations examine the event representations and reward-model training stages, while a human assessment checks agreement with the teacher’s scores. Together, these experiments provide useful initial evidence for the integrated system.

Critical weaknesses
  1. Step-level scoring does not clearly translate into step-level credit assignment. The reward model assigns a score to each memory operation, but Section 3.5 averages these scores into one trajectory reward. Equation (5) then uses the same trajectory-level advantage for every generated action token. A useful search and an incorrect note within the same trajectory therefore receive the same advantage signal. With rewards and masking fixed, the additional action index appears to regroup the token-level DAPO objective [1]. The paper should distinguish its reward-model contribution from a new optimization algorithm.

  2. The evaluation may mix improvements in memory management with improvements in the answerer. The memory agent produces working notes, which a separate answer model uses to generate the final response. Appendix C states that the LoCoMo answerer is fine-tuned on the training split, but does not clearly establish that every baseline uses the same checkpoint and inference configuration. The 1:1:8 split also needs to specify whether entire conversations are separated. These details are necessary to attribute the final QA gains to memory-policy learning.

  3. The benefit of learning a reward model is not isolated from the supervision used to create it. The reward model is first trained on rule-derived evidence scores and then aligned to a stronger model’s scoring sequences. The reported human agreement with the teacher is encouraging, but evaluates the teacher rather than fully establishing the trained reward model’s reliability. Removing its training stages also does not show whether it outperforms directly using the rules or teacher scores. A matched comparison is needed to justify this additional learned component.

  4. The scope of unified memory learning is narrower than the framing suggests. The persistent event bank is constructed by a separate compiler before the learned agent gathers query-specific working notes. It is therefore unclear which persistent memory-construction decisions are optimized jointly with retrieval. This distinction matters because Mem-T [2] already jointly learns memory construction and retrieval, while Fine-Mem [3] provides fine-grained feedback for memory operations. Both are cited, but the additional capability supplied by AiO-Mem’s unification needs clearer evidence.

  5. Bounded query processing does not imply history-independent total cost. The method limits the number of retrieval results and interaction steps, which bounds the text processed by the query-time agent. However, the full history must still be compiled and indexed, and the paper’s own complexity expression retains a history-dependent compilation term. The claim of cost invariance should identify the specific component being bounded and distinguish it from total or amortized system cost.

[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. 2025.

[2] Mem-T: Densifying Rewards for Long-Horizon Memory Agents. 2026.

[3] Fine-Mem: Fine-Grained Feedback Alignment for Long-Horizon Memory Management. 2026.

Questions for authors

Please refer to the weakness.

10 comments

Dear Reviewer DU1J,

We are deeply grateful for your valuable time, great suggestion and affirmation. Below are our detailed responses to your questions:

W1: Step-Level Credit Assignment and the Distinction from Token-Level DAPO

Step-level scoring does not clearly translate into step-level credit assignment. With rewards and masking fixed, the additional action index appears to regroup the token-level DAPO objective [1]. The paper should distinguish its reward-model contribution from a new optimization algorithm.

Thank you for raising such a professional and insightful question. We would like to clarify these distinctions below:

In AiO-Mem, step-level reward evaluation and trajectory-level policy optimization serve different purposes. The unified reward model evaluates individual memory actions, while PA-DAPO uses these evaluations to construct an action-balanced trajectory objective. This design addresses the interdependence of memory operations and their uneven frequencies and lengths. Let's clarify three aspects:

First, PA-DAPO's distinction from standard DAPO. The shared reward space introduces a challenge: frequent search and add operations may dominate sparse but critical stop decisions. PA-DAPO addresses this through action-balanced reward aggregation Eq\. 4 and a (group, action, token) optimization hierarchy Eq\. 5. Compared with DAPO, PA-DAPO explicitly structures process reward aggregation and policy optimization around memory actions, rather than treating the trajectory as a flat token sequence.

Second, the role of step-level rewards. Our reward model evaluates search, add, skip, and stop individually, while Eq. 44 balances their scores across action types to construct the trajectory reward. These process scores therefore shape the shared advantage through action-balanced aggregation, rather than producing separate advantages for individual actions.

Third, the choice of trajectory-level advantages. Memory actions are interdependent, and different rollouts produce different intermediate evidence states. Directly comparing local action scores across these states may introduce noisy credit signals. We therefore evaluate individual actions but optimize complete trajectories, using a shared advantage to coordinate the full memory interaction.

To further isolate the contribution of PA-DAPO, we conduct an additional ablation comparing it with standard DAPO and a variant using action-specific advantages:

Training schemeMulti-hop F1/B1Temporal F1/B1Open-domain F1/B1Single-hop F1/B1Overall F1/B1
Standard DAPO43.78 / 34.7065.79 / 57.8031.92 / 28.3663.91 / 58.2858.57 / 51.97
DAPO with step-level credit assignment40.88 / 32.7365.36 / 57.5725.56 / 21.1163.19 / 57.6257.14 / 50.74
PA-DAPO**(Ours)**47.10 / 38.8066.00 / 59.4033.70 / 30.5067.00 / 61.5061.00 / 55.00

As shown above, PA-DAPO improves Overall F1/B1 by 2.43/3.03 points over standard DAPO and 3.86/4.26 points over step-level credit assignment. Standard DAPO allows frequent actions to dominate, while step-level advantages introduce noisier updates due to varying intermediate evidence states. These results validate the effectiveness of our process-level supervision with shared trajectory advantages.

W2: Answerer Consistency and Conversation-Level Data Splitting

Appendix C states that the LoCoMo answerer is fine-tuned on the training split, but does not clearly establish that every baseline uses the same checkpoint and inference configuration. The 1:1:8 split also needs to specify whether entire conversations are separated.

Thank you for your suggestion. We would like to provide further details on the data splitting strategy and answer-model configuration. Below is our supplementary clarification:

II Data Splitting. We follow the conversation-level split used by Memory-R1 and Mem-T: conv-26 for training, conv-30 for validation, and conv-41 through conv-50 for testing. Each complete conversation, together with all its associated question–answer pairs, is assigned to a single split, ensuring no conversation-level overlap across splits.

IIII Answer Model Training. For a fair comparison, we follow the same setting as Mem-T, treating answer-model fine-tuning on the training split as part of the overall training pipeline. The case study in Sec. 4.5 further illustrates the contribution of our memory retrieval and working-memory construction mechanisms.

Once again, thank you for your valuable suggestions. We will incorporate these details into Appendix C of the revised manuscript.

W3: Value and Reliability of the Learned Reward Model Compared with Rules and Teacher Scores

The benefit of learning a reward model is not isolated from the supervision used to create it. Removing its training stages also does not show whether it outperforms directly using the rules or teacher scores. A matched comparison is needed to justify this additional learned component.

Thank you for your question. We conduct additional experiments to compare the learned RM with direct rule-derived and teacher-generated rewards, and report the results below:

Reward schemeMulti-hop F1/B1Temporal F1/B1Open-domain F1/B1Single-hop F1/B1Overall F1/B1
DeepSeek-V4-Flash scores42.61 / 34.5865.10 / 59.0926.73 / 22.2062.15 / 56.7457.10 / 50.95
rule-derived scores44.03 / 35.8465.44 / 57.6229.02 / 24.6263.67 / 58.4558.23 / 52.01
AiO-Mem w/o Evid. SFT41.40 / 33.6068.40 / 60.6028.60 / 24.5062.90 / 57.4057.90 / 51.60
AiO-Mem w/o Traj. DPO43.40 / 35.2066.50 / 59.5034.20 / 29.9064.00 / 58.4058.80 / 52.60
AiO-Mem47.10 / 38.8066.00 / 59.4033.70 / 30.5067.00 / 61.5061.00 / 55.00

As shown above, AiO-Mem improves Overall F1/B1 by 3.90/4.05 points over direct teacher scores and 2.77/2.99 points over rule-derived scores, with consistent improvements across all four question types. Removing Evid. SFT or Traj. DPO further reduces Overall F1/B1 by 3.10/3.40 and 2.20/2.40 points, respectively. These results demonstrate the added value of learning and aligning the RM, beyond directly using either supervision source.

We will incorporate these additional comparisons into Appendix E.2 of the revised manuscript. Once again, thank you again for your constructive suggestions!

Reviewer_Po7rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AiO-Mem, a unified framework for long-term memory-augmented question answering. Unlike existing approaches that optimize retrieval and memory construction separately, AiO-Mem models searching, writing, filtering, and stopping as sequential transitions over a shared evidence state. It first converts heterogeneous histories into a source-traceable, multi-view event bank combining raw text, summaries, query anchors, and event phrases. A memory policy then iteratively searches for relevant events and decides whether to add them to a compact working memory, discard them, or terminate retrieval. To provide fine-grained supervision, the authors train a trajectory-conditioned reward model through evidence-grounded supervised fine-tuning and trajectory-level DPO. Its step-level scores are aggregated into an action-balanced trajectory reward, which is used by PA-DAPO to optimize the memory policy. Experiments on LoCoMo, MemoryAgentBench, and PersonaMem show improvements over static memory systems and RL-trained memory agents, while the ablation results highlight the importance of working-memory construction, multi-view retrieval, and process-level reward modeling.

Critical strengths

Unified memory-policy formulation. AiO-Mem jointly models retrieval, memory writing, filtering, and termination as actions over a shared evidence state, enabling better coordination than pipelines that optimize these stages separately.

Fine-grained and evidence-grounded supervision. The trajectory-conditioned reward model evaluates every intermediate action rather than relying only on final-answer correctness. Its combination of rule-based SFT, trajectory-level DPO, and source-linked evidence improves credit assignment and memory faithfulness.

Strong and comprehensive empirical results. AiO-Mem achieves competitive overall performance across LoCoMo, MemoryAgentBench, and PersonaMem. The ablation studies support the contribution of its main components, while the efficiency analysis reports substantially lower retrieval and memory-construction token costs.

Critical weaknesses

The main weakness is the limited methodological novelty. Most components of AiO-Mem, including event-based memory organization, multi-view retrieval with reciprocal rank fusion, sequential memory actions, reward-model training through SFT and DPO, and DAPO-based policy optimization, are adaptations or combinations of established techniques. The primary contribution therefore appears to be integrating these components into a unified evidence-state and reward framework rather than introducing a fundamentally new learning algorithm.

Questions for authors

See weakness

5 comments

Dear Reviewer Po7r,

We are deeply grateful for your valuable time and constructive feedback! Below are our detailed responses to your questions:

W1: Clarification on Contribution Positioning.

Most components of AiO-Mem, including event-based memory organization, multi-view retrieval with reciprocal rank fusion, sequential memory actions, reward-model training through SFT and DPO, and DAPO-based policy optimization, are adaptations or combinations of established techniques.

Thank you for raising this question. In the field of agent memory, there are indeed works that adopt similar building blocks. LightMem [1] and EverMemOS [2] employ event-based memory organization, while Search-R1 [3], Fine-Mem [4], and Mem-T [5] explore learning sequential memory operations.

However, AiO-Mem shifts memory learning from stage-wise optimization to unified evidence-state learning, while PA-DAPO addresses the resulting challenge of balancing credit across heterogeneous action types. Together, these contributions go beyond integrating existing components by introducing a unified learning formulation and a corresponding optimization method.

Let's clarify these contributions from two perspectives: problem formulation (what is learned) and policy optimization (how it is optimized).

From stage-wise memory optimization to unified evidence-state learning

We summarize the learning objectives and reward mechanisms of representative memory methods in the table below:

MethodRetrievalConstructionStoppingStep-level Evaluation
(a) Single-stage Optimization with Final-answer Rewards
Search-R1 [3]final answer—final answer—
MemAgent [6]—final answer——
Memory-R1 [7]fixedfinal answer——
(b) Multi-stage Optimization with Final-answer Rewards
MEM1 [8]final answerfinal answerfinal answer—
AtomMem [9]final answerfinal answer——
(c) Stage-specific Process Feedback
Mem-α [10]fixedLocal Process Feedback + final answer—Is each edit valid?
Fine-Mem [4]fixedLocal Process Feedback + final answer—Can memory answer chunk-level QA?
TrustMem [11]fixedLocal Process Feedback + final answer—Is each edit faithful to its input?
Mem-T [5]Local Process Feedback + final answerRetrieval-derived FeedbackLocal Process Feedback + final answerHow much gold evidence is retrieved?
(d) Ours
AiO-Memshared reward spaceshared reward spaceshared reward spaceWhat does this action add to the current evidence state?

It can be observed that existing methods either optimize retrieval and construction separately or provide process feedback limited to specific stages. In contrast, AiO-Mem evaluates and coordinates heterogeneous memory actions within a shared reward space through two key designs:

  • Shared Evidence State. We formulate memory interaction as an MDP where a single policy jointly controls retrieval, writing, filtering, and stopping Sec\. 3\.3. Source-traceable events and notes enable evidence evaluation against the current working memory Sec\. 3\.2. Removing working-memory construction causes the largest ablation drop **61\.0 → 50\.9 F1**, Sec\. 4\.3, Table 2.

  • Unified Process Reward Modeling. Our trajectory-conditioned reward model directly scores search, add, skip, and stop through evidence-grounded supervision and trajectory-level semantic alignment Sec\. 3\.4. It captures evidence contributions and semantic errors beyond evidence-ID matching, as illustrated in Appendix E.2. Removing evidence grounding or trajectory-level alignment reduces Overall F1 by 3.1 and 2.2 points, respectively Sec\. 4\.3, Table 2.

Credit allocation across heterogeneous action types

The shared reward space introduces a distinct optimization challenge: balancing credit across actions with different frequencies and lengths. Search and add may occur repeatedly, while stop occurs only once but determines whether the collected evidence is sufficient. Without balancing, frequent or longer actions can dominate the training signal. PA-DAPO handles this through two complementary designs:

  • Reward-level balancing. Step rewards are averaged within each action type and then equally aggregated across the types present in a trajectory Eq\. 4. This prevents frequent actions from dominating the trajectory reward.

  • Action-aware optimization. We organize policy optimization into a (group, action, token) hierarchy Eq\. 5, applying a shared trajectory advantage while normalizing contributions at the action-step level.

The experimental results support these designs. Appendix E.3 shows coordinated improvements in Search, Extract, and Answer-Decision rewards. Appendix E.4 further reports consistent gains across the evidence flow: Search Recall 811˙→904˙81\.1 → 90\.4, Working-memory Recall 708˙→784˙70\.8 → 78\.4, and Answer F1 558˙→610˙55\.8 → 61\.0.

To further assess PA-DAPO's contribution, we conducted an additional ablation study comparing PA-DAPO with standard DAPO and DAPO with action-specific advantages:

Training schemeMulti-hopTemporalOpen-domainSingle-hopOverall
Standard DAPO43.78 / 34.7065.79 / 57.8031.92 / 28.3663.91 / 58.2858.57 / 51.97
DAPO w/ step-level credit assignment40.88 / 32.7365.36 / 57.5725.56 / 21.1163.19 / 57.6257.14 / 50.74
PA-DAPO (ours)47.10 / 38.8066.00 / 59.4033.70 / 30.5067.00 / 61.5061.00 / 55.00

As observed, PA-DAPO improves Overall F1/B1 by 2.43/3.03 points over standard DAPO and 3.86/4.26 points over action-specific advantages. These results support our design of balancing action-level process rewards while optimizing complete trajectories, rather than assigning separate advantages to individual actions.

We will further clarify these methodological contributions in Appendix B and incorporate the additional results into Appendix E.

Thank you again for your kind and constructive suggestions!

Please let us know if you have any further questions. If you find that our response addresses some of your questions, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.

Best Regards,

Authors of All-in-One Memory

References

[1] Jizhan Fang et al. LightMem: Lightweight and Efficient Memory-Augmented Generation. ICLR, 2026.

[2] Chuanrui Hu et al. EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoning. ACL, 2026.

[3] Bowen Jin et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv:2503.09516, 2025.

[4] Weitao Ma et al. Fine-Mem: Fine-Grained Feedback Alignment for Long-Horizon Memory Management. ACL, 2026.

[5] Yanwei Yue et al. Mem-T: Densifying Rewards for Long-Horizon Memory Agents. arXiv:2601.23014, 2026.

[6] Hongli Yu et al. MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-Based Memory Agent. ICLR, 2026.

[7] Sikuan Yan et al. Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning. ACL, 2026.

[8] Zijian Zhou et al. MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents. ICLR, 2026.

[9] Yupeng Huo et al. AtomMem: Learnable Dynamic Agentic Memory with Atomic Memory Operation. EMNLP 2026.

[10] Yu Wang et al. Mem-α: Learning Memory Construction via Reinforcement Learning. arXiv:2509.25911, 2025.

[11] Tianyu Yang et al. TrustMem: Learning Trustworthy Memory Consolidation for LLM Agents with Long-Term Memory. ICML 2026.

AuthorsOpenReview ↗

Additional Clarifications and Follow-Up for Reviewer Po7r

Dear Reviewer Po7r,

We highly appreciate the constructive comments and insightful suggestions you have shared with us!

As the discussion period has only a few days left, in order for us to have sufficient time to address any additional questions you may have, we kindly encourage you to engage in the ongoing discussion and share any further insights or clarifications you may have.

If you find that our response addresses some of your concerns, would you kindly consider raising your rating score for our paper?

Thank you very much for your time and consideration. We look forward to hearing from you.

Best Regards,

Authors of All-in-One Memory

Reviewer_dpTqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AiO-Mem learns when to search, add notes, skip evidence, and stop within a shared memory process. It builds an event bank with source references and multiple retrieval views, then trains a reward model using evidence-based supervision and trajectory-level preference alignment. The resulting step scores are averaged within action types and combined into a trajectory reward for PA-DAPO. Experiments on LoCoMo, MemoryAgentBench, and PersonaMem report gains over static memory systems and learned memory agents.

Critical strengths
  1. Treating retrieval and memory construction as decisions over the current evidence state is a useful formulation. The value of another search depends on what has already been retained, and a retrieved passage helps only if the agent preserves the information needed to answer. The shared state makes these dependencies explicit.
  2. The evaluation includes tasks beyond the conversational QA setting used for training and examines intermediate evidence retrieval and memory quality. The reward-model ablations and the semantic comparisons in Appendix E.2 provide evidence that the gains involve how information is processed, rather than only whether the final answer is correct.
Critical weaknesses

The paper takes useful steps toward measuring an action's contribution. Section 3.4 defines uncovered gold evidence and rewards newly retrieved or written evidence. Appendix E.2 also gives examples where trajectories with identical evidence-ID metrics receive different semantic scores, including cases involving repetition and incorrect attribution. These results address part of my concern, but they do not yet establish how reliably the reward distinguishes a useful action from a plausible action that adds nothing needed for the answer. The examples are limited, and evidence coverage alone cannot establish whether an action was necessary. A controlled evaluation that inserts redundant searches or removes useful notes would test this more directly. It would also be useful to check whether valid evidence outside the annotated set receives appropriate credit.

The optimization objective leaves a separate question about credit assignment. Equations (4) and (5) aggregate step scores into a trajectory reward and use one group-relative advantage for all actions in that trajectory. This provides a learning signal, but actions with different individual scores still share the same advantage. The reported gains could therefore come from evaluating complete trajectories more accurately without requiring more precise attribution within a trajectory. The existing component ablations do not isolate these explanations. Given related work such as Mem-T and Fine-Mem, a comparison holding the reward model and rollout budget fixed would make the contribution of PA-DAPO easier to assess.

Questions for authors
  1. Can you evaluate reward preferences on controlled trajectory pairs that insert a redundant search, remove a useful note, or substitute valid evidence absent from the gold annotations? Appendix E.2 provides helpful examples; a broader test would show how consistently reward changes track changes in the evidence needed to answer.
  2. With the same reward model and rollout budget, what changes when individual action scores are used to construct action-specific advantages instead of one shared trajectory advantage? A comparison with terminal-answer rewards would further clarify how much improvement comes from richer supervision and how much comes from the way PA-DAPO assigns credit.
4 comments

Dear Reviewer dpTq,

We are deeply grateful for your valuable time and insightful feedback! Below are our detailed responses to your questions:

W1 & Q1: Reward Reliability for Action Contributions and Valid Evidence Beyond Gold Annotations

A controlled evaluation that inserts redundant searches or removes useful notes would test this more directly. Can you evaluate reward preferences on controlled trajectory pairs that insert a redundant search, remove a useful note, or substitute valid evidence absent from the gold annotations?

Thank you for raising such a professional and insightful question. We conducted four controlled evaluations of our reward model RMRM: Redundant Search Insertion Red\. Search, 100 samples, Useful Note Removal (Note Removal, 100 samples), Plausible Non-Gold Evidence Substitution Non−GoldEvid,˙45samplesNon-Gold Evid\., 45 samples, and Irrelevant Evidence Substitution Irrel\. Evid\., 45 samples. The results are reported below:

Controlled Perturbation|Search|Add|Skip|Stop|Number of Cases (↓ / = / ↑)
Red. Search|0.94 → 0.61|0.89 → 0.89|0.90 → 0.90|0.95 → 0.95|99 / 0 / 1
Note Removal|0.94 → 0.94|0.91 → 0.15|0.81 → 0.15|0.93 → 0.15|100 / 0 / 0
Non-Gold Evid.|0.85 → 0.85|0.88 → 0.66|—|0.86 → 0.49|39 / 6 / 0
Irrel. Evid.|0.85 → 0.85|0.66 → 0.51|—|0.49 → 0.39|41 / 2 / 2

Together, these controlled comparisons directly demonstrate that our RM provides reliable, action-specific, and evidence-sensitive process rewards:

  • For redundant search insertion, the average search reward drops substantially 09˙4→06˙10\.94 → 0\.61, while other action rewards remain unchanged. Rewards decrease in 99/100 cases, indicating that the RM effectively penalizes unnecessary searches.

  • For useful note removal, the add, skip, and stop rewards drop sharply 09˙1/08˙1/09˙3→01˙5/01˙5/01˙50\.91/0\.81/0\.93 → 0\.15/0\.15/0\.15. 100/100 cases show reward reductions, indicating that the RM recognizes the importance of preserving useful evidence in working memory, even when retrieval results remain unchanged.

  • For plausible non-gold and irrelevant evidence substitution, rewards decrease in 39/45 cases when replacing original evidence with plausible non-gold evidence, and in 41/45 cases when further replacing it with irrelevant evidence. The add reward decreases progressively 08˙8→06˙6→05˙10\.88 → 0\.66 → 0\.51, as does the stop reward 08˙6→04˙9→03˙90\.86 → 0\.49 → 0\.39. These results suggest that the RM distinguishes different levels of evidence quality rather than relying solely on gold annotations.

We highly appreciate the constructive comments and insightful suggestions you have offered for our work. We will incorporate these additional experimental results into Appendix E.2 of the revised manuscript.

W2 & Q2: Step-Level Credit Assignment and Controlled Evaluation of PA-DAPO

With the same reward model and rollout budget, what changes when individual action scores are used to construct action-specific advantages instead of one shared trajectory advantage?

Thank you for your question. We adopt a shared trajectory advantage for two reasons. First, search, add, skip, and stop are interdependent, making trajectory-level optimization suitable for coordinating memory operations. Second, different rollouts produce different intermediate evidence states, making direct comparisons of action-specific scores less straightforward.

PA-DAPO therefore evaluates individual actions but optimizes complete trajectories. Action-balanced aggregation incorporates the scores of each action type, while group-relative advantages compare trajectories from the same query and history.

To quantitatively assess the contributions of process-level supervision and advantage construction, we conduct an additional ablation study comparing terminal-answer F1 rewards, process rewards with action-specific advantages, and PA-DAPO:

Training StrategiesMulti-hop F1 / B1Temporal F1 / B1Open-domain F1 / B1Single-hop F1 / B1Overall F1 / B1
Terminal-answer F1 reward41.20 / 32.7062.52 / 57.3132.80 / 29.4158.16 / 52.0754.31 / 48.12
Action-specific advantage40.88 / 32.7365.36 / 57.5725.56 / 21.1163.19 / 57.6257.14 / 50.74
PA-DAPO (Ours)47.10 / 38.8066.00 / 59.4033.70 / 30.5067.00 / 61.5061.00 / 55.00

As shown in the table, PA-DAPO substantially improves Overall F1/B1 over both terminal-answer rewards 543˙1/481˙2→610˙0/550˙054\.31/48\.12 → 61\.00/55\.00 and action-specific advantages 571˙4/507˙4→610˙0/550˙057\.14/50\.74 → 61\.00/55\.00. This further confirms that PA-DAPO's process rewards outperform terminal-answer rewards, while shared trajectory advantages utilize process scores more effectively than action-specific advantages.

Thank you very much for your constructive suggestions!

Please let us know if you have any further questions. If you find that our response addresses some of your questions, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.

Best Regards,

Authors of All-in-One Memory

AuthorsOpenReview ↗

Additional Clarifications and Follow-Up for Reviewer dpTq

Dear Reviewer dpTq,

We sincerely appreciate your valuable time, constructive feedback, and insightful suggestions!

As the discussion period is approaching its end, we would greatly appreciate the opportunity to address any remaining questions or concerns. We warmly welcome your further feedback and would be happy to provide additional clarifications during the remaining discussion period.

If you find that our response addresses some of your concerns, would you kindly consider raising your rating score for our paper?

Thank you very much for your time and consideration. We look forward to hearing from you.

Best Regards,

Authors of All-in-One Memory

AuthorsOpenReview ↗

Updated Manuscript and Summary of Revisions

Dear Reviewer dpTq,

We sincerely appreciate your thoughtful feedback and constructive suggestions, which have been invaluable in strengthening our work!

We have uploaded a revised manuscript PDF, with all additions and modifications highlighted in blue for your convenience. We would like to summarize the main revisions as follows:

  1. Reward Model Reliability. We have added four controlled evaluations covering redundant search insertion, useful note removal, plausible non-gold evidence substitution, and irrelevant evidence substitution, together with action-specific reward changes and corresponding statistics (Appendix G.1).

  2. Process Rewards and Advantage Construction. We have included comparisons with Outcome-only F1 Reward and Action-level Advantage, further clarifying the contributions of process-level supervision and shared trajectory advantages to PA-DAPO (Appendix G.2).

  3. Other Revisions. In response to suggestions from other reviewers, we have also incorporated comparisons with standard DAPO and direct reward sources, clarified the experimental protocol, and refined the computational complexity and efficiency analyses (Appendices G.2–G.3 and C.2; Section 4.4; Appendix F).

We are truly grateful for your valuable suggestions and hope that these revisions have adequately addressed your concerns and improved the clarity and rigor of our work. If you have any further questions or concerns, please feel free to let us know, we remain fully committed to addressing them with our utmost effort.

Best regards,

Authors of All-in-One Memory

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.