- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AiO-Mem learns how to gather evidence from a long history before answering a question. It first compiles the history into source-traceable events with several retrieval representations. A memory agent then repeatedly searches this event bank, adds useful information to working notes, skips unhelpful results, or stops. A separate reward model evaluates these operations using the full interaction trajectory. Its step scores are combined into a trajectory reward, which trains the agent through PA-DAPO. Experiments on LoCoMo, MemoryAgentBench, and PersonaMem report gains over static and RL-trained memory systems.
Critical strengths
-
The memory operations are explicit and grounded in source evidence. The agent maintains a question, working notes, and retrieved candidates, and acts through search, add, skip, and stop operations. Event locators connect the resulting notes to the original history. This makes the workflow inspectable and provides a concrete basis for evaluating whether an operation contributes useful evidence.
-
The evaluation covers several forms of memory use. The paper tests conversational memory, broader memory-agent tasks, and dynamic user profiling rather than relying on a single benchmark. Ablations examine the event representations and reward-model training stages, while a human assessment checks agreement with the teacher’s scores. Together, these experiments provide useful initial evidence for the integrated system.
Critical weaknesses
-
Step-level scoring does not clearly translate into step-level credit assignment. The reward model assigns a score to each memory operation, but Section 3.5 averages these scores into one trajectory reward. Equation (5) then uses the same trajectory-level advantage for every generated action token. A useful search and an incorrect note within the same trajectory therefore receive the same advantage signal. With rewards and masking fixed, the additional action index appears to regroup the token-level DAPO objective [1]. The paper should distinguish its reward-model contribution from a new optimization algorithm.
-
The evaluation may mix improvements in memory management with improvements in the answerer. The memory agent produces working notes, which a separate answer model uses to generate the final response. Appendix C states that the LoCoMo answerer is fine-tuned on the training split, but does not clearly establish that every baseline uses the same checkpoint and inference configuration. The 1:1:8 split also needs to specify whether entire conversations are separated. These details are necessary to attribute the final QA gains to memory-policy learning.
-
The benefit of learning a reward model is not isolated from the supervision used to create it. The reward model is first trained on rule-derived evidence scores and then aligned to a stronger model’s scoring sequences. The reported human agreement with the teacher is encouraging, but evaluates the teacher rather than fully establishing the trained reward model’s reliability. Removing its training stages also does not show whether it outperforms directly using the rules or teacher scores. A matched comparison is needed to justify this additional learned component.
-
The scope of unified memory learning is narrower than the framing suggests. The persistent event bank is constructed by a separate compiler before the learned agent gathers query-specific working notes. It is therefore unclear which persistent memory-construction decisions are optimized jointly with retrieval. This distinction matters because Mem-T [2] already jointly learns memory construction and retrieval, while Fine-Mem [3] provides fine-grained feedback for memory operations. Both are cited, but the additional capability supplied by AiO-Mem’s unification needs clearer evidence.
-
Bounded query processing does not imply history-independent total cost. The method limits the number of retrieval results and interaction steps, which bounds the text processed by the query-time agent. However, the full history must still be compiled and indexed, and the paper’s own complexity expression retains a history-dependent compilation term. The claim of cost invariance should identify the specific component being bounded and distinguish it from total or amortized system cost.
[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. 2025.
[2] Mem-T: Densifying Rewards for Long-Horizon Memory Agents. 2026.
[3] Fine-Mem: Fine-Grained Feedback Alignment for Long-Horizon Memory Management. 2026.
Questions for authors
Please refer to the weakness.
10 comments
W2: Answerer Consistency and Conversation-Level Data Splitting
Appendix C states that the LoCoMo answerer is fine-tuned on the training split, but does not clearly establish that every baseline uses the same checkpoint and inference configuration. The 1:1:8 split also needs to specify whether entire conversations are separated.
Thank you for your suggestion. We would like to provide further details on the data splitting strategy and answer-model configuration. Below is our supplementary clarification:
Data Splitting. We follow the conversation-level split used by Memory-R1 and Mem-T: conv-26 for training, conv-30 for validation, and conv-41 through conv-50 for testing. Each complete conversation, together with all its associated question–answer pairs, is assigned to a single split, ensuring no conversation-level overlap across splits.
Answer Model Training. For a fair comparison, we follow the same setting as Mem-T, treating answer-model fine-tuning on the training split as part of the overall training pipeline. The case study in Sec. 4.5 further illustrates the contribution of our memory retrieval and working-memory construction mechanisms.
Once again, thank you for your valuable suggestions. We will incorporate these details into Appendix C of the revised manuscript.
W3: Value and Reliability of the Learned Reward Model Compared with Rules and Teacher Scores
The benefit of learning a reward model is not isolated from the supervision used to create it. Removing its training stages also does not show whether it outperforms directly using the rules or teacher scores. A matched comparison is needed to justify this additional learned component.
Thank you for your question. We conduct additional experiments to compare the learned RM with direct rule-derived and teacher-generated rewards, and report the results below:
| Reward scheme | Multi-hop F1/B1 | Temporal F1/B1 | Open-domain F1/B1 | Single-hop F1/B1 | Overall F1/B1 |
|---|---|---|---|---|---|
| DeepSeek-V4-Flash scores | 42.61 / 34.58 | 65.10 / 59.09 | 26.73 / 22.20 | 62.15 / 56.74 | 57.10 / 50.95 |
| rule-derived scores | 44.03 / 35.84 | 65.44 / 57.62 | 29.02 / 24.62 | 63.67 / 58.45 | 58.23 / 52.01 |
| AiO-Mem w/o Evid. SFT | 41.40 / 33.60 | 68.40 / 60.60 | 28.60 / 24.50 | 62.90 / 57.40 | 57.90 / 51.60 |
| AiO-Mem w/o Traj. DPO | 43.40 / 35.20 | 66.50 / 59.50 | 34.20 / 29.90 | 64.00 / 58.40 | 58.80 / 52.60 |
| AiO-Mem | 47.10 / 38.80 | 66.00 / 59.40 | 33.70 / 30.50 | 67.00 / 61.50 | 61.00 / 55.00 |
As shown above, AiO-Mem improves Overall F1/B1 by 3.90/4.05 points over direct teacher scores and 2.77/2.99 points over rule-derived scores, with consistent improvements across all four question types. Removing Evid. SFT or Traj. DPO further reduces Overall F1/B1 by 3.10/3.40 and 2.20/2.40 points, respectively. These results demonstrate the added value of learning and aligning the RM, beyond directly using either supervision source.
We will incorporate these additional comparisons into Appendix E.2 of the revised manuscript. Once again, thank you again for your constructive suggestions!
Dear Reviewer DU1J,
We are deeply grateful for your valuable time, great suggestion and affirmation. Below are our detailed responses to your questions:
W1: Step-Level Credit Assignment and the Distinction from Token-Level DAPO
Thank you for raising such a professional and insightful question. We would like to clarify these distinctions below:
In AiO-Mem, step-level reward evaluation and trajectory-level policy optimization serve different purposes. The unified reward model evaluates individual memory actions, while PA-DAPO uses these evaluations to construct an action-balanced trajectory objective. This design addresses the interdependence of memory operations and their uneven frequencies and lengths. Let's clarify three aspects:
First, PA-DAPO's distinction from standard DAPO. The shared reward space introduces a challenge: frequent search and add operations may dominate sparse but critical stop decisions. PA-DAPO addresses this through action-balanced reward aggregation Eq\. 4 and a (group, action, token) optimization hierarchy Eq\. 5. Compared with DAPO, PA-DAPO explicitly structures process reward aggregation and policy optimization around memory actions, rather than treating the trajectory as a flat token sequence.
Second, the role of step-level rewards. Our reward model evaluates search, add, skip, and stop individually, while Eq. 4 balances their scores across action types to construct the trajectory reward. These process scores therefore shape the shared advantage through action-balanced aggregation, rather than producing separate advantages for individual actions.
Third, the choice of trajectory-level advantages. Memory actions are interdependent, and different rollouts produce different intermediate evidence states. Directly comparing local action scores across these states may introduce noisy credit signals. We therefore evaluate individual actions but optimize complete trajectories, using a shared advantage to coordinate the full memory interaction.
To further isolate the contribution of PA-DAPO, we conduct an additional ablation comparing it with standard DAPO and a variant using action-specific advantages:
As shown above, PA-DAPO improves Overall F1/B1 by 2.43/3.03 points over standard DAPO and 3.86/4.26 points over step-level credit assignment. Standard DAPO allows frequent actions to dominate, while step-level advantages introduce noisier updates due to varying intermediate evidence states. These results validate the effectiveness of our process-level supervision with shared trajectory advantages.