- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MEM-SONAR, a long-term memory framework that jointly learns memory organization and retrieval through alternating reinforcement learning. An Organizer extracts atomic facts and assigns them hierarchical Semantic Paths, while a Navigator searches the resulting memory tree to answer questions. Both roles share the same LLM and are optimized alternately using downstream QA rewards. Experiments on LoCoMo, LongMemEval, and NarrativeQA demonstrate improvements over existing memory systems.
Critical strengths
-
Treating memory organization as a learnable capability optimized for downstream retrieval is a promising idea. The coupling between organization and navigation is intuitive and potentially valuable for long-term LLM agents.
-
MEM-SONAR outperforms GAM on LoCoMo by 9.19 and 7.17 F1 points with Qwen3-4B and Qwen2.5-7B, respectively. Results on LongMemEval and NarrativeQA further support the potential of the approach.
Critical weaknesses
- The central claim is that alternating optimization enables mutually reinforcing improvements in organization and navigation. However, Figure 3 does not establish that alternating training is superior to simpler alternatives. Controlled comparisons against Organizer-only RL, Navigator-only RL, joint optimization, and sequential training under matched compute budgets are needed. Since both roles share model parameters, improvements may also reflect general RL training rather than the proposed bidirectional mechanism.
- The framework focuses on incremental fact insertion but does not adequately address outdated or contradictory memories, ambiguous fact placement, or access to facts relevant to multiple semantic categories. These issues are important for persistent memory systems.
Questions for authors
In Equation (6), the Organizer is optimized using episode-local QA rewards, whereas LoCoMo multi-hop questions require synthesizing information across multiple sessions. Given the relatively modest multi-hop improvements in Table 1, could the authors clarify to what extent episode-local memory organization contributes to cross-episode multi-hop retrieval, beyond the improvements learned by the Navigator?
3 comments
Response to the Reviewer(Question1)
[Q3] Clarification on the Contribution of Episode-Local Memory Organization to Cross-Session Multi-Hop Retrieval
We thank the reviewer for this insightful question. We clarify how episode-local memory organization improves cross-session multi-hop retrieval from both mechanistic and experimental perspectives.
Mechanistic Perspective: Local Evidence Supervision and Subtree Reachability to Global Reachability. Cross-session multi-hop retrieval essentially involves jointly searching multiple episode-local subtrees within the global memory tree. When constructing the episode-local training question set , a multi-hop question is included whenever at least part of its supporting evidence appears in the current episode, thereby providing a supervision signal for local memory organization. This encourages the Organizer to preserve key local facts required for multi-hop reasoning and optimize the corresponding subtree structure. As the accessibility of evidence within individual subtrees improves, the overall accessibility of evidence across multiple subtrees also increases. This is further supported by the cross-pairing experiments in Figure 3: when keeping the Navigator fixed, simply replacing the initial memory tree from Cycle 1 with the optimized tree from Cycle 9 yields a substantial improvement in overall retrieval performance.
Experimental Perspective: Controlled Comparison of Evidence Hit Rates on Multi-Hop Questions. To isolate the contribution of memory organization from improvements in the Navigator, we constructed memory trees using Organizer checkpoints from three training stages (tree-cycle1, tree-cycle5, and tree-cycle9). We then evaluated these trees using fixed Navigator models before training (Training-free) and after training (After-training), measuring the evidence hit rate on Multi-Hop questions:
| Organizer | Training-free | After-training | Average |
|---|---|---|---|
| tree-cycle1 | 48.29% | 48.59% | 48.44% |
| tree-cycle5 | 56.69% | 57.94% | 57.32% |
| tree-cycle9 | 56.89% | 60.57% | 58.73% |
The results show that, with either the training-free or the trained Navigator held fixed, the evidence hit rate on cross-session multi-hop questions consistently improves as Organizer training progresses. These findings demonstrate that episode-local memory organization training effectively improves evidence retrieval for cross-session multi-hop reasoning, independently of improvements in the Navigator.
Thank you to the authors for the detailed response and additional experiments. I appreciate the effort to address my concerns, particularly the new controlled comparisons of training strategies.
The rebuttal has improved my assessment of the paper, and I would have considered raising my score if the rating scale were more granular. However, given this year's grading scale, I believe 3 remains an appropriate rating for the paper.
Response to the Reviewer(Weakness 1 and Weakness 2)
[Weakness 1] Controlled Comparison of Training Schedules under Equal Compute Budgets
We thank the reviewer for this important suggestion. To rigorously validate the necessity of multi-round alternating optimization, we conducted additional controlled experiments comparing different training schedules under the same total training budget (matched training steps). We report the Overall F1 scores at five evenly spaced checkpoints:
Controlled Experimental Findings and Mechanistic Analysis
Organizer-only RL vs. Navigator-only RL. Allocating the entire training budget to either component leads to performance limitations. For Navigator-only RL, the initial memory tree lacks an effective organization structure, limiting retrieval performance regardless of improvements in navigation. Conversely, Organizer-only RL learns improved semantic organization patterns, but the unadapted Navigator cannot effectively exploit these newly learned structures or provide sufficiently informative feedback to distinguish better organizations.
Sequential Training (Org → Nav). We also evaluated a single-pass sequential schedule that allocates half of the training budget to Organizer training, followed by Navigator training. Despite using the same total budget, this approach underperforms multi-round alternating optimization, indicating that a single sequential pass is insufficient to fully capture the benefits of iterative co-adaptation.
Joint Optimization. In persistent memory systems, the memory tree must be constructed before cross-session retrieval can be performed. Directly coupling tree construction, tree search, and question answering into a single jointly optimized process introduces two major challenges: (i) ambiguous credit assignment due to extremely long trajectories and sparse terminal rewards, and (ii) severe environment non-stationarity, as parameter updates may continuously alter the memory tree topology. These factors increase training variance and hinder convergence. Consistently, Joint Optimization exhibits unstable performance, with Overall F1 fluctuating between 37.70 and 39.00 throughout training. In contrast, our alternating strategy optimizes one component while keeping the other fixed, thereby simplifying credit assignment and stabilizing the retrieval environment.
Training Dynamics across Checkpoints. The checkpoint trajectories further reveal distinct optimization behaviors. Organizer-only RL exhibits limited improvement throughout training. Navigator-only RL achieves substantial early gains by adapting to the initial static tree, but its improvements gradually diminish as training progresses. In contrast, our alternating optimization achieves consistent, monotonic improvements across all five checkpoints, without performance degradation. As training proceeds, a stronger Navigator provides more informative feedback for improving memory organization, while an improved Organizer produces clearer structures that facilitate subsequent navigation. This bidirectional improvement enables our method to surpass the performance limits of both single-component and sequential training schedules, ultimately achieving the highest Overall F1 of 53.24.
[Weakness 2] Discussion on Outdated/Contradictory Memories, Ambiguous Fact Placement, and Multi-Category Access
We thank the reviewer for highlighting these important challenges in persistent memory systems. The resolution of outdated, ambiguous, or contradictory memories has been extensively explored in prior work (e.g., Memory-R1 and Mem0) through explicit memory editing operations, such as Update, Delete, and Merge.
The primary focus of our work is to address the lack of implicit memory priors. Through bidirectional alternating reinforcement learning between the Organizer and Navigator, our method enables the model to autonomously learn hierarchical memory structures aligned with downstream retrieval needs, together with consistent navigation strategies.
Our implicit memory prior learning mechanism is orthogonal and complementary to these explicit memory maintenance mechanisms. Therefore, these explicit memory editing operations can be readily incorporated into the Organizer's action space within our framework to address outdated, ambiguous, or contradictory memories through memory maintenance and conflict resolution.