acceptodds
Under review as a conference paper at ICLR 2027

Mem-Sonar: Self-Organizing and Self-Navigating Memory via Alternating Reinforcement Learning

Abstract

Long-term memory systems enable large language models (LLMs) to retain and use information beyond their context windows. However, external memory often remains an opaque black box: without prior awareness of how its contents are organized, the LLM must rely on exploratory queries and repeated retrieval attempts. We refer to this problem as lack of memory priors. We introduce MEM-SONAR, which models memory organization and retrieval as two trainable capabilities of the same LLM. The Organizer generates Semantic Paths to organize atomic facts into a memory tree, while the Navigator follows these paths to retrieve evidence based on downstream questions. To improve and align memory organization and retrieval, we further propose multi-round alternating reinforcement learning. Retrieval performance provides feedback for improving memory organization, while the resulting structures support subsequent navigation training. Through this bidirectional learning process, the model progressively learns an organization strategy suited to its own retrieval behavior, developing an implicit memory prior that guides memory access. Experiments show that MEM-SONAR outperforms the compared baselines across multiple benchmarks and backbones. Further analyses demonstrate sustained improvements in both organization and retrieval, supporting the effectiveness of their bidirectional learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-term Memory, Large Language Models, Memory Organization, Memory Navigation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_K2fmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MEM-SONAR, a long-term memory framework that jointly learns memory organization and retrieval through alternating reinforcement learning. An Organizer extracts atomic facts and assigns them hierarchical Semantic Paths, while a Navigator searches the resulting memory tree to answer questions. Both roles share the same LLM and are optimized alternately using downstream QA rewards. Experiments on LoCoMo, LongMemEval, and NarrativeQA demonstrate improvements over existing memory systems.

Critical strengths
  1. Treating memory organization as a learnable capability optimized for downstream retrieval is a promising idea. The coupling between organization and navigation is intuitive and potentially valuable for long-term LLM agents.

  2. MEM-SONAR outperforms GAM on LoCoMo by 9.19 and 7.17 F1 points with Qwen3-4B and Qwen2.5-7B, respectively. Results on LongMemEval and NarrativeQA further support the potential of the approach.

Critical weaknesses
  1. The central claim is that alternating optimization enables mutually reinforcing improvements in organization and navigation. However, Figure 3 does not establish that alternating training is superior to simpler alternatives. Controlled comparisons against Organizer-only RL, Navigator-only RL, joint optimization, and sequential training under matched compute budgets are needed. Since both roles share model parameters, improvements may also reflect general RL training rather than the proposed bidirectional mechanism.
  2. The framework focuses on incremental fact insertion but does not adequately address outdated or contradictory memories, ambiguous fact placement, or access to facts relevant to multiple semantic categories. These issues are important for persistent memory systems.
Questions for authors

In Equation (6), the Organizer is optimized using episode-local QA rewards, whereas LoCoMo multi-hop questions require synthesizing information across multiple sessions. Given the relatively modest multi-hop improvements in Table 1, could the authors clarify to what extent episode-local memory organization contributes to cross-episode multi-hop retrieval, beyond the improvements learned by the Navigator?

3 comments
AuthorsOpenReview ↗

Response to the Reviewer(Weakness 1 and Weakness 2)

[Weakness 1] Controlled Comparison of Training Schedules under Equal Compute Budgets

We thank the reviewer for this important suggestion. To rigorously validate the necessity of multi-round alternating optimization, we conducted additional controlled experiments comparing different training schedules under the same total training budget (matched training steps). We report the Overall F1 scores at five evenly spaced checkpoints:

Training ScheduleCkpt 1Ckpt 2Ckpt 3Ckpt 4Ckpt 5
Organizer-only RL39.9240.0041.3041.3542.80
Navigator-only RL38.2643.3945.0048.4049.72
Sequential Training (Org → Nav)39.9240.0045.7949.0350.39
Joint Optimization38.1038.5037.7038.3039.00
Multi-round Alternating (Ours)45.1047.1448.1752.1953.24

Controlled Experimental Findings and Mechanistic Analysis

  1. Organizer-only RL vs. Navigator-only RL. Allocating the entire training budget to either component leads to performance limitations. For Navigator-only RL, the initial memory tree lacks an effective organization structure, limiting retrieval performance regardless of improvements in navigation. Conversely, Organizer-only RL learns improved semantic organization patterns, but the unadapted Navigator cannot effectively exploit these newly learned structures or provide sufficiently informative feedback to distinguish better organizations.

  2. Sequential Training (Org → Nav). We also evaluated a single-pass sequential schedule that allocates half of the training budget to Organizer training, followed by Navigator training. Despite using the same total budget, this approach underperforms multi-round alternating optimization, indicating that a single sequential pass is insufficient to fully capture the benefits of iterative co-adaptation.

  3. Joint Optimization. In persistent memory systems, the memory tree must be constructed before cross-session retrieval can be performed. Directly coupling tree construction, tree search, and question answering into a single jointly optimized process introduces two major challenges: (i) ambiguous credit assignment due to extremely long trajectories and sparse terminal rewards, and (ii) severe environment non-stationarity, as parameter updates may continuously alter the memory tree topology. These factors increase training variance and hinder convergence. Consistently, Joint Optimization exhibits unstable performance, with Overall F1 fluctuating between 37.70 and 39.00 throughout training. In contrast, our alternating strategy optimizes one component while keeping the other fixed, thereby simplifying credit assignment and stabilizing the retrieval environment.

  4. Training Dynamics across Checkpoints. The checkpoint trajectories further reveal distinct optimization behaviors. Organizer-only RL exhibits limited improvement throughout training. Navigator-only RL achieves substantial early gains by adapting to the initial static tree, but its improvements gradually diminish as training progresses. In contrast, our alternating optimization achieves consistent, monotonic improvements across all five checkpoints, without performance degradation. As training proceeds, a stronger Navigator provides more informative feedback for improving memory organization, while an improved Organizer produces clearer structures that facilitate subsequent navigation. This bidirectional improvement enables our method to surpass the performance limits of both single-component and sequential training schedules, ultimately achieving the highest Overall F1 of 53.24.


[Weakness 2] Discussion on Outdated/Contradictory Memories, Ambiguous Fact Placement, and Multi-Category Access

We thank the reviewer for highlighting these important challenges in persistent memory systems. The resolution of outdated, ambiguous, or contradictory memories has been extensively explored in prior work (e.g., Memory-R1 and Mem0) through explicit memory editing operations, such as Update, Delete, and Merge.

The primary focus of our work is to address the lack of implicit memory priors. Through bidirectional alternating reinforcement learning between the Organizer and Navigator, our method enables the model to autonomously learn hierarchical memory structures aligned with downstream retrieval needs, together with consistent navigation strategies.

Our implicit memory prior learning mechanism is orthogonal and complementary to these explicit memory maintenance mechanisms. Therefore, these explicit memory editing operations can be readily incorporated into the Organizer's action space within our framework to address outdated, ambiguous, or contradictory memories through memory maintenance and conflict resolution.

AuthorsOpenReview ↗

Response to the Reviewer(Question1)

[Q3] Clarification on the Contribution of Episode-Local Memory Organization to Cross-Session Multi-Hop Retrieval

We thank the reviewer for this insightful question. We clarify how episode-local memory organization improves cross-session multi-hop retrieval from both mechanistic and experimental perspectives.

Mechanistic Perspective: Local Evidence Supervision and Subtree Reachability to Global Reachability. Cross-session multi-hop retrieval essentially involves jointly searching multiple episode-local subtrees T(i)\mathcal{T}^{(i)} within the global memory tree. When constructing the episode-local training question set Q(i)Q^{(i)}, a multi-hop question is included whenever at least part of its supporting evidence appears in the current episode, thereby providing a supervision signal for local memory organization. This encourages the Organizer to preserve key local facts required for multi-hop reasoning and optimize the corresponding subtree structure. As the accessibility of evidence within individual subtrees improves, the overall accessibility of evidence across multiple subtrees also increases. This is further supported by the cross-pairing experiments in Figure 3: when keeping the Navigator fixed, simply replacing the initial memory tree from Cycle 1 with the optimized tree from Cycle 9 yields a substantial improvement in overall retrieval performance.

Experimental Perspective: Controlled Comparison of Evidence Hit Rates on Multi-Hop Questions. To isolate the contribution of memory organization from improvements in the Navigator, we constructed memory trees using Organizer checkpoints from three training stages (tree-cycle1, tree-cycle5, and tree-cycle9). We then evaluated these trees using fixed Navigator models before training (Training-free) and after training (After-training), measuring the evidence hit rate on Multi-Hop questions:

OrganizerTraining-freeAfter-trainingAverage
tree-cycle148.29%48.59%48.44%
tree-cycle556.69%57.94%57.32%
tree-cycle956.89%60.57%58.73%

The results show that, with either the training-free or the trained Navigator held fixed, the evidence hit rate on cross-session multi-hop questions consistently improves as Organizer training progresses. These findings demonstrate that episode-local memory organization training effectively improves evidence retrieval for cross-session multi-hop reasoning, independently of improvements in the Navigator.

Thank you to the authors for the detailed response and additional experiments. I appreciate the effort to address my concerns, particularly the new controlled comparisons of training strategies.

The rebuttal has improved my assessment of the paper, and I would have considered raising my score if the rating scale were more granular. However, given this year's grading scale, I believe 3 remains an appropriate rating for the paper.

Reviewer_VmtyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The method introduced in this paper is built around two models that share the same parameters and jointly maintain a memory tree. The Organizer extracts atomic facts from the input, builds semantic paths, and places the facts into a hierarchical tree. The Navigator traverses the tree from the root, selects branches level by level according to the question and the branch labels, and retrieves the relevant evidence. The two models are trained in a multi round alternating reinforcement learning loop. The Organizer generates several candidate memory trees. The Navigator attempts to answer downstream questions using these trees. The resulting question answering performance serves as a reward signal that updates the Organizer’s organization strategy. This cycle lets retrieval performance guide the organization strategy and, in turn, a better organized memory support more effective retrieval. Evaluation on several benchmarks with LLaMA3 and Qwen2 models shows the best performance. Ablation studies show that this method has better performance than vector based storage.

Critical strengths
  • Joint optimization of organization and retrieval addresses the lack of memory priors
  • The hierarchical organization of facts that reduces the black box nature of flat vector structures
  • RL uses downstream QA performance as a reward, which means the model’s organization is customized to its retrieval needs.
  • Ablations show this methos is better than vector space storage.
Critical weaknesses
  • The training of this method that combines bidirectional learning with RL seems very complex and likely sensitive to hyperparameters
  • The inference cost seems high and there is no analysis of computation costs
  • Results are shown for LLaMA3 and Qwen2 which is limited; the paper does not consider other model families
  • There is no scope/failure case analysis
Questions for authors
  • What are the computational costs (training and inference) ?
  • Are there failure modes?
  • How many cycles are there in the multi-round alternating RL?
  • Can you elaborate on the hyperparameter choices?
  • What is the trade-off between inference cost and retrieval, and when do we start observing diminishing returns?
  • Can the method be extended to other types of knowledge?
2 comments
AuthorsOpenReview ↗

Response to the Reviewer

Weakness 1 & Question 4 — Rationale for Hyperparameter Selection

Thank you for suggesting that we further explain the rationale behind our hyperparameter choices. The appendix already summarizes the main training and inference hyperparameters. In the revised manuscript, we have added explanations covering three aspects in Appendix A.2: training-time sampling and decoding strategies, granularity and scale control in memory construction, and generation budgets for different operations. These additions clarify the design rationale behind these parameters and their roles during training and inference, improving the interpretability and reproducibility of our experimental configuration.


Weakness 2 & Question 1 — Training and Inference Costs

We implement reinforcement learning using the Slime framework, combining Megatron-LM for parameter optimization with SGLang for rollout generation. Training and rollout inference run in parallel on a single node equipped with eight NVIDIA H20 GPUs. Completing all nine training cycles takes approximately three days (72 hours). Assuming that all eight GPUs remain allocated throughout the run, the total computational cost is approximately 576 GPU-hours. The overall training cost is manageable, and the training can be readily reproduced with standard academic computing resources.

Inference cost. As described in Section 4.4 (Figure 4) and Appendices A.3 and A.5, our framework decouples offline memory construction from online question answering and retrieval. The memory tree for each long conversation needs to be constructed only once, offline when the data are written, and can then be reused for all subsequent questions. During evaluation, we use vLLM to deploy the trained model checkpoints on eight NVIDIA H20 GPUs and process requests concurrently. Across the six long, multi-session conversations in the LoCoMo test set, the average end-to-end time per conversation, including memory-tree construction and retrieval and answering for all associated questions, is only three minutes. During online question answering, the default configuration consumes approximately 40,000 tokens per question, including both input and output tokens. This cost primarily arises from model-generated tree-navigation decisions and multi-round branch retrieval. MEM-SONAR occupies the highest-performance end of the Candidate Frontier shown in Figure 4(a).


Weakness 3 & Question 6 — Cross-Model Generalization

We have validated the effectiveness of MEM-SONAR in the paper using two models from the Qwen and Llama families.

To further evaluate its applicability across model families, we conducted additional LoCoMo experiments using GLM-4-9B-0414, with five cycles of alternating reinforcement learning. Overall F1 reached 37.0, 43.0, 46.7, 48.2, and 51.5 over the five successive cycles, yielding a cumulative improvement of 14.5 percentage points.

These results further demonstrate that MEM-SONAR’s performance gains are not limited to the Qwen family: the GLM model also exhibits sustained improvements, validating the framework’s applicability across model families.


Weakness 3 & Question 6 — Cross-Task Generalization and Supplementary HotpotQA Evaluation

To investigate MEM-SONAR’s generalization beyond long-term conversational memory, we added a cross-paradigm evaluation on HotpotQA (Dev Distractor) during the rebuttal period. We selected 500 questions through stratified sampling by question type and difficulty, treating the ten Wikipedia paragraphs as episodes and their sentences as input units for memory-tree construction.

Unlike the benchmarks in the main paper, which emphasize temporal evolution and state tracking across multi-turn interactions, HotpotQA is a non-conversational, knowledge-intensive multi-hop reasoning task over encyclopedic text with substantial distractor noise. It includes numerous thematically similar and highly confounding distractor paragraphs and specifically tests cross-document reasoning through intermediate bridging entities (Bridge) and parallel attribute comparisons across multiple entities (Comparison).

Under this setting, MEM-SONAR with Qwen3-4B-Instruct achieved good performance, with 65.37 F1 on Bridge questions and 75.00 F1 on Comparison questions. Ten repeated runs yielded stable mean scores of 66.51 F1 / 61.71 BLEU-1.

Together with the main experiments in Table 1 and the additional benchmark evaluations in Table 2, these results provide broad task coverage. At the methodological level, MEM-SONAR’s alternating Organizer–Navigator framework does not rely on predefined task-specific heuristic rules and therefore inherently supports transfer to a broader range of knowledge-intensive tasks in principle.

AuthorsOpenReview ↗

Response to the Reviewer

Weakness 4 & Question 2 — Scope of Applicability and Failure Case Analysis

Scope of applicability. Thank you for raising this question. This work addresses the lack of effective memory priors in memory systems by introducing a unified framework for memory organization and navigation, together with its training method. Tables 1 and 2 validate the framework’s applicability and effectiveness across different types of memory tasks. For a broader range of memory tasks, the framework can be extended with domain-specific memory organization methods to accommodate additional memory scenarios.

Failure case analysis. Through an attribution analysis of incorrect predictions on the test set, we identified three typical failure modes:

  1. Path-localization errors for queries involving implicit associations. When a query lacks an explicit target entity or event name, the Navigator must locate memories using indirect clues. Insufficient understanding of semantic associations can lead it to select an incorrect path.

  2. Counting and aggregation errors over multi-hop evidence. For tasks requiring cross-session counting or exhaustive enumeration, even when relevant evidence has been retrieved, the model may still omit items, count them more than once, or aggregate them incorrectly in its final answer because of limitations in its own capabilities.

  3. Reasoning errors involving complex temporal relationships. For questions involving multiple time points or time-interval calculations, the model may still confuse event order or make temporal calculation errors when generating its final answer, even after being provided with the relevant retrieved evidence.

Analysis of the underlying causes. These three failure modes reflect the system’s capability boundaries at two stages: evidence localization during retrieval and information integration after retrieval. In the first case, the absence of explicit entity anchors means that the Navigator may require stronger reasoning capabilities to capture implicit semantic associations. The latter two types of errors occur after the target evidence has been retrieved and arise from the model’s limitations in precisely aggregating fragmented information and performing complex temporal reasoning.

Question 3 — Number of Training Cycles

Both Qwen3-4B-Instruct-2507 and Qwen2.5-7B-Instruct are trained for nine alternating cycles, each consisting of one Organizer training stage followed by one Navigator training stage.

Weakness 2 & Question 5 — Trade-off Between Inference Cost and the Number of Retrieval Rounds

Regarding the trade-off between inference cost and task performance, I believe we can address your question by drawing on some interesting findings from our work. Increasing retrieval rounds provides more opportunities to gather evidence while increasing the potential overhead of navigation and evidence processing. As shown in Fig. 4, increasing MEM-SONAR’s maximum retrieval rounds from 1 to 2 improves F1 from 56.14 to 57.14, whereas further increasing the limit to 5 only raises F1 to 57.45. Thus, overall performance exhibits clear diminishing returns beyond two rounds. For cost-sensitive scenarios, 1–2 retrieval rounds provide a practical option for retaining good QA performance under a limited retrieval budget.

Reviewer_N59aOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MEM-SONAR, a long-term memory framework in which a shared language model acts as both an Organizer and a Navigator. The Organizer extracts atomic facts and generates semantic paths that place them in an episode-based memory tree. The Navigator retrieves evidence through tree-constrained beam search and generates answers. The two capabilities are trained through alternating GRPO stages: downstream question-answering rewards supervise memory construction, and the resulting trees provide the environment for navigation training.

It reports improvements over locally evaluated baselines on LoCoMo with two backbones, together with additional results on LongMemEval and NarrativeQA. Checkpoint-pairing experiments and structural analyses investigate how memory organization and navigation change during training.

Critical strengths
  1. A useful formulation of trainable memory organization. Generating complete semantic paths provides an interpretable action space for memory placement. Coupling this representation with question-driven navigation is a reasonable way to make the memory structure responsive to downstream utility.

  2. Consistent reported improvements on the controlled LoCoMo comparison. Table 1 reports overall F1 improvements over GAM of 9.19 points with Qwen3-4B and 7.17 points with Qwen2.5-7B. The gains extend across all four evaluated question categories. Appendix A.1 explicitly documents the conversation-disjoint training/evaluation split and the conversation-level aggregation procedure.

  3. Useful analyses beyond aggregate accuracy. Figure 3 evaluates different Organizer–Navigator checkpoint combinations, providing evidence that later memory constructions can help a fixed Navigator and that later Navigators can better use fixed trees. Figure 5 examines memory content and retrieval structure, while Appendix A.4 studies beam-search sensitivity.

  4. Transparent qualification of some comparisons. The paper distinguishes externally reported Memory-R1 results from locally reproduced results and explicitly acknowledges limitations of the inference-cost measurements in Appendix A.3.

Critical weaknesses
  1. The experiments do not sufficiently isolate the benefit of alternating optimization. The checkpoint-pairing study in Figure 3 is informative, but every checkpoint has already undergone alternating training. It therefore demonstrates improvements within the proposed training trajectory without establishing whether repeated alternation is responsible for them. Moreover, Navigator training includes final-answer generation, so increased QA F1 can reflect better answer generation as well as better retrieval. Figure 5 compares gains under different memory representations, but does not provide a sufficiently specified control for the optimization schedule or report absolute before/after scores for each variant. A focused comparison against Navigator-only training on fixed initial trees, and against a single Organizer-then-Navigator schedule with a comparable training budget, would substantially strengthen the central claim. Evidence-retrieval measurements would also help distinguish improved memory access from improved answering. The current results support the effectiveness of the complete system more strongly than the proposed explanation in terms of an “implicit memory prior.”

  2. The transfer results are insufficiently specified to assess their comparability and reproducibility. Table 2 reports promising gains, but Appendix A.1 does not identify the LongMemEval configuration, evaluated sample counts, or the procedure used to compute accuracy. For NarrativeQA, the evaluated split, full-document versus summary setting, and document-to-episode construction are also unspecified. These choices materially affect task difficulty.

  3. The Organizer reward is not specified sufficiently for reproduction. Equation (6) evaluates candidate episode trees using a question set Q(i)Q^{(i)}, but the manuscript does not explain how these questions are obtained or assigned to individual sessions. This is consequential because the Organizer rollout unit is a single session, whereas some benchmark questions require evidence across sessions. The handling of sessions without eligible questions is also unclear. In particular, it should be stated whether the same question set is fixed across all candidate trees, including candidates that fail to retain relevant facts. The definition of RfmtR_{\mathrm{fmt}} and the value of λfmt\lambda_{\mathrm{fmt}} are also missing from the reported configuration. Finally, Figure 2 appears to depict a frozen Navigator during Organizer training, whereas Section 3.3 states that the roles share the current parameters updated during that stage. Clarifying whether “frozen” means a separate checkpoint or only exclusion of Navigator outputs from the direct optimization loss would resolve this ambiguity.

  4. The current cost analysis cannot establish a controlled performance–cost comparison. Appendix A.3 explicitly states that accuracy and token costs may be measured on different question sets, and that some sample counts and accounting boundaries are incompletely documented. The manuscript appropriately acknowledges this limitation, but Figure 4 therefore provides limited evidence about the practical trade-off. Varying the maximum retrieval-round limit also does not match actual inference budgets across methods. A comparison on the same evaluation questions with consistent token accounting, together with separately reported memory-construction and training costs, would make the practical implications clearer. This is a secondary concern relative to the attribution and evaluation issues above.

Questions for authors
  1. Can the authors provide a focused, budget-controlled comparison with Navigator-only training on fixed trees and with a single Organizer-then-Navigator training schedule? If available, evidence-retrieval measurements would help clarify which capability improves.

  2. How is Q(i)Q^{(i)} constructed, how many training questions are eligible, and how are cross-session questions and sessions without eligible questions handled? Is Q(i)Q^{(i)} fixed across candidate trees? Please also specify RfmtR_{\mathrm{fmt}}, λfmt\lambda_{\mathrm{fmt}}, and the Navigator checkpoint used for reward evaluation.

  3. For Table 2, what are the exact dataset configurations, splits, sample counts, evaluation procedures, and per-method backbones/checkpoints? Is MEM-SONAR evaluated through direct transfer from LoCoMo without additional training or target-benchmark selection?

  4. How were the inference beam settings and final checkpoint selected? Figure 6 evaluates a large beam sweep on the six evaluation conversations, and the reported K=k=9K=k=9 configuration is its best-performing setting. Please clarify whether this configuration was fixed independently of those evaluation results. Per-conversation results and information about training-run variability would also help assess robustness.

4 comments
AuthorsOpenReview ↗

Response to the Reviewer

Weakness 1 & Question 1 — Alternating Optimization and Evidence Retrieval

Thank you for your valuable comments. The performance of memory agents depends jointly on memory extraction and construction, evidence retrieval, and downstream information integration. Mainstream baselines in this field, such as Memory-R1, GAM, LightMem, and Mem-α, do not separately evaluate evidence retrieval hit rates. Following this practice, our main paper uses end-to-end QA F1 and BLEU-1 to assess overall performance.

To address your concerns and explicitly distinguish “multi-round alternating optimization” from “one-sided/sequential training,” as well as “retrieval capability” from “generation capability,” we conducted controlled comparisons under the same computational budget among Navigator-Only training on fixed initial trees, Sequential Org→Nav training, with the first half of the budget devoted to Organizer training for tree construction and the second half to Navigator training for retrieval, and Alternating MEM-SONAR training. We jointly measured answer F1 and evidence hit rate (the proportion of required evidence items that are successfully retrieved) across five training stages. The results are shown below.

Training StageNavigator-Only (F1 / Hit)Sequential Org→Nav (F1 / Hit)Alternating MEM-SONAR (F1 / Hit)
Stage 10.3826 / 0.59630.3992 / 0.61550.4510 / 0.6877
Stage 20.4339 / 0.63720.4000 / 0.61680.4714 / 0.6832
Stage 30.4500 / 0.64690.4579 / 0.69550.4817 / 0.6816
Stage 40.4840 / 0.71710.4903 / 0.74070.5219 / 0.7621
Stage 50.4972 / 0.72630.5039 / 0.74640.5324 / 0.7516

Across all training schemes, answer F1 improves consistently, while evidence hit rates exhibit overall upward trends. These results demonstrate that our method enhances the model's ability to locate actual supporting evidence in self-organized memory trees. Navigator-Only training is constrained by the suboptimal branching structure of the initial memory trees, leading to diminishing performance gains in later stages. In Sequential Org→Nav training, although the observed metrics remain relatively stable during the first two stages when only the Organizer is optimized, this process provides a better structural foundation for Navigator training in the subsequent three stages. In contrast, multi-round alternating training repeatedly aligns tree construction and retrieval, promoting bidirectional improvements in both capabilities, and achieves the highest final answer F1 and evidence hit rate under the same budget.

Absolute Performance Before and After Training Across Memory Representations. Following the reviewer's suggestion, we further report the absolute F1 and BLEU-1 scores before and after training for all four memory variants in Figure 5(a).

Memory VariantBefore F1Before BLEU-1After F1After BLEU-1
Fact + Tree39.9835.2558.8653.53
Fact + Vector39.2934.8444.3139.60
Raw + Tree42.3437.2057.7952.33
Raw + Vector42.2136.8450.9245.07

Regardless of whether extracted facts or raw content are used, tree-based variants outperform their vector-based counterparts in both F1 and BLEU-1 after training. Notably, Fact + Tree achieves both the highest post-training performance and the largest training gains. These results further support the effectiveness of combining atomic facts with learnable hierarchical memory organization.

Failure Case Analysis. We further analyzed cases where evidence retrieval failed and found that, for questions whose required evidence was not successfully retrieved, answer F1 showed no improvement before and after training. This observation further indicates that our performance gains do not simply arise from improvements in the model's answer-generation capability.

Taken together, the controlled training-schedule comparison demonstrates that multi-round alternating optimization improves both final answer F1 and the retrieval hit rate of supporting evidence. These findings collectively support our central claim that MEM-SONAR learns more effective implicit memory priors through alternating optimization of memory organization and retrieval strategies, thereby improving memory access rather than relying on improvements in final-answer generation capability. We hope this response addresses your concerns.

AuthorsOpenReview ↗

Response to the Reviewer

Weakness 2 & Question 3 — Transfer and Evaluation Settings

Thank you for your suggestions regarding the transfer and evaluation settings in Table 2. We have added the detailed settings to Appendix A.1. MEM-SONAR uses the Cycle-9 Organizer and Cycle-9 Navigator checkpoints trained on LoCoMo with Qwen3-4B-Instruct-2507. Both checkpoints remain fixed across the two target benchmarks, without additional training. Inference prompts are adapted to the task and input formats; checkpoints, prompt variants, and inference hyperparameters are not selected based on target-benchmark evaluation results.

For LongMemEval, we use the cleaned LongMemEval-S configuration and evaluate all 500 questions, comprising 470 answerable and 30 unanswerable questions, while preserving session boundaries and timestamps. All methods use the same general correctness-judging prompt, with Qwen3-4B-Instruct-2507 as the judge. Based on the question and reference answer, the judge determines whether the prediction expresses the required core facts; accuracy is the number of correct predictions divided by all 500 questions.

For NarrativeQA, we use full documents from the test split, shuffle the documents with seed 42, select 30 documents, and include all 884 associated questions. Documents are divided at sentence boundaries into episodes of approximately 1,500 words. For each question, F1 and BLEU-1 independently take the maximum over the two reference answers, followed by an equally weighted average across questions.

Regarding model configurations, GAM, MemGPT, SimpleMem, and BM25 RAG use Qwen3-4B-Instruct-2507; LightMem uses Qwen3-30B-A3B-Instruct-2507; and Mem-α uses Memalpha-4B for memory writing and Qwen3-32B for answering. The appendix also specifies each method’s retrieval model, inference adaptations, question alignment, and procedures for handling missing predictions and judging failures.

Weakness 3 & Question 2 — Organizer Reward and Evaluation Protocol

Thank you for highlighting the ambiguities in the Organizer reward. Q(i)\mathcal{Q}^{(i)} is drawn directly from annotated QA pairs in the LoCoMo training set, without generating additional questions, and assigned to sessions using the original evidence IDs. For Single-Hop and Temporal questions, all annotated evidence must belong to the current session; for Multi-Hop questions, at least one annotated evidence item must belong to it. Under these criteria, 592 distinct training questions are eligible. Cross-session multi-hop questions can be assigned to multiple relevant sessions and scored against their original reference answers, providing proxy feedback for local memory organization.

Each session’s question set is determined before scoring and remains fixed across all eight candidate trees. Assignment depends only on the original session and annotated evidence; omitting facts from a candidate tree does not remove the corresponding questions from the scoring denominator. The QA reward is the mean token-level F1 over this fixed set, ranging from 0 to 1. No empty question sets occurred in our training data. If one occurs, the QA reward component is zero and provides no within-group advantage signal from that component.

We have added the format-reward definition and weight to Appendix A.2. Deterministic character and tag matching checks the required field markers and structural symbols in fact-extraction outputs, and verifies that path tags form matching pairs enclosing non-empty content. Rfmt=1R_{\mathrm{fmt}}=1 if all relevant outputs in an organization trajectory pass these checks, and 0 otherwise. We set λfmt=0.2\lambda_{\mathrm{fmt}}=0.2 and also report it in the hyperparameter table.

At cycle kk, the Organizer stage is initialized from the previous Navigator checkpoint. Reward evaluation uses the shared parameters of the current rollout policy, rather than a separate Navigator copy fixed throughout the stage. Navigator retrieval and answer outputs are excluded from the direct optimization loss during Organizer training. In the subsequent revision of the main text, we will provide clearer diagrams and more detailed explanations of question assignment, parameter sharing, and the scope of direct optimization to eliminate ambiguity around the term “frozen.”

AuthorsOpenReview ↗

Response to the Reviewer

Weakness 4 — Consistency of Evaluation Samples and Cost Accounting

Thank you for your suggestions regarding evaluation sample coverage and the cost-accounting protocol. In response, we have remeasured the average inference token cost per question using a common set of evaluation questions aligned across methods and a unified accounting protocol.

We have redrawn Figure 4 accordingly. Because the average token consumption per question for each method remains stable, the main trends after remeasurement are largely consistent with those in the original figure. We have also updated Appendix A.3 to clarify the aligned evaluation sample coverage and cost-accounting boundaries.

In addition, we have supplemented the appendix with offline memory-construction costs and training costs. We appreciate the reviewer’s valuable suggestions. These additions make the performance–cost analysis more rigorous and transparent and facilitate a more comprehensive assessment of the practical value of our method.


Question 4 — Independent Selection of Beam Settings and the Final Checkpoint (Cycle 9)

Thank you for your attention to experimental rigor.

First, regarding checkpoint selection and the stopping criterion, given that multi-round alternating reinforcement learning is a novel training paradigm, we predefined a fixed budget of nine alternating training cycles based on the available computational resources before starting the experiments. At the end of training, we strictly used the last checkpoint directly for evaluation. In fact, the training trajectory shows that the model continues to improve steadily at Cycle 9, with room for further gains. This directly confirms that Cycle 9 is a cutoff imposed by the predefined computational budget, rather than a performance peak selected based on the test set.

Second, regarding the beam configuration (K=k=9K=k=9), we fixed this default setting before formal testing, following preliminary probing using only a subset drawn from the training set and considering both retrieval coverage and inference time. This selection was entirely independent of the final test set.

The extensive grid sweep over the evaluation set in Figure 6 serves primarily as a post-hoc sensitivity analysis to show readers how performance varies across search widths, rather than as a basis for hyperparameter tuning. Moreover, the results in Figure 6 clearly show that performance differences across (K,k)(K,k) configurations are small and that performance generally lies on a flat plateau, demonstrating the robustness of our method’s gains.

To complement this sensitivity analysis, we will report detailed per-conversation performance and variability across independent training runs in the appendix after consolidating the evaluation results and completing the additional training experiments.

Dear reviewer N59a,

Thanks for your time and effort in reviewing ICLR2027 submissions. The authors have submitted their responses to your review. Please take the time to read, raise any further comments, and discuss them with the authors. Thanks very much!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.