acceptodds
Under review as a conference paper at ICLR 2027

MemOrbit: Traceable Dual-Loop Memory Learning for Lifelong Agents

Abstract

Agents operating across a sequence of tasks must learn both how to use past experience and which experiences to retain. These decisions are coupled: retained memories shape the evidence available for future tasks, while an agent’s memory-use behavior shapes the feedback available for updating those memories. We propose MemOrbit, a traceable dual-loop reinforcement learning (RL) framework for jointly optimizing memory-management behavior and persistent memory. The policy loop uses RL to learn memory-access and management decisions expressed as tool calls. The memory loop uses RL to update entry utilities from citation-attributed task feedback, guiding retention and memory updates. The loops share task outcomes but assign credit separately to policy decisions and persistent entries. Within each training step, the agent searches and reads a fixed, versioned memory snapshot, with provenance-bound citations linking its answer to the entries read. Persistent-memory changes are committed only at step boundaries, making successive policy–memory states auditable. Experiments on MemoryBench show that MemOrbit improves performance over existing memory-augmented agent approaches, demonstrating that explicit credit assignment over both agent behavior and persistent memory enables more adaptive and reliable long-horizon learning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Memory Agent, Dual Loop, RL

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_9huTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MemOrbit, which aims to enable long-running LLM agents to learn both how to search, retrieve, and manage external memories and which memory entries to retain, update, or forget. It consists of two coupled loops: the Policy Loop uses reinforcement learning to optimize memory interaction strategies, while the Memory Loop updates memory utility scores based on task outcomes and interaction traces, such as memory retrieval and citation. The paper also uses versioned memory snapshots, paired evaluation of candidate memory states, and step-boundary commits to improve consistency and traceability. Experiments on MemoryBench show improvements over several memory baselines.

Critical strengths
  1. Meaningful problem. Long-running agents need to learn not only how to use existing memories but also how to maintain them. Studying memory use and memory evolution within a unified framework is well motivated.
  2. Interpretability and auditability. Versioned memory snapshots, explicit retrieval and citation traces, and step-boundary commits make memory updates easier to track and analyze.
  3. Clear presentation. The paper is generally well written and easy to follow.
Critical weaknesses
  1. Limited novelty. MemRL already learns memory utility from environmental feedback, while Memory-R1 uses reinforcement learning to learn memory management and utilization policies. Given these overlaps, the paper should more clearly distinguish its methodological contributions from prior work, particularly in the utility update mechanism and the interaction between the two learning loops.
  2. Limited generalizability. The evaluation is mainly based on MemoryBench and its experimental setup. Even though the benchmark covers multiple tasks, the results do not necessarily establish that the method generalizes to other long-horizon agent settings.
  3. Limited statistical evidence. MemOrbit outperforms MemRL by only 0.003 on the main metric. How many independent random seeds were used, and are variances, confidence intervals, or significance tests reported for each baseline? Also, MemOrbit performs worse than MemRL on WritingPrompts. How does this result support the paper's claim of consistent improvements?
  4. Insufficient implementation details. Some important details about the memory operations (Write, Update, Merge, and Forget) remain unclear, which limits reproducibility and makes it difficult to assess the proposed mechanism. For example, how is the utility of a merged entry initialized or recalculated? Is the utility of deleted entries retained? How many trajectories and samples are used for paired evaluation, and how is generation randomness controlled? The authors should provide these details.
Questions for authors

See the weaknesses above.

Reviewer_sAGPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a dual-loop learning framework designed to co-optimize memory usage and memory evolution in language agent systems. The framework uses reinforcement learning to update tool actions (such as searching, reading, citing, and managing memory entries) while leveraging rollouts to collect data on entry retrieval and citation to estimate "memory utility." Evaluated on the MemoryBench benchmark, the proposed approach demonstrates gains over existing baselines. Additional experiments suggest that joint optimization outperforms independent training of these components and that the model learns non-trivial memory utility values.

Critical strengths
  1. The paper addresses a relevant and important problem, which is of interest to the community.
  2. The paper is generally well organized and easy to follow.
  3. Experiments show improvements over several benchmarks.
Critical weaknesses
  1. The core methodology offers limited novelty over existing agent memory and standard RL frameworks. While co-optimizing memory usage and evolution is an interesting direction, the actual implementation closely resembles standard policy optimization for tool actions. The primary addition, using retrieval and citation outcomes to update a scalar "memory utility" is straightforward, and the manuscript leaves it unclear how these utility scores are downstream-utilized to actively guide or refine memory operations.
  2. In the memory utility formulation, rewards depend on the relative improvement of a candidate memory compared to its prior version based on retrieval and citation events. This setup introduces two major issues: a) An improvement in the candidate score demonstrates the efficacy of the macro update process, but it does not isolate or prove the individual credit of specific retrieved or cited entries. The paper fails to detail the exact mechanism by which these specific entries drive the changes in the store. b) When a memory undergoes several concurrent modifications during rollouts, the framework provides no mechanism to disentangle credit or attribute gains to individual edits vs. noise.
  3. The presentation needs further polish. Section 4.1 explicitly promises a reporting of computational overhead, yet these metrics are missing from the draft. It contains multiple incorrect or broken citations, which undermines the completeness and polish expected of a conference submission.
  4. Some priori related work are missing: [1] Yu, Hongli, et al. "Memagent: Reshaping long-context llm with multi-conv rl-based memory agent." International Conference on Learning Representations. Vol. 2026. 2026. [2] Huo, Yupeng, et al. "Atommem: Learnable dynamic agentic memory with atomic memory operation." arXiv preprint arXiv:2601.08323 (2026).
Questions for authors

line 101: missing citation.

Reviewer_8EqGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: To address the problem of how an agent can improve across a sequence of tasks by learning both how to use memory and what to keep in memory, the authors propose a reinforcement learning (RL) framework that jointly learns these two aspects.

Contributions: Previous papers have explored different ways to learn memory use, including using RL. For example, MemRL learns which memories to use and continually updates its memory. MemOrbit uses RL to train the policy that generates memory-interaction actions, including search, read, citation, and memory-management proposals.

Validation: The authors conduct experiments on the MemoryBench dataset.

Critical strengths

Use RL to train the policy that generates memory-interaction actions, including search, read, citation, and memory-management proposals.

Critical weaknesses
  1. The main results in Table 1 show that the performance gap between the proposed MemOrbit and MemRL is small. If the gain is small, why do we have to add training costs for it? In addition, there is a calculation error: (0.427 + 0.943 + 0.881 + 0.945 + 0.948 + 0.872 + 0.861 + 0.723) / 8 = 0.825, not 0.828.
Questions for authors
  1. Could the authors check the reported average? (0.427 + 0.943 + 0.881 + 0.945 + 0.948 + 0.872 + 0.861 + 0.723) / 8 = 0.825, rather than 0.828.
Reviewer_WRqVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

MemOrbit jointly adapts an agent’s memory-interaction policy and its persistent memory. The policy loop applies group-relative policy gradients to memory-related actions. The memory loop updates entry utilities from the reward difference between the candidate and previous memory snapshots, and assigns credit only to entries that were both retrieved and cited. The authors argue that frozen snapshots, provenance-bound citations and synchronized commits make the update process traceable. Experiments on MemoryBench Open-Domain with a Qwen3-8B backbone aim to demonstrate improved task performance, complementary benefits from the two loops, and the predictive value of the learned utilities.

Critical strengths
  1. The problem is well motivated.
  2. The proposed factorial ablation, longitudinal retention analysis, and memory-removal interventions target meaningful aspects of the contribution.
Critical weaknesses
  1. The headline improvement is small, lacks uncertainty estimates, and depends on an unclear aggregation procedure. Lines 298–300 describe averaging across eight reporting groups, but the displayed scores yield an unweighted mean of 0.825 rather than the reported 0.828. Please clarify whether Macro uses equal group weights or sample-count weights and reconcile this discrepancy. Even if 0.828 is correct, the improvement over MemRL is only 0.003, with no uncertainty estimates to establish its robustness. The claim of “consistent gains” also needs qualification: MemOrbit underperforms MemRL on BigBang, WritingBench, and WritingPrompts, and underperforms the no-memory agent on WritingPrompts (0.723 vs. 0.783) and HelloBench (0.872 vs. 0.886). These regressions warrant discussion.

  2. The added value of policy training over a strong single-loop method is unclear. MemRL keeps the LLM frozen and learns only memory utilities, yet it reaches 0.825. That is above the authors’ policy-only (0.792) and memory-only (0.786) variants and ties MemOrbit under equal-weight recomputation. The ablation values appear only in Fig. 3(a), which is labeled “Illustrative values,” and Sec. 4.6 reports 0.828 for a fixed-policy memory loop. The authors should confirm which ablation values come from actual runs, reconcile the two memory-only results, and show that policy training adds a reliable benefit over MemRL relative to its (unreported) compute cost.

  3. The novelty over cited work is insufficiently established. The memory loop resembles MemRL (Zhang et al.) in combining semantic filtering, utility-aware selection, and exponential-moving-average utility updates. Provenance-based memory credit is already explored by MemQ (Liao et al.), while source-indexed policy credit routing appears in ECHO (Xie et al.). The policy loop likewise uses familiar group-relative advantages and tool-observation masking, as in Search-R1 (Jin et al.). MemOrbit’s potential contribution lies in combining these components with paired bank-level reward differences and synchronized updates, but the paper does not clearly establish the methodological advance or isolate the benefits of these distinguishing mechanisms.

Questions for authors
  1. How was the Macro column in Table 1 computed? Equal-weight averaging gives approximately 0.825 for both MemOrbit and MemRL, while item-weighted averaging gives approximately 0.791 and 0.792, respectively. Neither reproduces MemOrbit’s reported 0.828.
  2. Why does memory hurt on WritingPrompts and HelloBench?
  3. How is memory credit computed and used? In Eq. 14, which rollout defines I(m,τ)I(m,\tau), which tasks generate the paired evaluations, and how are added, unchanged, and removed entries handled? Please specify how Q(m)Q(m) controls retrieval and retention, and provide evidence that citation-gated paired feedback improves over simpler reward-based or frequency-based utility estimates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.