acceptodds
Under review as a conference paper at ICLR 2027

From Passive Retrieval to Active Memory Navigation: Learning to Use Memory as a Structured Action Space

Abstract

Long-term user memory is essential for personalized conversational agents, but exposing memory only as pre-selected context limits the agent’s control over when and how to access user history. We introduce NapMem, a framework for active memory navigation that treats long-term user memory as a structured action space. NapMem organizes user histories into a multi-granularity pyramid of raw conversations, typed memory records, topic tracks, and user profiles, with provenance links connecting abstractions to supporting evidence. These levels are exposed through memory tools, and reinforcement learning over memory-tool trajectories trains the agent to decide whether to access memory, which level to consult, and when to refine or stop navigation based on intermediate evidence. On PersonaMem-v2, LongMemEval, and LoCoMo, a 9B NapMem agent achieves the highest average score among the evaluated systems, outperforming a 397B variant without memory-navigation training. Besides, the learned policy maintains performance on the evaluated non-memory reasoning and tool-use tasks, while reducing the memory-call rate. Ablation and trajectory analyses support the value of coupling structured memory with a learned, query-conditioned navigation policy

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent memory, reinforcement learning in agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_noVLOpenReview ↗
Rating
1(−1)
Confidence
4(+1)
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces NapMem, a framework that organizes long-term user memory into a multi-level structure and uses reinforcement learning to train AI agents to decide when and how to navigate memory, improving personalized question answering and retrieval efficiency.

Critical strengths
  1. The paper presents an effective framework for active long-term memory navigation.
  2. The experiments show improved accuracy and more efficient memory retrieval.
Critical weaknesses

1. The improvements may come from RL training rather than the proposed hierarchical memory navigation.

2. NapMem performs much worse on adversarial/unanswerable questions in LoCoMo.

3. The evaluation uses limited test users and only 100 LongMemEval questions, raising concerns about generalization.

Questions for authors

1. How can you show that the improvements come from hierarchical memory navigation rather than RL training alone?

2. Why does NapMem perform much worse on adversarial/unanswerable questions in LoCoMo?

5 comments

We thank the reviewer for recognizing the effectiveness of active navigation for long-term memory. We address the three concerns below with one new experiment and additional analysis.

W1 & Q1: Contribution of components

NapMem is a co-designed framework, where the memory pyramid provides a structured and provenance-linked interface, and RL teaches the 9B model how to navigate this interface effectively.

To test whether RL alone explains the gains, we trained a new baseline, Flat-GRPO, from the same 9B base model with the same GRPO settings, but with a flat interface that searches raw dialogue only. The table compares it with full NapMem and two ablations from Table 5. All values are accuracies (%).

MethodRL-trainedMemory accessLoCoMoLongMemEvalPersonaMem-v2Avg.
NapMemyespyramid + navigation59.9280.3349.3263.19
Flat-GRPOyesflat search over raw dialogue53.0575.6743.5557.42
w/o navigationyespyramid, no tools48.3768.0045.8854.08
w/o RLnopyramid + navigation52.0972.3320.7548.39
  1. RL alone does not reproduce the gains. With both agents trained by GRPO, NapMem outperforms Flat-GRPO on all three benchmarks. Flat-GRPO also stays below Mem0, the strongest baseline without policy training, whereas RL over the pyramid exceeds it.
  2. Navigation over the hierarchy matters. The w/o navigation variant gives the same RL-trained model a fixed context from the pyramid (the top five records, the top-ranked topic file, and the user profile) instead of tools, and is 9.11 points lower on average. Restricting the trained agent to records only, or removing the topic and profile levels, also lowers the average .
  3. The pyramid alone is not sufficient either. The w/o RL variant gives the base 9B model the full pyramid and the same navigation tools. Its lower accuracy shows that the 9B model cannot use this interface reliably without RL training.

The two components are complementary: the pyramid makes it possible to move from a summary to its supporting dialogue, and RL teaches the agent when that extra evidence is worth retrieving. We will add the Flat-GRPO comparison to the revision.

W2 & Q2: Adversarial/unanswerable questions in LoCoMo

The large gap is specific to this category and has an identifiable cause. Adversarial questions are scored on abstention: the correct response is to recognize that the history does not support the question's premise. Our reward scores answer correctness and memory-use appropriateness but has no dedicated abstention term, and answerable questions make up most of the training mixture. The policy is therefore rewarded mainly for gathering evidence and committing to an answer. This tendency helps on questions that require composing evidence but works against abstention.

A qualitative analysis of LoCoMo trajectories shows two concrete failure modes:

  • Answering from a near-miss memory. The agent retrieves a closely related memory and answers before verifying that it supports the question's premise. For example, when asked which film motivated Evan, the agent retrieved Sam's account of watching The Godfather and attributed that experience to Evan.
  • Inherited attribution errors. In some cases, a speaker-attribution error in a memory summary carried through to the final answer.

In both cases the agent reaches related memory but does not check it against the question's premise. The remedy follows from the cause that an explicit abstention criterion in the reward. We will add this analysis and the failure cases to the revision as a stated limitation.

W3: LongMemEval subset

We held LongMemEval out of policy training. Each question has its own long, multi-session conversation history, making evaluating all 500 questions requires building 500 separate memory banks per method. To limit construction costs, we randomly sampled 100 questions and evaluated all methods on the same subset.

LongMemEval question typeFull benchmark, n (%)Evaluated subset, n (%)
Single-session user70 (14.0%)14 (14.0%)
Single-session assistant56 (11.2%)12 (12.0%)
Single-session preference30 (6.0%)6 (6.0%)
Multi-session133 (26.6%)26 (26.0%)
Temporal reasoning133 (26.6%)27 (27.0%)
Knowledge update78 (15.6%)15 (15.0%)

The 100-question subset closely represents the question-type distribution of the full benchmark. All six types appear in proportions within 0.8 percentage points of their full-benchmark shares, supporting that using the subset scores to estimate each method’s overall accuracy on the full 500-question LongMemEval benchmark. We will include the sampling procedure and the distribution above in the revision.

Very good rebuttal, I will maintain my score

Thank you for your positive feedback on our rebuttal. We appreciate your time and consideration. If there are any remaining concerns that led you to maintain your score, we would be grateful for the opportunity to clarify them.

Reviewer_5FCoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents NapMem, a framework that equips an LLM agent with user context retrieval tools over a hierarchical memory system. The memory hierarchy contains raw conversations, structured memory records, topic tracks, and user profiles, with provenance links connecting different levels. The authors train the agent with GRPO to decide when to access memory, which level to retrieve from, and when to stop. Experiments on LoCoMo, LongMemEval, and PersonaMem-v2 show that the trained 9B model outperforms existing memory systems and an untrained 397B model using the same memory architecture. The learned policy also reduces unnecessary memory calls.

Critical strengths
  1. Hierarchical memory systems that preserve access to raw conversations are an important research direction for long-term personalization. The proposed architecture is a practical implementation that future work can build on.
  2. The proposed recipe is intuitive, and the performance is strong across multiple memory benchmarks.
  3. The empirical evaluation is reasonably thorough. The authors compare against established memory systems and provide ablations on navigation, memory granularity, and RL training to verify the method's effectiveness.
Critical weaknesses
  1. The memory maintenance mechanism could be difficult to scale. Maintaining consistency across multiple abstraction levels, especially under continuous updates and conflicting information, could be challenging.
  2. The paper only trains the agent to read memory but not writing to it. Training the LLM to maintain and retrieve memory jointly [1, 2] is also an intuitive approach and could be more efficient. For example, using the same agent for reading and writing could allow memory operations to share the conversation context and cached prefix within a session. It would be helpful to discuss the trade-offs of this design choice.
  3. A simpler retrieval baseline would help establish the benefit of the proposed architecture. For example a standard RAG pipeline could be compared. It could be cheaper to construct and maintain.

References

[1] Jiang et al. PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory. arXiv 2025.

[2] Wang et al. Mem-α: Learning Memory Construction via Reinforcement Learning. arXiv 2025.

Questions for authors

Please refer to the weaknesses.

2 comments

We thank the reviewer for the constructive assessment and for recognizing the value of retaining access to raw conversations. We respond to the three weaknesses in turn.

W1: Scalability and consistency of memory maintenance

We agree that keeping several abstraction levels consistent under continuous updates is the main maintenance challenge for a hierarchical memory. NapMem addresses it in two ways: each update is kept local, and the original evidence is never overwritten.

Local, bounded updates. Construction is incremental, so each update processes only the new session. For each proposed record, the mechanism retrieves the five most similar existing records and decides whether to add the new record, skip a duplicate, update an outdated record, or merge related ones. Related records are grouped into at most 20 topic files per user. When a fact changes, the writer edits the relevant topic file, and the profile is refreshed after topic updates under a fixed length budget.

Recoverable inconsistency. Updates and merges retain the source-message IDs and timestamps of the records they replace, and the raw-conversation layer is append-only. A stale or conflicting summary therefore never destroys evidence: the reader can trace a revised fact back to the exchanges that support it.

We will state the update rules and discuss the potential limitations in the revision.

W2: Training the reader but not the writer

Joint training of reading and writing is a reasonable alternative, and we see the two designs as making different trade-offs. We kept the writer fixed for three reasons.

  1. The two skills have different objectives. The writer decides which details of a conversation to preserve for future, unknown questions. The reader locates the evidence needed for one particular question and decides when it has enough. These objectives can diverge: a question about a user's current city gives the writer little incentive to retain when they moved.
  2. Credit assignment. If a later answer is wrong, its score alone cannot tell whether the writer omitted the date or the reader failed to find it. Fixing the writer gives the reader an unambiguous learning signal. The choice costs little in NapMem, because the raw-conversation layer is retained in full: nothing is lost at write time, and the remaining problem is one of access.
  3. Modularity. The reader can be replaced without rebuilding users' memories. In our experiments, the same banks built by a Qwen3.5-9B writer served the 9B, 122B, and 397B readers.

Our results are consistent with this choice. AgeMem trains a single agent to both write and retrieve memory, and it reaches an average of 40.18 against 63.19, indicating that concentrating the learning signal on reading is effective.

We agree with the reviewer on what our design gives up. A unified agent can share the session context and cached prefix between writing and reading, whereas NapMem's writer runs as a separate offline pass. That pass adds no interaction latency, but it has its own compute cost. The two directions are compatible. We will add this trade-off and a comparison with [1, 2] to the Discussion, and we plan to explore joint training with separate credit for preserving useful evidence and for retrieving it when needed.

W3: Comparison with standard RAG

We added two standard RAG baselines that index raw conversations directly, with no memory construction. One indexes individual messages. The other indexes chunks of up to 512 tokens with a 64-token overlap within each session. Both retrieve the top 20 items once and pass them to the answering model. The table reports mean accuracy (%) under the evaluation protocol of Table 1.

MethodLoCoMoLongMemEvalPersonaMem-v2
Raw RAG, individual messages38.9158.3310.06
Raw RAG, 512-token chunks53.5154.3317.23
NapMem59.9280.3349.32

Direct indexing avoids the work of constructing records, topic files, and profiles. Compared with the stronger raw RAG variant on each dataset, NapMem gains 6.41 percentage points on LoCoMo, 22.00 on LongMemEval, and 32.09 on PersonaMem-v2. We will add both baselines to the revision.

[1] Jiang et al. PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory. arXiv 2025.

[2] Wang et al. Mem-α: Learning Memory Construction via Reinforcement Learning. arXiv 2025.

Thank you for your response. I remain my positive rating.

Reviewer_F7mAOpenReview ↗
Rating
4
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

NapMem combines a provenance-linked memory hierarchy with GRPO-trained tool navigation. Evaluation covers memory QA, personalization, and non-memory tasks.

Critical strengths
  1. The interface connects compact abstractions to underlying evidence, providing a useful design for adaptive memory access.

  2. Component ablations and held-out LongMemEval results support practical value beyond model scaling alone. These findings motivate weak acceptance.

Critical weaknesses
  1. Evaluation independence needs clarification: LoCoMo converts shared dialogues into speaker-specific users, then splits users. Without grouping both speakers together, shared histories could cross splits. This is a potential confound, not established leakage.

  2. Efficiency statistics condition on successful runs, potentially comparing different query subsets. This limits conclusions about deployment cost.

Questions for authors
  1. Were both speakers from each LoCoMo dialogue assigned to the same split? Please provide the grouping protocol.

  2. Were the navigation and granularity ablations retrained under matched budgets, or evaluated by disabling components after training?

  3. Does the latency advantage persist over all attempted queries, including failures, and after accounting for memory construction and input tokens?

2 comments

We thank the reviewer for the careful reading and for recognizing the value of linking compact memories to their supporting evidence. We answer the three questions below; each has led to a clarification or an addition in the revision.

W1 & Q1: LoCoMo split protocol

Yes. Both speakers of a LoCoMo dialogue are always assigned to the same split, so no dialogue history is shared across splits.

The split is made on the original conversations, before any speaker-specific user is created:

  1. Each original two-speaker conversation, together with all of its question–answer pairs, is assigned to exactly one of the training, validation, or test splits.
  2. Only then do we create the two user–assistant views of that conversation: one with speaker A as the user and B as the assistant, and one with the roles reversed. Both views inherit the split of their source conversation.

The splits are therefore disjoint in speaker identities, dialogue histories, and the question–answer pairs derived from them. We apologize that Appendix A incorrectly states a 6/2/2 user split. The actual LoCoMo split is 2/1/7 (train/validation/test). We will correct this in the revision and state the conversation-level assignment explicitly.

W2 & Q3: Efficiency over all attempted queries

We recomputed mean online latency over all attempted queries, including incorrect and invalid answers, and added input tokens and memory construction time as requested.

MethodOnline latency (s/query)Input tokens/queryBuild time (s/user)Amortized time (s/query) at 100 queries/user
NapMem4.098,535.961815.90
Zep6.36≥1,243.3183814.74
Mem06.381,230.821597.97
MemOS8.411,562.33679.08
MemoryOS31.89≥4,934.9513333.22

Input tokens are summed over all model calls for a query and averaged over queries; "≥" marks lower bounds where some retrieval-stage token counts were not logged. The last column adds one hundredth of the per-user build time in Table 4 to the online latency.

  • All attempts. NapMem has the lowest mean online latency, about 36% below the next two systems (4.09 s vs. 6.36 s for Zep and 6.38 s for Mem0).
  • Input tokens. NapMem does consume more input tokens than the single-retrieval systems, because the context of a multi-step trajectory is counted again at every step. Its latency is still lower because it generates fewer output tokens than Mem0, and generation length is a major driver of latency. We will report input tokens next to latency so that this trade-off is visible in the paper.
  • Construction. NapMem builds faster than Zep and is faster online than every system in the table. Its additional build time is recovered after about 10 queries per user relative to Mem0 and about 27 relative to MemOS.

We will add this table and revise Figure 3 to report latency over all attempts for the methods with complete logs.

Q2: How the ablations were run

The navigation and granularity ablations restrict the trained policy at inference; they are not retrained. All three use the same RL-trained 9B checkpoint as full NapMem:

  • w/o navigation: the model has no retrieval tools and receives a fixed context (the top five records, the top-ranked topic file, and the user profile).
  • records-only tools: only records and their tools are exposed.
  • w/o upper levels: raw conversations and records are exposed with their tools; topic tracks and the profile are removed.

The separate w/o RL baseline is the base 9B model with the full pyramid and tool interface.

Thank you for your response. I maintain my positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.