acceptodds
Under review as a conference paper at ICLR 2027

3D-Mem: Learning to Organize Evidence for Long-Horizon Conversational Memory

Abstract

A long-horizon conversational agent can retain the information needed to answer a question yet fail to recover it within a compact context. Relevant details may remain buried in coarse summaries, disconnected across sessions, or obscured by redundant memories. We address this gap between information retention and evidence access with 3D-Mem, a framework that learns how to organize conversational evidence for future retrieval. Its central mechanism couples task-feedback-guided graph-operator induction with a provenance-linked evidence index: a PPO-trained controller selects graph edits, an LLM designer refines the operator library from training failures, and the resulting topology supplies retrieval signals grounded in source segments. Hierarchical reranking projects these signals back to the supporting text, while utility gating and diversity pruning assemble the answer context. Trained on LoCoMo with Qwen, the controller and operator library are frozen for evaluation across two benchmarks and two backbones. On LoCoMo with Qwen, 3D-Mem improves overall F1 over A-Mem by 5.58 points. With Qwen, it uses 24.2% fewer retrieved tokens than Mem0; holding graph architecture and PPO budget fixed, freezing the initial operator library lowers F1 by 1.11 points. These results support adapting memory organization to downstream evidence needs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
large language model agents, long-term conversational memory, adaptive memory organization, graph-based retrieval, memory skill evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wnGCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose 3D-Mem, a framework that learns how to organize conversational evidence for future retrieval. 3D-Mem employs a PPO controller to select graph-edit operators based on task feedback, an LLM designer to evolve the operator library from QA failures, and provenance anchors to project graph relevance back to source text chunks. Coupled with hierarchical reranking, utility gating, and diversity pruning, this framework effectively organizes and assembles evidence, achieving competitive results on the LoCoMo and LongMemEval-S benchmarks.

Critical strengths
  1. The paper proposes a 3D-Mem, a framework that learns how to organize conversational evidence for future retrieval.
  2. 3D-Mem achieves competitive performance against multiple baselines on the LoCoMo and LongMemEval-S benchmarks.
Critical weaknesses
  1. 3D-Mem use PPO for skill selection has already been explored in MemSkill. The authors merely narrow the scope from general memory skills to graph-operation skills, without providing a comparative study of the two strategies within 3D-Mem, which makes the novelty unconvincing. Furthermore, the combination of a multi-scale temporal axis and a semantic topology axis has been extensively discussed in existing works such as MAGMA, MemOS, and TiMem. 3D-Mem essentially stacks existing hierarchical summarization and graph retrieval mechanisms without introducing fundamental architectural innovations.
  2. Authors claim that 3D-Mem can recover evidence within a "compact context" in abstract. However, according to the results in Table 4, the average retrieved token cost of 3D-Mem is 1259, which is significantly higher than the baseline MemSkill (605.60). This raises my concern of whether the higher F1 performance is merely achieved at the expense of consuming substantially more tokens. Additionally, the authors should provide a comprehensive cost profiling, such as comparing the token and time costs of different baselines across the memory construction, retrieval, and generation stages, to give a fair evaluation of efficiency.
  3. 3D-Mem appears to be a patchwork of stacked modules that lack overall logical coherence. Essentially, the system relies heavily on fixed heuristic rules. 4.Eq.8 introduces an embedding-based diversity threshold δ, but no experiments demonstrating how different values of this threshold affect the trade-off between F1 performance and token cost.
  4. Retrieval budgets (Sec.4.1) for the graph and L1–L5 layers are hard-coded as '4, 16, 8, 4, 2, 1'. What is the rationale behind these specific numbers? The paper provides no explanation for this design, nor are there any ablation studies to demonstrate the effectiveness or sensitivity of these hard-coded values.
Questions for authors

Please see weakness

Reviewer_BU15OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes 3D-Mem, a long-horizon conversational memory framework built on the observation that an agent can retain information yet fail to retrieve it when needed — details get buried in summaries, scattered across sessions, or crowded out by redundant entries. Memory is organized along two axes: a multi-level temporal hierarchy (from raw conversation segments up to a user profile) cross-linked with a knowledge graph whose nodes and edges keep pointers back to the raw source text. The distinctive part is that memory organization is learned: a small RL-trained controller decides which graph-editing operation to apply to each incoming piece of information, while an LLM-based "designer" periodically adds or refines the available operations based on what kinds of questions the system got wrong during training. At answer time, the graph is used as an index — retrieval scores are projected back onto the raw source text via the provenance pointers, then filtered for usefulness and redundancy before assembling the answer context.

The system is trained on part of the LoCoMo benchmark with a Qwen backbone, then frozen and evaluated on LoCoMo and LongMemEval with two different backbones. Headline results: +5.58 overall F1 over A-Mem on LoCoMo, the best overall scores on LongMemEval with Qwen, and the best answer quality at about 24% fewer retrieved tokens than Mem0. Ablations test removal of each major mechanism, freezing the learned operation library, and replacing the RL controller with an untrained LLM selector.

Critical strengths
  1. Unusually transparent and honest reporting, which makes the system's behavior genuinely checkable. The paper reports run-to-run variance for its own method (Tables 5, 13), a deployment cost breakdown (Table 4), the full library of graph operations and how it evolved during training (Tables 6–7), all prompts and RL settings (Appendices B, E.6, Tables 8–9), and even results that go against it — including a comparison where simply keeping the full conversation history beats 3D-Mem (Table 11). Most memory-system papers omit most of this. It matters because the method is largely reproducible from the paper alone, and because it makes it possible to pinpoint which components actually carry the performance — which is exactly what the weaknesses below rely on.

  2. The paper's best idea — using the graph as an index into the raw conversation rather than as a store of answers — is well-motivated and cleanly tested. Every fact written into the graph must cite the raw text passages it came from, and at answer time the graph's relevance signals are projected back onto those original passages (Table 6; Eq. 7). A dedicated ablation isolates exactly this mechanism, holding the stored memory fixed and switching only the projection off, and shows it helps (Table 5: 0.513 → 0.503). Combined with achieving the best answer quality while retrieving ~24% fewer tokens than Mem0 (Table 4), this substantiates the paper's central quality-versus-context-cost argument.

Critical weaknesses
  1. The paper's own ablations suggest that learning is not what drives the performance. The headline gain is +5.58 F1 over A-Mem (Table 2), but removing the entire learned evolution mechanism costs only 2.5 points (Table 5) — so a variant with no learning at all still beats A-Mem comfortably. The learning-specific effects are each about 1.1 F1 (freezing the operator library, Section 4.3; replacing PPO with an untrained LLM selector, Table 10), which is barely above the system's own three-run standard deviation of 0.92 (Table 13), and both are single runs. Given that the title and framing rest on "learning to organize," this is the paper's central claim left essentially unsupported.

  2. The main comparisons are not apples-to-apples, and results outside the training condition are much weaker. 3D-Mem is trained on LoCoMo conversations 1–6 and tested on 9–10 of the same benchmark, while all baselines are training-free (Appendix E.1); the test set is only 2 conversations / 314 questions, a limitation the authors themselves note (Appendix G.3). The QA prompt also forces answers under 5–6 words while baselines keep their own prompts, inflating token-overlap metrics like F1/BLEU-1 (Appendix E.4). Where the training distribution doesn't match, baselines often win: on LongMemEval-S, 3D-Mem loses badly on Preference (5.88 vs. 16.41) and Assistant (42.98 vs. 77.99) questions, and with LLaMA, A-Mem has the best overall F1 (Table 3). Closely related graph-memory systems (Zep/Graphiti, HippoRAG) are neither compared nor cited. The superiority claim therefore only holds in the one condition the system was trained for.

Questions for authors
  1. (Clarification) Table 13 reports a three-run SD of 0.0092 F1 for the full system, while the operator-library ablation (Section 4.3) is a single-run delta of 0.0111 with no uncertainty. Can you provide repeated runs for the fixed-initial-library variant and for Table 10's LLM-as-policy selector, so the two learning-specific contrasts can be assessed against run noise?
  2. (Clarification) How were the retrieval scoring constants in Eq. 7 / Appendix E.3 set (ω1=0.49, ω2=0.30, w_g=0.21, w_text=0.7, M=2, the 0.3 L1 threshold, mts_prior_alpha/beta=1.0, mts_exploration_bonus=1.5)? Were they tuned on LoCoMo validation conversations 7-8, and if so, were baselines given any comparable tuning budget?
  3. (Suggestion) Re-run the headline LoCoMo and LongMemEval-S comparisons with a common QA prompt (identical answer-length constraint) for all methods, or report L-J-only comparisons as primary, since Appendix E.4 acknowledges the answer-length confound and F1/BLEU-1 are length-sensitive.
  4. (Clarification) In Table 2, how were baseline numbers obtained — re-run by you with Qwen3-30B-A3B/LLaMA3.3-70B and GPT-4o-mini judging, or transcribed from prior papers? If re-run, were retrieval budgets matched to 3D-Mem's (Section 4.1: 4,16,8,4,2,1), and why do budgets differ between LoCoMo and LongMemEval-S for your method?
  5. (Clarification) For the LLM-as-a-Judge protocol (Appendix E.5): the judge prompt instructs the judge to "be generous" twice. Did you measure agreement between GPT-4o-mini judgments and human labels, or judge sensitivity across judges, given that several headline claims (e.g., best overall L-J with LLaMA on LongMemEval-S, Table 3) rest on L-J margins of 0.2 points?
  6. (Suggestion) Position the work against Zep/Graphiti (arXiv:2501.13956) and HippoRAG, which also build graph-indexed memories with provenance to source utterances and report LoCoMo/LongMemEval results; at minimum, Appendix A should discuss them.
  7. (Clarification) Do you intend to release code and the induced operator libraries? The paper contains prompts and hyperparameters but no code or data release statement, and components such as the Meta-Reflector clustering (Section 3.3.2, K-means over query representations: how many clusters, which features?) and the Gumbel-Top-k rollout details are not fully specified.
Reviewer_hDBBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents 3D-Mem, a memory system for answering questions about long conversation histories. Its main idea is that storing information is not enough that the system also needs to organize it so the right evidence can be found later. It combines summaries at different time scales with a graph linking people, events, and dialogue segments. Experiments on LoCoMo and LongMemEval-S with two language models report improvements in answer quality and more compact answer contexts.

Critical strengths

3D mem with five-level temporal memory is kind of motivated and interesting. The graph-based ablation holds stored memory fixed which providing useful evidence retrieval. I think it‘s efficient for conversational QA.

Critical weaknesses
  1. I find the novelty limited, mainly because the central ideas are already present in closely related work. MemSkill has the same controller–executor–designer loop for selecting and evolving memory skills, and in fact, this kind of loop has been widely use in many workflows; TiMem provides the five-level temporal hierarchy; and LiCoMemory already uses graph and session relevance to recover original dialogue chunks. What 3D-Mem adds is applying evolving skills to source-linked graph edits and feeding graph scores directly into chunk reranking. That could be useful, but the current experiments do not show clearly why this combination is more than a straightforward extension of these methods.
  2. Inconsistent result and confound gains are listed in my question.
  3. Your main method is deploy a graph structure to lead the retrieval and rerank on MemSkill, failure-driven and iterative reflection is not kind of novel. So you have to rethink or reorganize whats your core mechanism or ability that you added to 3DMem distinguishes from others.
Questions for authors
  1. I find some inconsistent reporting which you have to explain. In Table 3, Qwen–MemoryOS category scores imply 242/500 correct judgments (48.4%), whereas overall L-J is 50.8%, equivalent to 254/500. Rounding cannot explain this discrepancy. Table 2's Qwen–Ours L-J of 76.79% is also incompatible with a single binary evaluation of all 314 questions; different denominators or averaging require clarification. LiCoMemory F1 additionally differs between Tables 2 and 4 (38.38% versus 36.39%) which I double check and make sure they are the same method on same dataset. Please reconcile the tables.
  2. I really have doubts about the gains which shown in the main table when I see the exact prompt of each task. For example, explicitly instructs the model that “‘X is my weakness’ means loves or favorite X” and “‘I limit myself to X per day’ means loves X". Maybe it is well Additional, which you write in Appendix E.4 that acknowledges that answer prompts differ across methods. Without a shared-prompt evaluation and ablations of these rules, the paper does not isolate whether the gains arise from improved memory organization or just a better prompt.
  3. In Figure 2, the ablation results show one typical example is the significance of L1. So, could you please explain what the purpose of your L2-L5 design is for both sequential tasks and other memory tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.