acceptodds
Under review as a conference paper at ICLR 2027

UnfoldMem: Unfolding Multimodal Conversational Memory via Iterative Multi-Agent Interaction

Abstract

Long-term memory has become an increasingly important component of modern AI systems, enabling agents to leverage information from past interactions. More recent studies have shifted from passive retrieval toward active memory navigation, enabling agents to iteratively access relevant information from long-term memory. However, these approaches conduct both memory retrieval and evidence interpretation within the shared context, which entangles retrieval process with reasoning towards answer, making focused evidence interpretation increasingly difficult. To address this limitation, we propose UnfoldMem, a multi-agent framework that decouples memory exploration from evidence interpretation for multimodal conversational memory. Given a user query, the main agent progressively explores memory by selecting memory access tools and formulates focused sub-questions. Each sub-question is answered by a temporary sub-agent over question-specific context without further exploration, and its response guides main agent's subsequent memory access and final answer generation. Experiments on MemEye and Mem-Gallery demonstrate that UnfoldMem achieves state-of-the-art performance on multimodal long-term memory question answering. Extensive experiments and ablations further validate the effectiveness of each component of our framework and demonstrate its adaptive tool-use behavior across different questions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Term Memory, Multimodal Memory, Multi-Agent

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6MMJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UnfoldMem, which is a multi-agent framework for multimodal long-term conversational memory. The framework separates memory exploration from evidence interpretation. Specifically, a main agent iteratively selects memory access tools and formulates sub-questions, while temporary sub-agents interpret relevant conversational and visual evidence within isolated contexts. UnfoldMem combines session-level summaries with fine-grained text and image retrieval to support adaptive memory exploration. Experiments on two benchmarks, MemEye and Mem-Gallery, demonstrate improvements over existing RAG and agentic memory baselines.

Critical strengths

The paper addresses multimodal long-term memory utilization, particularly the challenges of retrieving and interpreting evidence distributed across multiple conversation sessions. Separating memory exploration from evidence interpretation provides a reasonable approach to managing growing context during iterative retrieval. The proposed method shows strong empirical performance across several benchmarks and analyses multiple settings to demonstrate the main claims. The additional case studies provide useful insights into the framework's behavior and limitations.

Critical weaknesses

Thanks for submitting the paper. The proposed method is insightful and significantly improves performance, however, there are several concerns:

  1. The proposed primarily combines established techniques, including session-level summaries, embedding-based retrieval, iterative tool selection, and sub-agent delegation. Its main contribution lies in separating memory exploration from evidence interpretation, but the necessity of this architecture remains unclear. In particular, could a single agent achieve comparable performance by iteratively formulating sub-questions, retrieving evidence, and interpreting the results? In addition, the ablation studies in Table 4 evaluate session summaries and memory access tools but do not directly examine the central main/sub-agent separation. A controlled single-agent baseline using the same memory representations, retrieval tools, and exploration budget would help determine whether the improvements arise from architectural decoupling or the underlying memory access mechanisms.
  2. Session summaries and the ask tool are closely interdependent: summaries guide session selection, while ask provides access to the selected sessions. Removing either component weakens the same session-level access pathway, making their individual contributions difficult to isolate. This may explain their nearly identical performance drops on Mem-Gallery (86.4 to 61.7 and 61.2, respectively). Although fine-grained search contributes more substantially on MemEye, its removal has a relatively small impact on Mem-Gallery (86.4 to 83.1). The authors should clarify how these results support the distinct contributions of each component across different memory settings.
  3. Despite achieving the highest overall score on MemGallery, the proposed method performs substantially worse on answer refusal (64.9) than all reported baselines. And all three ablated variants outperform the full system on this task. The authors should investigate whether this degradation reflects premature answering, over-reliance on partial evidence, or other limitations of the iterative exploration process.
  4. UnfoldMem relies on intermediate sub-agent responses to guide subsequent retrieval decisions. Errors in sub-question formulation or evidence interpretation may therefore propagate across iterations and misdirect further exploration. Appendix D.2.1 illustrates a case of retrieval drift toward an incorrect storyline. However, the paper provides limited analysis of how intermediate errors affect subsequent retrieval decisions or whether additional exploration can correct them.
  5. Several agentic memory baselines perform substantially worse than simpler RAG methods, particularly M2A and MMA on Mem-Gallery. Because these methods differ in memory representations and retrieval interfaces, it is unclear whether these gaps reflect limitations in their memory mechanisms or difficulty adapting to the benchmark. The authors should clarify the adaptations they made to these baselines and further analyse their unexpectedly low performance. In addition, the related work discusses M3-Agent and SimpleMem as representative approaches to active memory utilization, but neither is included in the experimental comparison. In particular, M3-Agent supports multi-turn reasoning with iterative memory access, making it directly relevant to the paper's claim that existing active navigation methods suffer from coupling retrieval and evidence interpretation. The authors should explain their exclusion and, if feasible, compare against these methods to better establish the advantages of UnfoldMem over existing active navigation approaches.
Questions for authors
  1. Could the authors provide a controlled comparison with a single-agent iterative navigation baseline using the same memory representations, retrieval tools, and exploration budget? This would help clarify whether the performance gains come from the proposed main/sub-agent separation rather than iterative retrieval or context management alone.
  2. Could the authors explain why closely related active memory navigation methods, particularly M3-Agent, are not included in the evaluation? If feasible, a comparison with M3-Agent would help establish the advantages of UnfoldMem over existing approaches.
  3. Could the authors further investigate the substantial performance degradation on Answer Refusal in Mem-Gallery, particularly why all three ablated variants outperform the full system on this task?
  4. Could the authors clarify the implementation and benchmark adaptation of M2A and MMA, especially the reasons for their substantially lower performance than simpler RAG baselines? Additional diagnostics on retrieval success and output validity would help assess the reliability of these comparisons.
Reviewer_hnifOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

UnfoldMem is a multi-agent framework for multimodal long-term memory. Instead of letting one agent handle both memory search and evidence interpretation in the same growing context, it uses a main agent to decide what to explore and temporary sub-agents to inspect focused pieces of memory. The system combines session summaries with text- and image-based retrieval, and the main agent can iteratively call different tools before producing the final answer.

Critical strengths

1. Good Writing and Organization

The paper is generally well written and easy to follow. The motivation, framework, experiments, and analyses are presented in a clear order.

2. Comprehensive Experimental Evaluation

The experiments are fairly extensive, covering two multimodal long-term memory benchmarks, multiple RAG and agentic-memory baselines, ablation studies, retrieval-quality analysis, token-usage analysis, tool-use patterns, and case studies.

3. Clear Implementation Details

The paper provides detailed implementation settings, prompts, tool schemas, and baseline configurations in the appendix. These details make the system easier to understand and improve the reproducibility of the work.

4. Strong Empirical Results

UnfoldMem achieves strong results on both MemEye and Mem-Gallery and consistently outperforms the reported non-oracle baselines across the main evaluation settings.

Critical weaknesses

1. Insufficient Validation of the Core Decoupling Motivation

The main motivation is that memory exploration and evidence interpretation should be separated, but the current ablations only remove components such as the session summary, ask tool, or search tools. There is no controlled baseline that keeps the same retrieved evidence, tool budget, and reasoning budget while letting a single agent interpret the evidence directly. Therefore, it is still unclear whether the gain really comes from the proposed decoupling itself.

2. Limited Academic Novelty and Insight

Most components, including session summaries, dense retrieval, tool-based exploration, and temporary LLM agents, are standard building blocks. The main novelty lies in how these components are organized rather than in a new memory or reasoning mechanism. The system works well, but the paper provides limited deeper insight into when and why this particular separation should work.

3. Error Propagation Through Intermediate Agent Outputs

Although raw evidence is isolated in temporary sub-agent contexts, the sub-agent’s textual interpretation is fed back to the main agent and directly affects later exploration. The failure cases show that an incorrect intermediate interpretation can redirect subsequent retrieval and eventually lead to a wrong answer. The paper does not analyze this cascading-error issue in much depth.

4. Clarity of Visual Presentation

The paper is generally readable, but Figure 2 is overly dense and difficult to follow. Too many components and small text are packed into a single figure, which makes the overall workflow harder to understand than necessary.

Questions for authors
  1. Could the authors provide a more direct ablation to show that the gain really comes from decoupling exploration and interpretation?

  2. Could the authors clarify what generalizable insight the proposed framework provides for multimodal memory systems?

  3. How often do incorrect sub-agent outputs affect later exploration, and is there any mechanism to correct such errors?

Reviewer_xXCJOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes UnfoldMem, a multi-agent framework for multimodal conversational memory that separates memory exploration from evidence interpretation. A main agent iteratively selects memory access tools and poses focused sub-questions, while temporary sub-agents interpret only the bounded, question-specific context exposed by each tool, with original images preserved. It is evaluated on MemEye and Mem-Gallery against retrieval and agentic memory baselines.

Critical strengths
  1. Preserving original images for sub-agents, rather than relying only on captions or summaries, is the right choice for pixel-level and scene-level visual questions, and the gains at both ends of the granularity axis support that choice.
  2. Experimental results are strong on both benchmarks, and with gains concentrated in compositional reasoning, visual reasoning, and conflict detection, which are the settings where distributed evidence should matter most.
  3. Token usage sits in an intermediate regime, higher than lightweight retrieval but substantially below the other agentic baselines that score lower, which makes the efficiency claim at least plausible at the token level.
  4. Tool use adapts to difficulty in a sensible way, with more calls for finer-grained and deeper questions, direct answers mainly for scene-level questions, and a shift from search tools for single detail questions toward session unfolding for multi-evidence questions.
Critical weaknesses
  1. The benchmark probes are artificial, so they do not show real conversational usefulness. The Figure 1 example counts Burger King ads by primary subject, and no natural memory recall query evaluation is reported.
  2. Memory construction is standard summarization plus embedding. It builds session summaries and round-level text and image embeddings while keeping the raw history, without consolidation, updating, or conflict handling.
  3. Utilization is a standard main-agent and sub-agent loop. The main agent picks tools and writes sub-questions, a temporary sub-agent reads a bounded context and is discarded, and the only twists are whole-session unfolding and original images.
  4. The strongest relevant baselines are missing, and only one backbone is used. M3-Agent and SimpleMem are cited in related work but not run, and all results use Qwen3.5-9B.
Questions for authors
  1. What would you claim as novel in your memory construction and utilization designs?
  2. What happens in a matched single-agent ablation with the same tools, summaries, and budget?
  3. How does it compare with M3-Agent and SimpleMem under the same backbone and budget? What about other stronger backbones?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.