- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the functional roles of multimodal information in agent memory systems. The authors introduce a four-level multimodal memory hierarchy that progressively integrates multimodal information into memory construction, storage and retrieval, and utilization. Experiments across 10 benchmarks, four task domains, and four MLLM backbones show that deeper multimodal integration does not consistently improve agent performance, with the optimal integration level varying across tasks and models.
Critical strengths
- Well-motivated and practically relevant research question. The paper addresses an important question in multimodal agent memory: whether increasing multimodal integration across different memory stages consistently improves downstream performance. This investigation is timely and provides useful insights for designing more effective multimodal memory systems.
- Clear and structured experimental framework. The proposed four-level hierarchy provides an intuitive and interpretable framework for examining multimodal integration across memory construction, storage and retrieval, and utilization. The progressive design facilitates comparisons between different integration levels within a consistent experimental setting.
Critical weaknesses
- Limited Novelty and Insufficient Differentiation from Prior Studies. The paper claims to present the first systematic study of multimodal integration across the agent memory pipeline. However, existing studies have already systematically investigated related aspects of multimodal agent memory. For example, Mem-Gallery [1] evaluates multimodal long-term memory across multiple functional dimensions and memory systems, while AOEB [2] examines multimodal retrieval capabilities in agent-oriented settings, including memory retrieval. Although the proposed four-level integration hierarchy provides a distinct experimental perspective, the novelty of its stage-specific analysis relative to these existing studies is not sufficiently established. The authors should more clearly articulate the unique insights enabled by their controlled integration hierarchy and clarify how their findings extend beyond prior evaluations of multimodal memory capabilities. [1] Bei, Yuanchen, et al. "Mem-gallery: Benchmarking multimodal long-term conversational memory for mllm agents." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026. [2] Zhang, Xin, et al. "AOEB: Benchmarking Agent-Oriented Multimodal Embeddings." Forty-third International Conference on Machine Learning.
- Insufficient Stage-Specific Analysis to Validate the Proposed Functional Roles. The paper identifies three functional roles of multimodal information, but the comparisons among L1, L2, L3, and L4 alone are insufficient to independently validate them. The proposed hierarchy evaluates only four cumulative configurations out of eight possible combinations, limiting the analysis of individual stage contributions and their interactions. More fine-grained, stage-specific ablation studies would provide stronger empirical support for the proposed functional roles.
Questions for authors
- How does the proposed four-level hierarchy differ from existing systematic evaluations of multimodal agent memory, such as Mem-Gallery (ACL 2026)? What new insights does it provide beyond prior studies?
- Why are only four cumulative configurations (L1, L2, L3, and L4) evaluated? Could additional non-cumulative configurations help isolate the contributions of individual memory stages and validate the three proposed functional roles?
- Are the performance differences between L1, L2, L3, and L4 statistically significant, particularly when the reported improvements are small? Could the authors provide standard deviations, confidence intervals, or significance tests?
- How reliable are the LLM-based assessments of historical experience relevance and evidence sufficiency? Can the authors provide human validation or inter-judge agreement results to support the proposed functional interpretations?
6 comments
Rebuttal (Q2)
Question 2
We understand your concern for “cumulative method desings are insufficient to isolate stage-specific contributions”. Generally, there are 8 possible combinations in total:
| Research Scope | Construction | Storage and Retrieval | Utilization |
|---|---|---|---|
| In our Research | |||
| 0 | 0 | 0 | |
| 1 | 0 | 0 | |
| 1 | 1 | 0 | |
| 1 | 1 | 1 | |
| Not in our Research | |||
| 0 | 1 | 0 | |
| 0 | 1 | 1 | |
| 0 | 0 | 1 | |
| 1 | 0 | 1 |
in which our paper covers the first 4 cases. As for 4 unexamined cases, they can be categorized into 2 types:
(1) Configurations 010 and 011
We exclude them because they violate the architectural causality of mainstream memory pipelines. In standard frameworks, the modality of Storage & Retrieval strictly depends on the output of Memory Construction. If the construction stage produces a text-only memory item (Construction = 0), the system physically lacks the visual data required to generate multimodal embeddings for retrieval (Retrieval = 1). While generative models could hypothetically synthesize visual outputs from text, this falls outside standard memory paradigms. Therefore, [0,1,*] configurations break the natural information flow and offer limited relevance to the practical design choices investigated in this work.
(2) Configurations L1′ (001) and L2′ (101)
Since memory storage remains text-only in these configurations, visual information at the utilization stage cannot be directly retrieved from stored visual memories. Instead, the text memories must retain metadata or references that allow the system to trace back to the original raw content and retrieve the corresponding visual evidence (e.g. MemVerse). We did not include this "source-level" lookup mechanism in our initial evaluation because it involves an alternative retrieval pathway beyond the conventional paradigm considered in this work. We thank the reviewer for this insightful suggestion and will incorporate both configurations into our supplementary experiments. We conduct experiments on 4 representative benchmarks and 2 models as shown in table below.
Qwen3-VL-32B
| Level | WebVoyager | AgentVista | Mem-Gallery | VideoMME |
|---|---|---|---|---|
| L1 | 55.55% | 11.61% | 55.65% | 61.60% |
| L2 | 47.89% | 9.81% | 55.99% | 72.22% |
| L3 | 49.04% | 10.72% | 65.21% | 72.39% |
| L4 | 42.67% | 9.09% | 63.06% | 74.18% |
| L1' (001) | 43.68% | 10.48% | 56.44% | 67.97% |
| L2' (101) | 43.99% | 8.57% | 54.68% | 74.35% |
GPT-5.4-nano
| Level | WebVoyager | AgentVista | Mem-Gallery | VideoMME |
|---|---|---|---|---|
| L1 | 41.00% | 5.61% | 55.24% | 56.37% |
| L2 | 40.23% | 9.35% | 54.19% | 61.93% |
| L3 | 36.78% | 7.48% | 61.51% | 61.27% |
| L4 | 23.75% | 7.17% | 60.59% | 64.87% |
| L1' (001) | 24.52% | 6.25% | 55.45% | 59.31% |
| L2' (101) | 26.05% | 9.24% | 54.18% | 63.56% |
Key findings are as follows:
Visual Utilization can work independently of prior visual integration
Visual utilization consistently yields performance gains on VideoMME across both backbones, even when both construction and retrieval remain text-only (L1' vs. L1). In contrast, it provides little or negative gains on WebVoyager, AgentVista, and Mem-Gallery. This demonstrates that visual utilization can provide additional benefits without prior visual integration, but such benefits are task-dependent.
Visual Information Serves Distinct Functional Roles Across Memory Stages
L2' (101) and L3 (110) both incorporate visual information during construction, but differ in the other stage where visual information is introduced: utilization in L2' and retrieval in L3. Thus, they have the same visual integration depth but differ in the placement of visual information. This comparison reveals contrasting stage preferences across tasks: Mem-Gallery benefits substantially more from visual retrieval (+10.53 and +7.33 pp across the two backbones), whereas VideoMME favors visual utilization (+1.96 and +2.29 pp).
These contrasting preferences further support the distinct functional roles of multimodal information: Experience Localization during retrieval and Evidence Resolution during utilization.
Rebuttal (Q3)
Question 3
We report mean accuracy, standard deviations over 3 independent runs, and 95% confidence intervals for the main experimental results. The standard deviations range from 0.17 to 4.49 percentage points across benchmarks. We give the statistics for Qwen3-VL-32B-Instruct on 4 benchmarks as follows:
| Level | WebVoyager | AgentVista | Mem-Gallery | VideoMME |
|---|---|---|---|---|
| L1 | 55.55% ± 2.70 [48.84, 62.26] | 11.61% ± 0.66 [9.97, 13.25] | 55.65% ± 0.45 [54.53, 56.77] | 61.60% ± 1.82 [57.08, 66.12] |
| L2 | 47.89% ± 1.92 [43.12, 52.66] | 9.81% ± 0.64 [8.22, 11.40] | 55.99% ± 0.60 [54.50, 57.48] | 72.22% ± 0.84 [70.13, 74.31] |
| L3 | 49.04% ± 4.49 [37.89, 60.19] | 10.72% ± 1.26 [7.59, 13.85] | 65.21% ± 0.18 [64.76, 65.66] | 72.39% ± 0.68 [70.70, 74.08] |
| L4 | 42.67% ± 2.55 [36.34, 49.00] | 9.09% ± 0.80 [7.10, 11.08] | 63.06% ± 0.30 [62.31, 63.81] | 74.18% ± 1.31 [70.93, 77.43] |
Standard Deviation Ranges are summarized as follows:
| Benchmark | Standard Deviation Range (pp) |
|---|---|
| WebVoyager | 1.22–4.49 |
| AgentVista | 0.31–1.99 |
| Mem-Gallery | 0.17–0.60 |
| VideoMME | 0.59–1.82 |
We treat differences within standard deviations as normal fluctuations. For example, on Mem-Gallery with Qwen3-VL-32B, the difference between L1 and L2 is only 0.34 percentage points (55.65% ± 0.45 vs. 55.99% ± 0.60), falling within the observed variability. In contrast, the 10.62-point improvement from L1 to L2 on VideoMME (61.60% ± 1.82 vs. 72.22% ± 0.84) substantially exceeds the observed variability, indicating a significant performance change associated with the integration level.
Despite these variations, the mean results retain non-monotonic performance patterns underlying our main findings.
Rebuttal (Q1)
Thank you for questions regarding our work novelty, isolation of stage-specific contributions, performance variability and human validation of LLM-judges. We will responde to them one by one.
Question 1
We clarify the distinction between our work and Mem-Gallery and AOEB from 3 perspectives, as summarized in the following table.
Research Dimension. Mem-Gallery evaluates different memory functional capabilities, including memory extraction, reasoning, and knowledge management, while AOEB focuses on agent-oriented retrieval capabilities across different tasks. In contrast, our work investigates where multimodal information should be introduced within the memory pipeline, covering memory construction, storage and retrieval, and utilization. Rather than evaluating what memory systems or retrieval models can achieve, we examine how introducing multimodal capabilities at different memory stages affects agent performance.
Research Scope. Mem-Gallery focuses on long-term multimodal conversations, while AOEB covers diverse agent-oriented retrieval scenarios, including deep QA, code, tool, reasoning, and memory retrieval. In contrast, our work spans diverse agent memory scenarios, including GUI interaction, visual reasoning, long-term conversation understanding, and video understanding. This broader coverage allows us to investigate whether the effects of multimodal integration generalize across different agent memory scenarios, rather than being limited to a particular application setting.
Research Paradigm. Mem-Gallery provides a benchmark for evaluating different multimodal memory systems, while AOEB also benchmarks embedding models for agent-oriented retrieval. In contrast, we conduct controlled empirical analyses by progressively introducing multimodal capabilities at different memory stages within shared memory frameworks. This allows us to isolate the contributions of individual pipeline stages and understand why multimodal integration helps in some scenario. Our goal is to provide a systematic empirical understanding of multimodal integration across memory pipeline stages, rather than proposing another memory method or benchmark.
Overall, we provide new insights on where multimodal information should be integrated, and why optimal integration levels vary across scenarios, what functional roles it serves at each memory stage,