acceptodds
Under review as a conference paper at ICLR 2027

More Multimodality Is Better? Uncovering the Functional Roles of Multimodal Information in Agent Memory

Abstract

Multimodal large language model (MLLM) agents increasingly rely on multimodal memory to store and reuse past experiences for long-horizon tasks, yet it remains unclear when, where, and how multimodal information benefits memory. We present the first systematic study of how different levels of multimodal integration within the memory pipeline affect agent performance. We establish a four-level multimodal memory hierarchy that progressively introduces multimodal capabilities into memory construction, storage and retrieval, and utilization, ranging from text-only to fully multimodal memory. Experiments are conducted under long-horizon settings across 4 representative memory-intensive task domains, 10 benchmarks, and 4 MLLM backbones. Our results reveal that deeper multimodal integration does not consistently improve performance. Instead, its benefits depend on whether newly introduced multimodal capabilities address remaining evidence gaps. Three critical roles of multimodal information are identified: a Semantic Enricher during memory construction, an Experience Locator during retrieval, and an Evidence Resolver during utilization—capturing distinct ways multimodal information supports downstream decision-making. Deeper integration, however, may also introduce distraction and yield limited or negative returns. These findings challenge the prevailing assumption that richer multimodal memories universally benefit agent performance and highlight the importance of stage-aware multimodal memory design.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Memory, Agent Memory, Multimodal Integration, Memory Pipeline, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UjNhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the functional roles of multimodal information in agent memory systems. The authors introduce a four-level multimodal memory hierarchy that progressively integrates multimodal information into memory construction, storage and retrieval, and utilization. Experiments across 10 benchmarks, four task domains, and four MLLM backbones show that deeper multimodal integration does not consistently improve agent performance, with the optimal integration level varying across tasks and models.

Critical strengths
  1. Well-motivated and practically relevant research question. The paper addresses an important question in multimodal agent memory: whether increasing multimodal integration across different memory stages consistently improves downstream performance. This investigation is timely and provides useful insights for designing more effective multimodal memory systems.
  2. Clear and structured experimental framework. The proposed four-level hierarchy provides an intuitive and interpretable framework for examining multimodal integration across memory construction, storage and retrieval, and utilization. The progressive design facilitates comparisons between different integration levels within a consistent experimental setting.
Critical weaknesses
  1. Limited Novelty and Insufficient Differentiation from Prior Studies. The paper claims to present the first systematic study of multimodal integration across the agent memory pipeline. However, existing studies have already systematically investigated related aspects of multimodal agent memory. For example, Mem-Gallery [1] evaluates multimodal long-term memory across multiple functional dimensions and memory systems, while AOEB [2] examines multimodal retrieval capabilities in agent-oriented settings, including memory retrieval. Although the proposed four-level integration hierarchy provides a distinct experimental perspective, the novelty of its stage-specific analysis relative to these existing studies is not sufficiently established. The authors should more clearly articulate the unique insights enabled by their controlled integration hierarchy and clarify how their findings extend beyond prior evaluations of multimodal memory capabilities. [1] Bei, Yuanchen, et al. "Mem-gallery: Benchmarking multimodal long-term conversational memory for mllm agents." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026. [2] Zhang, Xin, et al. "AOEB: Benchmarking Agent-Oriented Multimodal Embeddings." Forty-third International Conference on Machine Learning.
  2. Insufficient Stage-Specific Analysis to Validate the Proposed Functional Roles. The paper identifies three functional roles of multimodal information, but the comparisons among L1, L2, L3, and L4 alone are insufficient to independently validate them. The proposed hierarchy evaluates only four cumulative configurations out of eight possible combinations, limiting the analysis of individual stage contributions and their interactions. More fine-grained, stage-specific ablation studies would provide stronger empirical support for the proposed functional roles.
Questions for authors
  1. How does the proposed four-level hierarchy differ from existing systematic evaluations of multimodal agent memory, such as Mem-Gallery (ACL 2026)? What new insights does it provide beyond prior studies?
  2. Why are only four cumulative configurations (L1, L2, L3, and L4) evaluated? Could additional non-cumulative configurations help isolate the contributions of individual memory stages and validate the three proposed functional roles?
  3. Are the performance differences between L1, L2, L3, and L4 statistically significant, particularly when the reported improvements are small? Could the authors provide standard deviations, confidence intervals, or significance tests?
  4. How reliable are the LLM-based assessments of historical experience relevance and evidence sufficiency? Can the authors provide human validation or inter-judge agreement results to support the proposed functional interpretations?
6 comments
AuthorsOpenReview ↗

Rebuttal (Q1)

Thank you for questions regarding our work novelty, isolation of stage-specific contributions, performance variability and human validation of LLM-judges. We will responde to them one by one.

Question 1

We clarify the distinction between our work and Mem-Gallery and AOEB from 3 perspectives, as summarized in the following table.

Mem-GalleryAOEBOur Work
Research dimensionMemory Functional CapabilitiesAgent-oriented Retrieval CapabilitiesMultimodal Memory Pipeline stages
Research ScopeLong-term Multimodal ConversationAgent-oriented Retrieval ScenariosDiverse Agent Memory Scenarios
Research ParadigmMultimodal Memory Benchmark EvaluationEmbedding Model Benchmark EvaluationControlled Empirical Analysis

Research Dimension. Mem-Gallery evaluates different memory functional capabilities, including memory extraction, reasoning, and knowledge management, while AOEB focuses on agent-oriented retrieval capabilities across different tasks. In contrast, our work investigates where multimodal information should be introduced within the memory pipeline, covering memory construction, storage and retrieval, and utilization. Rather than evaluating what memory systems or retrieval models can achieve, we examine how introducing multimodal capabilities at different memory stages affects agent performance.

Research Scope. Mem-Gallery focuses on long-term multimodal conversations, while AOEB covers diverse agent-oriented retrieval scenarios, including deep QA, code, tool, reasoning, and memory retrieval. In contrast, our work spans diverse agent memory scenarios, including GUI interaction, visual reasoning, long-term conversation understanding, and video understanding. This broader coverage allows us to investigate whether the effects of multimodal integration generalize across different agent memory scenarios, rather than being limited to a particular application setting.

Research Paradigm. Mem-Gallery provides a benchmark for evaluating different multimodal memory systems, while AOEB also benchmarks embedding models for agent-oriented retrieval. In contrast, we conduct controlled empirical analyses by progressively introducing multimodal capabilities at different memory stages within shared memory frameworks. This allows us to isolate the contributions of individual pipeline stages and understand why multimodal integration helps in some scenario. Our goal is to provide a systematic empirical understanding of multimodal integration across memory pipeline stages, rather than proposing another memory method or benchmark.

Overall, we provide new insights on where multimodal information should be integrated, and why optimal integration levels vary across scenarios, what functional roles it serves at each memory stage,

AuthorsOpenReview ↗

Rebuttal (Q2)

Question 2

We understand your concern for “cumulative method desings are insufficient to isolate stage-specific contributions”. Generally, there are 8 possible combinations in total:

Research ScopeConstructionStorage and RetrievalUtilization
In our Research
000
100
110
111
Not in our Research
010
011
001
101

in which our paper covers the first 4 cases. As for 4 unexamined cases, they can be categorized into 2 types:

(1) Configurations 010 and 011

We exclude them because they violate the architectural causality of mainstream memory pipelines. In standard frameworks, the modality of Storage & Retrieval strictly depends on the output of Memory Construction. If the construction stage produces a text-only memory item (Construction = 0), the system physically lacks the visual data required to generate multimodal embeddings for retrieval (Retrieval = 1). While generative models could hypothetically synthesize visual outputs from text, this falls outside standard memory paradigms. Therefore, [0,1,*] configurations break the natural information flow and offer limited relevance to the practical design choices investigated in this work.

(2) Configurations L1′ (001) and L2′ (101)

Since memory storage remains text-only in these configurations, visual information at the utilization stage cannot be directly retrieved from stored visual memories. Instead, the text memories must retain metadata or references that allow the system to trace back to the original raw content and retrieve the corresponding visual evidence (e.g. MemVerse). We did not include this "source-level" lookup mechanism in our initial evaluation because it involves an alternative retrieval pathway beyond the conventional paradigm considered in this work. We thank the reviewer for this insightful suggestion and will incorporate both configurations into our supplementary experiments. We conduct experiments on 4 representative benchmarks and 2 models as shown in table below.

Qwen3-VL-32B

LevelWebVoyagerAgentVistaMem-GalleryVideoMME
L155.55%11.61%55.65%61.60%
L247.89%9.81%55.99%72.22%
L349.04%10.72%65.21%72.39%
L442.67%9.09%63.06%74.18%
L1' (001)43.68%10.48%56.44%67.97%
L2' (101)43.99%8.57%54.68%74.35%

GPT-5.4-nano

LevelWebVoyagerAgentVistaMem-GalleryVideoMME
L141.00%5.61%55.24%56.37%
L240.23%9.35%54.19%61.93%
L336.78%7.48%61.51%61.27%
L423.75%7.17%60.59%64.87%
L1' (001)24.52%6.25%55.45%59.31%
L2' (101)26.05%9.24%54.18%63.56%

Key findings are as follows:

Visual Utilization can work independently of prior visual integration

Visual utilization consistently yields performance gains on VideoMME across both backbones, even when both construction and retrieval remain text-only (L1' vs. L1). In contrast, it provides little or negative gains on WebVoyager, AgentVista, and Mem-Gallery. This demonstrates that visual utilization can provide additional benefits without prior visual integration, but such benefits are task-dependent.

Visual Information Serves Distinct Functional Roles Across Memory Stages

L2' (101) and L3 (110) both incorporate visual information during construction, but differ in the other stage where visual information is introduced: utilization in L2' and retrieval in L3. Thus, they have the same visual integration depth but differ in the placement of visual information. This comparison reveals contrasting stage preferences across tasks: Mem-Gallery benefits substantially more from visual retrieval (+10.53 and +7.33 pp across the two backbones), whereas VideoMME favors visual utilization (+1.96 and +2.29 pp).

These contrasting preferences further support the distinct functional roles of multimodal information: Experience Localization during retrieval and Evidence Resolution during utilization.

AuthorsOpenReview ↗

Rebuttal (Q3)

Question 3

We report mean accuracy, standard deviations over 3 independent runs, and 95% confidence intervals for the main experimental results. The standard deviations range from 0.17 to 4.49 percentage points across benchmarks. We give the statistics for Qwen3-VL-32B-Instruct on 4 benchmarks as follows:

LevelWebVoyagerAgentVistaMem-GalleryVideoMME
L155.55% ± 2.70 [48.84, 62.26]11.61% ± 0.66 [9.97, 13.25]55.65% ± 0.45 [54.53, 56.77]61.60% ± 1.82 [57.08, 66.12]
L247.89% ± 1.92 [43.12, 52.66]9.81% ± 0.64 [8.22, 11.40]55.99% ± 0.60 [54.50, 57.48]72.22% ± 0.84 [70.13, 74.31]
L349.04% ± 4.49 [37.89, 60.19]10.72% ± 1.26 [7.59, 13.85]65.21% ± 0.18 [64.76, 65.66]72.39% ± 0.68 [70.70, 74.08]
L442.67% ± 2.55 [36.34, 49.00]9.09% ± 0.80 [7.10, 11.08]63.06% ± 0.30 [62.31, 63.81]74.18% ± 1.31 [70.93, 77.43]

Standard Deviation Ranges are summarized as follows:

BenchmarkStandard Deviation Range (pp)
WebVoyager1.22–4.49
AgentVista0.31–1.99
Mem-Gallery0.17–0.60
VideoMME0.59–1.82

We treat differences within standard deviations as normal fluctuations. For example, on Mem-Gallery with Qwen3-VL-32B, the difference between L1 and L2 is only 0.34 percentage points (55.65% ± 0.45 vs. 55.99% ± 0.60), falling within the observed variability. In contrast, the 10.62-point improvement from L1 to L2 on VideoMME (61.60% ± 1.82 vs. 72.22% ± 0.84) substantially exceeds the observed variability, indicating a significant performance change associated with the integration level.

Despite these variations, the mean results retain non-monotonic performance patterns underlying our main findings.

Reviewer_nyJmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies where multimodal information benefits agent memory. It introduces a cumulative hierarchy that enables multimodality during memory construction, storage and retrieval, and final utilization. Experiments span GUI, visual reasoning, conversation, and video tasks across multiple backbones and implementations. Results show that deeper integration does not consistently improve performance. It identifies semantic enrichment, experience localization, and visual evidence resolution as functional roles, moderated by benchmark composition and experience relevance.

Critical strengths
  1. Broad comparisons within a common experimental framework. The study varies multimodal integration within shared memory frameworks. Multiple task domains, backbones, repeated runs, and alternative implementations strengthen the evidence that richer multimodal memory is not uniformly preferable.

  2. An informative analysis beyond aggregate accuracy. Separating multimodal subsets from complete benchmarks exposes how sample composition changes the apparent value of visual memory. The paired audit distinguishes evidence availability from answer correctness, helping identify where multimodal processing is useful.

Critical weaknesses
  1. The cumulative hierarchy only partially isolates stage-specific effects. Adjacent comparisons capture gains conditional on earlier stages, leaving interactions and alternative combinations unexplored. The alternative utilization setting also replaces images with captions, making its equivalence to fully multimodal integration unclear.

  2. The explanatory audits lack sufficient validation. Single-model judgments of evidence sufficiency and relevance lack human agreement checks, while reported evidence loss may reflect judging instability. Transfer relevance is assessed using text-only retrieval, potentially missing visual retrieval patterns. Multimodal-only subsets also lack instance-level verification of visual necessity, weakening the explanatory conclusions.

Questions for authors
  1. Evaluate non-cumulative configurations or fixed-evidence interventions to distinguish stage-specific effects from interactions among construction, retrieval, and utilization.

  2. Clarify how caption-based utilization alternatives are classified and whether robustness conclusions hold with direct visual access preserved.

  3. Provide human calibration of evidence-support and transfer-relevance judgments, and repeat the relevance audit using multimodal retrieval.

  4. Verify subset conclusions through image removal or replacement, and report uncertainty for smaller subsets and marginal differences.

7 comments
AuthorsOpenReview ↗

Rebuttal (Q1)

Thank you for suggestions regarding stage-specific effects, clarification of L4 alternative, validation of LLM-judge, retrieval relevance, and verification of multimodal subsets. We will respond to them one by one.

Question 1

We understand your concern for “cumulative method desings are insufficient to isolate stage-specific contributions”. Generally, there are 8 possible combinations in total:

Research ScopeConstructionStorage and RetrievalUtilization
In our Research
000
100
110
111
Not in our Research
010
011
001
101

in which our paper covers the first 4 cases. As for 4 unexamined cases, they can be categorized into 2 types:

(1) Configurations 010 and 011

We exclude them because they violate the architectural causality of mainstream memory pipelines. In standard frameworks, the modality of Storage & Retrieval strictly depends on the output of Memory Construction. If the construction stage produces a text-only memory item (Construction = 0), the system physically lacks the visual data required to generate multimodal embeddings for retrieval (Retrieval = 1). While generative models could hypothetically synthesize visual outputs from text, this falls outside standard memory paradigms. Therefore, [0,1,*] configurations break the natural information flow and offer limited relevance to the practical design choices investigated in this work.

(2) Configurations L1′ (001) and L2′ (101)

Since memory storage remains text-only in these configurations, visual information at the utilization stage cannot be directly retrieved from stored visual memories. Instead, the text memories must retain metadata or references that allow the system to trace back to the original raw content and retrieve the corresponding visual evidence (e.g. MemVerse). We did not include this "source-level" lookup mechanism in our initial evaluation because it involves an alternative retrieval pathway beyond the conventional paradigm considered in this work. We thank the reviewer for this insightful suggestion and will incorporate both configurations into our supplementary experiments. We conduct experiments on 4 representative benchmarks and 2 models as shown in table below.

Qwen3-VL-32B

LevelWebVoyagerAgentVistaMem-GalleryVideoMME
L155.55%11.61%55.65%61.60%
L247.89%9.81%55.99%72.22%
L349.04%10.72%65.21%72.39%
L442.67%9.09%63.06%74.18%
L1' (001)43.68%10.48%56.44%67.97%
L2' (101)43.99%8.57%54.68%74.35%

GPT-5.4-nano

LevelWebVoyagerAgentVistaMem-GalleryVideoMME
L141.00%5.61%55.24%56.37%
L240.23%9.35%54.19%61.93%
L336.78%7.48%61.51%61.27%
L423.75%7.17%60.59%64.87%
L1' (001)24.52%6.25%55.45%59.31%
L2' (101)26.05%9.24%54.18%63.56%

Key findings are as follows:

Visual Utilization can work independently of prior visual integration

Visual utilization consistently yields performance gains on VideoMME across both backbones, even when both construction and retrieval remain text-only (L1' vs. L1). In contrast, it provides little or negative gains on WebVoyager, AgentVista, and Mem-Gallery. This demonstrates that visual utilization can provide additional benefits without prior visual integration, but such benefits are task-dependent.

Visual Information Serves Distinct Functional Roles Across Memory Stages

L2' (101) and L3 (110) both incorporate visual information during construction, but differ in the other stage where visual information is introduced: utilization in L2' and retrieval in L3. Thus, they have the same visual integration depth but differ in the placement of visual information. This comparison reveals contrasting stage preferences across tasks: Mem-Gallery benefits substantially more from visual retrieval (+10.53 and +7.33 pp across the two backbones), whereas VideoMME favors visual utilization (+1.96 and +2.29 pp).

These contrasting preferences further support the distinct functional roles of multimodal information: Experience Localization during retrieval and Evidence Resolution during utilization.

AuthorsOpenReview ↗

Rebuttal (Q2)

Question 2

Thank you for this insightful observation. Indeed, caption-based utilization does not strictly qualify as "fully multimodal" (L4), since the downstream model receives textual descriptions rather than original visual inputs. We have revised our description in Section 4.2 to treat it as a caption-based modification of the L4 setup, rather than an L4 configuration itself.

In our survey of existing multimodal memory systems, we rarely see an L4 implementation beyond directly providing retrieved images to the downstream model. Nevertheless, converting retrieved images into captions is also a meaningful configuration, as it allows us to examine whether textual descriptions can serve as substitutes for direct visual access during memory utilization.

Importantly, our main L4 experiments preserve direct access to original images, and our primary findings are established under this setting.

AuthorsOpenReview ↗

Rebuttal (Q3)

Question 3

(1) human calibration

We conduct an independent human validation of both retrieval relevance and paired evidence-support judgments, as detailed in Appendix G in our revised paper. For the evidence-support audit, two human experts independently annotate 60 sampled queries (120 evidence-level judgments), covering all benchmark-transition combinations. We report inter-annotator agreement and the consistency of both LLM judges with human assessments. For Answer Support, the Human-Human, GPT-Human, and Claude-Human weighted Cohen's κ scores are 0.84, 0.76, and 0.69, respectively, with corresponding Sufficient agreement rates of 88.33%, 78.33%, and 76.67%. These results support the reliability of our LLM judges.

EvaluationHuman–HumanGPT–HumanClaude–Human
Relevance (0–3)0.820.700.73
Actionable (≥ 2)91.25%85.63%81.88%
Answer Support (0–3)0.840.760.69
Sufficient (= 3)88.33%78.33%76.67%

(2) relevance audit using multimodal retrieval

We extend our relevance audit along 2 dimensions. As for retrieval modality, we consider both text-only and multimodal retrieval. As for memory representation, we evaluate both original trajectories and their corresponding distilled insights from ReasoningBank under each setting. Results using GPT-5.4-mini are shown below:

RetrievalRepresentationWebVoyagerMind2WebAgentVistaMMSearch-Plus
Mean Relevance (0–3)
TextOriginal Trajectory0.640.900.320.20
TextDistilled Insight0.860.800.440.50
MultimodalOriginal Trajectory0.610.890.330.26
MultimodalDistilled Insight0.890.780.410.62
Actionable Relevance (%)
TextOriginal Trajectory4.220.85.83.2
TextDistilled Insight2.75.12.91.3
MultimodalOriginal Trajectory3.817.35.85.7
MultimodalDistilled Insight6.57.51.01.9

As shown in table, mean relevance (0-3) remains relatively low across benchmarks (0.20–0.90). Multimodal retrieval does not consistently improve actionable relevance over text-only retrieval. Moreover, distilled insights generally exhibit lower actionable relevance than original trajectories, suggesting that semantic relevance does not necessarily translate into actionable task-specific guidance. Evaluations using Claude Haiku 5.5 are presented in Table 7 in paper.

Reviewer_szYCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an empirical study of where multimodal information helps agent memory. It defines four cumulative configurations: text-only memory, multimodal construction with textual downstream memory, multimodal storage/retrieval, and direct use of retrieved visual evidence. These configurations are instantiated in ReasoningBank, Omni-SimpleMem, and WorldMM across four task domains, eight primary benchmarks plus two additional conversation benchmarks, and four backbones. The authors find non-monotonic performance and propose three stage-specific roles: semantic enrichment, experience location, and evidence resolution. Alternative implementations, multimodal-only subsets, retrieval-relevance audits, and paired answer-support judgments provide additional analyses.

Critical strengths
  1. A useful attempt at controlled comparison across otherwise heterogeneous memory systems. Existing work such as ReasoningBank focuses on distilled transferable strategies, while WorldMM combines complementary textual and visual memories with adaptive retrieval. Their published system-level scores do not directly reveal where multimodality helps. Holding a base framework fixed within each domain and progressively modifying memory stages is therefore informative. The four-backbone evaluation, alternative implementations, and three repeated runs per configuration are meaningful strengths. The consistent L3-to-L4 degradation on several GUI/visual-reasoning benchmarks deserves attention even though it does not by itself identify the mechanism.

  2. The analysis goes beyond the headline that more modalities can hurt. The distinction between transferring past experience and preserving evidence for later questions is useful, as is the separation between evidence availability and end-to-end answer correctness. The multimodal-only analysis changes MMRC's preferred configuration from L1 to L4, illustrating how aggregate results can hide benefits. This complements MemEye's emphasis on visual necessity and resistance to caption shortcuts. Appendix F also randomizes the evidence order, hides level identities, and matches the paired conditions to the agent's evidence, which are thoughtful controls for the diagnostic judge.

References: ReasoningBank; WorldMM; MemEye.

Critical weaknesses
  1. The stage-level explanation is not sufficiently separated from the chosen integration path and implementations. The four configurations cover (0,0,0), (1,0,0), (1,1,0), and (1,1,1), so each transition estimates an effect conditional on earlier stages already being enabled. They do not establish independent stage effects or their interactions. For example, exposing raw images after text-only retrieval could behave differently from exposing images after multimodal retrieval. This matters because WorldMM already motivates selective visual evidence use by the risk of losing details in text and introducing distracting frames; the additional contribution here must be a stronger explanation of when and where these effects occur. Appendix C also simplifies WorldMM by removing semantic memory, and the alternative-implementation experiments themselves show sizable changes. I therefore view the non-monotonicity result as supported for the evaluated pipelines, but the broader stage-based design conclusions as less established. A small number of targeted counterfactual configurations would be more persuasive than simply expanding the number of backbones.

  2. The diagnostics are suggestive rather than decisive evidence for the proposed mechanism. The historical-relevance audit evaluates trajectories retrieved by L1, whereas the visual-evidence degradation being explained often concerns L3 versus L4, which share a retrieval process that can differ from L1. Moreover, evaluating the source trajectory is not the same as evaluating the reusable strategy distilled from it, which is central to ReasoningBank. The paired support audit is better controlled, but relies on one reference-aware model judge without reported human agreement, and the paper presents aggregate support changes rather than showing whether individual evidence rescues predict actual answer corrections. Appendix C.3 reports three runs, which I appreciate, but uncertainty intervals are not reported for the comparisons driving the interpretation; several Table 3 changes are below one percentage point. MemEye already establishes the importance of visual necessity, so robust stage-specific explanatory evidence is particularly important to the paper's incremental contribution. These gaps currently lead me to weak rejection, rather than a rejection of the descriptive observations themselves.

Questions for authors
  1. Targeted evidence that could change the recommendation: On representative tasks, can you test a non-cumulative configuration such as visual utilization after text-only retrieval, with the same stored source content and evidence budget? Alternatively, can you explicitly narrow the stage claims to conditional effects along the evaluated path and demonstrate which conclusions remain justified?

  2. Mechanism check: Does the low incremental relevance found for L1 also hold for the actual L3/L4 retrieval sets? How do judgments differ when assessing the injected distilled memory rather than its source trajectory? An intervention that removes low-relevance retrieved images while holding other evidence fixed would directly test the proposed distraction explanation.

  3. Validation of the existing evidence: Can you report paired uncertainty estimates across runs/queries, with appropriate grouping for shared conversations or videos, and a human-checked sample of the answer-support judgments? In particular, does evidence rescue/loss predict answer correction/regression at the individual-query level, beyond the aggregate averages in Table 3?

  4. Clarification of scope relative to prior work: Which conclusions go beyond WorldMM's selective visual-memory rationale and MemEye's visual-necessity findings? Given the removal of WorldMM's semantic memory and the sensitivity to implementation, which claims concern the adapted pipelines specifically, and which are supported across memory architectures? I am not requesting a new state-of-the-art memory method; the key issue is the strength and scope of the study's explanatory contribution.

6 comments
AuthorsOpenReview ↗

Rebuttal (Q1)

Thank you for suggestions regarding stage-specific contributions, mechanism validation, the reliability of empirical evidence, and the positioning our work. We will respond to them one by one.

Question 1: Targeted evidence that could change the recommendation

We understand your concern for “cumulative method desings are insufficient to isolate stage-specific contributions”. Generally, there are 8 possible combinations in total:

Research ScopeConstructionStorage and RetrievalUtilization
In our Research
000
100
110
111
Not in our Research
010
011
001
101

in which our paper covers the first 4 cases. As for 4 unexamined cases, they can be categorized into 2 types:

(1) Configurations 010 and 011

We exclude them because they violate the architectural causality of mainstream memory pipelines. In standard frameworks, the modality of Storage & Retrieval strictly depends on the output of Memory Construction. If the construction stage produces a text-only memory item (Construction = 0), the system physically lacks the visual data required to generate multimodal embeddings for retrieval (Retrieval = 1). While generative models could hypothetically synthesize visual outputs from text, this falls outside standard memory paradigms. Therefore, [0,1,*] configurations break the natural information flow and offer limited relevance to the practical design choices investigated in this work.

(2) Configurations L1′ (001) and L2′ (101)

Since memory storage remains text-only in these configurations, visual information at the utilization stage cannot be directly retrieved from stored visual memories. Instead, the text memories must retain metadata or references that allow the system to trace back to the original raw content and retrieve the corresponding visual evidence (e.g. MemVerse). We did not include this "source-level" lookup mechanism in our initial evaluation because it involves an alternative retrieval pathway beyond the conventional paradigm considered in this work. We thank the reviewer for this insightful suggestion and will incorporate both configurations into our supplementary experiments. We conduct experiments on 4 representative benchmarks and 2 models as shown in table below.

Qwen3-VL-32B

LevelWebVoyagerAgentVistaMem-GalleryVideoMME
L155.55%11.61%55.65%61.60%
L247.89%9.81%55.99%72.22%
L349.04%10.72%65.21%72.39%
L442.67%9.09%63.06%74.18%
L1' (001)43.68%10.48%56.44%67.97%
L2' (101)43.99%8.57%54.68%74.35%

GPT-5.4-nano

LevelWebVoyagerAgentVistaMem-GalleryVideoMME
L141.00%5.61%55.24%56.37%
L240.23%9.35%54.19%61.93%
L336.78%7.48%61.51%61.27%
L423.75%7.17%60.59%64.87%
L1' (001)24.52%6.25%55.45%59.31%
L2' (101)26.05%9.24%54.18%63.56%

Key findings are as follows:

Visual Utilization can work independently of prior visual integration

Visual utilization consistently yields performance gains on VideoMME across both backbones, even when both construction and retrieval remain text-only (L1' vs. L1). In contrast, it provides little or negative gains on WebVoyager, AgentVista, and Mem-Gallery. This demonstrates that visual utilization can provide additional benefits without prior visual integration, but such benefits are task-dependent.

Visual Information Serves Distinct Functional Roles Across Memory Stages

L2' (101) and L3 (110) both incorporate visual information during construction, but differ in the other stage where visual information is introduced: utilization in L2' and retrieval in L3. Thus, they have the same visual integration depth but differ in the placement of visual information. This comparison reveals contrasting stage preferences across tasks: Mem-Gallery benefits substantially more from visual retrieval (+10.53 and +7.33 pp across the two backbones), whereas VideoMME favors visual utilization (+1.96 and +2.29 pp). These contrasting preferences further support the distinct functional roles of multimodal information: Experience Localization during retrieval and Evidence Resolution during utilization.

AuthorsOpenReview ↗

Rebuttal (Q2)

Question 2: Mechanism check

(1) Incremental relevance in L3/L4 retrieval sets

Yes, the low incremental relevance also holds for L3/L4 retrieval sets. To validate this, we extend our relevance analysis along two dimensions: retrieval modality (text-only vs. multimodal) and memory representation (original trajectories vs. distilled insights from ReasoningBank). Results using GPT-5.4-mini are shown below:

RetrievalRepresentationWebVoyagerMind2WebAgentVistaMMSearch-Plus
Mean Relevance (0–3)
TextOriginal Trajectory0.640.900.320.20
TextDistilled Insight0.860.800.440.50
MultimodalOriginal Trajectory0.610.890.330.26
MultimodalDistilled Insight0.890.780.410.62
Actionable Relevance (%)
TextOriginal Trajectory4.220.85.83.2
TextDistilled Insight2.75.12.91.3
MultimodalOriginal Trajectory3.817.35.85.7
MultimodalDistilled Insight6.57.51.01.9

Mean relevance remains relatively low across benchmarks (0.20–0.90), and multimodal retrieval does not consistently improve actionable relevance over text-only retrieval. Moreover, distilled insights often exhibit lower actionable relevance than their source trajectories, suggesting that semantic relevance does not necessarily translate into actionable task-specific guidance. Evaluations using Claude Haiku 5.5 are additionally reported in Table 7 of the revised paper.

(2) Intervention-based evidence for the distraction explanation

We propose a distraction explanation for the observation in Section 5.2 that moving from L3 to L4 consistently degrades performance across GUI interaction and visual reasoning benchmarks. To further investigate this explanation, we conduct two controlled interventions on L4: removing retrieved images (L4_v2) and replacing them with meaningless white noise (L4_v3), while keeping other evidence fixed.

LevelWebVoyagerAgentVista
L155.55% ± 2.7011.61% ± 0.66
L247.89% ± 1.919.81% ± 0.64
L349.04% ± 4.4910.72% ± 1.26
L442.67% ± 2.259.09% ± 0.80
L4_v2 (image removal)48.66% ± 2.0110.28% ± 0.71
L4_v3 (white noise replacement)41.76% ± 3.369.58% ± 0.58

Removing images recovers performance, bringing WebVoyager and AgentVista close to their respective L3 results. In contrast, replacing images with white noise does not yield similar improvements, with performance remaining close to the original L4. These findings provide intervention-based evidence consistent with the distraction explanation, suggesting that retrieved visual content may introduce distracting information during memory utilization. The effect is less pronounced on AgentVista, potentially due to its smaller number of retrieved images.

Question 3: Validation of the existing evidence

(1) Uncertainty Estimates

We report mean accuracy, standard deviations over 3 independent runs, and 95% confidence intervals for the main experimental results. The standard deviations range from 0.17 to 4.49 percentage points across benchmarks. We give the statistics for Qwen3-VL-32B-Instruct on 4 benchmarks as follows:

LevelWebVoyagerAgentVistaMem-GalleryVideoMME
L155.55% ± 2.70 [48.84, 62.26]11.61% ± 0.66 [9.97, 13.25]55.65% ± 0.45 [54.53, 56.77]61.60% ± 1.82 [57.08, 66.12]
L247.89% ± 1.92 [43.12, 52.66]9.81% ± 0.64 [8.22, 11.40]55.99% ± 0.60 [54.50, 57.48]72.22% ± 0.84 [70.13, 74.31]
L349.04% ± 4.49 [37.89, 60.19]10.72% ± 1.26 [7.59, 13.85]65.21% ± 0.18 [64.76, 65.66]72.39% ± 0.68 [70.70, 74.08]
L442.67% ± 2.55 [36.34, 49.00]9.09% ± 0.80 [7.10, 11.08]63.06% ± 0.30 [62.31, 63.81]74.18% ± 1.31 [70.93, 77.43]

Standard Deviation Ranges are summarized as follows:

BenchmarkStandard Deviation Range (pp)
WebVoyager1.22–4.49
AgentVista0.31–1.99
Mem-Gallery0.17–0.60
VideoMME0.59–1.82

Despite these variations, the results retain non-monotonic performance patterns underlying our main findings.

(2) human validation

We conduct an independent human validation of both retrieval relevance and paired evidence-support judgments, as detailed in Appendix G in our revised paper. For the evidence-support audit, two human experts independently annotate 60 sampled queries (120 evidence-level judgments), covering all benchmark-transition combinations. We report inter-annotator agreement and the consistency of both LLM judges with human assessments. For Answer Support, the Human-Human, GPT-Human, and Claude-Human weighted Cohen's κ scores are 0.84, 0.76, and 0.69, respectively, with corresponding Sufficient agreement rates of 88.33%, 78.33%, and 76.67%. These results support the reliability of our LLM judges.

EvaluationHuman–HumanGPT–HumanClaude–Human
Relevance (0–3)0.820.700.73
Actionable (≥ 2)91.25%85.63%81.88%
Answer Support (0–3)0.840.760.69
Sufficient (= 3)88.33%78.33%76.67%

(3) individual-query level evidence support

We add an instance-level analysis in Table 8 in our revised paper to examine whether evidence rescue is associated with answer correction beyond aggregate performance comparisons. Specifically, we compare answer correction rates between queries with and without evidence rescue across the three memory transitions. The results show consistently higher correction rates in the Rescue group, with improvements of 29.1, 25.1, and 17.3 percentage points for L1→L2, L2→L3, and L3→L4, respectively. These findings demonstrate that evidence rescue is strongly associated with answer correction at the individual-query level, providing further support for the functional benefits of the corresponding memory transitions.

TransitionRescueNon-RescueDifference
L1→L251.0%22.0%+29.1 pp
L2→L343.3%18.2%+25.1 pp
L3→L430.0%12.6%+17.3 pp

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.