acceptodds
Under review as a conference paper at ICLR 2027

HiVe-Mem: Interleaving Horizontal Memory Expansion and Vertical Evidence Composition for Multimodal Agents

Abstract

As multimodal agents accumulate increasingly long interaction histories, effectively retrieving and utilizing past information becomes critical for long-term interaction. Existing memory systems typically rely on semantic retrieval over compressed memories and expose predefined representations, which may miss relevant memories and overlook heterogeneous evidence needs across queries. We introduce \hivemem, a multimodal memory framework that interleaves Horizontal Memory Expansion with Vertical Evidence Composition. For horizontal memory expansion, \hivemem organizes memory episodes into a relational graph using textual and visual anchors, expanding semantically retrieved seeds with related neighbors to recover complementary information. For vertical evidence composition, each retrieved memory retains multiple evidence forms, including compact summaries, captions, grounded visual views, raw images, and interaction history. A lightweight trainable selector independently composes an evidence subset for each retrieved memory, enabling query-adaptive utilization of heterogeneous multimodal information. This two-dimensional design jointly broadens memory coverage and adapts the evidence contributed by individual memories. Experiments across three multimodal memory benchmarks show that \hivemem consistently improves task performance across base models while reducing token consumption by about 90% compared with the most efficient competing multimodal baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Agent Memory, Graph, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rDm6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

HiVe-Mem combines Horizontal Memory Expansion over an episode affinity graph with Vertical Evidence Composition, which selects evidence forms for each retrieved memory. It addresses question answering over long multimodal interaction histories and reports higher average answer quality with substantially lower inference-token consumption than the evaluated multimodal baselines.

Critical strengths
  • Compared with the evaluated multimodal baselines, the complete framework improves average answer quality across three benchmarks and two base models while substantially reducing reported inference-token consumption.

  • The evidence selector transfers from Qwen3-VL-4B-Instruct to GPT-5-mini without target-model retraining, providing meaningful evidence of cross-model utility.

Critical weaknesses
  • The Vertical Evidence Composition ablation confounds adaptive selection with access to richer evidence. Its summary-only variant removes raw text, images, and other evidence forms, so the drop from 57.96 to 44.47 LJ does not isolate the contribution of the learned per-memory composition policy.

  • Horizontal Memory Expansion is compared with summary-based semantic retrieval while additionally using fine-grained textual and visual anchors. Although the retrieval count is matched, the improvement does not distinguish the value of cross-episode graph relations from the value of richer retrieval cues, leaving the specifically relational contribution unresolved.

  • The evidence selector judges the need for richer evidence from only the query and compact summary, despite possible omission of the relevant details from that summary. Query-type selection rates show modality preferences but do not establish whether these per-memory decisions track the need for information found only in the richer evidence forms.

  • HiVe-Mem’s advantage over closely related adaptive multimodal memory remains unclear. Omni-SimpleMem already combines graph-augmented retrieval with budget-controlled expansion from summaries to detailed and raw evidence. Although HiVe-Mem introduces episode-level affinities and learned per-memory composition, the reported comparisons do not establish the benefit of these distinctions over that existing design.

Questions for authors

See Weakness

Reviewer_YgHKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HiVe-Mem addresses retrieval coverage and evidence utilization in multimodal agent memory. It expands semantically retrieved memory episodes through graph links based on textual and visual anchors, then uses a learned, query-conditioned selector to choose summaries, captions, image crops, and raw text or images from each episode. Across three benchmarks, it achieves average LLM-Judge scores of 57.96 and 64.69 with Qwen3-VL-4B and GPT-5-mini, respectively, while reporting lower token consumption than the evaluated multimodal baselines.

Critical strengths
  1. This work is timely and well motivated. Expanding memory coverage and adapting the evidence contributed by each memory address complementary limitations of multimodal retrieval.

  2. Evaluation across three benchmarks and two backbones supports the method’s effectiveness. Transferring the selector to GPT-5-mini without retraining provides additional evidence of applicability across models.

Critical weaknesses
  1. The efficiency evaluation does not separately characterize graph construction and update costs or their scaling with memory size. It is unclear whether the graph is rebuilt or updated incrementally as new episodes arrive, limiting the assessment of efficiency for a growing memory.

  2. Equation (2) defines anchor matching through set intersections, while Appendix E.1 produces structured entity–attribute–value records. The paper does not explain which fields determine a match or how equivalent but differently worded values are handled. This clarification matters because matching determines graph connectivity.

  3. In Table 2, removing vertical composition also restricts each memory to its summary. The resulting performance drop therefore does not isolate the learned selector’s contribution from the benefit of access to richer evidence.

  4. The paper lacks qualitative examples showing how the retrieved episodes and selected evidence contribute to the final answer. Such examples would help illustrate how the method addresses the stated retrieval and evidence-utilization limitations.

Questions for authors
  1. Is the graph updated incrementally or rebuilt when new episodes arrive? Please clarify the update procedure and report how graph construction and update costs scale with the number of episodes.

  2. What constitutes an anchor match in Equation (2)? Which fields of the structured records must match, and are values compared exactly, normalized, or matched semantically?

  3. Does the learned selector outperform a fixed evidence-selection policy using the same available evidence forms under a comparable token budget?

  4. Can you provide qualitative examples showing how graph expansion recovers supporting memories missed by semantic retrieval and how the selected evidence contributes to the final answer?

Reviewer_jfZjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HiVe-Mem addresses retrieval and evidence selection in long multimodal interaction histories. It constructs episode memories containing summaries, captions, raw text/images, grounded image crops, and textual/visual anchors. Horizontal expansion supplements semantic retrieval with graph neighbors ranked by query-relevant, IDF-weighted shared anchors. A PPO-trained MLP then makes per-episode binary decisions over five evidence forms, using answer F1 and an optional token penalty as its reward. Evaluation covers Mem-Gallery, the Lifelong subset of WorldMemArena, and H2HMem, excluding adversarial settings. The paper reports improved mean F1/LJ and lower token consumption with Qwen3-VL-4B-Instruct and a selector transferred to GPT-5-mini, together with component ablations and evidence-use analyses.

Critical strengths
  1. A coherent, practical decomposition of multimodal memory use. The framework distinguishes which episodes to retrieve from which evidence each episode should expose. Textual and cross-modal anchor overlaps provide an interpretable expansion mechanism, while the small selector permits different evidence combinations within a query (Sections 3.1-3.3, pp. 3-5). This is a useful systems contribution for avoiding a uniform representation across heterogeneous memories.

  2. Encouraging results with meaningful existing controls. Table 1 (p. 6) reports mean LJ of 57.96 and 64.69 for the two backbones, compared with the strongest competing multimodal means of 48.35 and 60.71. The GPT-5-mini evaluation transfers selector weights without additional training. The horizontal ablation also keeps the number of retrieved episodes fixed: replacing expansion with semantic retrieval reduces LJ from 57.96 to 56.51 (Table 2, p. 8). Thus, the paper supplies evidence beyond simply retrieving more memories. These are point-estimate improvements, not demonstrated statistical significance.

Critical weaknesses
  1. The main vertical-composition ablation does not isolate the proposed adaptivity. “w/o Vertical Composition” supplies only compact summaries, reducing LJ from 57.96 to 44.47 (Section 4.4 and Table 2, pp. 7-8). This removes access to raw text, images, captions and grounded views at the same time as removing the selector. The comparison therefore demonstrates the value of richer evidence, but does not establish how much benefit comes from query-adaptive composition over a fixed multimodal bundle. Removing individual evidence types shows their utility but does not resolve this confound. Figures 4 and 6 establish different selection frequencies, not the downstream advantage of those decisions. A comparison against a validation-chosen fixed bundle and an all-evidence setting, holding retrieved episodes constant and reporting quality and tokens, would directly test the central contribution.

  2. The reported quality-cost advantage is not sufficiently auditable from the experimental protocol. The 6:2:2 split is explicitly stated (Section 4.1, p. 6; Appendix A.3, p. 13), but its unit and sample counts are unspecified. It is unclear whether test queries share histories, users or images with selector-training queries, so the generalization setting cannot be determined; this is a reporting ambiguity, not a claim of leakage. Likewise, “total input and output tokens” does not specify whether memory construction, caption/crop generation, auxiliary runtime calls or only answering are counted. Appendix A.1 describes VLM-based preprocessing, making these boundaries consequential. The stated 91.6%/89.8% reductions agree arithmetically with Table 1, but their operational scope remains unclear. A common Top-7 setting is helpful, yet does not establish comparable evidence volume across different memory representations. These omissions limit interpretation of the paper's principal accuracy/efficiency claim.

Questions for authors
  1. Limited additional validation: With the same retrieved episodes, how does the learned selector compare with a validation-chosen fixed multimodal evidence bundle and with including all available evidence? Please report F1/LJ and tokens, and a matched-budget comparison where feasible.
  2. Clarification: What is the splitting unit, what are the train/validation/test counts, and which histories/users/images overlap across partitions? Is the selector trained separately for each benchmark or on pooled training data?
  3. Clarification: Which operations and image-token conventions contribute to Table 1's Tok.? Please separate offline construction/training from online answering and auxiliary calls, and state whether the same accounting applies to all methods.
  4. Clarification: Does transfer reuse the same memory bank, summaries and retrieved episodes? Figures 4 and 6 report different visual-dominant raw-image selection rates, 35.2% versus 30.2%, although the selector is transferred and inference uses a 0.5 threshold (Table 3). Changes in selector inputs could explain this; specifying them would make the transfer result interpretable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.