acceptodds
Under review as a conference paper at ICLR 2027

Remembering Through Change: Provenance-Preserving Hypergraph Memory for Multimodal Agents

Abstract

Long-term memory for multimodal agents remains challenging, as it must preserve fine-grained visual evidence while tracking entity evolution across sessions. Existing abstraction-based memories compress images into text, often discarding visually discriminative details, whereas pixel-preserving memories retain raw observations but struggle to reliably associate diverse observations with persistent entity identities and track their evolving states. To address these limitations, we propose MHG-Mem, a multimodal hypergraph memory that jointly organizes entity identity, state evolution, and provenance to visual evidence. Specifically, MHG-Mem represents distinct entity mentions and their observed facts as individual nodes, grouping identity-consistent mentions and facts through occurrence hyperedges. It then synthesizes aspect-specific evolution states from these grouped observations and links them to supporting entities, facts, and source evidence through evolutionary and evidential hyperedges. To navigate this structured memory, we introduce hypergraph-based agentic search, which adaptively selects an entry layer, traverses typed hyperedges across memory levels, and inspects source observations when fine-grained verification is required. These designs support both detailed visual verification and reasoning over cross-session state changes. Extensive experiments across three benchmarks demonstrate the effectiveness of MHG-Mem, improving performance by 8.20-9.98% over the strongest baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
agent memory, multimodal large language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LHWdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies long-term memory for multimodal agents, with a particular focus on tracking how persistent entities evolve across multiple sessions while retaining access to the original visual evidence. The authors argue that existing abstraction-based memory systems may discard fine-grained visual details, whereas pixel-preserving systems often fail to associate visually diverse observations with the same entity or to organize its state changes over time.

To address this problem, the paper proposes MHG-Mem, an entity-centric multimodal hypergraph memory. It represents entity mentions, atomic facts, and aspect-specific state evolution as different nodes, and connects them using typed occurrence, evolution, and evidence hyperedges. A hypergraph-based agentic search procedure selects an appropriate entry layer, traverses relations across memory levels, and inspects the original dialogue or image when fine-grained verification is needed. The method is evaluated on MemEye, Mem-Gallery, and WorldMemArena-S with three backbone models. The reported results consistently outperform the included memory baselines, and the ablations attribute the gains to entity grouping, explicit evolution modeling, provenance links, and adaptive search.

Critical strengths
  1. The paper identifies a meaningful gap at the intersection of multimodal memory and temporal state tracking. Prior work has studied multimodal memory and evolving or temporal memory separately, but it remains difficult to associate visually diverse observations with a persistent entity, track its changes across sessions, and retain the original evidence needed for fine-grained verification. This is a practically relevant failure mode for long-running multimodal agents.

  2. The proposed representation is coherent and is supported by reasonably comprehensive component ablations. The occurrence, evolution, and evidence hyperedges have clear functional roles, and the agentic retrieval procedure can move between compact state summaries and original observations. The consistent improvements across three benchmarks and three backbone models suggest that this combination is useful within the evaluated setting.

Critical weaknesses

The novelty is limited and is primarily an A+B-style integration. Entity-centric memory, multimodal evidence retention, temporal state tracking, provenance links, hypergraph-based organization, and agentic retrieval have each been explored in prior work. The main novelty lies in combining these ideas for multimodal entity evolution. This is a sensible design choice, but the paper does not establish a sufficiently large conceptual or algorithmic advance beyond the combination itself. In particular, a stronger comparison against the closest graph-based or provenance-aware multimodal memory systems, as well as against simple extensions that add temporal tracking to an existing multimodal memory baseline, would be necessary to demonstrate that the proposed hypergraph formulation is essential.

  1. The evidence for generality, scalability, and robust performance is limited. The method is closely tailored to benchmarks in which entity evolution and evidence tracing are important, and the evaluation does not establish how well the full pipeline transfers to substantially different domains, longer and noisier histories, or open-ended agent environments. The system also depends on several model-generated stages, including entity association, fact extraction, evolution synthesis, and agentic retrieval. Errors in entity linking or state summarization may propagate through the graph, but these intermediate components are not independently evaluated. The additional construction, storage, and retrieval costs are also not compared with the baselines on an accuracy-cost frontier. Consequently, it remains unclear whether the observed gains justify the added complexity or generalize beyond the current evaluation setting.
Questions for authors
  1. Can the authors provide a controlled comparison that isolates the proposed joint representation from a simple combination of existing techniques? For example, how does MHG-Mem compare with (i) a strong multimodal memory baseline augmented with an explicit temporal state table, and (ii) a temporal or graph memory baseline augmented with access to the original images and provenance pointers, under the same backbone, information access, and model-call budget? This comparison would materially affect my assessment of whether the typed hypergraph provides benefits beyond an A+B integration.

  2. How robust is MHG-Mem to errors in entity association and evolution synthesis? Please report the accuracy of entity linking, state-update extraction, and evidence-edge construction, together with an error analysis showing how these errors affect downstream QA. A controlled perturbation experiment that introduces incorrect identity links or conflicting state updates would also clarify whether the method remains reliable in realistic noisy histories.

  3. Can the authors provide stronger evidence of scalability and generalization? In particular, please report performance and cost as the number of sessions, entities, and observations grows, including memory construction calls, storage size, retrieval latency, and token usage. Results on an additional domain or a less evolution-centric benchmark would help determine whether the method is a generally useful memory architecture or mainly a specialized solution for the evaluated task structure.

2 comments
AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer LHWd,

As the rebuttal deadline is approaching, we are eager to know whether our responses adequately address your concerns. We would be happy to clarify any additional questions you may have.

We have also incorporated additional experiments, analyses, and clarifications into the revised PDF, including:

  • Comparisons with graph-based, provenance-aware, and temporally extended baselines to validate the contributions of MHG-Mem (Appendix D.6).
  • Generalization to additional benchmarks (Appendix D.7).
  • Analyses of entity linking errors, hallucinations in evolution synthesis, and error propagation (Appendices D.8, E.6, and E.7).
  • Stage-wise cost breakdown (Appendix D.2).
  • Analyses of memory size, average per-sample token consumption, and QA performance in relation to the numbers of sessions, entities, and evolution states (Appendix D.3).

We would like to express our sincere gratitude for the time and effort you have dedicated to reviewing our work. It has been a pleasure engaging with you throughout this process.

Best regards,

The Authors

The current comparison does not disentangle the benefit of the proposed hypergraph memory from that of additional inference-time computation. Full MHG-Mem consumes 31,407 tokens, compared with 16,762 tokens for UniversalRAG, while agentic search alone contributes 14,696 additional tokens and 5.10 Judge points. A compute-matched UniversalRAG baseline equipped with iterative agentic retrieval is therefore necessary to determine whether the remaining gain arises from the proposed memory representation and traversal mechanism, rather than from a larger inference budget.

Reviewer_LiRGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing abstraction-based memories compress images into text, often discarding visually discriminative details, whereas pixel-preserving memories retain raw observations but struggle to reliably associate diverse observations with persistent entity identities and track their evolving states. To address these limitations, the authors propose MHG-Mem, a multimodal hypergraph memory that jointly organizes entity identity, state evolution, and provenance to visual evidence.

Critical strengths

S1:The motivation is well-articulated. The problem framing clearly exposes the failure modes of existing approaches, abstraction discarding visual cues and pixel-preserving lacking identity binding. S2: The paper gets consistent empirical gains. 8.20–9.98% improvements across three benchmarks and three backbones is convincing, with the gains holding on both EM/F1 and LLM-judge metrics. The ablations cleanly separates memory-side contributions and retrieval side contribution. Meanwhile, the paper has honest cost analysis, which acknowledges ~2× token cost over multimodal baselines, with an ablation showing the cost-performance trade-off.

Critical weaknesses

W1: The paper appears to simply plug various components into a hypergraph framework. However, it fails to articulate the actual benefits that the hypergraph formulation brings. For instance, whether it accelerates training and inference, enables data quality management through data dependencies, or offers theoretical optimization guarantees. Without such justification, the work feels more like an incremental repackaging. The paper would benefit significantly from a deeper methodological analysis that highlights the unique advantages of the hypergraph structure. W2: The method needs four LLM calls per round, which takes heavy VLM/LLM dependence in construction. W3: Eq. 4 uses max of textual/visual similarity. A single close-but-distinct entity mention (e.g., two similar-looking products) could trigger spurious merging. No analysis of coreference error rates. Meanwhile, the evolution synthesis is brittle. LLM-generated "change" detection may miss subtle diffs or hallucinate changes. The retention rule (≥2 distinct values) is too coarse?

Questions for authors

Q1:What is the entity linking error rate on a held-out subset, and how does it propagate to downstream QA? Q2:Why does 17.5% of queries force-submit, and what characterizes these cases? Q3:How does MHG-Mem handle entity coreference conflicts (e.g., two mentions with conflicting attributes)?

1 comment

I have raised my score to 3. Thanks for the authors' response.

Reviewer_ySxaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose MHG-Mem, a long-term memory system for multimodal AI agents, it links text, images, and changes over time for the same entity, while preserving access to the original evidence. This helps agents track evolving information and verify details, improving performance across three benchmarks at the cost of higher token usage.

Critical strengths
  1. The paper addresses a practical challenge: preserving visual details while tracking how the same entity changes over time. The proposed memory structure is well aligned with this problem.
  2. Results across three benchmarks and three backbone models show consistent improvements in average judge scores. Ablation studies also support the benefits of entity linking, evolution tracking, and source inspection.
Critical weaknesses
  1. State sequences follow observation order, which may differ from actual event order.
  2. Handling of delayed uploads and historical information lacks dedicated evaluation.
Questions for authors

A user may upload an older image after providing a recent observation. Although dates are preserved in textual facts, it remains unclear how the system reconstructs the correct event timeline in such cases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.