acceptodds
Under review as a conference paper at ICLR 2027

MemVista: Benchmarking Long-Term Memory for Multimodal Agents in Evolving Projects

Abstract

Multimodal agents working on long-running projects must move beyond traditional memory compression and retrieval methods to dynamically track evolving project states. We present MemVista, a benchmark designed to evaluate long-term multimodal memory in project-oriented interactions spanning multiple interdependent sessions. Our automated pipeline utilizes project-specific state schemas and sequences of state operations to guide data generation and establish ground truth for memory evaluation. Observable state facts are embedded in images as fine-grained attributes and relationships, with state-driven edits applied to maintain visual continuity over time. LLM-simulated users progressively reveal state facts through multi-session interactions with a tool-augmented agent, generating conversation histories that interleave user text and images with multimodal tool outputs. We define four task families to assess recall of currently valid facts, reconstruction of changes and their stated causes, abstention when a fact is not yet established or is no longer valid, and multi-hop reasoning across interrelated facts. We systematically compare visual memory representations and context-management strategies across long-context, retrieval-based, and agentic memory systems. MemVista provides a controlled testbed for investigating how agents maintain, update, and reason over multimodal information throughout evolving projects.

Primary area:
datasets and benchmarks
Keywords:
Memory Benchmark, Multimodal Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cctpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MemVista benchmarks long-term multimodal memory in evolving projects. A hidden state program guides state changes, synthetic images, and multi-session conversations with tool outputs. Questions evaluate current-state recall, change trajectories, temporal boundaries, and reasoning across related facts. Experiments compare long-context, retrieval, and agentic memory systems under image, caption, and text-only inputs. Results highlight difficulties in preserving historical changes, rejecting obsolete evidence, and retaining fine-grained visual information.

Critical strengths
  1. A well-motivated evaluation of temporal validity. The state program distinguishes current values, historical changes, removals, and unestablished facts. Linking these transitions to establishing messages makes the benchmark more diagnostic than ordinary factual recall. The program and evidence annotations remain hidden from evaluated systems.

  2. Useful coverage of multimodal project histories. Evolving images, accepted changes, unconfirmed proposals, and tool-generated distractors create meaningful challenges for memory systems. Aligned input views reveal when visual evidence is necessary or interferes with textual reasoning.

Critical weaknesses
  1. The reliability of realized evidence is insufficiently quantified. Deterministic replay ensures state consistency but not visual fidelity. Human inspection and evidence-only checks lack reported coverage, error rates, or answerability measures. Shared state programs may also mask unsupported evidence, while limited project diversity raises uncertainty.

  2. System comparisons conflate memory quality with evidence access. Systems differ in visual access, memory formats, construction models, and retrieval budgets. Some use original images, while others rely on textual memories, including flattened MIRIX outputs. These differences complicate attribution of performance gains to memory quality, motivating comparisons with matched visual access and evidence budgets.

Questions for authors
  1. Please quantify the human audit of image-state consistency, visual continuity, textual leakage, and reference answerability. How accurately can annotators or models answer when supplied with the correct evidence messages?

  2. How do the system rankings change when original images are reattached through provenance links wherever supported, using a common answer model and matched evidence budgets?

  3. Report per-project results and confidence intervals based on project-level resampling rather than treating questions from the same history as independent.

  4. Distinguish failures to retain historical evidence from failures to retrieve or interpret it using the available operation-to-message annotations.

Reviewer_cVVAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MEMVISTA, a benchmark for evaluating long-term multimodal memory in evolving projects. It uses a state-driven pipeline to generate multi-session interactions with changing project information and evaluates existing memory systems on state recall, tracking, and reasoning.

Critical strengths

The paper addresses an important challenge in long-term agent memory: tracking evolving information rather than simply retrieving past facts.

The paper compares long-context models, retrieval-based methods, and agentic memory systems, providing useful insights into their strengths and limitations.

Critical weaknesses
  1. Although MEMVISTA contains lengthy conversations and multiple project sessions, the paper does not sufficiently analyze whether these characteristics introduce meaningful long-term memory challenges.

  2. While the benchmark is constructed around project goals and evolving states, it remains unclear to what extent the evaluation tasks require understanding cross-session project dependencies rather than retrieving and reasoning over individual state variables.

  3. The paper's motivation and benchmark setting overlap with several recent studies. In particular, RealMem already investigates long-term project-oriented interactions and evolving project states, while MemoryArena evaluates interdependent tasks across multiple sessions. MemEye and MemLens also explore multimodal memory, visual state changes, and temporal reasoning.

  4. The benchmark contains lengthy conversations and evolving project states, but the experiments do not sufficiently examine how memory performance changes with evidence distance, conversation length, or cross-session dependencies.

  5. The main findings largely confirm existing observations, such as information loss during memory compression and the importance of preserving visual evidence.

Questions for authors

Refer to the critical weaknesses.

Reviewer_bLoNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MemVista, a benchmark for long-term multimodal memory in evolving projects. For each of 20 synthetic projects across 5 domains, a hidden state program drives image generation and editing and LLM-simulated multi-session dialogues with a tool-augmented agent. This yields histories of 40+ sessions and over 300K tokens per project. Questions are categorized into four families: Snapshot, Trajectory, Boundary, and Reasoning. Evaluating long-context VLMs, retrieval systems, and agentic memory systems, the authors find that long-context models lead on average, that rankings flip on Boundary questions, and that raw images help visual questions while hurting non-visual ones.

Critical strengths
  • The motivation makes a lot of sense: testing memory in long, multimodal, tool-using projects where facts dynamically get added, updated, or deleted over time, rather than just doing static needle-in-a-haystack recall.
  • The interaction histories are quite long and realistic. Keeping visual continuity across image edits and including tool-generated distractor images also adds nice realism.
  • Evaluation covers long-context models, retrieval methods, and agentic memory systems under a shared answering backbone, and tests three different input views
  • The deterministic validation checks and provenance documentation in the appendix are thorough.
Critical weaknesses
  • Dataset scale is pretty small and there are no error bars. There are only 20 projects in total (4 per domain), and the paper never actually states the exact total number of evaluation questions.
  • **Heavy model entanglement with the GPT-5.6 family. There's a pretty obvious risk of self-preference and style bias here. You really need to check the rankings with a judge from a different model family and report inter-judge agreement.
  • No closed-book baseline to verify that questions actually require memory. Some of the numbers make me suspect that Reasoning questions can be answered from general world knowledge or prompt priors alone: for instance, simple BM25 gets 41.6 on Reasoning despite only getting 13.5 on Snapshot. Also, Reasoning is graded on a pretty loose 5-level scale by the LLM judge.
  • Inconsistencies in task format description. The main text says Snapshot, Trajectory, and Boundary are short-answer questions, whereas the appendix talks about multiple-choice options and balanced answer positions.
Questions for authors
  1. What is the exact total number of questions per task family across the 20 projects? Can you add bootstrap confidence intervals over questions/projects in Table 3?
  2. Which model acts as the tool-augmented assistant during history generation? How do the Table 3 rankings hold up if you use a non-OpenAI judge?
  3. How well does a closed-book (no-history) baseline score on each task family, especially on Reasoning?
  4. For long-context baselines, how do you handle projects (for example, 335K tokens + ~95 images) that exceed the model's context limit or image-count limit?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.