- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MemVista benchmarks long-term multimodal memory in evolving projects. A hidden state program guides state changes, synthetic images, and multi-session conversations with tool outputs. Questions evaluate current-state recall, change trajectories, temporal boundaries, and reasoning across related facts. Experiments compare long-context, retrieval, and agentic memory systems under image, caption, and text-only inputs. Results highlight difficulties in preserving historical changes, rejecting obsolete evidence, and retaining fine-grained visual information.
Critical strengths
-
A well-motivated evaluation of temporal validity. The state program distinguishes current values, historical changes, removals, and unestablished facts. Linking these transitions to establishing messages makes the benchmark more diagnostic than ordinary factual recall. The program and evidence annotations remain hidden from evaluated systems.
-
Useful coverage of multimodal project histories. Evolving images, accepted changes, unconfirmed proposals, and tool-generated distractors create meaningful challenges for memory systems. Aligned input views reveal when visual evidence is necessary or interferes with textual reasoning.
Critical weaknesses
-
The reliability of realized evidence is insufficiently quantified. Deterministic replay ensures state consistency but not visual fidelity. Human inspection and evidence-only checks lack reported coverage, error rates, or answerability measures. Shared state programs may also mask unsupported evidence, while limited project diversity raises uncertainty.
-
System comparisons conflate memory quality with evidence access. Systems differ in visual access, memory formats, construction models, and retrieval budgets. Some use original images, while others rely on textual memories, including flattened MIRIX outputs. These differences complicate attribution of performance gains to memory quality, motivating comparisons with matched visual access and evidence budgets.
Questions for authors
-
Please quantify the human audit of image-state consistency, visual continuity, textual leakage, and reference answerability. How accurately can annotators or models answer when supplied with the correct evidence messages?
-
How do the system rankings change when original images are reattached through provenance links wherever supported, using a common answer model and matched evidence budgets?
-
Report per-project results and confidence intervals based on project-level resampling rather than treating questions from the same history as independent.
-
Distinguish failures to retain historical evidence from failures to retrieve or interpret it using the available operation-to-message annotations.