- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes DSV-Mem, a new multimodal benchmark that evaluates Dense Stateful Visual Memory. Compared to prior works, DSV-Mem focuses on professional scenarios, whereas others focus on informal everyday interactions. The benchmark comprises 10 different professional scenarios (i.e., long conversations), up to 50k tokens, that result in 1000 distinct questions (100 per scenario). Each conversation comprises structured artifacts that get revised multiple times by user updates. Questions span 5 different categories about the current and past states, derived states, history changes, and state conflicts. Results on 27 different model configurations show that the benchmark is challenging, with models failing to approach human performance. Additionally, the authors found that (i) models have a harder time tracking update histories of values compared to their final states; (ii) closed-weight models perform better than their open counterparts; (iii) memory-management methods perform below the baseline.
Critical strengths
- The benchmark evaluates a timely problem that prior work overlooked (professional scenarios vs. informal chats).
- Data curation is well validated by both human agreement and automatic checks.
- Each question provides much more candidate visual evidence compared to existing benchmarks.
- Results are validated by checking their variance, robustness to prompting, two-stage judge verification, human-judge agreement on a subset of answers, cross-judge robustness, and OCR control.
Critical weaknesses
- Limited conversations. While the benchmark features lots of questions, they actually come from a handful of conversations. Thus, it can only capture a restricted set of different cases. It would have been better, in my opinion, to propose fewer questions per conversation, but evaluate MLLMs in multiple situations.
- The framing does not match the goal. Compared to prior work [1], I would have expected multi-conversation histories. Instead, the paper proposes one relatively long conversation. As a result, it seems more of a "long context understanding" type of benchmark, rather than a benchmark that evaluates MLLM agent memory. This is exacerbated by the fact that the Haystack experiment shows that adding more tokens does not make the task harder, which questions whether the benchmark evaluates information localization (i.e., model memory) or reasoning about updates (i.e., reasoning abilities).
[1] Ren, Xiyu, et al. "Memlens: Benchmarking multimodal long-term memory in large vision-language models." arXiv preprint arXiv:2605.14906 (2026).
Questions for authors
- Could the authors test models on a clean, ordered list of updates without distractors and check whether the accuracy stays low?
- Could the authors evaluate GPT-5/6 and Opus-5.x deployed on Codex and Claude Code, respectively? Even if for a subset of questions, I think this is important to establish the current state of the art.
6 comments
Rebuttal Part 2/3
W2. ... This is exacerbated by the fact that the Haystack experiment shows that adding more tokens does not make the task harder, which questions whether the benchmark evaluates information localization (i.e., model memory) or reasoning about updates (i.e., reasoning abilities).
R2.2 This is a good point. We agree with the reviewer's observation: the main difficulty lies in handling updates, not in finding information among more tokens (Sec. 4.4); however, we kindly note that this does not make DSV-Mem a reasoning benchmark, since handling updates is a core memory ability in prior work: MemLens [1] lists "knowledge update" and "answer refusal" among its five memory abilities, and LongMemEval [2] (ICLR 2025) lists "knowledge updates" and "abstention". DSV-Mem studies these abilities in depth for professional workflows, through its Current State, Past State, Change History, and Conflict/Refusal categories (Sec. 3).
More precisely, answering a DSV-Mem question takes three steps (Sec. 3, Sec. 4.2): (i) finding and reading the relevant evidence; (ii) maintaining the state, i.e., retaining the updates and reconciling them into what is currently valid; and (iii) reasoning over the resolved state. The flat haystack result shows that the difficulty of DSV-Mem does not come from context length: if it did, adding more context would make it harder (Sec. 4.4, Fig. 7a). The main difficulty lies in step (ii), and four results show that it is distinct from the reasoning in step (iii): 1) the errors are memory failures: 89.7% of errors are state-reconstruction errors, led by lost or collapsed updates (30%) and stale values (27.1%), while downstream computation accounts for only 1.0% (Sec. 4.2, Fig. 5); 2) answering from the resolved state is easy: when the required values are provided, accuracy reaches 93% (Appendix C.4); 3) a memory structure helps: our self-ledger, which explicitly records and reconciles updates, improves accuracy by +13.6 pp on Gemini 3.7 Flash and +26.0 pp on Kimi K3 (Sec. 5); and 4) more reasoning does not solve it: raising GPT-5.5 from no reasoning to the highest effort improves accuracy from 15.5% to only 32.5%, and chain-of-thought prompting adds only +1.2 pp on Gemini 3.7 Flash and none on GPT-5.5 (Table 2; Sec. 5; Appendix D.8).
Overall, step (ii) is a core ability that professional workflows require; it is distinct from downstream reasoning, and it is what DSV-Mem is designed to measure.
Q1. Could the authors test models on a clean, ordered list of updates without distractors and check whether the accuracy stays low?
A1. Thank you for the suggestion. We have now conducted this experiment on Gemini 3.7 Flash and GPT-5.5. We replace the conversation with a clean, chronologically ordered list of the initial values and all updates relevant to the question in text, including corrections and withdrawn updates, with all other content removed. In terms of the three steps in R2.2, this removes step (i) and keeps steps (ii) and (iii). We compare it with the original conversation and with our intervention that removes the state changes while keeping the full multimodal conversation (Sec. 4.3), on the three categories for which removing statefulness is well defined (Current, Past, and Derived State), as in Fig. 6 and Fig. 12:
| Accuracy (%) | Original conversation | Clean, ordered list of updates | Statefulness removed |
|---|---|---|---|
| Gemini 3.7 Flash | 52.7 | 67.4 | 95.9 |
| GPT-5.5 | 41.4 | 60.4 | 93.3 |
As shown in the table, the accuracy does not stay low, and it does not reach the level without state changes either: 1) the clean list improves accuracy by +14.7 and +19.0 pp, so finding and reading the updates in a long, dense multimodal conversation with competing artifact revisions is a real part of the difficulty, as the reviewer suggested; and 2) a gap of 28.5 and 32.9 pp remains even when every update is laid out in order, so reconciling the updates is the larger part. This is consistent with our finding that state evolution is the dominant factor of difficulty (Sec. 4.3; Fig. 6, Fig. 12).
(continued in Part 3/3)
Rebuttal Part 3/3
Q2. Could the authors evaluate GPT-5/6 and Opus-5.x deployed on Codex and Claude Code, respectively? Even if for a subset of questions, I think this is important to establish the current state of the art.
A2. Thank you for the suggestion. We agree that this is important, and we have now evaluated the GPT and Claude families on the full benchmark in the updated draft, through their APIs: 12 configurations, for 42 in total (Table 2; Sec. 4.5; Appendix D.8). We discuss the Codex and Claude Code setting after the results.
| Model (medium effort) | Current | Past | Derived | Change History | Conflict/ Refusal | Category Avg. | Total Avg. |
|---|---|---|---|---|---|---|---|
| GPT-5.6 Luna | 26.0 ± 1.7 | 17.3 ± 4.6 | 12.8 ± 0.6 | 13.6 ± 1.7 | 8.3 ± 0.6 | 15.6 ± 0.7 | 13.7 ± 0.6 |
| GPT-5.6 Terra | 39.7 ± 1.2 | 26.7 ± 1.5 | 24.5 ± 1.5 | 20.0 ± 2.2 | 18.0 ± 2.6 | 25.8 ± 1.2 | 23.0 ± 1.5 |
| GPT-5.6 Sol | 64.3 ± 1.2 | 44.7 ± 1.5 | 48.5 ± 1.3 | 36.4 ± 3.5 | 23.1 ± 2.2 | 43.4 ± 0.8 | 39.1 ± 1.2 |
| GPT-5.5 | 53.3 ± 1.5 | 39.3 ± 3.1 | 36.5 ± 1.3 | 24.9 ± 1.2 | 11.7 ± 1.4 | 33.2 ± 0.0 | 28.2 ± 0.3 |
| Claude Opus 4.8 | 43.3 ± 1.5 | 45.0 ± 1.0 | 33.5 ± 1.3 | 20.6 ± 2.1 | 54.4 ± 2.2 | 39.4 ± 0.8 | 36.3 ± 1.0 |
| Claude Opus 4.7 | 33.3 ± 0.6 | 38.0 ± 2.0 | 23.0 ± 1.5 | 15.4 ± 1.7 | 42.8 ± 1.4 | 30.5 ± 0.7 | 27.8 ± 0.8 |
| Claude Sonnet 4.6 | 29.0 ± 2.0 | 12.3 ± 0.6 | 22.5 ± 1.3 | 11.7 ± 1.8 | 26.8 ± 2.1 | 20.5 ± 0.7 | 19.4 ± 0.9 |
| Gemini 3.7 Flash | 61.7 ± 1.5 | 46.3 ± 2.1 | 50.0 ± 1.7 | 27.8 ± 1.2 | 36.4 ± 2.5 | 44.4 ± 0.4 | 39.6 ± 0.8 |
| Human | 87.0 ± 1.0 | 88.0 ± 1.5 | 80.0 ± 0.6 | 75.0 ± 3.0 | 88.0 ± 3.0 | 83.6 ± 0.9 | 81.8 ± 1.1 |
As shown in the table, the best models of all three frontier families stay below 40% (Gemini 3.7 Flash 39.6%, GPT-5.6 Sol 39.1%, Claude Opus 4.8 36.3%), against 81.8% for humans. More reasoning effort does not close the gap either: raising GPT-5.5 from no reasoning to the highest effort improves accuracy from 15.5% to only 32.5% (Appendix D.8, Fig. 13). The full results, including the reasoning-effort sweep, are in Table 2 of the updated draft.
Regarding Codex and Claude Code, we kindly note that DSV-Mem follows the protocol of prior memory benchmarks, including MemLens [1] and MemEye [3]: base models are evaluated directly in the conversational setting through their APIs, receiving the multi-turn conversation and answering the question as the final turn, while memory-augmented systems are evaluated separately. This also ensures a fair comparison across model families. Because a coding harness wraps the model in its own memory and context-management mechanisms, it belongs to the class of memory-augmented systems rather than base models. To evaluate this class in a controlled, model-agnostic, and reproducible way, we benchmark four memory-management methods alongside our self-ledger (Table 2, Sec. 5).
We have incorporated the GPT and Claude results into the updated draft, and we sincerely hope our responses and the new results could resolve the reviewer's concerns. We are also happy to address further questions from the reviewer.
References
[1] Ren et al., MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models, arXiv 2026.
[2] Wu et al., LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory, ICLR 2025.
[3] Guo et al., MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory, arXiv 2026.
[4] Bei et al., Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents, ACL 2026.
[5] Maharana et al., Evaluating Very Long-Term Conversational Memory of LLM Agents, ACL 2024.
Rebuttal Part 1/3
Thank you for your detailed feedback and insightful questions. We are encouraged that you found that DSV-Mem "evaluates a timely problem that prior work overlooked", that the data curation is "well validated", and that each question provides "much more candidate visual evidence compared to existing benchmarks". We have uploaded an updated draft (new content in blue), which now includes the GPT and Claude families, and we address your concerns point by point below.
R1. Thank you for raising this point. We kindly note that this setting is in line with recent memory benchmarks: MemEye [3] is built on 8 tasks, Mem-Gallery [4] (ACL 2026) on 20 scenarios, and LoCoMo [5] (ACL 2024) on 10 conversations. Each workflow in DSV-Mem is also deep rather than short: on average 212 turns and 24 dense visual artifacts (up to 50K tokens).
On the number of cases, the 100 questions of a workflow do cover multiple situations: they target different fields, time points, and question categories, and the updates they depend on are drawn from a library of 27 transition types (Sec. 3, Appendix B.1). The benchmark is also not limited to these ten workflows: our generation harness can extend it to new ones, each going through the same expert review and validation (Sec. 3, Appendix B, Appendix C.4).
We also kindly note that the goal of DSV-Mem is to diagnose how MLLM agents maintain dense, evolving visual state, and the ten workflows already expose this failure clearly and consistently: eight of the ten scenarios are below 50% for the best model (Appendix D.5, Table 6), 89.7% of errors are state-reconstruction errors (Sec. 4.2), and the same findings hold across the GPT, Claude, and Gemini families (Sec. 4.5).
R2.1 Thank you for the thoughtful comment and for mentioning MemLens [1]. We already cited and discussed this work in our paper (Sec. 1, Sec. 2), compare with it directly (Table 1), and follow its evaluation setting (Sec. 4). We kindly note that MemLens evaluates its models in the same way as DSV-Mem: the history is given to the model as one complete input, and memory-augmented agents are evaluated separately, answering from their own memory. DSV-Mem covers both as well: the unaugmented models themselves, and the memory-management methods Mem0, SimpleMem, LightMem, and MIRIX (Table 2), together with our self-ledger (Sec. 5).
We also kindly point out that a professional workflow is one sustained project with many stages, so we model it as a single conversation. For a model that receives the full history, this does not change what has to be remembered. First, had we split the current conversation into several sessions, the model would still receive the same content, and the evidence for a question would still be spread over many turns and many versions of the artifacts (7.78 candidate visual artifacts per question; Table 1). Second, adding question-irrelevant content around the conversation, as a history of mixed conversations would, also changes accuracy little (Sec. 4.4, Fig. 7a). DSV-Mem therefore evaluates agent memory in the same sense as prior work.
(continued in Part 2/3)