acceptodds
Withdrawn from ICLR 2027

DSV-Mem: Evaluating multimodal memory in professional workflows for MLLM agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Conversational MLLM agents are increasingly expected to assist in professional workflows: from AI research and engineering design to product management and business operations. Despite this growing need, this capability has not been well explored: existing benchmarks largely focus on informal, everyday interactions and personal-life scenarios that typically feature photographic natural images, isolated, static artifacts, and recall-oriented questions. In contrast, professional scenarios often involve structured, information-heavy artifacts that undergo frequent revisions and authority updates, as well as compositional queries that require reconciling many artifact versions in memory while tracking state precisely. To address these challenges, we introduce DSV-Mem, a carefully curated benchmark for evaluating Dense Stateful Visual Memory. DSV-Mem comprises a diverse suite of expert-reviewed scenarios and 1,000 questions across five user-oriented categories (Current State, Past State, Derived State, Change History, and Conflict/Refusal). Moreover, a Hartley-inspired criterion is introduced to favor questions with broader visual-evidence inspection demands. To support further study, we introduce a generation harness that produces high-quality, domain-specific evaluation suites by decoupling state-transition synthesis from conversation filling. Rigorous evaluation over 27 configurations spanning frontier, open-weight models and popular memory management methods reveals that even the strongest baseline scores below 45% on DSV-Mem. Comprehensive analysis surfaces several important findings: 1) multimodality and information density both contribute to difficulty, but state evolution, particularly the number of governing updates, is the dominant tested factor. Raw conversation/haystack length, OCR and arithmetic are not the primary bottlenecks; 2) models frequently fail to verify user premises against prior state updates before answering; 3) popular interventions such as increased reasoning effort and memory management methods yield limited gains, whereas state-aware designs prove more effective.

Primary area:
datasets and benchmarks
Keywords:
Conversational Agent, LLMs, MLLM, Memory, Long-Context Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_H2VROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DSV-Mem, a new multimodal benchmark that evaluates Dense Stateful Visual Memory. Compared to prior works, DSV-Mem focuses on professional scenarios, whereas others focus on informal everyday interactions. The benchmark comprises 10 different professional scenarios (i.e., long conversations), up to 50k tokens, that result in 1000 distinct questions (100 per scenario). Each conversation comprises structured artifacts that get revised multiple times by user updates. Questions span 5 different categories about the current and past states, derived states, history changes, and state conflicts. Results on 27 different model configurations show that the benchmark is challenging, with models failing to approach human performance. Additionally, the authors found that (i) models have a harder time tracking update histories of values compared to their final states; (ii) closed-weight models perform better than their open counterparts; (iii) memory-management methods perform below the baseline.

Critical strengths
  1. The benchmark evaluates a timely problem that prior work overlooked (professional scenarios vs. informal chats).
  2. Data curation is well validated by both human agreement and automatic checks.
  3. Each question provides much more candidate visual evidence compared to existing benchmarks.
  4. Results are validated by checking their variance, robustness to prompting, two-stage judge verification, human-judge agreement on a subset of answers, cross-judge robustness, and OCR control.
Critical weaknesses
  1. Limited conversations. While the benchmark features lots of questions, they actually come from a handful of conversations. Thus, it can only capture a restricted set of different cases. It would have been better, in my opinion, to propose fewer questions per conversation, but evaluate MLLMs in multiple situations.
  2. The framing does not match the goal. Compared to prior work [1], I would have expected multi-conversation histories. Instead, the paper proposes one relatively long conversation. As a result, it seems more of a "long context understanding" type of benchmark, rather than a benchmark that evaluates MLLM agent memory. This is exacerbated by the fact that the Haystack experiment shows that adding more tokens does not make the task harder, which questions whether the benchmark evaluates information localization (i.e., model memory) or reasoning about updates (i.e., reasoning abilities).

[1] Ren, Xiyu, et al. "Memlens: Benchmarking multimodal long-term memory in large vision-language models." arXiv preprint arXiv:2605.14906 (2026).

Questions for authors
  1. Could the authors test models on a clean, ordered list of updates without distractors and check whether the accuracy stays low?
  2. Could the authors evaluate GPT-5/6 and Opus-5.x deployed on Codex and Claude Code, respectively? Even if for a subset of questions, I think this is important to establish the current state of the art.
6 comments
AuthorsOpenReview ↗

Rebuttal Part 1/3

Thank you for your detailed feedback and insightful questions. We are encouraged that you found that DSV-Mem "evaluates a timely problem that prior work overlooked", that the data curation is "well validated", and that each question provides "much more candidate visual evidence compared to existing benchmarks". We have uploaded an updated draft (new content in blue), which now includes the GPT and Claude families, and we address your concerns point by point below.

W1. Limited conversations. While the benchmark features lots of questions, they actually come from a handful of conversations. Thus, it can only capture a restricted set of different cases. It would have been better, in my opinion, to propose fewer questions per conversation, but evaluate MLLMs in multiple situations.

R1. Thank you for raising this point. We kindly note that this setting is in line with recent memory benchmarks: MemEye [3] is built on 8 tasks, Mem-Gallery [4] (ACL 2026) on 20 scenarios, and LoCoMo [5] (ACL 2024) on 10 conversations. Each workflow in DSV-Mem is also deep rather than short: on average 212 turns and 24 dense visual artifacts (up to 50K tokens).

On the number of cases, the 100 questions of a workflow do cover multiple situations: they target different fields, time points, and question categories, and the updates they depend on are drawn from a library of 27 transition types (Sec. 3, Appendix B.1). The benchmark is also not limited to these ten workflows: our generation harness can extend it to new ones, each going through the same expert review and validation (Sec. 3, Appendix B, Appendix C.4).

We also kindly note that the goal of DSV-Mem is to diagnose how MLLM agents maintain dense, evolving visual state, and the ten workflows already expose this failure clearly and consistently: eight of the ten scenarios are below 50% for the best model (Appendix D.5, Table 6), 89.7% of errors are state-reconstruction errors (Sec. 4.2), and the same findings hold across the GPT, Claude, and Gemini families (Sec. 4.5).

W2. The framing does not match the goal. Compared to prior work [1], I would have expected multi-conversation histories. Instead, the paper proposes one relatively long conversation. As a result, it seems more of a "long context understanding" type of benchmark, rather than a benchmark that evaluates MLLM agent memory.

R2.1 Thank you for the thoughtful comment and for mentioning MemLens [1]. We already cited and discussed this work in our paper (Sec. 1, Sec. 2), compare with it directly (Table 1), and follow its evaluation setting (Sec. 4). We kindly note that MemLens evaluates its models in the same way as DSV-Mem: the history is given to the model as one complete input, and memory-augmented agents are evaluated separately, answering from their own memory. DSV-Mem covers both as well: the unaugmented models themselves, and the memory-management methods Mem0, SimpleMem, LightMem, and MIRIX (Table 2), together with our self-ledger (Sec. 5).

We also kindly point out that a professional workflow is one sustained project with many stages, so we model it as a single conversation. For a model that receives the full history, this does not change what has to be remembered. First, had we split the current conversation into several sessions, the model would still receive the same content, and the evidence for a question would still be spread over many turns and many versions of the artifacts (7.78 candidate visual artifacts per question; Table 1). Second, adding question-irrelevant content around the conversation, as a history of mixed conversations would, also changes accuracy little (Sec. 4.4, Fig. 7a). DSV-Mem therefore evaluates agent memory in the same sense as prior work.

(continued in Part 2/3)

AuthorsOpenReview ↗

Rebuttal Part 2/3

W2. ... This is exacerbated by the fact that the Haystack experiment shows that adding more tokens does not make the task harder, which questions whether the benchmark evaluates information localization (i.e., model memory) or reasoning about updates (i.e., reasoning abilities).

R2.2 This is a good point. We agree with the reviewer's observation: the main difficulty lies in handling updates, not in finding information among more tokens (Sec. 4.4); however, we kindly note that this does not make DSV-Mem a reasoning benchmark, since handling updates is a core memory ability in prior work: MemLens [1] lists "knowledge update" and "answer refusal" among its five memory abilities, and LongMemEval [2] (ICLR 2025) lists "knowledge updates" and "abstention". DSV-Mem studies these abilities in depth for professional workflows, through its Current State, Past State, Change History, and Conflict/Refusal categories (Sec. 3).

More precisely, answering a DSV-Mem question takes three steps (Sec. 3, Sec. 4.2): (i) finding and reading the relevant evidence; (ii) maintaining the state, i.e., retaining the updates and reconciling them into what is currently valid; and (iii) reasoning over the resolved state. The flat haystack result shows that the difficulty of DSV-Mem does not come from context length: if it did, adding more context would make it harder (Sec. 4.4, Fig. 7a). The main difficulty lies in step (ii), and four results show that it is distinct from the reasoning in step (iii): 1) the errors are memory failures: 89.7% of errors are state-reconstruction errors, led by lost or collapsed updates (30%) and stale values (27.1%), while downstream computation accounts for only 1.0% (Sec. 4.2, Fig. 5); 2) answering from the resolved state is easy: when the required values are provided, accuracy reaches 93% (Appendix C.4); 3) a memory structure helps: our self-ledger, which explicitly records and reconciles updates, improves accuracy by +13.6 pp on Gemini 3.7 Flash and +26.0 pp on Kimi K3 (Sec. 5); and 4) more reasoning does not solve it: raising GPT-5.5 from no reasoning to the highest effort improves accuracy from 15.5% to only 32.5%, and chain-of-thought prompting adds only +1.2 pp on Gemini 3.7 Flash and none on GPT-5.5 (Table 2; Sec. 5; Appendix D.8).

Overall, step (ii) is a core ability that professional workflows require; it is distinct from downstream reasoning, and it is what DSV-Mem is designed to measure.

Q1. Could the authors test models on a clean, ordered list of updates without distractors and check whether the accuracy stays low?

A1. Thank you for the suggestion. We have now conducted this experiment on Gemini 3.7 Flash and GPT-5.5. We replace the conversation with a clean, chronologically ordered list of the initial values and all updates relevant to the question in text, including corrections and withdrawn updates, with all other content removed. In terms of the three steps in R2.2, this removes step (i) and keeps steps (ii) and (iii). We compare it with the original conversation and with our intervention that removes the state changes while keeping the full multimodal conversation (Sec. 4.3), on the three categories for which removing statefulness is well defined (Current, Past, and Derived State), as in Fig. 6 and Fig. 12:

Accuracy (%)Original conversationClean, ordered list of updatesStatefulness removed
Gemini 3.7 Flash52.767.495.9
GPT-5.541.460.493.3

As shown in the table, the accuracy does not stay low, and it does not reach the level without state changes either: 1) the clean list improves accuracy by +14.7 and +19.0 pp, so finding and reading the updates in a long, dense multimodal conversation with competing artifact revisions is a real part of the difficulty, as the reviewer suggested; and 2) a gap of 28.5 and 32.9 pp remains even when every update is laid out in order, so reconciling the updates is the larger part. This is consistent with our finding that state evolution is the dominant factor of difficulty (Sec. 4.3; Fig. 6, Fig. 12).

(continued in Part 3/3)

AuthorsOpenReview ↗

Rebuttal Part 3/3

Q2. Could the authors evaluate GPT-5/6 and Opus-5.x deployed on Codex and Claude Code, respectively? Even if for a subset of questions, I think this is important to establish the current state of the art.

A2. Thank you for the suggestion. We agree that this is important, and we have now evaluated the GPT and Claude families on the full benchmark in the updated draft, through their APIs: 12 configurations, for 42 in total (Table 2; Sec. 4.5; Appendix D.8). We discuss the Codex and Claude Code setting after the results.

Model (medium effort)CurrentPastDerivedChange HistoryConflict/ RefusalCategory Avg.Total Avg.
GPT-5.6 Luna26.0 ± 1.717.3 ± 4.612.8 ± 0.613.6 ± 1.78.3 ± 0.615.6 ± 0.713.7 ± 0.6
GPT-5.6 Terra39.7 ± 1.226.7 ± 1.524.5 ± 1.520.0 ± 2.218.0 ± 2.625.8 ± 1.223.0 ± 1.5
GPT-5.6 Sol64.3 ± 1.244.7 ± 1.548.5 ± 1.336.4 ± 3.523.1 ± 2.243.4 ± 0.839.1 ± 1.2
GPT-5.553.3 ± 1.539.3 ± 3.136.5 ± 1.324.9 ± 1.211.7 ± 1.433.2 ± 0.028.2 ± 0.3
Claude Opus 4.843.3 ± 1.545.0 ± 1.033.5 ± 1.320.6 ± 2.154.4 ± 2.239.4 ± 0.836.3 ± 1.0
Claude Opus 4.733.3 ± 0.638.0 ± 2.023.0 ± 1.515.4 ± 1.742.8 ± 1.430.5 ± 0.727.8 ± 0.8
Claude Sonnet 4.629.0 ± 2.012.3 ± 0.622.5 ± 1.311.7 ± 1.826.8 ± 2.120.5 ± 0.719.4 ± 0.9
Gemini 3.7 Flash61.7 ± 1.546.3 ± 2.150.0 ± 1.727.8 ± 1.236.4 ± 2.544.4 ± 0.439.6 ± 0.8
Human87.0 ± 1.088.0 ± 1.580.0 ± 0.675.0 ± 3.088.0 ± 3.083.6 ± 0.981.8 ± 1.1

As shown in the table, the best models of all three frontier families stay below 40% (Gemini 3.7 Flash 39.6%, GPT-5.6 Sol 39.1%, Claude Opus 4.8 36.3%), against 81.8% for humans. More reasoning effort does not close the gap either: raising GPT-5.5 from no reasoning to the highest effort improves accuracy from 15.5% to only 32.5% (Appendix D.8, Fig. 13). The full results, including the reasoning-effort sweep, are in Table 2 of the updated draft.

Regarding Codex and Claude Code, we kindly note that DSV-Mem follows the protocol of prior memory benchmarks, including MemLens [1] and MemEye [3]: base models are evaluated directly in the conversational setting through their APIs, receiving the multi-turn conversation and answering the question as the final turn, while memory-augmented systems are evaluated separately. This also ensures a fair comparison across model families. Because a coding harness wraps the model in its own memory and context-management mechanisms, it belongs to the class of memory-augmented systems rather than base models. To evaluate this class in a controlled, model-agnostic, and reproducible way, we benchmark four memory-management methods alongside our self-ledger (Table 2, Sec. 5).

We have incorporated the GPT and Claude results into the updated draft, and we sincerely hope our responses and the new results could resolve the reviewer's concerns. We are also happy to address further questions from the reviewer.

References

[1] Ren et al., MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models, arXiv 2026.

[2] Wu et al., LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory, ICLR 2025.

[3] Guo et al., MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory, arXiv 2026.

[4] Bei et al., Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents, ACL 2026.

[5] Maharana et al., Evaluating Very Long-Term Conversational Memory of LLM Agents, ACL 2024.

Reviewer_moPVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops DSV-Mem, a benchmark curated to evaluate the performance of multi-modal models in professional applications that involve many turns of conversations. The benchmark is synthetically generated.

Critical strengths

I think the professional tasks in the benchmark are pretty interesting and well motivated. It's also helpful to have long-turn interactions. The authors provide human validations of the LLM judge. The ablations also provide useful insights.

Critical weaknesses

I'm not sure about the ecological validity of the tasks since they are all synthetically generated. It'd be helpful if some of the tasks are derived from real case studies.

Related to the first comment, how do we know that the conversations in the tasks are realistic?

Why are GPT and Claude not included in the evaluation? They are the most popular models.

Questions for authors

See above. I'm happy to increase my score if they are well addressed.

3 comments
AuthorsOpenReview ↗

Rebuttal Part 1/3

We genuinely appreciate your thoughtful review and constructive comments. We are glad that you found the professional tasks "interesting and well motivated", the long-turn interactions "helpful", and that "the ablations also provide useful insights". We have uploaded an updated draft (new content in blue), which now includes the GPT and Claude families, and we address each of your points below.

W1. I'm not sure about the ecological validity of the tasks since they are all synthetically generated. It'd be helpful if some of the tasks are derived from real case studies.

R1.1 Thank you for raising this important point. As the reviewer has recognized, our work is motivated by the fact that MLLM agents are increasingly deployed in long professional workflows, where they must keep track of dense visual artifacts that are revised many times (Sec. 1). Existing multimodal memory benchmarks focus on informal, everyday conversations with static images, and to the best of our knowledge none evaluates this capability (Sec. 1, Fig. 1). DSV-Mem is designed to fill this gap, and in this sense we believe our setting is valid and meaningful.

To keep the tasks faithful to real professional practice, we have further applied three checks on their realism.

1) Every scenario starts from a human-authored professional workflow. For each scenario, a human author supplies the initial artifact, a note describing the professional workflow, and an inventory of the changes that matter in that domain (Appendix B.2, Step 1). The ten workflows follow established professional procedures, such as the analysis of experimental results in ML research, the postmortem of an SRE incident, and the pre-filing reconciliation of a tax return (Appendix C.2).

2) Domain experts confirmed their realism. Six domain experts reviewed every scenario, including its initial artifacts and its transition trajectory, for professional realism, appropriate terminology and units, plausible updates, and correctly propagated consequences (Sec. 3, Appendix C.4).

3) Automated checks are built into our generation harness. Every update is checked for consistency with the current state before it is accepted, and the complete trajectory and dialogue are then audited for state continuity, coherence, and correctly propagated consequences (Sec. 3; Appendix B.2, Steps 3–5; Appendix C.4).

The resulting tasks are also natural for people: three independent auditors agree with the gold answers on 98.3% of an audited sample of questions (Sec. 3), and humans reach 81.8% while the best model stays below 40% (Table 2), so the difficulty is specific to current models.

W1. ... they are all synthetically generated. ...

R1.2 We kindly note that creating all content from scratch is a deliberate choice, for two reasons. (i) Real records are not publicly available. Records in these domains, such as tax returns or incident reports, are private by nature and not publicly available, so a benchmark that can be openly released has to be constructed. No real records, documents, or personal data were used at any stage (Ethics Statement). (ii) Existing work also adopts this approach. Recent memory benchmarks generate their conversations and then validate them with humans (LoCoMo [1], Mem-Gallery [2], MemLens [3]). In our case, this design additionally provides the verified state after every update, which evaluating state tracking requires (Sec. 3).

(continued in Part 2/3)

AuthorsOpenReview ↗

Rebuttal Part 2/3

W2. Related to the first comment, how do we know that the conversations in the tasks are realistic?

R2. Thank you for the question. Following your comment, we conducted a human study of conversation realism. Ten PhD-level raters each reviewed three randomly assigned scenarios, with the full conversation and all artifacts, and rated four aspects on a 1–5 scale (1 = clearly implausible, 5 = highly realistic). The results are now in the updated draft (Sec. 3; Appendix C.4, Table 3):

AspectMean rating (1–5)
Professional plausibility of the work4.9
Naturalness of the dialogue4.3
Coherence of the progression4.5
Plausibility of the updates4.3
Overall4.5

As shown in the table, the conversations are rated as realistic on all four aspects (4.5 out of 5 overall). This is consistent with how the conversations are built and checked: 1) by construction: the harness first builds and validates the sequence of updates, and a two-model role-play then realizes it as a coherent professional conversation (Sec. 3; Appendix B.2, Step 5); 2) by automated checks: every conversation is checked for coherence, non-repetition, and consistency with the validated trajectory (Appendix B.2, Step 5); and 3) by expert review: six domain experts reviewed every scenario for professional realism, appropriate terminology and units, plausible updates, and correctly propagated consequences (Sec. 3, Appendix C.4).

(continued in Part 3/3)

AuthorsOpenReview ↗

Rebuttal Part 3/3

W3. Why are GPT and Claude not included in the evaluation? They are the most popular models.

R3. Thank you for the suggestion. We were not able to include these results at submission time, and we have now added them: the updated draft evaluates 12 GPT and Claude configurations, for 42 configurations in total (Table 2; Sec. 4.5; Appendix D.8).

ModelEffortCurrentPastDerivedChange HistoryConflict/ RefusalCategory Avg.Total Avg.
GPT-5.6 LunaMedium26.0 ± 1.717.3 ± 4.612.8 ± 0.613.6 ± 1.78.3 ± 0.615.6 ± 0.713.7 ± 0.6
GPT-5.6 TerraMedium39.7 ± 1.226.7 ± 1.524.5 ± 1.520.0 ± 2.218.0 ± 2.625.8 ± 1.223.0 ± 1.5
GPT-5.6 SolMedium64.3 ± 1.244.7 ± 1.548.5 ± 1.336.4 ± 3.523.1 ± 2.243.4 ± 0.839.1 ± 1.2
GPT-5.6 SolMedium-Pro60.0 ± 1.049.0 ± 1.047.2 ± 2.534.5 ± 2.623.2 ± 1.442.8 ± 0.538.2 ± 0.7
GPT-5.5None26.0 ± 2.022.0 ± 3.613.5 ± 1.015.7 ± 1.89.9 ± 1.017.4 ± 1.515.5 ± 1.2
GPT-5.5Low43.0 ± 1.024.7 ± 1.523.7 ± 2.020.7 ± 1.912.0 ± 2.124.8 ± 0.721.7 ± 0.9
GPT-5.5Medium53.3 ± 1.539.3 ± 3.136.5 ± 1.324.9 ± 1.211.7 ± 1.433.2 ± 0.028.2 ± 0.3
GPT-5.5High60.7 ± 1.543.7 ± 4.041.2 ± 3.627.5 ± 0.412.9 ± 1.837.2 ± 0.231.5 ± 0.6
GPT-5.5XHigh58.0 ± 3.048.3 ± 1.543.8 ± 2.327.4 ± 1.713.9 ± 1.338.3 ± 0.232.5 ± 0.6
Claude Opus 4.8Medium43.3 ± 1.545.0 ± 1.033.5 ± 1.320.6 ± 2.154.4 ± 2.239.4 ± 0.836.3 ± 1.0
Claude Opus 4.7Medium33.3 ± 0.638.0 ± 2.023.0 ± 1.515.4 ± 1.742.8 ± 1.430.5 ± 0.727.8 ± 0.8
Claude Sonnet 4.6Medium29.0 ± 2.012.3 ± 0.622.5 ± 1.311.7 ± 1.826.8 ± 2.120.5 ± 0.719.4 ± 0.9
Gemini 3.7 FlashMedium61.7 ± 1.546.3 ± 2.150.0 ± 1.727.8 ± 1.236.4 ± 2.544.4 ± 0.439.6 ± 0.8
Human87.0 ± 1.088.0 ± 1.580.0 ± 0.675.0 ± 3.088.0 ± 3.083.6 ± 0.981.8 ± 1.1

As shown in the table, our findings are consistent across all three frontier families:

1) DSV-Mem is challenging for every family. The best models stay below 40% (Gemini 3.7 Flash 39.6%, GPT-5.6 Sol 39.1%, Claude Opus 4.8 36.3%), against 81.8% for humans. 2) The same factor drives the difficulty. The three controlled interventions replicate on GPT-5.5: removing statefulness gives the largest gain (+43.2 to +56.0 pp), far above density reduction (+13.7 pp) and oracle visual-to-text (+9.5 pp) (Appendix D.8 and Fig. 12 in the updated draft). 3) More reasoning effort does not close the gap. Raising GPT-5.5 from no reasoning to the highest effort improves accuracy from 15.5% to only 32.5% (Appendix D.8, Fig. 13). 4) The families fail in different ways. GPT-5.6 Sol leads on Current State and Change History, while Claude Opus 4.8 has the best Conflict/Refusal accuracy (54.4%), which shows the value of reporting all five categories.

Q. See above.

A. Please refer to R1.1, R1.2, R2, and R3 above. In short, every scenario is modeled on established professional practice and confirmed by domain experts, human raters find the conversations realistic (4.5 out of 5), and the GPT and Claude families are now included in the updated draft with consistent findings.

Flag For Ethics Review: Yes

Ethics. Thank you for the careful consideration. We would like to kindly clarify that DSV-Mem contains no real records, documents, or personal data: all artifact content is created from scratch by the research team, and the benchmark contains no personal, confidential, or proprietary information (Ethics Statement; please also refer to R1.2). We would be glad to address any specific concern the reviewer may have.

We have incorporated these new results into the updated draft, and we sincerely hope they could resolve the reviewer's concerns.

References

[1] Maharana et al., Evaluating Very Long-Term Conversational Memory of LLM Agents, ACL 2024.

[2] Bei et al., Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents, ACL 2026.

[3] Ren et al., MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models, arXiv 2026.

Reviewer_FmotOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DSV-Mem, a benchmark for evaluating multimodal memory in professional workflows involving dense visual artifacts and evolving states. The benchmark contains 1,000 questions across ten professional scenarios and five categories, covering current and historical states, derived information, change history, and conflict detection.The authors construct the benchmark using a generation framework that separates state-transition construction from dialogue generation.

Critical strengths
  1. The paper addresses a limitation of existing multimodal memory benchmarks, which often focus on factual recall or relatively static visual information. Evaluating how models maintain and reconcile changing visual information is relevant to long-running workflows.

  2. Separating state-transition construction from dialogue generation is a useful design choice. Explicit transition operators, state-consistency checks, and evidence tracking provide a structured way to generate verifiable instances. The authors also report automated checks, independent human auditing, and domain-expert reviews.

Critical weaknesses
  1. Although the benchmark contains 1,000 questions, they are generated from only ten constructed professional workflows, and multiple questions share the same underlying conversation and state trajectory.

  2. Unaugmented models receive the complete multimodal conversation, whereas memory-management methods answer using retrieved memories and question-attached images. Although historical visual information is provided during memory construction, the information available at answer time differs. The observed degradation may therefore reflect both state-tracking limitations and information loss during memory construction or retrieval.

  3. Removing statefulness also changes the reasoning requirements of the task, so the reported interventions do not necessarily provide directly comparable estimates of each difficulty factor.

Questions for authors
  1. Given the substantial performance variation across the ten scenarios, can the authors analyze how the number and types of state transitions affect difficulty, and explain how representative the constructed workflows are of the intended professional applications?

  2. How do differences in historical-evidence access affect the comparison with existing memory-management methods? Would the conclusions remain similar if these methods could revisit the original visual artifacts during answering?

3 comments
AuthorsOpenReview ↗

Rebuttal Part 1/3

We sincerely thank you for your careful review and insightful comments, which have greatly improved our work. We are encouraged that you found that DSV-Mem "addresses a limitation of existing multimodal memory benchmarks" and that separating state-transition construction from dialogue generation is "a useful design choice". We have uploaded an updated draft (new content in blue) and address your concerns point by point below.

W1. Although the benchmark contains 1,000 questions, they are generated from only ten constructed professional workflows, and multiple questions share the same underlying conversation and state trajectory.

R1. Thank you for raising this point. We kindly note that this setting is in line with recent memory benchmarks: LoCoMo [1] (ACL 2024) is built on 10 conversations, MemEye [2] on 8 tasks, and Mem-Gallery [3] (ACL 2026) on 20 scenarios. Each workflow in DSV-Mem is also deep rather than short: on average 212 turns and 24 dense visual artifacts (up to 50K tokens), and its 100 questions target different fields, time points, and question categories (Sec. 3, Appendix C.2).

As the reviewer noted, DSV-Mem is built with a generation framework that separates state-transition construction from dialogue generation (Sec. 3, Appendix B). The benchmark itself is therefore not limited to these ten workflows: it can be extended to new ones, each going through the same expert review and validation (Appendix C.4).

We also kindly note that the goal of DSV-Mem is to diagnose how MLLM agents maintain dense, evolving visual state, and the ten workflows already expose this failure clearly and consistently: 1) across workflows: eight of the ten scenarios are below 50% for the best model (Appendix D.5, Table 6); 2) with the same cause: 89.7% of errors are state-reconstruction errors, which is confirmed by human annotation (Sec. 4.2, Fig. 5; Appendix D.3); 3) under controlled interventions: on the same questions, removing statefulness raises accuracy to 95.9% (+43.2 pp; Sec. 4.3, Fig. 6); and 4) with stable results: three rewrites of each question change accuracy by less than one point (Appendix C.5).

To further validate this point, we have now also evaluated the other two frontier families, GPT and Claude, in the updated draft (Sec. 4.5, Table 2, Appendix D.8):

Model (medium effort)CurrentPastDerivedChange HistoryConflict/ RefusalCategory Avg.Total Avg.
GPT-5.6 Luna26.0 ± 1.717.3 ± 4.612.8 ± 0.613.6 ± 1.78.3 ± 0.615.6 ± 0.713.7 ± 0.6
GPT-5.6 Terra39.7 ± 1.226.7 ± 1.524.5 ± 1.520.0 ± 2.218.0 ± 2.625.8 ± 1.223.0 ± 1.5
GPT-5.6 Sol64.3 ± 1.244.7 ± 1.548.5 ± 1.336.4 ± 3.523.1 ± 2.243.4 ± 0.839.1 ± 1.2
GPT-5.553.3 ± 1.539.3 ± 3.136.5 ± 1.324.9 ± 1.211.7 ± 1.433.2 ± 0.028.2 ± 0.3
Claude Opus 4.843.3 ± 1.545.0 ± 1.033.5 ± 1.320.6 ± 2.154.4 ± 2.239.4 ± 0.836.3 ± 1.0
Claude Opus 4.733.3 ± 0.638.0 ± 2.023.0 ± 1.515.4 ± 1.742.8 ± 1.430.5 ± 0.727.8 ± 0.8
Claude Sonnet 4.629.0 ± 2.012.3 ± 0.622.5 ± 1.311.7 ± 1.826.8 ± 2.120.5 ± 0.719.4 ± 0.9
Gemini 3.7 Flash61.7 ± 1.546.3 ± 2.150.0 ± 1.727.8 ± 1.236.4 ± 2.544.4 ± 0.439.6 ± 0.8
Human87.0 ± 1.088.0 ± 1.580.0 ± 0.675.0 ± 3.088.0 ± 3.083.6 ± 0.981.8 ± 1.1

As shown in the table, the best models of all three families stay below 40% (Gemini 3.7 Flash 39.6%, GPT-5.6 Sol 39.1%, Claude Opus 4.8 36.3%), against 81.8% for humans, and the same ordering of the three interventions replicates on GPT-5.5 (+43.2–56.0 pp from removing statefulness; Fig. 12). The diagnosis is therefore consistent across workflows and model families.

(continued in Part 2/3)

AuthorsOpenReview ↗

Rebuttal Part 2/3

W2. Unaugmented models receive the complete multimodal conversation, whereas memory-management methods answer using retrieved memories and question-attached images. Although historical visual information is provided during memory construction, the information available at answer time differs.

R2.1 Thank you for the careful reading, and we apologize for the confusion caused by our original wording. We would like to kindly clarify that "question-attached images" are images that are part of the question itself (284 of the 1,000 questions include one), and unaugmented models and memory-management methods both receive them. Hence both settings receive the exact same questions.

The only difference is that memory-management methods answer from their retrieved memories, which follows each method's own design: they first distill the history into compact memories and answer from what they retrieve. We follow each method's own pipeline and supply every historical image during memory construction, so no evidence is withheld. We have revised this wording in the updated draft (Sec. 4, Appendix D.1 and D.6).

W2. ... The observed degradation may therefore reflect both state-tracking limitations and information loss during memory construction or retrieval.

R2.2 This is a good point, and we agree that it is valuable to separate the two factors. In our paper, the performance of these methods reflects both: (i) the gap to the baseline comes from visual details lost during memory construction (Sec. 4.1), and (ii) the baseline itself is limited by state tracking (Sec. 4.2). To isolate the two factors, we ran an additional control: during memory construction, each method receives an oracle textual rendering of every visual artifact in place of its own reading of the images. This gives the methods full access to the content of the original artifacts.

GPT-5.5 (medium)CurrentPastDerivedChange HistoryConflict/ RefusalCategory Avg.Total Avg.
Baseline53.3 ± 1.539.3 ± 3.136.5 ± 1.324.9 ± 1.211.7 ± 1.433.2 ± 0.028.2 ± 0.3
+ Mem050.3 ± 2.336.7 ± 6.533.3 ± 3.222.6 ± 2.210.5 ± 1.230.7 ± 1.625.9 ± 1.3
+ SimpleMem55.0 ± 3.539.7 ± 2.338.5 ± 0.925.3 ± 1.712.4 ± 1.634.2 ± 1.029.1 ± 0.8
+ LightMem51.0 ± 3.037.7 ± 1.536.5 ± 2.226.4 ± 1.812.0 ± 2.032.7 ± 1.028.4 ± 1.0
+ MIRIX53.0 ± 1.035.3 ± 1.241.5 ± 0.924.9 ± 2.712.0 ± 1.133.3 ± 0.728.8 ± 1.0

As shown in the table, with oracle image information all four methods perform on par with the baseline (25.9–29.1 vs. 28.2). This isolates the two factors raised by the reviewer: (i) information loss during memory construction is a factor, since the gap to the baseline disappears once the visual details are preserved, which is consistent with the finding of MemLens [4] that memory agents lose visual details when compressing for storage; and (ii) state-tracking limitations remain, since even with the visual details preserved the methods stay at the baseline level. Our finding is thus consistent even when the methods have full access to the original artifacts.

(continued in Part 3/3)

AuthorsOpenReview ↗

Rebuttal Part 3/3

W3. Removing statefulness also changes the reasoning requirements of the task, so the reported interventions do not necessarily provide directly comparable estimates of each difficulty factor.

R3. Thank you for the thoughtful comment. The reviewer is right that removing statefulness also changes the reasoning required. We kindly note that answering a question under statefulness in itself requires reasoning over the updates, so removing statefulness naturally removes this reasoning as well. This is hence by design (Sec. 4.3), and it holds for all three interventions: multimodality requires reading values from dense visual artifacts and information density requires locating the relevant values among many others. Each intervention therefore measures the same thing in the same way: how much accuracy recovers when one property, together with the reasoning it requires, is removed.

All three interventions are evaluated on the same questions, and each is a minimal change: it alters only its own property and keeps the other two as in the original conversation (Fig. 6). The difficulty of each factor is therefore measured on the same footing, and we believe this setup provides a fair and informative way to identify which property matters most. The resulting ordering is also clear and consistent: removing statefulness improves accuracy by +43.2 pp, against +12.6 pp for density reduction and +8.2 pp for oracle visual-to-text, and the same ordering holds on GPT-5.5, as newly reported in the updated draft (Sec. 4.5, Fig. 12). A new control further supports this: replacing the conversation with a clean, ordered list of the updates keeps the reasoning over updates but removes the other two properties, and it improves accuracy by only +14.7 pp on Gemini 3.7 Flash (52.7% → 67.4%) and +19.0 pp on GPT-5.5 (41.4% → 60.4%). The original questions also show the same pattern without any intervention: accuracy declines as the number of state updates grows (Sec. 4.4, Fig. 7b), and 89.7% of errors are state-reconstruction errors (Sec. 4.2, Fig. 5). We have clarified this in Sec. 4.3 of the updated draft.

Q1. Given the substantial performance variation across the ten scenarios, can the authors analyze how the number and types of state transitions affect difficulty, ...

A1.1 Thank you for the suggestion. We kindly note that our paper already analyzes both. 1) Number of transitions (Sec. 4.4, Fig. 7b): accuracy declines as the number of governing state updates grows. Following the suggestion, we further extended this analysis to the GPT family in the updated draft (Appendix D.8, Table 7):

Accuracy (%)1–7 governing updates8+ governing updates
Gemini 3.7 Flash43.927.5
GPT-5.6 Sol42.130.2
GPT-5.531.120.1

As shown in the table, questions that depend on eight or more updates are 11 to 16 points harder on all three models. 2) Types of transitions (Sec. 4.4, Fig. 5c): we break down the errors by the type of state operation that is mishandled. Lost or collapsed updates (30% of all errors) and stale values (27.1%) are the largest failure modes, followed by update integration, entity tracking, and authority resolution. The variation across scenarios reflects their different mix of updates and artifact forms (Appendix D.5).

Q1. ... and explain how representative the constructed workflows are of the intended professional applications?

A1.2 Thank you for the question. As stated in Sec. 3 and Appendix C.4, six domain experts reviewed every scenario, including its initial artifacts and its transition trajectory, and confirmed its professional realism, appropriate terminology and units, plausible updates, and correctly propagated consequences.

Q2. How do differences in historical-evidence access affect the comparison with existing memory-management methods? Would the conclusions remain similar if these methods could revisit the original visual artifacts during answering?

A2. Please refer to R2.2. In short, when the methods are given oracle information of every original artifact, they perform on par with the baseline, so our conclusions remain the same.

We have incorporated these clarifications and the GPT and Claude results into the updated draft, and we sincerely hope our responses could resolve the reviewer's concerns.

References

[1] Maharana et al., Evaluating Very Long-Term Conversational Memory of LLM Agents, ACL 2024.

[2] Guo et al., MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory, arXiv 2026.

[3] Bei et al., Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents, ACL 2026.

[4] Ren et al., MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models, arXiv 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

Summary of changes in the updated draft

We thank all reviewers for their thoughtful and constructive feedback. We are glad that the reviewers found that DSV-Mem addresses a timely and well-motivated problem that existing benchmarks overlook (Fmot, moPV, H2VR), that its generation design is useful and its data well validated (Fmot, H2VR), and that its analysis and result validation are insightful and thorough (moPV, H2VR).

We have uploaded an updated draft (new content in blue). The main additions are:

  1. GPT and Claude families (Table 2; Sec. 4.5; Appendix D.8). We now evaluate 12 GPT and Claude configurations on the full benchmark, for 42 configurations in total. The best models of all three frontier families stay below 40%.
  2. Human study of conversation realism (Sec. 3; Appendix C.4, Table 3). Ten PhD-level raters rate the conversations as realistic on all four aspects (4.5 out of 5 overall).
  3. Update-count analysis on the GPT family (Appendix D.8, Table 7). Questions that depend on eight or more updates are 11 to 16 points harder on all three models.
  4. Clarifications (Sec. 4, Sec. 4.3, Sec. 4.4; Appendix D.1, D.5, D.6). We clarify the evaluation setting of the memory-management methods, the design of the controlled interventions, and what the benchmark stresses.

We address each reviewer's points in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.