acceptodds
Under review as a conference paper at ICLR 2027

VideoMemBench: How Well Do MLLMs Remember in Video Conversations?

Abstract

Multimodal large language models (MLLMs) can now process hour-long videos, making extended multi-turn dialogue over a single video a natural mode of use. Sustaining such a dialogue requires more than answering each question in isolation: the model must retrieve and reuse information established earlier in the conversation, both what it has seen and what it has said, an ability we term in-context video memory. Existing video QA benchmarks leave this ability unmeasured: their questions are mutually independent, never reference one another, and each is answerable from the video alone, so nothing ever requires carrying information across turns. We introduce VideoMemBench, in which each sample pairs a single video with a long multi-turn dialogue, whose turns are connected by typed cross-turn dependencies: an Episodic Dependency refers back to visual episodes described in earlier turns, probing memory of what the model has seen, while an Answer Dependency builds on the model's own earlier answer, probing memory of what the model has said. Every dependency is verified considering two properties: 1) necessity, meaning the question cannot be answered without its earlier turns, and 2) non-leakage, meaning the question does not restate the information from its earlier turns. To enforce these properties, we design a construction pipeline with three LLM roles: a planner, a generator, and a judger, with each dialogue reviewed by human experts. VideoMemBench comprises 326 videos with 2,476 turns, of which 1,387 depend on earlier turns, including multi-parent dependencies and long-range call-back. Among over 20 evaluated open- and closed-source MLLMs, Claude Opus 5 performs best with an overall score of only 45.7%, highlighting the challenge of in-context video memory. Our dataset and evaluation will be released.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Video understanding, Multimodal large language models, Video memory

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_uVfMOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VideoMemBench evaluates memory in multi-turn video dialogue. It contains 2,476 questions across 326 videos, with explicit episode and answer dependencies. The paper evaluates 25 models using their own previous responses as dialogue history.

Critical strengths
  • Explicit dependency graphs support more detailed analysis than turn-level accuracy alone.

  • The benchmark includes human review, human evaluation, and ablations on frame count and dialogue history.

Critical weaknesses
  • Questions are generated from densely sampled segments, but evaluation uses only 128 frames per video. For the longest video (e.g. 23.47 h), this is roughly one frame every 11 minutes. Missing evidence may be mistaken for memory failure.

  • Answer-dependent accuracy requires all answer ancestors to be correct. A correct current response can therefore receive no credit because of an earlier error. This limits its ability to distinguish memory failures from perception and reasoning errors.

  • The construction checks are useful, but stronger evaluation controls are needed to establish dependency necessity. Removing dialogue history or specific parent turns would test whether models actually need the annotated dependencies.

  • The novelty over existing multi-turn video benchmarks (MT-Video-Bench) is limited. Typed dependency graphs are the main addition, but their additional diagnostic value is not clearly established.

  • Some results also differ across experiments: Qwen3-VL-32B scores 17.2 in Table 2 but 15.4 in the history ablation.

Questions for authors
  • What fraction of questions is answerable from the 128 sampled frames, broken down by video duration? How do models perform with the annotated evidence segments?

  • How does answer-dependent performance change when evaluated only on turns with correct answer ancestors?

  • How much does performance drop when history or the designated parent turns are removed?

Reviewer_NhjnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes VideoMemBench, a benchmark for multi-turn dialogue over a single video where later questions depend on earlier turns. A dependency is either episodic (the question refers to a scene discussed earlier) or answer-based (the question needs an earlier answer). Dialogues are generated by a planner/generator/judge LLM pipeline that checks necessity, non-leakage and grounding, and then reviewed by humans. The final set has 326 videos and 2,476 turns. The authors evaluate 25 MLLMs and the best model reaches 45.7% overall, against 86.1% for humans.

Critical strengths
  • Testing questions that depend on earlier turns is a novel contribution, and separating episodic from answer dependencies makes sense.
  • The construction process is well documented, with all prompts in the appendix, and every dialogue is checked by a human.
  • Many models are evaluated, and the oracle-history experiment in Fig. 4(d) is useful.
Critical weaknesses
  • The benchmark does not isolate memory from other forms of conversation history. The full dialogue stays in context and the frames remain available, so an episodic turn only requires reading the earlier question. EpiDepAcc is within about 4 points of TurnAcc for most models, which suggests the difficulty comes from the visual question rather than from recalling earlier turns. A control with the parent information written into the question, and one with no history, would help separate the two.
  • The drop in AnsDepAcc comes mostly from chain crediting. A model that consistently builds on its own earlier but incorrect answer is still scored as wrong. Although the oracle-history experiment partially addresses this issue, the metric still does not separate memory failures from failures caused by incorrect earlier answers.
  • The automated checks miss some cases. The necessity probe is told not to guess about removed turns, so it may encourage abstention whenever a question refers to a removed turn.
  • Models get 128 frames for videos that average 28 minutes and run up to 23 hours, so evidence windows may contain no sampled frame. Grounding was verified on dense frames. There is also no no-video or chance baseline.
  • The planner, generator and judge (GPT-5.5, Gemini 3.5 Flash, Claude Opus 4.8) come from the same families as several evaluated models. Filtering on one judge's answers could affect the ranking, and the paper does not discuss this.
  • The claim that existing benchmarks leave this ability unmeasured is too strong. MT-Video-Bench already includes memory recall and object reference across turns.
Questions for authors
  • What is accuracy on dependent turns with no history, and with the parent content written into the question?
  • What is unchained accuracy on answer-dependent turns?
  • What fraction of turns have at least one of the 128 frames inside the evidence window?
  • How many turns did each check reject, and how many did reviewers revise or remove?
Reviewer_WqPjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VideoMemBench to study in-context video memory, evaluating Turn Accuracy, Episodic Dependency Accuracy, and Answer Dependency Accuracy across over 20 open- and closed-source MLLMs. The benchmark is built via an LLM-based plan-generate-judge pipeline with manual verification. The paper finds that MLLMs have difficulty maintaining and using dialogue context in video conversations.

Critical strengths
  1. The paper addresses an important problem. Evaluating video memory across dialogue turns is meaningful for long-video understanding, as it goes beyond single-turn video QA and probes whether models can retain and reuse information over extended interactions.

  2. The benchmark construction and evaluation are fairly reasonable. The paper provides detailed prompts, which improves reproducibility and helps readers understand how the benchmark is curated.

  3. The evaluation covers over 20 open- and closed-source MLLMs, and the results show that current models struggle to maintain and use dialogue context in video conversations.

Critical weaknesses
  1. The paper does not sufficiently distinguish what makes video in-context memory challenging compared with the multi-image memory discussed in related work. A more detailed discussion is needed to justify why video in-context memory requires a separate benchmark and problem formulation.

  2. Although the motivation is meaningful, the novelty of the paper is relatively limited. I can hardly identify clear novelty in the data curation and metric design. And the paper does not provide ablations on the data curation pipeline design.

  3. The paper reports that there is substantial room for MLLMs to improve in-context video memory, but it does not provide potential solutions or model-level contributions. This raises concerns about the value of the work beyond benchmark construction and empirical evaluation.

Questions for authors
  1. What are the success rates of Stage 1 planning and Stage 2 generation? What proportion of samples undergo LLM regeneration and expert review? These details help assess the effectiveness of the data curation pipeline.

  2. At line 462, the paper notes that “independent-turn accuracy fluctuates at intermediate.” Can the authors explain why this fluctuation occurs or provide additional experiments?

  3. Figure 4(b) is incomplete. The text is truncated.

Reviewer_6EGoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VideoMemBench, a benchmark for evaluating in-context video memory in multi-turn video conversations. Each sample pairs a long video with a multi-turn dialogue whose turns are linked by explicit typed cross-turn dependencies: Episodic Dependencies and Answer Dependencies. The benchmark contains 326 videos, 2,476 turns, and 1,387 dependent turns, and is constructed with a planner/generator/judge pipeline plus human expert review. The authors evaluate 25 MLLMs and report that the best model, Claude Opus 5, reaches only 45.7% overall, versus 86.1% for humans.

Critical strengths
  1. The formulation of dependency necessity, non-leakage, and visual grounding is sensible.
  2. The benchmark covers 326 videos, 31 domains, ~154 hours, and up to 23.47-hour videos. The evaluation includes 25 models, both closed- and open-source, plus a human baseline.
Critical weaknesses
  1. The major motivation of this benchmark is the absence of question dependency in existing video QA benchmarks. However, these prevalent video QA benchmarks [1,2] already contain question dependency.
  2. All models receive only 128 uniformly sampled frames across the entire video, even for videos as long as 23.47 hours. For such videos, 128 frames is extremely sparse, and some dependent turns may be unanswerable because the relevant visual evidence is simply not in the sampled frames.
  3. The planner, generator, and judge are all LLM-based, and the necessity/non-leakage/grounding checks are also performed by LLMs. Human experts review all retained dialogues, but the paper does not report quantitative human-validation statistics.
  4. 37% of questions are open-form, and GPT-5.4 mini is used as a judge. This model API will be deprecated, and the evaluation will be unavailable to reproduce. The paper also does not report agreement between this judge and human grading.

References:

[1] Yang, Z., Hu, Y., Du, Z., Xue, D., Qian, S., Wu, J., ... & Xu, C. (2025, May). Svbench: A benchmark with temporal multi-turn dialogues for streaming video understanding. In International Conference on Learning Representations (Vol. 2025, pp. 42522-42556).

[2] Pan Y, Xie Q, Zhang G, Wang ZM, Wen Y, Zhang Y, Hu H, Pan Z, Huang Y, Gan Z, Lin Y. Mt-video-bench: A holistic video understanding benchmark for evaluating multimodal llms in multi-turn dialogues. In Findings of the Association for Computational Linguistics: ACL 2026 2026 Jul (pp. 8105-8126).

Questions for authors

Please reply to weaknesses.

1 comment

Thanks for the explanation. However, those discussions and results don't address the proposed weaknesses. These weaknesses remain, or the paper is significantly revised, which doesn't support the acceptance of the current version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.