- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VideoMemBench evaluates memory in multi-turn video dialogue. It contains 2,476 questions across 326 videos, with explicit episode and answer dependencies. The paper evaluates 25 models using their own previous responses as dialogue history.
Critical strengths
-
Explicit dependency graphs support more detailed analysis than turn-level accuracy alone.
-
The benchmark includes human review, human evaluation, and ablations on frame count and dialogue history.
Critical weaknesses
-
Questions are generated from densely sampled segments, but evaluation uses only 128 frames per video. For the longest video (e.g. 23.47 h), this is roughly one frame every 11 minutes. Missing evidence may be mistaken for memory failure.
-
Answer-dependent accuracy requires all answer ancestors to be correct. A correct current response can therefore receive no credit because of an earlier error. This limits its ability to distinguish memory failures from perception and reasoning errors.
-
The construction checks are useful, but stronger evaluation controls are needed to establish dependency necessity. Removing dialogue history or specific parent turns would test whether models actually need the annotated dependencies.
-
The novelty over existing multi-turn video benchmarks (MT-Video-Bench) is limited. Typed dependency graphs are the main addition, but their additional diagnostic value is not clearly established.
-
Some results also differ across experiments: Qwen3-VL-32B scores 17.2 in Table 2 but 15.4 in the history ablation.
Questions for authors
-
What fraction of questions is answerable from the 128 sampled frames, broken down by video duration? How do models perform with the annotated evidence segments?
-
How does answer-dependent performance change when evaluated only on turns with correct answer ancestors?
-
How much does performance drop when history or the designated parent turns are removed?
Thanks for the explanation. However, those discussions and results don't address the proposed weaknesses. These weaknesses remain, or the paper is significantly revised, which doesn't support the acceptance of the current version.