- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Data video is a media form that integrates data visualization with video narrative, widely adopted in news reporting and business analysis. This paper proposes a benchmark to evaluate the models’ ability on data video understanding. Systematic evaluation of 19 mainstream MLLMs shows that there is a huge gap between the human performance and MLLMs. The analysis shows that the models perform worst on Causal Reasoning and Narrative Structure.
Critical strengths
-
The paper is well written. The motivation and methodology are clearly presented, and the figures and tables are well designed and easy to understand.
-
The authors introduce a benchmark for the novel scenario of data video understanding. The benchmark contains a comprehensive set of questions covering different capability levels and domains. The dataset construction process is well designed and reliable, with extensive human validation. The authors also provide thorough analyses in the experimental section.
Critical weaknesses
A. Do Level 3 questions truly evaluate narrative understanding?
(1) As shown in Figure 5, Gemini-3.1-Pro achieves nearly 60% accuracy on Level 3 with question-only input, which is very close to its performance with frames and subtitles. Does this suggest that these questions can be answered without relying on video information?
(2) Moreover, its question-only accuracy on Level 3 is even higher than on Levels 1 and 2. In contrast, when frames and subtitles are provided, Gemini-3.1-Pro's accuracy decreases significantly as the capability level increases. How can this contrasting trend be explained?
B. The evaluation uniformly samples 50 frames from each video. What happens if the frames containing the necessary evidence to answer a question are not sampled? Have the authors analyzed whether incorrect answers are caused by missing relevant frames or by limitations in the models' capabilities?
Questions for authors
Please refer to the weaknesses.
Summary of Revisions
Dear Reviewer s4WG,
Thank you again for your careful review. We have uploaded the revised manuscript, with all new and revised content marked in blue. Following your comments, we have made the following revisions, which make the evaluation more comprehensive, clarify the positioning of the paper, and make data construction and release more reliable and principled:
As the discussion period is about to close, we would be grateful to know whether these revisions address your concerns. If they do, we would appreciate it if you could take them into account in your final assessment.
Best regards,
The Authors