acceptodds
Under review as a conference paper at ICLR 2027

DataVista: Diagnosing Multimodal LLMs on Data Video Understanding

Abstract

Data video is a media form that integrates data visualization with video narrative, widely adopted in news reporting and business analysis. Compared with general video understanding, data video understanding places greater emphasis on accurately reading data from animated charts, integrating evidence across charts and time, and understanding how narrative organization and visual design communicate information. Yet existing benchmarks target either general videos or static charts, and data video understanding has not been systematically evaluated. We present DataVista, the first benchmark for data video understanding, containing 961 real-world data videos and 6,775 evaluation questions organized under a three-level progressive capability framework (data perception, temporal reasoning, narrative understanding) with 10 fine-grained question types across five topic domains. Systematic evaluation of 19 mainstream MLLMs shows that the best-performing model, Gemini-3.1-Pro, achieves 70.0% overall accuracy, far below the human expert score of 91.7%, with models performing worst on Causal Reasoning and Narrative Structure. Increasing frame counts and adding subtitles mainly benefit data perception and temporal reasoning, with limited gains in narrative understanding. Further analysis of model responses identifies typical failure modes in chart reading, evidence judgment, and instruction understanding.

Primary area:
datasets and benchmarks
Keywords:
data video understanding, multimodal large models, benchmark, data visualization, video reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_frjoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Data video is a media form that integrates data visualization with video narrative, widely adopted in news reporting and business analysis. This paper proposes a benchmark to evaluate the models’ ability on data video understanding. Systematic evaluation of 19 mainstream MLLMs shows that there is a huge gap between the human performance and MLLMs. The analysis shows that the models perform worst on Causal Reasoning and Narrative Structure.

Critical strengths
  1. The paper is well written. The motivation and methodology are clearly presented, and the figures and tables are well designed and easy to understand.

  2. The authors introduce a benchmark for the novel scenario of data video understanding. The benchmark contains a comprehensive set of questions covering different capability levels and domains. The dataset construction process is well designed and reliable, with extensive human validation. The authors also provide thorough analyses in the experimental section.

Critical weaknesses

A. Do Level 3 questions truly evaluate narrative understanding?

(1) As shown in Figure 5, Gemini-3.1-Pro achieves nearly 60% accuracy on Level 3 with question-only input, which is very close to its performance with frames and subtitles. Does this suggest that these questions can be answered without relying on video information?

(2) Moreover, its question-only accuracy on Level 3 is even higher than on Levels 1 and 2. In contrast, when frames and subtitles are provided, Gemini-3.1-Pro's accuracy decreases significantly as the capability level increases. How can this contrasting trend be explained?

B. The evaluation uniformly samples 50 frames from each video. What happens if the frames containing the necessary evidence to answer a question are not sampled? Have the authors analyzed whether incorrect answers are caused by missing relevant frames or by limitations in the models' capabilities?

Questions for authors

Please refer to the weaknesses.

Reviewer_s4WGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark DataVista for data video understanding. The benchmark is designed to have a three-level capability hierarchy that evaluates models’ data fact reading, cross-charts reasoning, and narrative understanding. The evaluation is performed on 19 MLLMs and the best model Gemini-3.1-Pro shows 21.7% performance gap below human.

Critical strengths
  • The benchmark contains 961 real-world and 6775 evaluation questions. The video category across 5 major domains and their sub-categories is also comprehensive.
  • Many design choices made for benchmark curation are well-motivated or supported by analysis, including contamination analysis, single- and multi-select QAs, and distractor design. The authors have also provided results for text-only, random selection, and frequent choice settings.
Critical weaknesses
  • Though authors have evaluated the benchmarks on 19 MLLMs, the open-source models span only two families and do not include strong models such as Qwen2.5-VL-72B (which is commonly reported as the strongest open-sourced model on several video understanding benchmarks). Analysis provided in the paper are all based on limited model families and sizes.
  • The paper claims that it is the “first” data video benchmark, but there seems to be other benchmarks ([1], [2], [3]) that investigate chart reading and predate the conference submission deadline.

[1] Wang, Bomiao, et al. DVBench: Benchmarking MLLMs for Understanding Dynamic Charts and Narratives in Data Videos. arXiv, 2026, doi:10.48550/arXiv.2608.29711.
[2] Kairui Hu, et al. 2026. Video-MMMU: Evaluating Knowledge Acquisition from Multidisciplinary Professional Videos. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 27798–27828, San Diego, California, United States. Association for Computational Linguistics.
[3] Mahbub, Ridwan, et al. "DATAREEL: Automated Data-Driven Video Story Generation with Animations." arXiv preprint arXiv:2604.25220 (2026).

Questions for authors
  • During candidate video pre-screening, could Gemini-3-Flash have excluded videos suitable for challenging QA because of its limited capability or frame sampling rate? Were any excluded videos reviewed by humans to assess this possibility?
1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Reviewer s4WG,

Thank you again for your careful review. We have uploaded the revised manuscript, with all new and revised content marked in blue. Following your comments, we have made the following revisions, which make the evaluation more comprehensive, clarify the positioning of the paper, and make data construction and release more reliable and principled:

  • Model coverage (W1): We expanded the evaluation from 19 to 35 models, with open-source models now spanning 10 families, including Qwen2.5-VL-72B. The results thus give a more complete picture of current open-source models: the best-performing open-source model, Gemma-4-31B, still trails Gemini-3.1-Pro by 8.9 percentage points. See Sections 4.1–4.2 and Appendix E.1.
  • "First" claim (W2): The abstract, introduction, and conclusion no longer describe DataVista as the first data video benchmark. Section 2 adds comparisons with DVBench, Video-MMMU, and DATAREEL, which more clearly delineate the positioning and contributions of DataVista.
  • Video pre-screening (Q1): Section 3.1 and Appendix B.1 add a cross-blind audit of 300 excluded videos by three annotators, which finds an exclusion accuracy of 99.7% and provides human verification of the reliability of pre-screening.
  • Video copyright and licensing (E1): The Ethics Statement describes the license types of the source videos and our tiered release policy: for videos under the Standard YouTube License, we provide only official YouTube links without redistributing the video files, and we are pursuing academic permissions through the publishers' official licensing channels. The Reproducibility Statement adds an official evaluation service and public leaderboard, so that the benchmark can be released in compliance with license terms while still supporting reproducible evaluation on the full dataset.

As the discussion period is about to close, we would be grateful to know whether these revisions address your concerns. If they do, we would appreciate it if you could take them into account in your final assessment.

Best regards,

The Authors

Reviewer_JGfbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DataVista, a benchmark for understanding data videos, meaning videos that tell a story with animated charts (news explainers, market reports, science channels). It contains 961 YouTube videos and 6,775 multiple-choice questions organized into three levels: reading data from charts, reasoning across charts and time, and understanding the narrative and design intent. Questions are written by annotators with a Gemini-assisted pipeline, filtered with a text-only screen, and double-reviewed. The authors evaluate 19 MLLMs; the best, Gemini-3.1-Pro, reaches 70.0% against a human score of 91.7%. More frames and subtitles help chart reading and temporal reasoning but barely move narrative understanding, and a manual study of 240 errors shows that models often read the chart correctly but fail to judge whether a fact supports a claim.

Critical strengths
  1. The data is real and well curated. The videos come from professional media, are mostly recent to limit contamination, and come with structured metadata (chart segments, narrative stages, argument chapters) that others can reuse.
  2. The question design is careful. Each of the ten question types has explicit rules for distractors, such as a single-chart trap for cross-chart questions and same-polarity options for quality judgments, and the text-only filter removes three quarters of candidates and is validated across five model families.
  3. The evaluation is complete for a benchmark paper: human baseline, random and majority baselines, video-level bootstrap intervals, frame-budget and input-ablation studies, a multi-select precision/recall breakdown, a contamination check, and an error analysis on two models.
  4. The main finding is clear and useful: more visual input fixes chart reading but not narrative understanding, so the gap at Level 3 is about judgment rather than perception.
Critical weaknesses
  1. The paper calls itself the first data video benchmark, but DVBench (arXiv 2608.29711, Findings of EMNLP 2026) already does this with 300 videos and 1,000 questions across narrative, animation, chart perception, chart reasoning, and subtitle alignment. DataVista is bigger and has more analysis, but it is the same task.
  2. Related work misses two groups of directly relevant benchmarks: knowledge and lecture video benchmarks that already contain many charts and slides (Video-MMMU, Video-MMLU, MMVU, SciVideoBench), and the current standard static chart reasoning benchmarks (CharXiv, ChartMuseum). Without them it is hard to see how much of DataVista's difficulty is specific to data videos.
  3. The same model builds and takes the test. Gemini-3-Flash does the video screening, the evidence pre-extraction, and the text-only filter, and it is also one of the 19 evaluated models. Questions that Gemini finds easy from text alone were removed, an advantage no other model family received.
  4. Human and model conditions differ. Humans watch the full video with audio while models get 50 silent frames. The paper's own Table 9 shows native video plus audio adds about 5 points for Gemini, so part of the 21.7-point human gap is input, not ability.
Questions for authors
  1. Questions from the same video are submitted together in one request. Did you check whether answering them one at a time changes accuracy? Later questions can reveal facts that help earlier ones.
  2. About 40% of videos are from Economy and 153 come from one channel. Does accuracy differ between CNBC videos and the rest, and would the ranking hold if each channel were capped?

Comments on the paper

2 comments

The rebuttal addresses Weakness B but the answers to Weakness A are quite vague and indirect. I understand analyzing the behavior of a specific model may be difficult. I would like to maintain my original score of Weak Accept, since I don't think the paper reaches the bar for a Strong Accept.

Thank you for the authors' response, which addressed most of my concerns. I have raised my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.