- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether multimodal LLMs give different answers when the same structured data is shown in different visual formats. The authors introduce StructViz-Bench, a new benchmark with 3,795 base questions across tables, time series, and graphs. The same data and question are kept fixed, while only the visual representation is changed. They evaluate several open and closed multimodal models and report large accuracy differences across formats in the original benchmark.
The paper then shows that a large part of this difference comes from the fact that some visualizations do not actually contain all the information needed to answer the question. For example, some plots only show averages, some graph views do not label nodes, and some text-based views are truncated. The authors therefore create a corrected version of the benchmark where the visualizations contain more complete information. On this corrected setup, the remaining format effect is smaller, but still present for tables and graphs for the main open model they test. They also compare image inputs with raw text, test different prompt styles, and try two mitigation methods: choosing the best format for each task and fine-tuning a model on multiple formats. Overall, the paper argues that model performance depends on how structured data is presented, but also shows that information availability itself is a major part of this effect.
Critical strengths
The paper introduces StructViz-Bench, a new controlled benchmark for studying visualization-format dependence across tabular, time-series, and graph data. The benchmark construction is systematic and accompanied by a deterministic generation and evaluation pipeline, making it potentially useful for future work.
Critical weaknesses
-
The main experiments do not cleanly test the paper's stated research question. In the original benchmark, visualization format is confounded with information availability: many representations omit, truncate, aggregate, or explicitly provide information required by the question. Therefore, the large headline performance gaps cannot be interpreted as visualization-format dependence alone; they substantially measure the difference between representations that contain the required information and those that do not. This is particularly consequential because these original experiments provide the broad seven-model evidence underlying the paper's strongest claims.
-
The corrected experiments are too limited to support the paper's broad claims. The corrected suite is the experiment that more directly addresses the stated research question, yet its cleanest residual-format results are obtained primarily with Qwen2.5-VL-7B. The residual gap falls to 11.3 pp for tables and 4.9 pp for graphs, while no comparable all-format residual can be established for time series because several representations remain inherently lossy. Thus, the evidence that properly controls for answerability does not have the cross-model and cross-modality breadth of the original, confounded evaluation.
-
Limited conceptual novelty and insight. Once the answerability confound is removed, the main takeaway becomes substantially more expected: models perform better when the representation contains and explicitly exposes the information needed to answer the question. The paper documents this carefully, but provides limited new understanding of the underlying causes of visualization dependence. In particular, it does not offer a convincing mechanism or theory explaining why information-equivalent visual representations lead to different model behavior; it mainly quantifies that such differences remain. Given that the cleaner residual effects are also considerably smaller than the headline gaps, I find the resulting conceptual contribution weaker than the paper's framing suggests.
Questions for authors
No questions.