acceptodds
Under review as a conference paper at ICLR 2027

StructViz-Bench: A Controlled Study of Visualization-Format Dependence in Multimodal LLM Reasoning over Structured Data

Abstract

Multimodal LLMs increasingly reason over structured data that reaches them as a picture—a bar chart, a heatmap, a node–link diagram—and the rendering is usually chosen incidentally by whatever tool drew the figure. We ask whether that arbitrary choice changes what the model concludes, and introduce StructViz-Bench, which treats visualization format as a controlled variable—fixing the underlying data and the question semantics and varying only how the data is drawn—and is, to our knowledge, the first to do so across structured modalities: tabular, time-series, and graph under one protocol. The benchmark comprises 3,795 base items rendered into 14 visualization types over four reasoning-depth categories. Evaluating several proprietary and open MLLMs, we establish that the choice matters, and by a wide margin: within-modality best–worst exact-match gaps reach 40.5pp, roughly half of all base questions (45–56% per model) change correctness under some format swap, and paired item-level tests are significant for all 12 model×modality comparisons (Bonferroni-corrected p<0.01). The effect is not an artifact of the scoring rule or of layout randomness, and it persists across model families, a 4.6× parameter increase, and all four prompt styles we test—including chain-of-thought, which never helps (4.4–14.6pp below each model's best non-CoT prompt). An answerability audit of every (question, rendering) pair then shows what these gaps are made of: about half of all pairs do not encode the answer at all (a bar chart of column means has no rows; a drawing of a large graph has no node labels), and on those, accuracy sits at prior level for every model. Re-rendering the suite so that every format encodes the answer and re-evaluating an open model, the answer-less renderings recover 16–23pp, the graph listing loses 10pp once it stops printing node degrees, and the rendering effect that remains for the same information is 11pp on tables and 5pp on graphs; none is identified for time series, where no rendering but the complete listing beats a question-only prior, and no graph rendering beats it at all for this model. Printing the queried quantity matters more than the chart type. For practice, much of the accuracy cost is recoverable by choosing the rendering: a learned lookup that picks the best rendering per question type, with no weight update, recovers up to 19.5pp over a random format and 2–8pp over the best fixed format per modality (object-level CIs exclude zero for all seven models; a hand rule that only picks an encoding format captures 0.4–2.7pp of it); and multi-format fine-tuning raises cross-format agreement in every modality (Consistency Rate up 15–25pp) and narrows the accuracy gap in two of three, while a λ=0 control shows that a consistency regulariser adds agreement but no accuracy, partly by making errors consistent. We release the benchmark, the deterministic generation pipeline, and all evaluation and mitigation code.

Primary area:
datasets and benchmarks
Keywords:
multimodal large language models, visualization format, structured data reasoning, benchmark, robustness, chart understanding, format sensitivity, answerability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_42xPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether multimodal LLMs give different answers when the same structured data is shown in different visual formats. The authors introduce StructViz-Bench, a new benchmark with 3,795 base questions across tables, time series, and graphs. The same data and question are kept fixed, while only the visual representation is changed. They evaluate several open and closed multimodal models and report large accuracy differences across formats in the original benchmark.

The paper then shows that a large part of this difference comes from the fact that some visualizations do not actually contain all the information needed to answer the question. For example, some plots only show averages, some graph views do not label nodes, and some text-based views are truncated. The authors therefore create a corrected version of the benchmark where the visualizations contain more complete information. On this corrected setup, the remaining format effect is smaller, but still present for tables and graphs for the main open model they test. They also compare image inputs with raw text, test different prompt styles, and try two mitigation methods: choosing the best format for each task and fine-tuning a model on multiple formats. Overall, the paper argues that model performance depends on how structured data is presented, but also shows that information availability itself is a major part of this effect.

Critical strengths

The paper introduces StructViz-Bench, a new controlled benchmark for studying visualization-format dependence across tabular, time-series, and graph data. The benchmark construction is systematic and accompanied by a deterministic generation and evaluation pipeline, making it potentially useful for future work.

Critical weaknesses
  1. The main experiments do not cleanly test the paper's stated research question. In the original benchmark, visualization format is confounded with information availability: many representations omit, truncate, aggregate, or explicitly provide information required by the question. Therefore, the large headline performance gaps cannot be interpreted as visualization-format dependence alone; they substantially measure the difference between representations that contain the required information and those that do not. This is particularly consequential because these original experiments provide the broad seven-model evidence underlying the paper's strongest claims.

  2. The corrected experiments are too limited to support the paper's broad claims. The corrected suite is the experiment that more directly addresses the stated research question, yet its cleanest residual-format results are obtained primarily with Qwen2.5-VL-7B. The residual gap falls to 11.3 pp for tables and 4.9 pp for graphs, while no comparable all-format residual can be established for time series because several representations remain inherently lossy. Thus, the evidence that properly controls for answerability does not have the cross-model and cross-modality breadth of the original, confounded evaluation.

  3. Limited conceptual novelty and insight. Once the answerability confound is removed, the main takeaway becomes substantially more expected: models perform better when the representation contains and explicitly exposes the information needed to answer the question. The paper documents this carefully, but provides limited new understanding of the underlying causes of visualization dependence. In particular, it does not offer a convincing mechanism or theory explaining why information-equivalent visual representations lead to different model behavior; it mainly quantifies that such differences remain. Given that the cleaner residual effects are also considerably smaller than the headline gaps, I find the resulting conceptual contribution weaker than the paper's framing suggests.

Questions for authors

No questions.

Reviewer_UfkHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces StructViz-Bench, a question-answering benchmark for assessing how visualization formats affect the performance of multimodal large language models. It includes 3,795 questions over tabular, time-series, and graph data rendered in 14 visualization formats. Evaluations of seven models show that accuracy can vary by up to 40.5% across visualization formats. An audit further shows that, in roughly half of the cases, the visualization does not contain sufficient information to answer the corresponding question. To address this issue, the authors re-render the visualizations to improve information availability and explore fine-tuning on multiple visualization formats to reduce performance variation across formats.

Critical strengths

The paper addresses an important question about the robustness of multimodal large language models by comparing their performance across different visualization formats of the same underlying data and questions.

Critical weaknesses
  1. Construct-validity issue. The expressiveness criterion requires a visualization to encode the information it is intended to convey (Mackinlay, 1986). However, the authors' own audit shows that, in roughly half of the cases, the rendered image does not contain sufficient information to answer the corresponding question. For such examples, an incorrect answer cannot be attributed to a model's ability to interpret the visualization. Consequently, the benchmark may conflate sensitivity to visualization format with differences in what information each format actually exposes.

  2. Weak evaluation. In Sec. 5, the authors re-render the visualizations to increase the fraction of answerable examples, but some corrected visualizations still do not contain all information required by the questions. This limits the fairness of comparisons across formats even after correction. Moreover, the corrected suite is evaluated on only two open-source models, and all models in the main evaluation were released between 2024 and mid-2025. The empirical evidence is therefore insufficient to establish that the reported trends generalize to stronger or more recent models.

  3. Limited external validity. The visualizations are generated using the authors' rendering pipeline, and the questions are template-based. This controlled setup is useful for isolating format effects, but it may not capture the diversity and noise of real-world visualizations and user questions. Similarly, the fine-tuning experiments are trained and evaluated within the benchmark setting, without validation on naturally occurring charts. It is therefore unclear how well the reported findings and mitigation strategy transfer to practical use cases.

  4. Poor presentation. The paper often presents many statistics and results in long, complex sentences with frequent parentheses and dashes. This makes the main research questions, key findings, and conclusions unnecessarily difficult to follow.

Reference J. Mackinlay. Automating the design of graphical presentations of relational information. ACM Transactions on Graphics, 5(2):110–141, 1986. doi: 10 .1145/22949.22950.

Questions for authors
  1. Many rendering parameters, such as displaying only the first 12 rows or labeling nodes only for graphs with fewer than 30 nodes, directly determine whether the information required by a question is visible. How were these parameters chosen?

  2. Did the rendering process explicitly consider the expressiveness criterion in visualization (Mackinlay, 1986)?

Reviewer_uMX6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces StructViz-Bench to study how visualization format affects multimodal LLM reasoning over structured data, including tabular, time-series, and graph data. The benchmark holds the underlying data and question semantics fixed while varying the visualization format. It includes 14 visualization types, synthetic and real-world data, and template-based questions organized into four reasoning categories: 1-hop, 2-hop, 3-hop, and counterfactual. The dataset yields 18K rendered instances per model. The authors provide programmatic and human answerability audits and release code to support reproducibility. Evaluations across multiple model families reveal substantial performance differences across formats, partly attributable to whether the rendering contains the information needed to answer the question and model preference. The authors also provide a way to select a format for each model using a lookup model learned from a training split, as well as multi-format fine-tuning to improve consistency across formats.

Critical strengths
  • The paper addresses the interesting challenge of how MLLMs interpret different visual representations of tabular, time-series, and graph data commonly encountered in academic and technical documents.
  • The dataset construction is clearly described and supported by released code for reproducibility. The authors also provide programmatic and human audits to assess data quality and answerability.
  • The evaluation spans multiple model families and experimental settings, supporting the credibility and robustness of the findings.
  • The authors provide explanations and follow-up experiments to investigate why models succeed or fail on the benchmark.
  • The authors introduce a lookup table learned from a training split to select a visualization format for each model and question type, providing a practical approach that could inform future work.
Critical weaknesses
  • Although the evaluation spans multiple model families and includes a comparison between Qwen2.5-VL-7B and 32B, within-family scaling experiments remain limited. Evaluating additional sizes and families would help establish whether the observed scaling trends generalize.
  • Although the question templates, source domains, and quality checks are documented, further analysis of how template design and answer distributions affect question difficulty would strengthen the benchmark’s interpretation.
  • The paper defines reasoning categories and evaluates CoT prompting, but a breakdown of CoT performance by reasoning category would help clarify whether its effects differ across question types. In particular, it would be useful to determine whether CoT helps on more complex questions when the necessary information is available in the rendering.
  • The answerability audits and follow-up experiments provide useful explanations of performance differences. However, a more systematic categorization of errors (categories analysis) would help distinguish failures further.
  • As an optional extension, the authors could investigate whether a model can convert an input visualization into its preferred format before answering. Sharing the results would strengthen the claim about the preferred format, especially captioning.
Questions for authors

Please address the weaknesses and the question below.

  • The text-only rendering performs strongly in several settings. Although the paper also evaluates raw textual inputs for Qwen2.5-VL-7B, could the authors further investigate whether extracting structured text from an input image before answering improves performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.