acceptodds
Under review as a conference paper at ICLR 2027

ChartSeer: A Benchmark for Inductive Reasoning over Charts

Abstract

Chart understanding is an important capability of multimodal large language models (MLLMs) for analyzing and reasoning over data visualizations. Existing chart understanding benchmarks largely focus on deductive reasoning, where the question specifies what relation to evaluate, and typically involve charts with relatively few data points. As a result, they largely overlook the evaluation of inductive reasoning over charts, which requires models to discover latent regularities across sufficient data points without being explicitly given the specific regularity. To address this gap, we introduce ChartSeer, a benchmark for evaluating inductive reasoning over charts. We construct ChartSeer from 1,438 data-rich charts collected from diverse public datasets and chart repositories, each paired with its underlying tabular data. Based on the underlying data, we identify and verify latent regularities and construct 3,929 questions covering two complementary capabilities: rule induction for identifying latent regularities in charts and example inference for generalizing from these regularities to new cases. We evaluate 14 state-of-the-art MLLMs and compare them with humans, finding that current models remain substantially behind human performance. Further analysis reveals that visual perception and rule induction remain two major bottlenecks. We hope ChartSeer facilitates future research on the inductive reasoning capabilities of MLLMs for chart understanding.

Primary area:
datasets and benchmarks
Keywords:
chart understanding, inductive reasoning, multimodal large language models, visual reasoning, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8wNfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ChartSeer, a benchmark for evaluating inductive reasoning over charts. It contains 1,438 charts and 3,929 question. Answer pairs covering trend, periodicity, distribution, and association.

The benchmark evaluates two capabilities: identifying underlying regularities and applying them to new cases. Experiments on 14 multimodal large language models show substantial gaps relative to human performance. Additional analyses using model-extracted tables, ground-truth tables, and reference rules suggest that both visual information extraction and rule induction contribute to model errors.

Critical strengths
  1. By focusing on the pattern induction in the charts and their application to new cases, it supplements the traditional chart-based question answering approach that mainly focuses on extracting, comparing and calculating numerical data.
  2. Documents substantial quality control: all 1,438 charts are manually inspected, and 400 QA pairs are independently audited by three experts for answer correctness and ambiguity.
  3. Evaluates 14 models, including proprietary models, general-purpose open-source models, and chart-specific models, and provides a matched human and model comparison on the same sampled questions in Figure 44.
Critical weaknesses
  1. The extracted-table versus gold-table comparison does not establish that content selection, formatting, and input length are controlled. Figure 6 shows reversals, such as GPT-5.4’s periodicity accuracy dropping from 35.0% to 27.5%, which require further analysis before attributing performance differences to extraction errors.
  2. Some reference-rule prompts explicitly provide the target class, as acknowledged in Appendix J. Improvements under this condition therefore cannot cleanly distinguish successful rule application from following an answer-containing instruction.
  3. The text-only baseline reaches 44.1% on distribution-based example inference, but random-choice, majority-label, and option-frequency baselines are missing. This leaves the contribution of guessing, label imbalance, and textual shortcuts unresolved.
Questions for authors
  1. Why does Gold Table underperform Extracted Table in some settings in Figure 6, such as GPT-5.4 on periodicity (27.5% vs. 35.0%)? Are the two table inputs matched in content coverage, formatting, and length? Could the authors provide an analysis or a controlled experiment that corrects extraction errors while preserving the extracted table’s structure, to distinguish data-correctness effects from representation effects?
  2. For some classification instances, Appendix J indicates that the reference-rule prompt supplies the predicted class and instructs the model to return it. In these cases, what reasoning remains for the model to perform? Could the authors clarify how gains from these instances support the bottleneck analysis, given that the intervention provides the classification result rather than only the rule needed to derive it?
2 comments

Q1: Why does Gold Table underperform Extracted Table in some settings? Are content coverage, formatting, and input length controlled?

A1: This may be because models do not fully utilize the provided information. We strictly control the question, answer format, and evaluation criterion, but models may not make better use of more accurate table data when they are provided through prompts. One possible way to improve their use of this information is fine-tuning. We will add this clarification in Section 4.3. Moreover, the extracted and gold tables have largely consistent structures. Therefore, the proposed control would largely duplicate our existing comparison setting. Nevertheless, we will add a discussion.


Q2: What reasoning remains when the reference-rule prompt supplies the target class, and how do gains from these instances support the bottleneck analysis?

A2: We argue that, as described in Section 4.3, Gold Table + Rule provides reference rules for models to perform example inference. Its purpose is to examine whether supplying the underlying regularity helps models apply it to new cases. Nevertheless, we will provide a more detailed explanation in the appendix.

Thank you for the clarification. My concerns remain:

  1. It remains unclear whether table content, formatting, and input length are controlled. The explanation that models fail to utilize accurate information remains speculative.
  2. The purpose of reference rules is clear, but what reasoning remains when the correct target class is already supplied is still unclear.

These concerns limit the conclusions supported by the bottleneck analysis. I will maintain my original score.

Reviewer_8CGeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ChartSeer benchmark with 1,438 charts and 3,929 questions to evaluate inductive reasoning through rule induction across four statistical regularities. The authors construct chart-table pairs, apply statistical filtering and human validation, and evaluate 14 multimodal models to reveal performance gaps due to challenges in visual perception and rule induction.

Critical strengths
  1. ChartSeer contains 1,438 charts from multiple sources with their underlying structured data and cover trend, periodicity, distribution, and association.

  2. The evaluation covers 14 multimodal models and investigates model behaviour under different inout conditions.

Critical weaknesses
  1. The core motivation of this work is the existing chart benchmarks primarily assess deductive reasoning and the inductive reasoning is largely under explored. However, the distinction is not sufficiently established against the prior works. Like ChartQAPro (Masry et al., 2025) explicitly includes hypothetical questions involving future trends and extrapolation, which overlap with this paper’s example inference formulation. Also, ChartMuseum (Tang et al., 2025), CharXiv (Wang et al., 2024), and InfoChartQA (Xie et al., 2025) also explore higher-level interpretation and reasoning over charts. The authors cite several of these benchmarks but do not adequately examine their task-level overlap, and ChartQAPro is missing from the discussion, can the authors answer this with a cleaner distinction to support their claims?

  2. I can see that most rule induction questions involve selecting a rule from predefined options, do you have evidence that establishes models actually infer the latent regularity rather than identify or eliminate candidate answers? Like comparison between MCQ and free-form rule induction on the same charts?

Questions for authors

See questions in the “Critical Weaknesses”

  1. Did you evaluate option-only or question-only baselines, answer-position distributions, and simple statistical heuristics that apply the ChartSeer's regularity definitions?

  2. What is the difference between interpolation from extrapolation? Any failure case study based on these two?

I will appreciate the answers to these questions in the rebuttals.

2 comments

Q1 & W1: How is ChartSeer distinct from prior benchmarks with higher-level reasoning and extrapolation questions?

A1: We argue that Table 1 already shows the substantial differences between our benchmark and existing work, including ChartMuseum, CharXiv, and InfoChartQA. Although these benchmarks include higher-level interpretation questions, ChartSeer systematically evaluates both identifying and applying statistically verified latent regularities.

Specifically, we examined ChartMuseum’s 1,162 QA records and identified only 38 questions (3.27%) involving inductive reasoning, with no evaluation of example inference. In CharXiv, only 1.65% of its 11,615 QA records involve inductive reasoning, all of which concern rule induction over a single trend regularity, with no evaluation of example inference. For InfoChartQA, the chart images have not been publicly released, preventing a detailed comparison of the datasets.

We also examined ChartQAPro’s 1,948 QA records and found that only 98 (5.03%) are hypothetical questions. Among these, only 21 (1.08% of all records) involve inductive reasoning, all of which focus on example inference within a single regularity family, without evaluating rule induction. In contrast, ChartSeer systematically evaluates both rule induction and example inference across four regularity families and 54 chart types on 3,929 QA records.


Q2 & W2: Do multiple-choice questions establish that models infer latent regularities rather than identify or eliminate candidate answers? Was a comparison with free-form rule induction conducted?

A2: We understand your concern that models may eliminate candidate answers rather than infer latent regularities. However, we believe evidence-based elimination itself involves evaluating competing regularities, because ruling out a candidate requires assessing whether it is consistent with the observed data. Using multiple-choice questions to evaluate reasoning also has precedent, as exemplified by Multimodal-CoT (Zhang et al., 2024). A comparison with free-form rule induction is reasonable but requires substantial human evaluation effort. Nevertheless, the 20 case studies included in our paper provide qualitative examples in which models derive regularities from chart evidence rather than merely eliminate answer options. We leave a matched comparison with free-form rule induction for future work.


Q3: Did you evaluate option-only or question-only baselines, answer-position distributions, and simple statistical heuristics that apply ChartSeer’s regularity definitions?

A3: We understand your concern about whether models can answer using options, textual biases, or simple statistical heuristics without understanding the chart. Our existing experiments directly examine this concern. Removing the chart reduces GPT-5.4’s accuracy from 53.3% to 22.0% on rule induction and from 37.7% to 18.6% on example inference (Table 2, p. 7). Replacing the chart while keeping the question and options unchanged also substantially reduces accuracy for both tested models (Appendix G.2, pp. 36–37). These results indicate that model performance substantially depends on chart-specific evidence.

Regarding statistical heuristics, our regularity criteria verify regularities using the underlying data (§3.2), whereas evaluated models must identify the relevant regularity from the chart and apply it to new cases. Our controlled experiments with gold tables and reference rules (§4.3) further examine these requirements. We therefore believe the existing evaluations adequately support our conclusions.


Q4: What is the difference between interpolation and extrapolation? Were failure cases compared between them?

A4: Interpolation predicts an unobserved case within the observed domain, whereas extrapolation predicts beyond the observed range or period. ChartSeer’s example inference covers both future predictions and predictions for new instances. We already analyze 160 matched rule induction–example inference pairs sharing the same chart and target regularity (Appendix H), alongside detailed failure cases (Appendix K).

I thank the authors for their clarifications.

However, I remain concerned about whether the multiple-choice formulation adequately captures genuine rule induction, and I find the evidence addressing this concern insufficient.

After considering the rebuttal and other reviewers' discussions, I will maintain my original rating.

Reviewer_BSKeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • The paper introduces ChartSeer, a benchmark of 1,438 charts and 3,929 questions covering trends, periodicity, distributions, and associations.
  • The benchmark pairs rule induction with example inference to evaluate both identifying regularities and applying them to new cases.
  • Experiments on 14 MLLMs show substantial gaps relative to humans, with additional analyses investigating perception and reasoning bottlenecks.
Critical strengths
  • The benchmark covers several regularity families and combines original charts with visualizations rendered from public data.
  • The scale of the dataset is viable for the task
Critical weaknesses
  • TSRBench and ChartQAPro address some of the same gaps in the existing benchmark; it would make sense to compare against them.
  • Multiple-choice rule identification and numerical prediction with a 5% tolerance differ in answer space and scoring.
  • The scope is not significantly different from the existing works.
  • The complexity of the showcased charts seems limited compared to those in the wild or in recent benchmarks.
Questions for authors
  • What were the initial disagreement and ambiguity rates in the 400-question semantic audit?
  • What capabilities does ChartSeer measure beyond TSRBench’s induction/prediction tasks and ChartQAPro’s extrapolation questions?
  • How sensitive are the conclusions to stronger specialist models, larger reasoning budgets, and tool-assisted chart analysis?
2 comments

Q1: What were the initial disagreement and ambiguity rates in the 400-question semantic audit?

A1: Appendix D (p. 29) describes the independent audit and adjudication. The initial disagreement and ambiguity rates were 5.5% and 5.3%, respectively. We will add these statistics.


Q2: What capabilities does ChartSeer measure beyond TSRBench’s induction/prediction tasks and ChartQAPro’s extrapolation questions?

A2: We argue that Table 1 already shows the substantial differences between our benchmark and existing work. Traditional benchmarks include only a small number of examples covering a single chart type for these tasks, whereas our benchmark uses diverse chart types with complementary regularities and evaluates models’ inductive reasoning capabilities systematically. This makes our benchmark fundamentally different from existing benchmarks.

We also examined ChartQAPro’s 1,948 QA records and found that only 98 (5.03%) are hypothetical questions. Among these, only 21 (1.08% of all records) involve inductive reasoning, all of which focus on example inference over charts covering a single regularity family, without evaluating rule induction. Similarly, TSRBench includes only 100 explicitly designated inductive reasoning questions (2.42% of all records), all of which focus on example inference over charts covering a single regularity family, without evaluating rule induction. Moreover, its visual inputs are limited to time-series line plots. In contrast, ChartSeer systematically evaluates both rule induction and example inference across four diverse regularity families and a broader range of chart types, with 54 chart types.


Q3: How sensitive are the conclusions to stronger specialist models, larger reasoning budgets, and tool-assisted chart analysis?

A3: We argue that our evaluation is reasonable for the following reasons. Regarding “stronger specialist models,” we evaluate ChartGemma and TinyChart, which achieve 26.9%/18.9% and 27.7%/15.5% on rule induction/example inference, respectively (Table 2). Regarding “tool-assisted chart analysis,” our Gold Table and Gold Table + Rule experiments indirectly examine its potential benefits, showing that reliable data access helps but does not eliminate reasoning errors (§4.3). Regarding “larger reasoning budgets,” we include o3 and o4-mini with their native reasoning configurations, which we consider adequate for the current evaluation. ChartQA’s 5% relative-error tolerance, while option-based questions use exact match, with the same criteria applied to humans (§4.1). We therefore believe our evaluation is appropriate for assessing the evaluated models under the reported settings.

Finally, numerical prediction follows ChartQA’s 5% relative-error tolerance, while option-based questions use exact match, with the same criteria applied to humans (§4.1). We therefore believe our evaluation is appropriate for assessing the evaluated models under the reported settings.

I thank the authors for the new insights. While A2 and A3 partially address my concerns with the paper overall, having read all reviews and responses so far, I still lean towards the original rating.

Comments on the paper

1 comment
Area_Chair_H16TOpenReview ↗

Reminder: Please Review the Author Response and Reconsider Your Final Decision

Dear Reviewers,

This is a gentle reminder to review the authors’ response to your comments. After considering their clarifications and rebuttals, I kindly ask you to reassess your review and update your final recommendation if warranted.

Thank you for your time and thoughtful consideration.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.