- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces ChartSeer, a benchmark for evaluating inductive reasoning over charts. It contains 1,438 charts and 3,929 question. Answer pairs covering trend, periodicity, distribution, and association.
The benchmark evaluates two capabilities: identifying underlying regularities and applying them to new cases. Experiments on 14 multimodal large language models show substantial gaps relative to human performance. Additional analyses using model-extracted tables, ground-truth tables, and reference rules suggest that both visual information extraction and rule induction contribute to model errors.
Critical strengths
- By focusing on the pattern induction in the charts and their application to new cases, it supplements the traditional chart-based question answering approach that mainly focuses on extracting, comparing and calculating numerical data.
- Documents substantial quality control: all 1,438 charts are manually inspected, and 400 QA pairs are independently audited by three experts for answer correctness and ambiguity.
- Evaluates 14 models, including proprietary models, general-purpose open-source models, and chart-specific models, and provides a matched human and model comparison on the same sampled questions in Figure 44.
Critical weaknesses
- The extracted-table versus gold-table comparison does not establish that content selection, formatting, and input length are controlled. Figure 6 shows reversals, such as GPT-5.4’s periodicity accuracy dropping from 35.0% to 27.5%, which require further analysis before attributing performance differences to extraction errors.
- Some reference-rule prompts explicitly provide the target class, as acknowledged in Appendix J. Improvements under this condition therefore cannot cleanly distinguish successful rule application from following an answer-containing instruction.
- The text-only baseline reaches 44.1% on distribution-based example inference, but random-choice, majority-label, and option-frequency baselines are missing. This leaves the contribution of guessing, label imbalance, and textual shortcuts unresolved.
Questions for authors
- Why does Gold Table underperform Extracted Table in some settings in Figure 6, such as GPT-5.4 on periodicity (27.5% vs. 35.0%)? Are the two table inputs matched in content coverage, formatting, and length? Could the authors provide an analysis or a controlled experiment that corrects extraction errors while preserving the extracted table’s structure, to distinguish data-correctness effects from representation effects?
- For some classification instances, Appendix J indicates that the reference-rule prompt supplies the predicted class and instructs the model to return it. In these cases, what reasoning remains for the model to perform? Could the authors clarify how gains from these instances support the bottleneck analysis, given that the intervention provides the classification result rather than only the rule needed to derive it?
2 comments
Thank you for the clarification. My concerns remain:
- It remains unclear whether table content, formatting, and input length are controlled. The explanation that models fail to utilize accurate information remains speculative.
- The purpose of reference rules is clear, but what reasoning remains when the correct target class is already supplied is still unclear.
These concerns limit the conclusions supported by the bottleneck analysis. I will maintain my original score.
Q1: Why does Gold Table underperform Extracted Table in some settings? Are content coverage, formatting, and input length controlled?
A1: This may be because models do not fully utilize the provided information. We strictly control the question, answer format, and evaluation criterion, but models may not make better use of more accurate table data when they are provided through prompts. One possible way to improve their use of this information is fine-tuning. We will add this clarification in Section 4.3. Moreover, the extracted and gold tables have largely consistent structures. Therefore, the proposed control would largely duplicate our existing comparison setting. Nevertheless, we will add a discussion.
Q2: What reasoning remains when the reference-rule prompt supplies the target class, and how do gains from these instances support the bottleneck analysis?
A2: We argue that, as described in Section 4.3, Gold Table + Rule provides reference rules for models to perform example inference. Its purpose is to examine whether supplying the underlying regularity helps models apply it to new cases. Nevertheless, we will provide a more detailed explanation in the appendix.