acceptodds
Under review as a conference paper at ICLR 2027

AdjuvantChart: Benchmarking Multi-Panel Relational Understanding in Real-World Scientific Charts

Abstract

Scientific chart understanding requires models not only to read numerical values but also to identify experimental conditions and integrate evidence across panels to draw conclusions. Existing benchmarks focus primarily on visual information extraction, numerical reasoning, and single-turn scientific question answering. They place less emphasis on how models organize and use chart evidence in real experimental contexts. We introduce AdjuvantChart, a scientific chart understanding benchmark built from published adjuvant research papers. Guided by domain experts’ needs in literature analysis, we design three tasks: structured data extraction, experimental relation verification, and cross-panel composition. For the latter two tasks, we construct question families grounded in traceable atomic facts, comprising local, compositional, and condition-variation questions. We evaluate both answers and supporting evidence to assess whether models consistently use the same underlying chart evidence across related questions. AdjuvantChart contains 600 high-quality composite figures from published papers. The experimental relation verification and cross-panel synthesis tasks include 380 question families and 1,134 question-answer pairs. In our evaluation, GPT-5.5 achieves the highest exact-match accuracy of 81.71% on the question-answering tasks, while Gemini-3.7-Flash achieves the highest RMS-F1 score of 54.92% on structured extraction. Joint evaluation of answers and supporting evidence changes the relative ranking of some models, indicating that evaluating final answers alone does not fully capture models’ ability to use evidence from scientific charts. AdjuvantChart provides a new testbed for evaluating multimodal models’ chart understanding and evidence use in experimental contexts.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific chart understanding, multimodal large language models, compositional reasoning, benchmark evaluation.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AFNUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces AdjuvantChart, a multi-chart understanding benchmark based on charts from adjuvant research papers. The benchmark defines three tasks: structured data extraction, experimental relationship verification, and cross-panel composition. The authors select the charts from the source papers and constructed 380 question families containing 1,134 question-answer pairs. The authors then evaluated different model configurations with two different settings: target panel provided and target panel needs to be identified by the model.

Critical strengths

1. Controlled question families based on shared evidence.

The authors are constructing related questions from shared atomic facts. This idea is interesting as they construct condition-variant questions that provides a relatively controlled settings for examining whether models can adjust their answers and supporting evidence when the question changes.

Critical weaknesses

1. Unclear benchmark formulation and experiment presentation, make it difficult to assess the paper's contribution.

The paper presentation is very difficult to follow.

The wording of the key tasks types is not very consistent throughput the paper, which makes it difficult to understand the connection between section 3's task definition and section 4's experimental set up. E.g. "Structured knowledge extraction" VS. "Structured data extraction." And for "cross-panel composition," it is unclear which part of the experiments in section 4 is especially targeted for this task. Based on section 3's definition, it is a multi-image QA task so it might be able to assume that the answer accuracy can be addressing this part, but this should have been clearly stated. The shared-evidence family analysis seems to be an interesting experiment and corresponds to the question creation process, but it is not stated in the main tasks of this benchmark.

The main text is also missing a complete example of the QA-pairs in the benchmark under the three evaluation tasks together with its source image, extracted atomic facts, and rules being applied.

The motivation of focusing on adjuvant research papers is not clearly explained in benchmark formulation. It is not justified why adjuvant research charts are specifically well-fitted for the three evaluation tasks comparing to other fields.

2. Underspecified question-generation pipeline and inadequate validation of the benchmark ground truth.

A main contribution of this benchmark should come from the question generation. Section 3.3 ntroduces executable programs that produce reference answers and supporting evidence from annotated atomic facts, and appendix c.1 also lists the rule patterns and operations in natural language for the program. However, it is very unclear what is the executable program as it is not clearly stated anywhere. The process of transforming facts to natural language questions is not specified.

As for validation, appendix b.3 describes high level validation steps like "fact verification", but it is unclear who is responsible for these steps and how the quality control is being executed. The AI use statement stated that generative AI assisted with dataset construction and review, but the manuscript does not clearly explain which construction and verification stages involved AI assistance versus independent human expert assessment. The authors explicitly wrote in b.3: "For question answering, program replay establishes consistency between the annotations and executable rules; it does not establish that the annotations correctly interpret the source figures." and "Independent expert adjudication remains incomplete in the archived version, including unresolved source-interpretation cases. The references therefore have not been established as fully expert-validated annotations." It is unclear what is the "archived version" or "references" here, and this poses great concern about the benchmark.

Questions for authors
  1. Could the authors provide a complete worked example in the main text showing how an original scientific figure is converted into annotated atomic facts, natural-language questions, reference answers, and supporting evidence?

  2. How are the natural-language questions and their corresponding executable programs constructed from the annotated atomic facts?

  3. Which steps of question generation and ground-truth validation are performed manually, programmatically, or with generative AI assistance?

Reviewer_raekOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AdjuvantChart, which is a benchmark for evaluating multimodal models’ ability to understand scientific charts beyond numerical extraction, and it focuses on experimental relationships, cross-panel reasoning, and evidence grounding. Experiments show that GPT-5.5 achieves the highest QA accuracy and the key takeaway is correct answers do not necessarily correspond to the right use of supporting evidence.

Critical strengths
  • This paper addresses an important problem, and is generally well written and well motivated.
  • The tasks are diverse and cover the practical perspectives of scientific understanding, and the benchmark includes real world scientific figures.
Critical weaknesses
  • The paper could better describe the difference compared with prior work such as CharXiv and others as they already evaluate scientific chart interpretations. Beyond this, the scientific domain seems kinda narrow which further limits whether the findings are generalizable to other domains.
  • Appendix B.3 mentioned that independent expert adjudication remains incomplete, this might also further hurt the reliability of the benchmark.
  • Some compositional questions involve parallel verification of multiple records rather than complex multi-step reasoning, thus the benchmark may not consistently distinguish cross-panel reasoning from independent information retrieval.
Questions for authors
  • What proportion of compositional questions requires dependent multi-step reasoning rather than independent verification of multiple chart observations?
Reviewer_gSMjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AdjuvantChart, a scientific chart understanding benchmark that integrates three tasks: structured data extraction, experimental relation verification and cross-panel composition. Both answers and supporting evidence are evaluated to reveal the model capability of evidence extraction and organization. Finally, the study conducts comprehensive experiments indicating that evaluating final answers alone does not fully capture models’ ability to use evidence from scientific charts.

Critical strengths
  1. The benchmark targets experimental condition binding and cross-panel evidence integration, which are important for scientific chart understanding.
  2. Traceable atomic facts and executable programs support controlled construction of linked question families.
  3. Evaluating both answers and supporting evidence provides a richer assessment than answer accuracy alone.
Critical weaknesses
  1. The conceptual novelty appears limited, as the benchmark mainly combines structured extraction, relation verification, and cross-panel reasoning. The paper does not clearly establish what fundamentally new evaluation capability this combination offers beyond existing benchmarks.
  2. The reliability of the reference annotations remains uncertain, as Appendix B.3 acknowledges incomplete expert adjudication and unresolved source interpretations. This is a significant concern for a benchmark centered on precise experimental semantics.
  3. The main text is overly descriptive and lacks a concrete worked example illustrating the atomic facts, linked questions, and expected evidence outputs. This makes the benchmark design difficult to follow.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.