acceptodds
Under review as a conference paper at ICLR 2027

OmniScience++: A Hierarchical Corpus for Evidence-Grounded Scientific Reasoning

Abstract

Scientific conclusions often depend on observations spread across multiple panels, measurements, and experimental conditions. Understanding these figures requires identifying the relevant evidence and determining how the observations support a conclusion. We introduce OmniScience++, a hierarchical corpus for evidence-grounded scientific reasoning, organizing 1.52 million figures into a figure–panel–subfigure structure with 10.02 million localized atomic-subfigure records. An evidence-first data engine turns this corpus into questions with explicit visual support. The engine selects observations and composes a reasoning program before generating the question and answer. Constraints on entities, conditions, and measurements guide evidence composition, while source regions and intermediate operations remain traceable. This construction supports scientific tasks such as comparing experimental outcomes and using an intermediate finding to select evidence for a subsequent inference. We evaluate these capabilities with a 600-question human-reviewed benchmark covering quantitative analysis, visual structure, and experimental relations. Across eight multimodal models, answer accuracy and evidence localization produce different rankings. Providing reference evidence crops improves answer accuracy by 2.36-17.70 percentage points across four models. By enabling the construction and evaluation of scientific reasoning tasks with explicit visual support, OmniScience++ provides a foundation for future AI for scientific research that connects conclusions to their underlying experimental evidence.

Primary area:
datasets and benchmarks
Keywords:
AI for Science, Image Caption, Scientific Image Dataset, Multi-modal Large Language Model, Document Analysis, Data Mining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_EzLtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces OmniScience++, an extension of OmniScience containing 10,021,541 atomic-subfigure records derived from approximately 1.52 million scientific figures. The resource links localized regions to source figures and paper metadata, with visual types, semantic tags and generated local captions. Its construction pipeline combines a task-adapted YOLO26x detector with recursive decomposition, semantic recursion gating, residual-region recovery and semantic annotation.

An evidence-first engine selects compatible observations and an intended reasoning program before generating questions, answers and region-linked reference steps. Evaluation comprises Eval-1K, with 1,000 figures and 12,785 reference boxes; an automated annotation audit; and a separate 600-question human-reviewed diagnostic benchmark. Eight multimodal models are evaluated for answer correctness and evidence localization and four receive reference evidence crops in an oracle condition. The paper reports extraction improvements, different answer and grounding rankings and increased overall answer accuracy with oracle crops. Its contributions concern resource construction and diagnostic evaluation, rather than a new answering model or a demonstrated training improvement (Sections 3–6).

Critical strengths
  1. Source-linked resource and inspectable construction records. Preserving source coordinates, paper associations, observations and intended operations is a useful design choice. Selecting evidence and a program before verbalization makes the intended construction inspectable rather than leaving it entirely implicit in a generated rationale. The scale and provenance structure give the resource potential reuse value, although they do not establish annotation correctness (Sections 3–4).

  2. Concrete extraction controls and complementary evaluation outputs. The paper includes a panel-trained detector baseline, task-specific detector refinement and SRG/LRR comparisons at multiple IoU thresholds. Eval-1K excludes the detector-refinement figures. Reporting answer accuracy separately from grounding, together with a same-model crop intervention, provides more information than answer accuracy alone. These are useful evaluation elements even though their scope is limited (Sections 5–6; Table 3).

Critical weaknesses
  1. The reliability and scope of the evidence supporting the resource and benchmark are insufficiently established.

Evaluation references and coverage. Eval-1K is described as sampled from figures excluded from detector refinement and evaluated against manually calibrated boxes. However, the manuscript does not sufficiently document the figure-selection procedure, annotation instructions for atomic boundaries, independent checking or disagreement resolution, or Eval-1K-specific coverage of small, densely annotated and visually complex regions (Section 6.1; Appendix A.2). Creating an evaluation set is not itself problematic. The issue is that the disclosed protocol does not allow a reliable assessment of reference consistency, selection effects, or the scope of generalization. Full-corpus composition statistics do not establish the composition of Eval-1K.

Scientific validity beyond box overlap. Table 3 measures agreement with reference boxes, not whether the extracted evidence preserves, or reliably links to, the labels, units, legends, scale bars and experimental conditions needed for a question. The source figure and context may mitigate crop omissions but their effectiveness is not quantified. Appendix B.2 appropriately considers boundary quality and factual reliability; however, its Qwen/GPT audit is explicitly not calibrated against human judgments. Schema consistency and numerical program checks also cannot certify the underlying extracted observations. The frequency of scientifically consequential annotation errors and their impact on downstream QA are not independently quantified by the reported evaluation. This is not a claim that a particular fraction of the 10.02M records is wrong.

QA selection and score comparability. The 600 questions are explicitly a curated diagnostic set, not a random audit of the corpus. Nevertheless, the candidate-pool size, rejection and correction counts and selection procedure are insufficiently reported (Sections 4.3–5.1; Appendix C.1). Human review may make the retained examples valid but it does not establish typical automated-generation quality or coverage of difficult cases. Additionally, Table 4 excludes refusals and unresolved answers separately for each model. The resulting conditional scores are not comparisons on an identical item set and the paper does not report enough retained-item counts or common-set analysis to assess sensitivity of the rankings (Section 5.2; Appendix C.2).

Statistical uncertainty and run-to-run stability. Tables 3–5 report point estimates without confidence intervals. Appendix A.2 specifies seed 0 but reports no multi-seed detector-training analysis. No repeated-run variability analysis is reported for the generative stages. Appendix C.4 leaves statistical uncertainty unassessed and Appendix F makes no significance claim. Thus, the stability of small extraction differences and oracle gains remains uncertain. Paired intervals accounting for shared source figures or papers and repeatability checks for trainable or stochastic components would help. Repeating identical deterministic inference would not. Missing uncertainty analysis compounds the existing validation gaps; it does not establish that the observed gains are insignificant.

Numerical consistency and stated limitations. Table 5 reports a −0.81 percentage-point quantitative oracle change for Claude, whereas Table 10 reports a +25.1% relative error reduction. Under the stated transformation, these cannot describe the same comparison and need reconciliation. Relevant caveats are acknowledged in Section 6.1 and Appendices B, C and F but Section 7 discusses mainly open-ended task coverage. The AI Use Statement also refers annotation-validation limitations to Section 7 without a corresponding discussion there. The implications of these limitations for the central contributions should be more clearly reflected in Section 7 and the conclusions.

Why this affects my recommendation. These concerns affect the validity and interpretation of the principal resource and benchmark evidence, not just presentation. Addressing them may require representative independent checks and additional analysis or rescoring, rather than merely more cautious wording. I do not infer deliberate selection bias or invalid data from the missing documentation.

  1. The incremental scientific contribution of the construction methodology is not convincingly demonstrated.

Differentiation from prior approaches. Structured visual QA generation is established by work such as GQA (Hudson and Manning, CVPR 2019); MultiChartQA (Zhu et al., NAACL 2025) studies comparative and sequential multi-chart reasoning; and SciEGQA (Yu et al., arXiv:2511.15090v2) combines scientific evidence grounding with automated QA construction. These are related settings, not identical systems or directly interchangeable benchmark scores. OmniScience++ could contribute through condition-aware composition and its integration with a large atomic-region resource. My concern is not missing citations, the use of existing components, or the absence of a new fundamental algorithm. It is the limited evidence establishing what important capability or insight this particular integration adds.

Controls do not isolate the central design choices. Every extraction configuration in Table 3 includes RD, so the contribution of recursion over a single-pass version of the same AS-YOLO checkpoint is unknown. The largest reported gain comes from additional atomic-region supervision. SRG improves the reported extraction results, while LRR further reduces omissions with a small decrease in strict-IoU F1. These results establish a local extraction tradeoff, not its effect on scientific evidence completeness or QA validity. More importantly, no matched comparison isolates evidence-first/program-first generation from direct region-conditioned generation, or tests the incremental contribution of the experimental compatibility constraints. Equation 3 states eligibility conditions but does not specify the sampling distribution or selection procedure sufficiently to assess reproducibility and coverage.

Contribution-specific evaluations and metrics are missing. Table 1 documents feature availability rather than feature reliability, breadth or practical advantage. Tables 4–5 characterize models answering the constructed questions; they do not compare construction methods. The reported evaluation does not quantify independently verified valid-QA yield, joint answer–evidence success or performance by reasoning-program family. Appendix F explicitly states that category-level gains cannot be attributed to the proposed select-then-lookup family. I therefore find little new understanding of the claimed compositional capabilities in these aggregate analyses. A comparison on the same source figures with comparable information access, supervision, generation budget and verification would help establish whether a reasonable direct or structured-generation alternative produces equally valid questions and evidence. Other convincing contribution-specific validation could also address this concern; comparing scores from different published benchmarks would not.

Scope of the resource-cost evidence. API expenditure and partial token accounting are legitimate implementation details and their exclusions are disclosed in Tables 3 and 12. They do not establish end-to-end computational efficiency or valid QA yield per unit of resource. I do not regard the absence of a local-VLM comparison as an independent rejection reason.

Why this affects my recommendation. I recommend rejection. The potential utility of the resource does not outweigh the unresolved validation gaps, the limited evidence for incremental value and the unquantified stability of the central results. These concerns affect the principal contributions and call for substantive validation or analysis rather than minor editorial clarification.

Questions for authors
  1. Eval-1K reference construction and coverage. How were the 1,000 figures selected and fixed relative to method development and how were atomic-region boundaries annotated and checked? Please clarify the instructions, annotator roles, handling of disagreements and whether references were drawn independently or corrected from model proposals. What existing evidence shows coverage and performance for small, densely annotated, or complex regions?

  2. Semantic reliability and QA selection. What independent checks against the original figures establish scientifically usable regions, correct observations and valid evidence–answer links? How many QA candidates were generated, rejected, corrected and retained to obtain the 600 questions? Please distinguish curated-benchmark validity from the reliability and yield of the automated generation process.

  3. Incremental benefit of the construction method. Are results available for the same AS-YOLO checkpoint without recursion and for a reasonable direct or structured QA-generation alternative on the same source figures with comparable information access, supervision, generation budget and verification? What evidence isolates the benefit of program-first selection and experimental compatibility constraints in valid question yield, evidence completeness, or compositional coverage?

  4. Comparable and internally consistent benchmark results. Please provide retained/refused/unresolved counts by model and category and clarify whether the conclusions hold on a common, adjudicated item set. How are the −0.81 pp change in Table 5 and +25.1% error reduction in Table 10 for Claude’s quantitative category reconciled? Can the existing per-item records support the stated compositional interpretation beyond category marginals?

  5. Statistical uncertainty and repeatability. Can the authors provide paired uncertainty estimates for the main extraction differences and oracle gains with shared-source dependence accounted for? What evidence is available on variability across detector training seeds or repeated stochastic generation? Please distinguish these robustness checks from fixed-seed reproducibility and operational retries.

2 comments

Thank you for the constructive review. Our responses to each questions are below.

Q1.

Thank you for the suggestion. As for construction of Eval-1K, we first manually refined approximately 6,000 figures. For each figure, one human annotator corrected model proposals, and two reviewers independently checked the resulting annotations. Acceptance required approval from both reviewers; disagreements were resolved through discussion among all three. We then split these figures into Eval-1K and the second-stage detector training set. Eval-1K was selected to approximately balance scientific disciplines, prioritizing figures containing multiple visual types. After selection, the figures and annotations underwent an additional manual review, during which nine ambiguous figures were replaced.

Eval-1K contains 1,000 figures with 12,785 reference boxes. Of these boxes, 15.4% occupy less than 1% of their source figure’s area. Among the figures, 18.5% contain at least 20 reference boxes, 49.2% contain at least three visual types, and 7.6% satisfy our defined scale-heterogeneity criterion.

Q2.

Thank you for the suggestions on presenting the benchmark construction and the questions generated by the data engine. QA pairs with problems were revised manually, and regenerated questions were reviewed manually again before the final 600 questions were fixed.

Table R1: Question revision and replacement rates

CategoryOriginal nQuestion unchangedInitial wording revisionLater condition clarificationFinal question changedDiscarded and replacedChanged or replaced
Quantitative analysis200191 (95.5%)4 (2%)1 (0.5%)5 (2.5%)4 (2%)9 (4.5%)
Visual structure200192 (96%)0 (0%)5 (2.5%)5 (2.5%)3 (1.5%)8 (4%)
Experimental relations200193 (96.5%)0 (0%)0 (0%)0 (0%)7 (3.5%)7 (3.5%)
Overall600576 (96%)4 (0.67%)6 (1%)10 (1.67%)14 (2.33%)24 (4%)

Q3

Thank you for the review on the experiments. As for the results not using recursion, we could not be able to obtain a decently good performance on subfigure level.

To compare our method with direct QA generation, we ran the following experiment, blind-reviewed by two researchers, with a third reviewer resolving conflicts.

We added a generation comparison with six configurations on the same 50 figures. Each configuration has 50 medium and 50 hard QA pairs.

  • A: the figure, context and caption are given directly to the LLM, which generates the QA, the evidence and the reference reasoning.
  • B: in addition to A's inputs, the evidence and sub-figures are given to the LLM, which generates directly.
  • C: generation by the OmniScience++ data engine.

The usability is defined as that question is answerable, answer is correct, difficulty is met, and boxes correctly cover the necessary evidence.

Table R2: Usability over 100 pre-assigned slots per configuration

GeneratorA: direct LLM generationB: evidence-assisted joint generationC: OmniScience++
GPT-6 Luna (medium)12%69%85%
GPT-6 Sol (high)41%86%96%

The results show that our evidence helps the LLM localize evidence and construct the reasoning better, and our complete pipeline achieves the best generation quality. In addition, with a lower-cost model, our pipeline reaches results comparable to group B, suggesting that our pipeline could be faster and cost less with a lighter-weight LLM.

Q4.

Thank you for the suggestions on presenting the benchmark results. We now consider refusals and invalid answers as incorrect, and judge disagreements went through a second round of review. The updated results are below.

Table R3: Main results on all 600 questions

modelJoint (answer × [email protected])AccuracyEvid. [email protected]Evid. [email protected]Evid. [email protected]refusalinvalid output
gpt-6-sol68.328279.8371.149.2800
gemini-3.8-flash35.6584.538.6834.3222.1171
claude-opus-5-546.6673.1750.0731.0910.431041
gpt-4o3.8435.178.870.25010
Kimi-k344.9975.555.3929.986.500
qwen3.8-max10.6168.6713.469.792.3630
GLM-5.3-flash34.3670.1745.2717.312.5209
DeepSeek-V4.1-Flash22.8867.1731.275.710.0810

The inconsistency is because Table 5 used paired questions while Table 10 used separately subsets due to refusal and invalid outputs; the corrected results use the same 600 questions, with refusals and invalid answers counted as incorrect.

Updated Table 10:

ModelQuantitative analysisVisual structureExperimental relations
Claude Opus 5.5−10.00%11.94%3.13%
GPT-4o25.45%10.88%33.33%
Qwen3.8-Max59.18%48.15%75.86%
Kimi-K33.45%8.99%27.59%
Reviewer_zbcKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OmniScience++ extends an existing scientific figure corpus with 10.02 million atomic-region records and a condition-aware question generation pipeline. Questions are generated after selecting observations and specifying a reference program for deriving the answer. The paper evaluates eight models on a 600-question human-reviewed benchmark using answer correctness and evidence localization. Its validation includes extraction ablations, annotation audits, and oracle-crop experiments.

Critical strengths
  1. Linking image regions, experimental conditions, and reference programs makes the evidence and answer construction inspectable, providing structured supervision beyond figure-level descriptions.

  2. The additional atomic-region supervision substantially improves extraction. Under the RD setting in Table 3, macro F1 at IoU 0.5 increases from 77.7 to 91.4. This is the clearest empirical support for the value of the additional annotations.

Critical weaknesses
  1. The benchmark’s differentiation from recent scientific QA work is insufficiently established.
  • SPUR (ACL 2026) evaluates fine-grained experimental images and cross-panel reasoning; SORBE (ICML 2026) constructs evidence-linked, multi-image reasoning tasks. And there are more related papers. Their settings differ from OmniScience++, which combines figure-only input, atomic regions, condition constraints, and reference programs. However, the paper does not sufficiently demonstrate what this combination adds to scientific figure evaluation.
  • Table 1 lists feature coverage, while the experiments provide little evidence of an additional capability or failure pattern that would establish the benchmark’s distinctive value.
  1. The experiments are insufficient to establish the benefit of the proposed QA construction.
  • Table 3 supports the extraction annotations, and oracle crops test the effect of supplying relevant regions. Neither comparison isolates the benefit of selecting programs before generating questions or enforcing experimental compatibility.
  • Tables 4–5 also aggregate results into three content categories without program-family or Basic/Advanced breakdowns. Appendix F acknowledges the limits of this analysis, but the reported results still leave unclear what the construction reveals about condition matching and compositional reasoning.
  1. The reliability of the final QA references is insufficiently documented.
  • Section 4.3 and Appendix A.3 report program checks, separate model judging, visual inspection of difficult items, and human review. These are useful safeguards. However, the paper does not sufficiently document reviewer expertise, how disagreements were resolved, or the errors remaining in observation–condition links, programs, and answers after review.
Questions for authors
  1. What specific capability or failure pattern does OmniScience++ reveal beyond the closest settings in SPUR, SORBE, and SciEGQA? What evidence supports that distinction after accounting for differences in input and answer format?

  2. What controlled comparison supports the benefit of program-first sampling or experimental-compatibility constraints? Can you also report the existing results by program family and Basic/Advanced level, with retained, refused, and unresolved counts per model, and check whether the main patterns persist on a common question set?

5 comments

Thank you for the careful review. Our responses to the weaknesses and questions are below.

W1/Q1:

Thank you for raising how our benchmark differs from recent scientific QA work. The main differences are:

  • vs. SPUR (multiple choice, accuracy only): we also require the evidence boxes in the figure.
  • vs. SORBE (pathology images, mainly qualitative questions, LLM-judged text reasoning): we also cover charts and quantitative questions, and score evidence boxes directly by IoU.
  • vs. SciEGQA (evidence is a whole paragraph, table or figure): we localize evidence down to the subfigures inside a figure.

Our joint score checks whether a model can support a correct answer with the right evidence. Verifiable evidence is what MLLMs need to help scientific literature understanding and research. On the same 600 questions and format, it reveals what these benchmarks cannot:

  • Gemini-3.8-Flash and GPT-6-Sol differ by 2.5 points in accuracy but by 32.7 in joint score, because 60% of Gemini's correct answers lack complete evidence. Accuracy alone could not tell them apart, as in SPUR.
  • Qwen3.8-Max answers 68.67% of the questions correctly, but 85.2% of those answers lack complete evidence. A text-based reasoning score like SORBE's could not check this.
  • Even with the whole figure visible, ground-truth subfigure crops raise Qwen3.8-Max by 18.67 points. Whole figure evidence in SciEGQA could not expose this bottleneck.

We will add these dimensions to Table 1.

Besides, OmniScience++ provides 10.02M subfigures from 1.52M figures across multiple disciplines, and a low-cost pipeline for building the resource in batches.

Table R1: Main results on all 600 questions (refusals and invalid outputs scored as incorrect)

modelJoint (answer × [email protected])AccuracyEvid. [email protected]Evid. [email protected]Evid. [email protected]refusalinvalid output
gpt-6-sol68.328279.8371.149.2800
gemini-3.8-flash35.6584.538.6834.3222.1171
claude-opus-5-546.6673.1750.0731.0910.431041
gpt-4o3.8435.178.870.25010
Kimi-k344.9975.555.3929.986.500
qwen3.8-max10.6168.6713.469.792.3630
GLM-5.3-flash34.3670.1745.2717.312.5209
DeepSeek-V4.1-Flash22.8867.1731.275.710.0810

Table R2: Answer × evidence on all 600 questions per model (Recall@IoU=0.5=1)

ModelAnswer ✓ / evidence ✓Answer ✓ / evidence ✗Answer ✗ / evidence ✓Answer ✗ / evidence ✗Correct answers with incomplete evidence
GPT-6-Sol39993674118.9%
Gemini-3.8-Flash203304177660.0%
Claude-Opus-5.52731662014137.8%
GPT-4o221892236789.6%
Kimi-K3260193588942.6%
Qwen3.8-Max613511717185.2%
GLM-5.3-Flash2062156111851.1%
DeepSeek-V4.1-Flash1272764814968.5%

Table R3: Oracle-evidence comparison (accuracy with ground-truth evidence crops; Δ relative to the same model without them)

ModelCategoryOracle acc.Δ acc. (pp)
Claude-Opus-5.5All74.331.17
Claude-Opus-5.5Experimental relations691
Claude-Opus-5.5Quantitative analysis83.5-1.5
Claude-Opus-5.5Visual structure70.54
GPT-4oAll49.8314.67
GPT-4oExperimental relations5622
GPT-4oQuantitative analysis5914
GPT-4oVisual structure34.58
Kimi-K3All78.332.83
Kimi-K3Experimental relations89.54
Kimi-K3Quantitative analysis860.5
Kimi-K3Visual structure59.54
Qwen3.8-MaxAll87.3318.67
Qwen3.8-MaxExperimental relations9322
Qwen3.8-MaxQuantitative analysis9014.5
Qwen3.8-MaxVisual structure7919.5

W2/Q2

Thank you for the suggestions on our experiments. To compare our method with direct QA generation, we ran the following experiment, blind-reviewed by two researchers, with a third reviewer resolving conflicts.

We added a generation comparison with six configurations on the same 50 figures. Each configuration has 50 medium and 50 hard QA pairs.

  • A: the figure, context and caption are given directly to the LLM, which generates the QA, the evidence and the reference reasoning.
  • B: in addition to A's inputs, the evidence and sub-figures are given to the LLM, which generates directly.
  • C: generation by the OmniScience++ data engine.

The usability is defined as that question is answerable, answer is correct, difficulty is met, and boxes correctly cover the necessary evidence.

Table R4: Usability over 100 pre-assigned slots per configuration

GeneratorA: direct LLM generationB: evidence-assisted joint generationC: OmniScience++
GPT-6 Luna (medium)12%69%85%
GPT-6 Sol (high)41%86%96%

The results show that our evidence helps the LLM localize evidence and construct the reasoning better, and our complete pipeline achieves the best generation quality. In addition, with a lower-cost model, our pipeline reaches results comparable to group B, suggesting that our pipeline could be faster and cost less with a lighter-weight LLM.

For the detailed model performance, we updated the tables below. Refusals and invalid outputs are scored as incorrect, and all models are now evaluated on all 600 benchmark questions. We will add the updated tables and information to the revised paper.

Table R1: as in our response to W1.

Table R5: results by Basic/Advanced level, question category.

Table R5a: By question category

ModelCategoryAcc.[email protected][email protected][email protected]Joint
GPT-6-SolExperimental relations90.585.7277.3149.1279.57
GPT-6-SolQuantitative analysis9493.4581.4557.788.62
GPT-6-SolVisual structure61.560.3254.5341.0236.77
Gemini-3.8-FlashExperimental relations9545.0340.7820.3143.98
Gemini-3.8-FlashQuantitative analysis9253.3345.8331.8351.33
Gemini-3.8-FlashVisual structure66.517.6816.3514.1811.65
Claude-Opus-5.5Experimental relations6850.3227.488.5148.22
Claude-Opus-5.5Quantitative analysis8569.0743.571366.4
Claude-Opus-5.5Visual structure66.530.8222.229.7825.37
GPT-4oExperimental relations348.180.7503.92
GPT-4oQuantitative analysis4512.87005.7
GPT-4oVisual structure26.55.57001.92
Kimi-K3Experimental relations85.560.0632.176.5853.53
Kimi-K3Quantitative analysis85.575.3741.337.1763.2
Kimi-K3Visual structure55.530.7316.455.7518.25
Qwen3.8-MaxExperimental relations7111.598.942.599.92
Qwen3.8-MaxQuantitative analysis75.521.514.833.517
Qwen3.8-MaxVisual structure59.57.35.614.9
GLM-5.3-FlashExperimental relations83.551.98203.3744.73
GLM-5.3-FlashQuantitative analysis7348.5819.581.8337.42
GLM-5.3-FlashVisual structure5435.2512.352.3720.93
DeepSeek-V4.1-FlashExperimental relations8029.735.790.2525.62
DeepSeek-V4.1-FlashQuantitative analysis72.545.337.5033.33
DeepSeek-V4.1-FlashVisual structure4918.743.8309.7

Comments on the paper

4 comments

Thank you for the detailed review. Our responses to each weakness are below.

W1

Thank you for the suggestions on reporting results. Refusals and invalid answers are counted as incorrect. For disagreements, a third human reviewer will decide independently.

Table R1: Main results on all 600 questions

modelJoint (answer × [email protected])AccuracyEvid. [email protected]Evid. [email protected]Evid. [email protected]refusalinvalid output
gpt-6-sol68.328279.8371.149.2800
gemini-3.8-flash35.6584.538.6834.3222.1171
claude-opus-5-546.6673.1750.0731.0910.431041
gpt-4o3.8435.178.870.25010
Kimi-k344.9975.555.3929.986.500
qwen3.8-max10.6168.6713.469.792.3630
GLM-5.3-flash34.3670.1745.2717.312.5209
DeepSeek-V4.1-Flash22.8867.1731.275.710.0810

Table R2: Results by question category

ModelCategoryAcc.[email protected][email protected][email protected]Joint
GPT-6-SolExperimental relations90.585.7277.3149.1279.57
GPT-6-SolQuantitative analysis9493.4581.4557.788.62
GPT-6-SolVisual structure61.560.3254.5341.0236.77
Gemini-3.8-FlashExperimental relations9545.0340.7820.3143.98
Gemini-3.8-FlashQuantitative analysis9253.3345.8331.8351.33
Gemini-3.8-FlashVisual structure66.517.6816.3514.1811.65
Claude-Opus-5.5Experimental relations6850.3227.488.5148.22
Claude-Opus-5.5Quantitative analysis8569.0743.571366.4
Claude-Opus-5.5Visual structure66.530.8222.229.7825.37
GPT-4oExperimental relations348.180.7503.92
GPT-4oQuantitative analysis4512.87005.7
GPT-4oVisual structure26.55.57001.92
Kimi-K3Experimental relations85.560.0632.176.5853.53
Kimi-K3Quantitative analysis85.575.3741.337.1763.2
Kimi-K3Visual structure55.530.7316.455.7518.25
Qwen3.8-MaxExperimental relations7111.598.942.599.92
Qwen3.8-MaxQuantitative analysis75.521.514.833.517
Qwen3.8-MaxVisual structure59.57.35.614.9
GLM-5.3-FlashExperimental relations83.551.98203.3744.73
GLM-5.3-FlashQuantitative analysis7348.5819.581.8337.42
GLM-5.3-FlashVisual structure5435.2512.352.3720.93
DeepSeek-V4.1-FlashExperimental relations8029.735.790.2525.62
DeepSeek-V4.1-FlashQuantitative analysis72.545.337.5033.33
DeepSeek-V4.1-FlashVisual structure4918.743.8309.7

Table R3: Results by basic/advanced

ModelLevelAcc.[email protected][email protected][email protected]Joint
GPT-6-SolBasic8482.4274.0351.6971.33
GPT-6-SolAdvanced8077.2468.1746.8765.3
Gemini-3.8-FlashBasic8542.9838.5922.9638.96
Gemini-3.8-FlashAdvanced8434.3930.0521.2632.35
Claude-Opus-5.5Basic7556.1334.413.0652.76
Claude-Opus-5.5Advanced71.334427.777.8140.57
GPT-4oBasic34.3312.340.1704.86
GPT-4oAdvanced365.40.3302.83
Kimi-K3Basic7661.3435.338.1148.54
Kimi-K3Advanced7549.4324.644.8941.44
Qwen3.8-MaxBasic70.3315.4310.71312.93
Qwen3.8-MaxAdvanced6711.498.871.738.28
GLM-5.3-FlashBasic73.3347.0117.092.6736.7
GLM-5.3-FlashAdvanced6743.5317.532.3832.02
DeepSeek-V4.1-FlashBasic6834.796.170.1723.36
DeepSeek-V4.1-FlashAdvanced66.3327.755.25022.41

W2

Thank you for the suggestions on our experiments. To compare our method with direct QA generation, we ran the following experiment, blind-reviewed by two researchers, with a third reviewer resolving conflicts.

We added a generation comparison with six configurations on the same 50 figures. Each configuration has 50 medium and 50 hard QA pairs.

  • A: the figure, context and caption are given directly to the LLM, which generates the QA, the evidence and the reference reasoning.
  • B: in addition to A's inputs, the evidence and sub-figures are given to the LLM, which generates directly.
  • C: generation by the OmniScience++ data engine.

The usability is defined as that question is answerable, answer is correct, difficulty is met, and boxes correctly cover the necessary evidence.

Table R4: Usability over 100 pre-assigned slots per configuration

GeneratorA: direct LLM generationB: evidence-assisted joint generationC: OmniScience++
GPT-6 Luna (medium)12%69%85%
GPT-6 Sol (high)41%86%96%

The results show that our evidence helps the LLM localize evidence and construct the reasoning better, and our complete pipeline achieves the best generation quality. In addition, with a lower-cost model, our pipeline reaches results comparable to group B, suggesting that our pipeline could be faster and cost less with a lighter-weight LLM.

W3

Thank you for the suggestions on the human review. Our human reviews were carried out by two Master/PhD students in related STEM fields; when they disagreed, a third independent researcher reviewed it. All benchmark-related checks hid non-essential information such as the answering model. For Eval-1K, we manually checked every figure and its sub-figure results and replaced figures with ambiguity.

W4

Thank you for the suggestions on presented the joint performance. The joint score checks whether a model can back a correct answer with the right evidence in the figure. Verifiable evidence is what MLLMs need to support scientific literature understanding and research. Our results show that accuracy alone hides this. Gemini-3.8-Flash and GPT-6-Sol differ by only 2.5 points in accuracy but by 32.7 in joint score, because 60% of Gemini's correct answers lack complete evidence. Qwen3.8-Max answers 68.67% of the questions correctly, but 85.2% of those answers lack complete evidence. Finding the right subfigure is itself a bottleneck that giving the ground-truth subfigure crops raises Qwen3.8-Max's accuracy by 18.67 points (Table R5).

Table R5: Oracle-evidence comparison

ModelCategoryOracle acc.Δ acc. (pp)
Claude-Opus-5.5All74.331.17
Claude-Opus-5.5Experimental relations691
Claude-Opus-5.5Quantitative analysis83.5-1.5
Claude-Opus-5.5Visual structure70.54
GPT-4oAll49.8314.67
GPT-4oExperimental relations5622
GPT-4oQuantitative analysis5914
GPT-4oVisual structure34.58
Kimi-K3All78.332.83
Kimi-K3Experimental relations89.54
Kimi-K3Quantitative analysis860.5
Kimi-K3Visual structure59.54
Qwen3.8-MaxAll87.3318.67
Qwen3.8-MaxExperimental relations9322
Qwen3.8-MaxQuantitative analysis9014.5
Qwen3.8-MaxVisual structure7919.5

W5

Thank you for the suggestion on the Eval-1K setup. The original split keeps Eval-1K separate from the second-stage refinement data. The first stage used coarser panel-level supervision and did share source figures with Eval-1K. To rule out this factor, we additionally trained YOLO and AS-YOLO with both stages fully separated from Eval-1K and reran the evaluation. After the update, every AS-YOLO detection metric changes by less than 1 percentage point, and the best configuration for each metric is unchanged, which further supports the robustness of the main conclusions of the paper. Updated tables are in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.