- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces OmniScience++, an extension of OmniScience containing 10,021,541 atomic-subfigure records derived from approximately 1.52 million scientific figures. The resource links localized regions to source figures and paper metadata, with visual types, semantic tags and generated local captions. Its construction pipeline combines a task-adapted YOLO26x detector with recursive decomposition, semantic recursion gating, residual-region recovery and semantic annotation.
An evidence-first engine selects compatible observations and an intended reasoning program before generating questions, answers and region-linked reference steps. Evaluation comprises Eval-1K, with 1,000 figures and 12,785 reference boxes; an automated annotation audit; and a separate 600-question human-reviewed diagnostic benchmark. Eight multimodal models are evaluated for answer correctness and evidence localization and four receive reference evidence crops in an oracle condition. The paper reports extraction improvements, different answer and grounding rankings and increased overall answer accuracy with oracle crops. Its contributions concern resource construction and diagnostic evaluation, rather than a new answering model or a demonstrated training improvement (Sections 3–6).
Critical strengths
-
Source-linked resource and inspectable construction records. Preserving source coordinates, paper associations, observations and intended operations is a useful design choice. Selecting evidence and a program before verbalization makes the intended construction inspectable rather than leaving it entirely implicit in a generated rationale. The scale and provenance structure give the resource potential reuse value, although they do not establish annotation correctness (Sections 3–4).
-
Concrete extraction controls and complementary evaluation outputs. The paper includes a panel-trained detector baseline, task-specific detector refinement and SRG/LRR comparisons at multiple IoU thresholds. Eval-1K excludes the detector-refinement figures. Reporting answer accuracy separately from grounding, together with a same-model crop intervention, provides more information than answer accuracy alone. These are useful evaluation elements even though their scope is limited (Sections 5–6; Table 3).
Critical weaknesses
- The reliability and scope of the evidence supporting the resource and benchmark are insufficiently established.
Evaluation references and coverage. Eval-1K is described as sampled from figures excluded from detector refinement and evaluated against manually calibrated boxes. However, the manuscript does not sufficiently document the figure-selection procedure, annotation instructions for atomic boundaries, independent checking or disagreement resolution, or Eval-1K-specific coverage of small, densely annotated and visually complex regions (Section 6.1; Appendix A.2). Creating an evaluation set is not itself problematic. The issue is that the disclosed protocol does not allow a reliable assessment of reference consistency, selection effects, or the scope of generalization. Full-corpus composition statistics do not establish the composition of Eval-1K.
Scientific validity beyond box overlap. Table 3 measures agreement with reference boxes, not whether the extracted evidence preserves, or reliably links to, the labels, units, legends, scale bars and experimental conditions needed for a question. The source figure and context may mitigate crop omissions but their effectiveness is not quantified. Appendix B.2 appropriately considers boundary quality and factual reliability; however, its Qwen/GPT audit is explicitly not calibrated against human judgments. Schema consistency and numerical program checks also cannot certify the underlying extracted observations. The frequency of scientifically consequential annotation errors and their impact on downstream QA are not independently quantified by the reported evaluation. This is not a claim that a particular fraction of the 10.02M records is wrong.
QA selection and score comparability. The 600 questions are explicitly a curated diagnostic set, not a random audit of the corpus. Nevertheless, the candidate-pool size, rejection and correction counts and selection procedure are insufficiently reported (Sections 4.3–5.1; Appendix C.1). Human review may make the retained examples valid but it does not establish typical automated-generation quality or coverage of difficult cases. Additionally, Table 4 excludes refusals and unresolved answers separately for each model. The resulting conditional scores are not comparisons on an identical item set and the paper does not report enough retained-item counts or common-set analysis to assess sensitivity of the rankings (Section 5.2; Appendix C.2).
Statistical uncertainty and run-to-run stability. Tables 3–5 report point estimates without confidence intervals. Appendix A.2 specifies seed 0 but reports no multi-seed detector-training analysis. No repeated-run variability analysis is reported for the generative stages. Appendix C.4 leaves statistical uncertainty unassessed and Appendix F makes no significance claim. Thus, the stability of small extraction differences and oracle gains remains uncertain. Paired intervals accounting for shared source figures or papers and repeatability checks for trainable or stochastic components would help. Repeating identical deterministic inference would not. Missing uncertainty analysis compounds the existing validation gaps; it does not establish that the observed gains are insignificant.
Numerical consistency and stated limitations. Table 5 reports a −0.81 percentage-point quantitative oracle change for Claude, whereas Table 10 reports a +25.1% relative error reduction. Under the stated transformation, these cannot describe the same comparison and need reconciliation. Relevant caveats are acknowledged in Section 6.1 and Appendices B, C and F but Section 7 discusses mainly open-ended task coverage. The AI Use Statement also refers annotation-validation limitations to Section 7 without a corresponding discussion there. The implications of these limitations for the central contributions should be more clearly reflected in Section 7 and the conclusions.
Why this affects my recommendation. These concerns affect the validity and interpretation of the principal resource and benchmark evidence, not just presentation. Addressing them may require representative independent checks and additional analysis or rescoring, rather than merely more cautious wording. I do not infer deliberate selection bias or invalid data from the missing documentation.
- The incremental scientific contribution of the construction methodology is not convincingly demonstrated.
Differentiation from prior approaches. Structured visual QA generation is established by work such as GQA (Hudson and Manning, CVPR 2019); MultiChartQA (Zhu et al., NAACL 2025) studies comparative and sequential multi-chart reasoning; and SciEGQA (Yu et al., arXiv:2511.15090v2) combines scientific evidence grounding with automated QA construction. These are related settings, not identical systems or directly interchangeable benchmark scores. OmniScience++ could contribute through condition-aware composition and its integration with a large atomic-region resource. My concern is not missing citations, the use of existing components, or the absence of a new fundamental algorithm. It is the limited evidence establishing what important capability or insight this particular integration adds.
Controls do not isolate the central design choices. Every extraction configuration in Table 3 includes RD, so the contribution of recursion over a single-pass version of the same AS-YOLO checkpoint is unknown. The largest reported gain comes from additional atomic-region supervision. SRG improves the reported extraction results, while LRR further reduces omissions with a small decrease in strict-IoU F1. These results establish a local extraction tradeoff, not its effect on scientific evidence completeness or QA validity. More importantly, no matched comparison isolates evidence-first/program-first generation from direct region-conditioned generation, or tests the incremental contribution of the experimental compatibility constraints. Equation 3 states eligibility conditions but does not specify the sampling distribution or selection procedure sufficiently to assess reproducibility and coverage.
Contribution-specific evaluations and metrics are missing. Table 1 documents feature availability rather than feature reliability, breadth or practical advantage. Tables 4–5 characterize models answering the constructed questions; they do not compare construction methods. The reported evaluation does not quantify independently verified valid-QA yield, joint answer–evidence success or performance by reasoning-program family. Appendix F explicitly states that category-level gains cannot be attributed to the proposed select-then-lookup family. I therefore find little new understanding of the claimed compositional capabilities in these aggregate analyses. A comparison on the same source figures with comparable information access, supervision, generation budget and verification would help establish whether a reasonable direct or structured-generation alternative produces equally valid questions and evidence. Other convincing contribution-specific validation could also address this concern; comparing scores from different published benchmarks would not.
Scope of the resource-cost evidence. API expenditure and partial token accounting are legitimate implementation details and their exclusions are disclosed in Tables 3 and 12. They do not establish end-to-end computational efficiency or valid QA yield per unit of resource. I do not regard the absence of a local-VLM comparison as an independent rejection reason.
Why this affects my recommendation. I recommend rejection. The potential utility of the resource does not outweigh the unresolved validation gaps, the limited evidence for incremental value and the unquantified stability of the central results. These concerns affect the principal contributions and call for substantive validation or analysis rather than minor editorial clarification.
Questions for authors
-
Eval-1K reference construction and coverage. How were the 1,000 figures selected and fixed relative to method development and how were atomic-region boundaries annotated and checked? Please clarify the instructions, annotator roles, handling of disagreements and whether references were drawn independently or corrected from model proposals. What existing evidence shows coverage and performance for small, densely annotated, or complex regions?
-
Semantic reliability and QA selection. What independent checks against the original figures establish scientifically usable regions, correct observations and valid evidence–answer links? How many QA candidates were generated, rejected, corrected and retained to obtain the 600 questions? Please distinguish curated-benchmark validity from the reliability and yield of the automated generation process.
-
Incremental benefit of the construction method. Are results available for the same AS-YOLO checkpoint without recursion and for a reasonable direct or structured QA-generation alternative on the same source figures with comparable information access, supervision, generation budget and verification? What evidence isolates the benefit of program-first selection and experimental compatibility constraints in valid question yield, evidence completeness, or compositional coverage?
-
Comparable and internally consistent benchmark results. Please provide retained/refused/unresolved counts by model and category and clarify whether the conclusions hold on a common, adjudicated item set. How are the −0.81 pp change in Table 5 and +25.1% error reduction in Table 10 for Claude’s quantitative category reconciled? Can the existing per-item records support the stated compositional interpretation beyond category marginals?
-
Statistical uncertainty and repeatability. Can the authors provide paired uncertainty estimates for the main extraction differences and oracle gains with shared-source dependence accounted for? What evidence is available on variability across detector training seeds or repeated stochastic generation? Please distinguish these robustness checks from fixed-seed reproducibility and operational retries.
2 comments
Q3
Thank you for the review on the experiments. As for the results not using recursion, we could not be able to obtain a decently good performance on subfigure level.
To compare our method with direct QA generation, we ran the following experiment, blind-reviewed by two researchers, with a third reviewer resolving conflicts.
We added a generation comparison with six configurations on the same 50 figures. Each configuration has 50 medium and 50 hard QA pairs.
- A: the figure, context and caption are given directly to the LLM, which generates the QA, the evidence and the reference reasoning.
- B: in addition to A's inputs, the evidence and sub-figures are given to the LLM, which generates directly.
- C: generation by the OmniScience++ data engine.
The usability is defined as that question is answerable, answer is correct, difficulty is met, and boxes correctly cover the necessary evidence.
Table R2: Usability over 100 pre-assigned slots per configuration
| Generator | A: direct LLM generation | B: evidence-assisted joint generation | C: OmniScience++ |
|---|---|---|---|
| GPT-6 Luna (medium) | 12% | 69% | 85% |
| GPT-6 Sol (high) | 41% | 86% | 96% |
The results show that our evidence helps the LLM localize evidence and construct the reasoning better, and our complete pipeline achieves the best generation quality. In addition, with a lower-cost model, our pipeline reaches results comparable to group B, suggesting that our pipeline could be faster and cost less with a lighter-weight LLM.
Q4.
Thank you for the suggestions on presenting the benchmark results. We now consider refusals and invalid answers as incorrect, and judge disagreements went through a second round of review. The updated results are below.
Table R3: Main results on all 600 questions
| model | Joint (answer × [email protected]) | Accuracy | Evid. [email protected] | Evid. [email protected] | Evid. [email protected] | refusal | invalid output |
|---|---|---|---|---|---|---|---|
| gpt-6-sol | 68.32 | 82 | 79.83 | 71.1 | 49.28 | 0 | 0 |
| gemini-3.8-flash | 35.65 | 84.5 | 38.68 | 34.32 | 22.11 | 7 | 1 |
| claude-opus-5-5 | 46.66 | 73.17 | 50.07 | 31.09 | 10.43 | 104 | 1 |
| gpt-4o | 3.84 | 35.17 | 8.87 | 0.25 | 0 | 1 | 0 |
| Kimi-k3 | 44.99 | 75.5 | 55.39 | 29.98 | 6.5 | 0 | 0 |
| qwen3.8-max | 10.61 | 68.67 | 13.46 | 9.79 | 2.36 | 3 | 0 |
| GLM-5.3-flash | 34.36 | 70.17 | 45.27 | 17.31 | 2.52 | 0 | 9 |
| DeepSeek-V4.1-Flash | 22.88 | 67.17 | 31.27 | 5.71 | 0.08 | 1 | 0 |
The inconsistency is because Table 5 used paired questions while Table 10 used separately subsets due to refusal and invalid outputs; the corrected results use the same 600 questions, with refusals and invalid answers counted as incorrect.
Updated Table 10:
| Model | Quantitative analysis | Visual structure | Experimental relations |
|---|---|---|---|
| Claude Opus 5.5 | −10.00% | 11.94% | 3.13% |
| GPT-4o | 25.45% | 10.88% | 33.33% |
| Qwen3.8-Max | 59.18% | 48.15% | 75.86% |
| Kimi-K3 | 3.45% | 8.99% | 27.59% |
Thank you for the constructive review. Our responses to each questions are below.
Q1.
Thank you for the suggestion. As for construction of Eval-1K, we first manually refined approximately 6,000 figures. For each figure, one human annotator corrected model proposals, and two reviewers independently checked the resulting annotations. Acceptance required approval from both reviewers; disagreements were resolved through discussion among all three. We then split these figures into Eval-1K and the second-stage detector training set. Eval-1K was selected to approximately balance scientific disciplines, prioritizing figures containing multiple visual types. After selection, the figures and annotations underwent an additional manual review, during which nine ambiguous figures were replaced.
Eval-1K contains 1,000 figures with 12,785 reference boxes. Of these boxes, 15.4% occupy less than 1% of their source figure’s area. Among the figures, 18.5% contain at least 20 reference boxes, 49.2% contain at least three visual types, and 7.6% satisfy our defined scale-heterogeneity criterion.
Q2.
Thank you for the suggestions on presenting the benchmark construction and the questions generated by the data engine. QA pairs with problems were revised manually, and regenerated questions were reviewed manually again before the final 600 questions were fixed.
Table R1: Question revision and replacement rates