acceptodds
Under review as a conference paper at ICLR 2027

ScienceDoc: Multidisciplinary Multimodal Evidence-Grounded Question Answering for Scientific Documents

Abstract

Multimodal large language models (MLLMs) are increasingly applied to document understanding and scientific question answering (QA). However, current evaluation remains limited by narrow disciplinary coverage, insufficient assessment of multimodal evidence grounding, and limited evaluation of cross-page and multi-document reasoning. We introduce ScienceDoc, an evidence-grounded multimodal question answering benchmark built from 693 scientific documents, containing 2,200 questions across 8 major disciplines and 61 scientific fields. ScienceDoc includes 4 task types, namely General, Unanswerable, Reasoning, and Multi-Document, together with 21 discipline-specific question types, covering multimodal content from text, equations, tables, and figures. All initial questions are designed by human experts, with expert-annotated answers and manually labeled evidence pages, supporting high-quality hierarchical evaluation of answer correctness and evidence grounding. Experiments on 11 representative MLLMs reveal challenges in multi-document QA and large performance variation on unanswerable questions. The study findings also indicate significant differences across scientific disciplines and gaps between answer accuracy and evidence localization, suggesting that conventional answer accuracy alone is insufficient for evaluating trustworthy scientific document question answering.

Primary area:
datasets and benchmarks
Keywords:
Scientific Document Understanding, Benchmark Dataset, Multimodal Large Language Model, Trustworthy Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HFLjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ScienceDoc is an evidence-grounded scientific document QA benchmark with 2,200 questions over 693 documents, covering eight disciplines and four tasks: General, Unanswerable, Reasoning, and Multi-Document QA. Experts design initial questions and annotate answers and evidence pages, with model-assisted refinement and at least two human checks per item. Evaluation of 11 MLLMs reports answer accuracy and page-level evidence precision, recall and F1, highlighting multi-document difficulty, uneven abstention, and answer–evidence mismatches.

Critical strengths
  1. Useful coverage and task design. The benchmark extends scientific document QA beyond predominantly CS/NLP settings, with eight disciplines, multi-document integration, and explicit unanswerable questions. Joint answer/evidence reporting can reveal limitations that aggregate answer accuracy misses (Sections 3–5).

  2. Substantial annotation effort. Expert-defined questions, at least two independent human checks per item, and fact/source-necessity checks provide a stronger construction protocol than unverified synthetic QA. The detailed audit workflow and qualitative cases make the intended evaluation concrete (Section 3.3; Appendices A–D).

Critical weaknesses
  1. Evaluation reliability needs stronger validation. Despite independent human checks, inter-annotator agreement and judge–human agreement are not quantified. The paper acknowledges that alternative valid evidence pages may receive lower scores, but does not measure the impact on scores or model rankings.

  2. Positioning against close prior work remains incomplete. M3SciQA already evaluates multimodal, multi-document scientific QA. Although cited, it is absent from Table 1. ScienceDoc’s broader disciplinary coverage is valuable, but its additional diagnostic contribution needs clearer comparison with this close predecessor. (M3SciQA)

  3. The reported input configurations remain unclear. Appendix C.4 lists full, oracle, and qa_field page-selection policies without identifying which produced the main results. Per-model context/image limits, output budgets, and resource-failure rates also need reporting to distinguish capability limitations from input and deployment constraints.

Questions for authors
  1. Please report expert agreement, judge-human calibration, and sensitivity to alternative valid evidence sets. How do these affect benchmark scores and model rankings?

  2. What distinct capabilities or failure modes does ScienceDoc reveal beyond M3SciQA? Please include this close comparator in Table 1 and clarify the additional contribution.

  3. What page-input policies and per-model resource limits were used? Would the main findings persist with a stronger frontier model, such as Opus 5.5? A tool-augmented baseline, such as Codex or Claude Code, would also be informative if evaluated separately with explicit tools and budgets.

Reviewer_FC8sOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ScienceDoc, an evidence-grounded, multidisciplinary, and multimodal question answering (QA) benchmark for scientific documents. The dataset consists of 2,200 QA pairs derived from 693 source documents, spanning 8 major disciplines and 61 fine-grained scientific fields. The benchmark is uniquely structured to include 4 task types: General, Reasoning, Multi-Document, and Unanswerable questions. A key feature of ScienceDoc is its hierarchical evaluation, which assesses both the semantic correctness of the answer and the model's ability to precisely locate the supporting evidence pages. The authors evaluate 11 representative Multimodal Large Language Models (MLLMs), finding that while models can achieve high answer accuracy, their evidence localization (Evidence F1) often lags significantly behind, and tasks like multi-document integration and unanswerable question detection remain highly challenging.

Critical strengths
  1. Separation of Answer Accuracy and Evidence Grounding: The benchmark makes a highly consequential contribution by evaluating evidence localization (E-Precision, E-Recall, E-F1) alongside standard answer accuracy. The qualitative case studies brilliantly illustrate why this is necessary: for instance, a model like Qwen3-VL-4B can generate a semantically correct answer while indiscriminately citing all 51 pages of a document. This proves that relying solely on answer accuracy is insufficient for evaluating true document comprehension in scientific QA.
  2. Robust Task Diversity Exposing MLLM Flaws: The inclusion of Multi-Document and Unanswerable tasks exposes critical weaknesses in state-of-the-art models. The finding that accuracy on unanswerable questions varies wildly from 2.50% (InternVL3.5-8B) to 91.00% (Claude Sonnet 5) provides an excellent, rigorous test of hallucination resistance and a model's ability to recognize insufficient evidence.
Critical weaknesses
  1. Misalignment Between Claimed Diversity and Actual Dataset Scale: The paper strongly promotes its broad coverage of 8 disciplines and 61 scientific fields. However, the total dataset size is only 2,200 questions. This results in a very sparse distribution of questions across these fine-grained fields. For example, the entire Quantitative Finance domain contains only 86 questions, and Quantitative Biology contains only 114. Drawing definitive conclusions about a model's domain-specific strengths and weaknesses (as attempted in Table 3 and Figure 5b) is statistically fragile with such small sample sizes per field. 2 . Unvalidated Reliability of LLM-as-a-Judge for Niche Science: The semantic correctness of the free-form answers is evaluated using an LLM-as-a-Judge approach (prompting a model to output CORRECT/INCORRECT). Given the highly specialized, multidisciplinary nature of the dataset—which includes complex equations, physics thresholds, and quantitative biology metrics—relying on an LLM to accurately judge semantic equivalence without a reported human-agreement validation study is a significant methodological risk. False positives or false negatives by the judge model could easily skew the benchmark results.
Questions for authors
  1. Given the sparse distribution of questions in certain disciplines (e.g., only 86 questions for Quantitative Finance), how statistically significant are the domain-specific performance variations reported in Table 3 and Figure 5b? Would you consider aggregating some of these fields or adding confidence intervals to the domain-specific breakdown to prevent over-interpretation of the results?
    2.How did you validate the reliability of the LLM-as-a-Judge? Did you conduct a human-annotator agreement study on a subset of the judge's outputs to ensure it accurately assesses semantic equivalence for highly complex, mathematically heavy, or discipline-specific answers? If so, what was the agreement rate?
  2. For the multi-document task, do the concatenated PDFs retain the formatting and metadata of the original separate PDFs in a way that might provide artificial cues to the models regarding document boundaries?
1 comment

Thank you for the additional experiments, particularly the human validation of the LLM judge, which substantially addresses my concern about evaluation reliability. However, the significance tests for selected model pairs within two disciplines do not sufficiently resolve my concern about the statistical reliability of the broader domain-specific conclusions, and uncertainty in these comparisons remains insufficiently characterized. Given the emphasis placed on disciplinary differences in the paper, I maintain my weak-reject recommendation.

Reviewer_pgLaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ScienceDoc evaluates question answering over scientific documents using 2,200 questions from 693 documents across eight disciplines. It includes General, Unanswerable, Reasoning, and Multi-Document tasks, with reference answers and page-level evidence annotations. The evaluation provides PDF pages as images to 11 multimodal models and scores answer correctness and evidence selection separately. The results show variation across tasks and disciplines, particularly in abstention and evidence localization.

Critical strengths

S1. The dataset covers eight disciplines and includes questions about both single and multiple documents, providing material for comparing model performance across scientific documents. The answer and evidence-page annotations allow subsequent studies to compare answering and evidence selection on the same questions without reconstructing these annotations.

S2. The paper checks both whether a model answers correctly and whether its cited pages match the annotations. This reveals problems that overall accuracy can hide: a model can answer correctly but cite the wrong pages, or perform well on answerable questions but struggle to abstain when information is missing. These distinctions make the evaluation more informative.

S3. The paper is well written. The presentation is clear and easy to follow.

Critical weaknesses

W1. The paper does not clearly explain what this benchmark measures beyond existing work. The introduction emphasizes disciplinary differences, evidence support, and reasoning across documents, then interprets the results as weaknesses in these capabilities (pp. 1–2). The analysis mainly reports score differences across models, tasks, and disciplines, without showing which errors arise from scientific content and which reflect problems common to long-document QA. Table 1 lists benchmark features but does not compare the cited M3SciQA, which appears closely related from the description in this paper. Readers therefore see a new set of evaluation results, but it remains unclear what they learn beyond existing scientific-document QA benchmarks.

W2. The experiments do not show where the models struggle. The authors render PDF pages as images, so models must read the pages and locate information before answering. The resulting scores reflect both the input format and the demands of the questions. I could not find results comparing this setup with structured text and tables alongside retained figures, or with retrieval followed by answering over the same documents. Appendix C.4 lists full, oracle, and qa-field page-selection modes, but does not specify the settings used for each model in the main table or clearly report performance with only gold evidence pages. Without these comparisons, it is difficult to tell whether models struggle to answer the scientific questions or to read and locate the information in the supplied input.

W3. The evidence-page evaluation has a related problem. The pages a model lists at the end do not establish which pages it used to answer: if other pages were available, the answer could draw on information that was not cited. The current page-level metrics check agreement with annotations, but do not test whether the citations provide enough evidence for the answer. If the model answers again in a fresh context containing only its cited pages, does it retain the original accuracy? This would help assess whether the declared evidence is sufficient. Even unchanged accuracy would not establish that the first answer actually relied on those pages.

Questions for authors

Q1. What specific capability does ScienceDoc add to the evaluation provided by existing scientific-document QA benchmarks? Please clarify the distinction from M3SciQA and identify the results that support the interpretation of the observed errors as limitations in scientific understanding or reasoning.

Q2. For each model in the main table, which page-selection policy, page limit, resizing or tiling settings, and overflow handling were used? Were all selected images supplied in one request? Are there results using structured text and tables with retained figures, or retrieval over the same documents? Such comparisons would help determine how strongly the conclusions depend on the chosen input pipeline; their document coverage, parsing losses, and computational budgets should be specified.

Q3. Was the listed oracle policy evaluated? For the same answerable questions, what are the answer accuracies with full-document inputs and gold-evidence-page inputs, broken down by task and model? A large gap would indicate a substantial contribution from evidence selection or input burden. Remaining errors would still require separating within-page perception from understanding and reasoning. Unanswerable questions should be treated separately because providing empty gold evidence changes the task.

1 comment
Reviewer_pgLaOpenReview ↗

Follow-up on Authors' Response

Thank you for the detailed response and additional experiments. However, my core concerns remain unresolved:

  1. New insights over M3SciQA: Broader disciplinary coverage and more task types do not automatically yield new insights. The response still does not clearly show what specific conclusions arise from multidisciplinary complexity that cannot already be observed in NLP-focused benchmarks like M3SciQA.

  2. Page-level evidence contribution: In the Gold Evidence Pages experiment, providing gold pages brings clear gains only on Multi-Document QA, while offering no improvement (and even drops for several models) on single-document General and Reasoning tasks. This does not fully support the claimed contribution of page-level evidence annotations.

Therefore, I will maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.