- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ScienceDoc is an evidence-grounded scientific document QA benchmark with 2,200 questions over 693 documents, covering eight disciplines and four tasks: General, Unanswerable, Reasoning, and Multi-Document QA. Experts design initial questions and annotate answers and evidence pages, with model-assisted refinement and at least two human checks per item. Evaluation of 11 MLLMs reports answer accuracy and page-level evidence precision, recall and F1, highlighting multi-document difficulty, uneven abstention, and answer–evidence mismatches.
Critical strengths
-
Useful coverage and task design. The benchmark extends scientific document QA beyond predominantly CS/NLP settings, with eight disciplines, multi-document integration, and explicit unanswerable questions. Joint answer/evidence reporting can reveal limitations that aggregate answer accuracy misses (Sections 3–5).
-
Substantial annotation effort. Expert-defined questions, at least two independent human checks per item, and fact/source-necessity checks provide a stronger construction protocol than unverified synthetic QA. The detailed audit workflow and qualitative cases make the intended evaluation concrete (Section 3.3; Appendices A–D).
Critical weaknesses
-
Evaluation reliability needs stronger validation. Despite independent human checks, inter-annotator agreement and judge–human agreement are not quantified. The paper acknowledges that alternative valid evidence pages may receive lower scores, but does not measure the impact on scores or model rankings.
-
Positioning against close prior work remains incomplete. M3SciQA already evaluates multimodal, multi-document scientific QA. Although cited, it is absent from Table 1. ScienceDoc’s broader disciplinary coverage is valuable, but its additional diagnostic contribution needs clearer comparison with this close predecessor. (M3SciQA)
-
The reported input configurations remain unclear. Appendix C.4 lists full, oracle, and qa_field page-selection policies without identifying which produced the main results. Per-model context/image limits, output budgets, and resource-failure rates also need reporting to distinguish capability limitations from input and deployment constraints.
Questions for authors
-
Please report expert agreement, judge-human calibration, and sensitivity to alternative valid evidence sets. How do these affect benchmark scores and model rankings?
-
What distinct capabilities or failure modes does ScienceDoc reveal beyond M3SciQA? Please include this close comparator in Table 1 and clarify the additional contribution.
-
What page-input policies and per-model resource limits were used? Would the main findings persist with a stronger frontier model, such as Opus 5.5? A tool-augmented baseline, such as Codex or Claude Code, would also be informative if evaluated separately with explicit tools and budgets.
Thank you for the additional experiments, particularly the human validation of the LLM judge, which substantially addresses my concern about evaluation reliability. However, the significance tests for selected model pairs within two disciplines do not sufficiently resolve my concern about the statistical reliability of the broader domain-specific conclusions, and uncertainty in these comparisons remains insufficiently characterized. Given the emphasis placed on disciplinary differences in the paper, I maintain my weak-reject recommendation.