acceptodds
Desk rejected at ICLR 2027

SciDocBench: A Workflow-Centered Benchmark and Data Pipeline for Scientific Document Understanding

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Scientific papers require models to integrate evidence across text, equations, figures, tables, code, and datasets while preserving its provenance. Existing benchmarks typically assess these capabilities separately, making it difficult to determine how well models support realistic scientific-reading workflows. We introduce SCIDOCBENCH, a workflow-centered benchmark containing 124 expert-authored and difficulty-screened questions across seven research-assistant capability groups, 19 subtasks, and five scientific domains. Each question is instantiated in four matched settings formed by pairing its bilingual variants with the All Images First and Markdown Interleaved document representations, yielding 496 evaluation instances. The strongest evaluated model, Claude-Opus-5, scores 62.6 out of 100, with remaining gaps in evidence localization, structured information extraction, cross-document synthesis, and robustness to document representation. To convert these diagnostics into scalable training signals, we introduce SCIDOCIR, a structured representation of scientific document objects, layout and cross-reference relations, and provenance. Using SCIDOCIR, we construct SCIDOCDATASET, which contains 4K supervised fine-tuning instances and 10K reinforcement-learning instances, built on 14 verifiable training subtasks. Posttraining Qwen3.6-27B on task-aligned data improves its SCIDOCBENCH score from 40.03 to 45.33 with supervised fine-tuning and to 45.74 with subsequent reinforcement learning. Both adapted models preserve DocVQA and InfoVQA performance and improve ChartQA accuracy over the original model by 0.80 and 3.40 points, respectively. Together, SCIDOCBENCH, SCIDOCIR, and SCIDOCDATASET form an evaluation-to-training framework that connects capability diagnosis with verifiable training-data construction for scientific-document assistants.

Primary area:
datasets and benchmarks
Keywords:
scientific document understanding, multimodal large language models, benchmarks and evaluation, evidence grounding, synthetic training data

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KaMzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SciDocBench, a workflow-centered benchmark with 124 expert-authored, difficulty-screened VQA for scientific document understanding. SciDocBench is built on verifiable deliverables rather than just answers. Additionally, the paper also introduces SciDocIR and SciDocDataset. Using these, post-trained Qwen3.6-27b achieves a better performance successfully.

Critical strengths
  1. The paper focuses on verifiable workflow outputs rather than answer correctness alone, which could help pinpoint exactly where the model performs well or poorly.
  2. The proposed benchmark is expected-validated to ensure the quality.
  3. Beyond identifying the problem using the benchmark, the paper also proposes a potential solution to alleviate the problem. The effectiveness of the solution is demonstrated in experiments.
Critical weaknesses
  1. The proposed benchmark appears to target LLMs rather than agents. In that case, the results in Table 2 do not seem particularly poor to me (For pure LLMs, getting 60+ on a new benchmark is actually quite high), and I believe the scores could be higher with a harness. If they are indeed higher under agentic evaluation, then the core issue becomes whether the identified problem in SciDocBench is actually a problem.
  2. The SciDocDataset is solely validated on Qwen3.6-27B. Will the gain still hold when post-training on other models?
Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Tool-Assisted Evaluation: Harness Gains, Answer Coverage, and Remaining Challenges

Thank you for asking whether tool access changes how we should read the scores in Table 2. A harness can make long documents easier to work with by keeping relevant passages and page images available across turns. We tested GPT-5.6-Sol with Codex and Claude Opus 4.8 with Claude Code to see how much this helps with tasks such as finding evidence for a claim, connecting findings across papers, and reconstructing the order of shuffled document fragments.

Table 2 reports direct scores of 61.00 for GPT-5.6-Sol and 53.41 for Claude Opus 4.8. Using the same 496 views and assigning zero to missing or unusable answers, we obtained 62.37 with Codex and 55.10 with Claude Code.

Among usable answers, Claude Opus 4.8 averaged 56.39 across 488/496 views in three archived regrades of the direct answers. Claude Code averaged 64.46 across 424/496 views. On the 417 views where both produced usable answers, the scores were 59.13 and 65.15, respectively (paired gain 6.02 points, 95% CI 1.33 to 10.44, resampling original questions). The two sets of answers were graded in separate batches. In a concurrent comparison on a fixed subset of 100 views, the 68 complete Claude pairs averaged 59.14 and 60.45 (gain 1.31, 95% CI −2.56 to 5.63). The concurrent Codex comparison yielded 61.77 and 62.98 on 488 complete pairs (gain 1.21, 95% CI −1.37 to 3.92). We will report coverage beside the scores. With 72 views missing usable Claude Code answers, scores for individual capabilities are sensitive to which views returned an answer.

On a table verification question, Claude Code cropped the relevant regions and recomputed the averages, yielding the correct comparison. On a citation question, it made eight crops and still omitted citations and changed some author names. We will include these cases in the revision to illustrate how tools helped the model inspect evidence and where assembling a complete answer remained difficult.

AuthorsOpenReview ↗

Extending Task-Aligned Adaptation to Ministral 3 8B

Thank you for suggesting that we test whether the training gains carry over to other model families. We had considered this comparison for the original study, but prioritized Qwen3.6-27B given the available time and the stability of our SFT and RL implementations for that model. We have now added experiments with Ministral 3 8B.

We chose Ministral 3 8B as a relatively recent open multimodal model with a 256K-token context window. Its long context was particularly useful for SciDocBench, where questions often require reading substantial scientific documents containing both text and figures. All 496 views fit within its context window while preserving the native image-text order and reserving 32,768 tokens for the answer.

We first evaluated the unmodified BF16 model and inspected its answers. The errors revealed difficulties in recovering the logical order of paper fragments, distinguishing the roles of cited work from the surrounding evidence, and following precise output requirements. This analysis guided our selection of public SciDocDataset training examples and construction of additional examples from separate source material.

We applied LoRA to the language model's query and output projections, using rank 8, alpha 16, dropout 0, AdamW with a learning rate of 5 × 10⁻⁵, and an effective batch size of 30 (microbatch 1, gradient accumulation 30). The BF16 base weights, vision encoder, and multimodal projector remained frozen.

With the same 32,768-token output limit and fixed VLMEvalKit evaluator, the score increased from 31.59 to 33.73 across all 496 views, a paired gain of 2.13 points (95% CI 0.35 to 3.94, resampling the 124 original questions). Two examples illustrate the improvements. In a Chinese paper-fragment ordering question, the baseline placed an experiment fragment before the abstract. The adapted model recovered the complete sequence of abstract, introduction, method, experiment, and conclusion, improving the rule score from 0.40 to 1.00. In an English citation-role question, it correctly distinguished work mentioned in related work from a method used for initialization and experimental comparison. Specifically, it changed LVSM's role from “Background, Comparison” to “Background” and VGGT's from “Background, Method” to “Method, Comparison,” improving the rule score from 0.40 to 0.80. It still assigned an extra role to another citation. These cases show progress in reconstructing document structure and connecting citations to their actual use in a paper, with evidence-based role assignment remaining a useful direction for further training.

Thank you for detailed responses. I will raise my score to 3 after taking the new results into consideration.

Reviewer_sQZQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

By combining Chinese/English languages with two document representations, the authors derive 496 evaluation instances, enriching the evaluation dimensions, and employ three types of graders for scoring. In addition to the benchmark, they contribute SciDocIR and SciDocDataset, and evaluate 13 closed- and open-source models, with results that validate the capability of the strongest models. They further post-train Qwen3.6-27B, demonstrating the dataset's value. Overall, this is a solid benchmark paper, though several concerns remain: the sample size is small relative to the large number of task categories, LLM-based grading may introduce bias, and the post-training analysis is not thorough enough. In particular, I wonder why Qwen3.6-27B was chosen instead of Qwen3.8-27B.

Critical strengths
  1. Although the original dataset contains only 116 papers and 124 questions, the authors expand the instances through different combinations and obtain evaluation results from corresponding perspectives. For most models, the difference between representations is larger than the difference between languages, and the two axes interact with or even reverse each other. The reliability of the review process is also audited.
  2. Through a structured intermediate representation and a verifiable task-construction mechanism, this work explicitly ensures that benchmark questions are not converted into training samples. In addition to the benchmark, the authors also contribute SciDocIR and SciDocDataset.
Critical weaknesses

However, the most significant issue still stems from the data size. As the authors acknowledge, the 496 instances are matched variants rather than independent questions; it is therefore questionable how much value the model profile over 19 sub-tasks really provides, and several of the analyses lack strong statistical support.

The post-training on Qwen3.6-27B is valuable, but the design of the training experiments is fairly simple, and no comparison is made against an equally sized general document-QA corpus. As a result, whether the observed model gains actually arise from the verifiable SciDocIR construction itself deserves deeper discussion. Meanwhile, the improvement of roughly 5 points over the 496 matched instances corresponds to only a handful of underlying questions and may not be statistically significant. Here I am also curious why Qwen3.8-27B was not used instead.

Although verifiability is presented as the key selling point, 66 of the 124 questions are graded by an LLM judge while only 1 is verified by execution, so concerns about self-bias remain unresolved. Finally, SciDocIR depends on layout parsing/OCR/LaTeX alignment, whose errors propagate into task generation and training data; this aspect should also be quantified further.

Questions for authors
  1. Since the benchmark contains only 124 underlying questions, please report the question count for each capability group and sub-task, and add bootstrap confidence intervals along with paired significance tests for (a) the ranking gaps on the leaderboard and (b) the language and representation gaps.
  2. For the post-training experiments, would it be possible to include a comparison against an equally sized general-purpose corpus, so as to better substantiate the value of the proposed data?
  3. The bias introduced by LLM-based grading should be taken into account; the authors could report human–machine agreement or employ judge models from different model families.
  4. Since SciDocIR depends on layout parsing/OCR/LaTeX alignment, can you quantify how often upstream parsing errors change a question's answerability or a training target's validity?
4 comments
AuthorsOpenReview ↗

Human Review and Reliability of Semantic Scoring

Thank you for asking us to clarify the limited use of execution-based verification and the reliability of semantic grading. The evaluation uses deterministic rules for 57 original questions, semantic judgment for 66, and execution-based verification for one.

In our current implementation, each execution-graded question requires a separate sandbox image and task-specific verification tests. Developing and validating these environments takes additional care to ensure reliable execution and reproducible scoring. We also consider whether the task is well suited to executable verification. These considerations have kept the number of execution-graded questions small in the current release. We will add more such questions in future benchmark updates, alongside validated sandbox environments and verification procedures.

For semantic questions, experts check the supporting evidence and reference answers, and the evaluator follows a task-specific rubric specifying the required content and partial-credit criteria. Appendix C.1 reports a human audit. The blind-review interface in Figure 7 shows the question, reference answer, anonymized response, and grading rubric. Model identities, automated scores, and judge rationales remain hidden until the reviewer completes all assessments. This helps reduce expectations associated with particular models and anchoring on the automated judgment.

Human–judge agreement was 92%, with a quadratic-weighted κ of 0.866, under the incorrect, partially correct, and correct mapping. Table 16 reports both mappings and their agreement results. We will reference this audit directly in the evaluation section and clarify the evidence and verification procedures available for each type of task.

AuthorsOpenReview ↗

Question Counts and Statistical Uncertainty

Thank you for emphasizing that the interpretation of capability profiles should reflect the number of underlying questions. We will report the original-question counts alongside the scores and use original questions as the unit for uncertainty estimates. In the revised taxonomy, Groups A–G contain 34, 23, 27, 17, 10, 4, and 9 original questions, respectively, for a total of 124. The seven Dataset Lineage questions move from Group A to Group G because their focus on dataset provenance fits more naturally with the other provenance-oriented tasks.

To quantify overall uncertainty, we analyzed the five selected GPT-5.6-Luna runs on the same 460 scored views from 122 original questions. The mean was 52.26, with a run-to-run standard deviation of 0.35 points. Resampling original questions 10,000 times, while keeping each question’s available views and all five run scores together, gave an overall 95% bootstrap interval of 46.65–57.78. The standard deviation describes variation across these five runs on the fixed set of views, while the interval describes sensitivity to question composition for this Luna subset.

We also added paired uncertainty estimates for a representative model using its original Table 2 scores. For Qwen3.8-27B, across all 124 original questions, the interleaved-minus-images-first difference was −11.39 points in English (95% bootstrap CI [−17.75, −5.32]) and −2.76 points in Chinese ([−10.44, 4.82]). Paired question-level sign-flip tests, with Holm correction across seven language and representation contrasts for this model, gave adjusted p-values of 0.0049 and 0.942, respectively. The language-by-representation contrast was 8.63 points, with a 95% interval of [−0.02, 17.37] and an adjusted p-value of 0.279. We will report all seven contrasts in the appendix and keep this discussion specific to the evaluated model. The archived scores include one original question with execution-evaluator errors recorded as zero in all four settings. Excluding that question gave an English layout difference of −11.49 points (95% CI [−17.68, −5.38], adjusted p = 0.0035).

We will revise Section 5.2 to present capability scores as descriptive profiles of the evaluated questions and describe the leaderboard ordering through the observed point estimates. In Section 5.3, we will report the Gemini-3.6-Flash format differences as observed point estimates and remove the inference of a language-by-representation interaction. In the next benchmark expansion, we aim to bring every capability group to at least 20 original questions.

Thank you to the authors for their responses, which have addressed some of my concerns. However, several issues remain unresolved, including quantifying the impact of upstream errors, the gap between the emphasis on executability and the evaluation methods actually used, and the lack of fully reported statistical analyses, which the authors indicate will be included in the revised version. Most importantly, the central concern remains: the number of underlying questions is small, and their distribution across task categories is uneven.

Nevertheless, I consider the benchmark construction and evaluation analysis to be reasonably sound overall. I therefore maintain my original overall rating while raising my assessment of soundness. I also ask the editors to take the above concerns and recommendations into account.

AuthorsOpenReview ↗

Reply to Reviewer sQZQ

Thank you for your thoughtful and constructive feedback, and for raising your assessment of soundness. We are glad that our responses have addressed some of your concerns. We have since completed an additional training experiment to provide further evidence for your question about general document QA data.

We trained the public Qwen3.6-27B checkpoint on 510 English DocVQA training examples with images first and final-answer supervision for one epoch. We applied LoRA to the language model’s linear layers with rank 32, alpha 64, and dropout 0.05, using AdamW with a learning rate of 1.5 × 10⁻⁵, weight decay 0.01, a cosine schedule with 5% warmup, and an effective batch size of 16. The base weights and vision components remained frozen. We evaluated the final adapter on all 496 SciDocBench views using native image-text order, a 32,768-token output limit, greedy decoding, and the fixed VLMEvalKit evaluator.

The DocVQA adapter scored 42.04, with complete scoring coverage of 496/496 views; its newly evaluated starting checkpoint scored 43.32. For reference, the SciDocDataset SFT experiment scored 45.33, using the training mixture with teacher explanations described in Appendix F. We will present the training-set sizes, supervision formats, and optimization settings alongside these results so that the scope of each experiment is explicit.

We also take your concern about the small and uneven distribution of underlying questions seriously. In the next version of SciDocBench, we will prioritize increasing the question count and improving the balance across capability groups and evaluation methods, including adding more tasks suitable for execution-based evaluation. Developing challenging questions with clear evidence and reliable reference answers takes careful construction and validation. We are committed to giving this process the time and attention it needs, and we sincerely appreciate your suggestions in helping us improve the benchmark.

Reviewer_Qh9POpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors introduce SCIDOCBENCH, a workflow-centered benchmark for scientific document understanding comprising 124 expert-authored, difficulty-screened questions over seven capability groups, 19 subtasks, and five domains, each instantiated in four matched settings (English/Chinese × All Images First/Markdown Interleaved) for 496 instances. They further propose SCIDOCIR, a structured representation of document objects and provenance, and SCIDOCDATASET (4K SFT, 10K RL instances), and show that post-training Qwen3.6-27B raises its benchmark score from 40.03 to 45.33 (SFT) and 45.74 (SFT+GRPO).

Critical strengths
  1. The workflow framing is a real gap: evaluating intermediate deliverables such as evidence location, definition-in-context, and consistency checking is more informative for scientific assistants than answer-only QA, and the example in Figure 1(a) makes the distinction concrete.

  2. The dual-axis design is the most valuable part of the paper. The finding that nine models favor All Images First while four favor Markdown Interleaved, with swings up to 9.02 points and rank changes across settings (Sec. 5.3), is a genuinely useful caution against single-representation leaderboards.

  3. Evaluation reliability is taken seriously: the GPT-5.4-mini judge is audited against a blind human expert on 100 responses with 92.0% agreement, quadratic-weighted κ = 0.866, and a bootstrapped Krippendorff's α of 0.854 [0.757, 0.924] (Sec. 3.5). This is more rigorous than most benchmark papers.

Critical weaknesses
  1. The analysis granularity far exceeds what 124 questions can support. The taxonomy spans 7 groups × 19 subtasks × 5 domains, so most cells contain one or two questions, yet Sec. 5.2–5.3 draw conclusions at the level of individual capability groups, individual models, and even language × representation interactions (e.g. Gemini-3.6-Flash reversing by 3.79 and 2.07 points). Apart from the judge-reliability audit, no confidence interval, standard error, or significance test appears anywhere in the results. Differences of 1–4 points on ~18 questions per group are not distinguishable from noise, and the prose ("Document perception and scientific extraction form common bottlenecks") asserts more than the data can carry. Please report per-group sample sizes and uncertainty.

  2. GRPO contributes almost nothing. SFT yields 45.33 and SFT+GRPO yields 45.74, a difference of 0.41 points on a 124-question benchmark — well within the noise implied by W1 — despite 10K RL instances being a headline component of SCIDOCDATASET. Either report evidence that this gain is real (multiple seeds, variance, per-group breakdown) or temper the claim that the RL split is a useful contribution.

  3. Difficulty screening introduces a selection bias that is never discussed. Candidates are discarded when all four of Claude Opus 4.8, Gemini 3.1 Pro, ChatGPT 5.5, and Kimi K2.5 answer correctly (Sec. 3.5), so every retained question is one that at least one of these systems failed. The leaderboard is then topped by successors of those same systems (Claude-Opus-5, GPT-5.6-Sol, Gemini-3.6-Flash). The benchmark is therefore adversarially filtered against specific model families, which affects both absolute scores and cross-family comparisons.

Questions for authors

No

5 comments
AuthorsOpenReview ↗

Group Sizes, Statistical Uncertainty, and Interpretation of Capability Scores

In the revised taxonomy, Groups A–G contain 34, 23, 27, 17, 10, 4, and 9 original questions, respectively, for a total of 124. We moved the seven Dataset Lineage questions from Group A to Group G because they concern dataset provenance and therefore align more naturally with the other provenance-oriented tasks in Group G. This reassignment reflects a conceptual refinement of taxonomy. We will report the updated counts alongside the corresponding scores in Table 2. Each original question appears in four matched settings, enabling within-question comparisons across languages and document representations.

To quantify overall uncertainty, we evaluated GPT-5.6-Luna five times on the same 460 scored views from 122 original questions. The mean was 52.26, with a run-to-run standard deviation of 0.35 points. Resampling original questions 10,000 times while keeping each question’s available views and all five run scores together gave an overall 95% bootstrap interval of 46.65–57.78. The standard deviation describes repeatability on the fixed set of views, while the interval describes sensitivity to question composition for this Luna subset.

We will revise Section 5.2 to frame the capability scores as descriptive profiles of the evaluated questions. For example, 12 of 13 models scored lower on Scientific Information Extraction than on Evidence Alignment and Verification. In Section 5.3, we will present the Gemini-3.6-Flash format differences as observed point estimates and remove the inference of a language-by-representation interaction. In the next benchmark expansion, we aim to bring every capability group to at least 20 original questions.

AuthorsOpenReview ↗

SFT and GRPO Results: Gains, Uncertainty, and the Role of the RL Split

We agree that the incremental gain from GRPO is small and uncertain. On SciDocBench, task-aligned SFT raised the Qwen3.6-27B score from 40.03 to 45.33. Adding GRPO raised it to 45.74, a further gain of 0.41 points, with a paired 95% bootstrap interval of [−1.79, 2.52] points obtained by resampling the 124 original questions. We will revise the abstract and training discussion to reflect the uncertainty of this incremental gain.

SciDocBench is the main contribution of the paper. Even the highest-scoring model reaches 62.60 out of 100 and makes errors on concrete scientific-document tasks, such as identifying which metrics belong to a specified section or finding all inconsistencies in a results table. Our SFT and GRPO experiments are initial attempts to improve performance on these tasks. The SFT result shows that the evaluated model can make progress with task-aligned supervision, while the small and uncertain gain from GRPO leaves open how to improve these capabilities further.

SciDocBench provides a common evaluation for testing whether future methods improve specific capabilities, such as evidence localization and numerical verification. The RL split provides verifiable training targets for exploring approaches such as capability-specific curricula and rewards for intermediate verification steps. We will present the RL split as a resource for these investigations and the current GRPO result as an initial baseline.

AuthorsOpenReview ↗

Difficulty Screening, Expert Verification, and Selection Bias

Thank you for highlighting how the choice of screening models can influence benchmark difficulty and comparisons across model families. When we began the project in March 2026, our goal was to identify scientific-document tasks that remained challenging for strong models. We therefore screened expert-written questions with Claude Opus 4.8, Gemini 3.1 Pro, ChatGPT 5.5, and Kimi K2.5, spanning four model families, and removed candidates that all four solved. Each remaining candidate underwent independent review by a second expert, who checked the supporting evidence, reference answer, clarity of the requested output, and correctness labels assigned to the screening responses. The review interface in Appendix C, Figure 6 supports explicit acceptance, revision, and rejection decisions.

Model-assisted difficulty screening also has precedents in established benchmarks. SimpleQA required an incorrect response from at least one of four screening systems. Humanity’s Last Exam tested submissions against frontier models and forwarded questions that passed its difficulty checks to expert review. MMLU-Pro removed questions answered correctly by more than four of eight screening models and used GPT-4-Turbo to construct additional plausible distractors.

We recognize that both automated assessment and human judgment are susceptible to errors and bias, which motivated our combination of model feedback, independent expert verification, and a separate audit of semantic scoring. In the audit, agreement with GPT-5.4-mini was 92.0% across incorrect, partially correct, and correct ratings, Table 16 reports these results and the score mappings, while Figure 7 shows the blind-review interface.

Our leaderboard also shows substantial remaining difficulty for models outside the screening panel: Qwen3.8-Max scored 57.15, and MiMo-V2.5 scored 40.16. We will expand Section 3.5 to explain the rationale for the screening panel and discuss its potential influence on the retained question distribution. Comparing these results with performance on a sample drawn before screening would help quantify the panel’s effect on absolute scores and model rankings.

Comments on the paper

1 comment
Area_Chair_geVSOpenReview ↗

Please Respond to the Authors’ Rebuttal

Dear Reviewers,

We are now roughly halfway through the Author–Reviewer Discussion period (which runs through 18 November). The authors have submitted their response and, in some cases, a revised manuscript.

Could you please take a moment to:

  • Read the authors’ rebuttal and any updated version of the paper;
  • Leave a reply to the authors on the points you raised, indicating whether your concerns have been addressed.

Even if your evaluation remains unchanged, a short reply confirming your final position is very helpful, as it allows me to track each reviewer’s stance and write an accurate meta-review.

If you are unable to participate further, please let me know as soon as possible.

Thank you for your time and careful reviewing.

Best regards,

Area Chair, ICLR 2027

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.