- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SciDocBench, a workflow-centered benchmark with 124 expert-authored, difficulty-screened VQA for scientific document understanding. SciDocBench is built on verifiable deliverables rather than just answers. Additionally, the paper also introduces SciDocIR and SciDocDataset. Using these, post-trained Qwen3.6-27b achieves a better performance successfully.
Critical strengths
- The paper focuses on verifiable workflow outputs rather than answer correctness alone, which could help pinpoint exactly where the model performs well or poorly.
- The proposed benchmark is expected-validated to ensure the quality.
- Beyond identifying the problem using the benchmark, the paper also proposes a potential solution to alleviate the problem. The effectiveness of the solution is demonstrated in experiments.
Critical weaknesses
- The proposed benchmark appears to target LLMs rather than agents. In that case, the results in Table 2 do not seem particularly poor to me (For pure LLMs, getting 60+ on a new benchmark is actually quite high), and I believe the scores could be higher with a harness. If they are indeed higher under agentic evaluation, then the core issue becomes whether the identified problem in SciDocBench is actually a problem.
- The SciDocDataset is solely validated on Qwen3.6-27B. Will the gain still hold when post-training on other models?
Questions for authors
Please see weaknesses.
3 comments
Extending Task-Aligned Adaptation to Ministral 3 8B
Thank you for suggesting that we test whether the training gains carry over to other model families. We had considered this comparison for the original study, but prioritized Qwen3.6-27B given the available time and the stability of our SFT and RL implementations for that model. We have now added experiments with Ministral 3 8B.
We chose Ministral 3 8B as a relatively recent open multimodal model with a 256K-token context window. Its long context was particularly useful for SciDocBench, where questions often require reading substantial scientific documents containing both text and figures. All 496 views fit within its context window while preserving the native image-text order and reserving 32,768 tokens for the answer.
We first evaluated the unmodified BF16 model and inspected its answers. The errors revealed difficulties in recovering the logical order of paper fragments, distinguishing the roles of cited work from the surrounding evidence, and following precise output requirements. This analysis guided our selection of public SciDocDataset training examples and construction of additional examples from separate source material.
We applied LoRA to the language model's query and output projections, using rank 8, alpha 16, dropout 0, AdamW with a learning rate of 5 × 10⁻⁵, and an effective batch size of 30 (microbatch 1, gradient accumulation 30). The BF16 base weights, vision encoder, and multimodal projector remained frozen.
With the same 32,768-token output limit and fixed VLMEvalKit evaluator, the score increased from 31.59 to 33.73 across all 496 views, a paired gain of 2.13 points (95% CI 0.35 to 3.94, resampling the 124 original questions). Two examples illustrate the improvements. In a Chinese paper-fragment ordering question, the baseline placed an experiment fragment before the abstract. The adapted model recovered the complete sequence of abstract, introduction, method, experiment, and conclusion, improving the rule score from 0.40 to 1.00. In an English citation-role question, it correctly distinguished work mentioned in related work from a method used for initialization and experimental comparison. Specifically, it changed LVSM's role from “Background, Comparison” to “Background” and VGGT's from “Background, Method” to “Method, Comparison,” improving the rule score from 0.40 to 0.80. It still assigned an extra role to another citation. These cases show progress in reconstructing document structure and connecting citations to their actual use in a paper, with evidence-based role assignment remaining a useful direction for further training.
Thank you for detailed responses. I will raise my score to 3 after taking the new results into consideration.
Tool-Assisted Evaluation: Harness Gains, Answer Coverage, and Remaining Challenges
Thank you for asking whether tool access changes how we should read the scores in Table 2. A harness can make long documents easier to work with by keeping relevant passages and page images available across turns. We tested GPT-5.6-Sol with Codex and Claude Opus 4.8 with Claude Code to see how much this helps with tasks such as finding evidence for a claim, connecting findings across papers, and reconstructing the order of shuffled document fragments.
Table 2 reports direct scores of 61.00 for GPT-5.6-Sol and 53.41 for Claude Opus 4.8. Using the same 496 views and assigning zero to missing or unusable answers, we obtained 62.37 with Codex and 55.10 with Claude Code.
Among usable answers, Claude Opus 4.8 averaged 56.39 across 488/496 views in three archived regrades of the direct answers. Claude Code averaged 64.46 across 424/496 views. On the 417 views where both produced usable answers, the scores were 59.13 and 65.15, respectively (paired gain 6.02 points, 95% CI 1.33 to 10.44, resampling original questions). The two sets of answers were graded in separate batches. In a concurrent comparison on a fixed subset of 100 views, the 68 complete Claude pairs averaged 59.14 and 60.45 (gain 1.31, 95% CI −2.56 to 5.63). The concurrent Codex comparison yielded 61.77 and 62.98 on 488 complete pairs (gain 1.21, 95% CI −1.37 to 3.92). We will report coverage beside the scores. With 72 views missing usable Claude Code answers, scores for individual capabilities are sensitive to which views returned an answer.
On a table verification question, Claude Code cropped the relevant regions and recomputed the averages, yielding the correct comparison. On a citation question, it made eight crops and still omitted citations and changed some author names. We will include these cases in the revision to illustrate how tools helped the model inspect evidence and where assembling a complete answer remained difficult.