- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents ToolSciVer, a framework for multimodal scientific claim verification. It combines SciEviKit, a toolkit for extracting table rows/columns, parsing charts, and inspecting image regions, with a two-stage training procedure: on-policy self-distillation using reference tool calls as privileged information, followed by GRPO with rewards for answer correctness, formatting, tool-call validity, and efficiency. Evaluation covers SciVer and MuSciClaims using three approximately 4B-scale VLM backbones. The paper reports average improvements of 8.16 and 7.92 percentage points over non-tool CoT, respectively. Additional experiments examine warm-up strategies, tool efficiency, evidence acquisition, verification given gold evidence, and transfer to external chart and figure benchmarks.
Critical strengths
-
A useful system design with improvements across multiple backbones and benchmarks. SciEviKit addresses concrete difficulties in scientific verification: extracting precise table entries, recovering chart content, and inspecting localized figure evidence. Table 2 reports higher overall accuracy than the compared existing methods for all three backbones on both datasets. Table 12 also shows that OPSD warm-up improves over tool-enabled GRPO without warm-up across these settings. This breadth makes the proposed combination promising beyond a single backbone or benchmark.
-
The evaluation distinguishes evidence acquisition from reasoning with evidence. Tables 5 and 6 provide complementary diagnostics: REAR increases from 83.00% to 90.00%, while verification accuracy with identical gold evidence increases from 60.33% to 71.67%. The latter includes a paired significance test. The external chart/figure evaluation also reports improvements without tool-use instructions. These analyses strengthen the empirical contribution by examining more than final task accuracy alone.
Critical weaknesses
-
W1 — The reported evaluation does not establish an independent model-selection protocol. Section 4 and Appendix B.3 state that checkpoints are evaluated every 50 GRPO steps and that the checkpoint with the highest average evaluation accuracy is selected. Appendix B.1 specifies training and final evaluation/test sets but does not identify a separate validation set. Consequently, it is unclear whether the reported evaluation data also influenced checkpoint selection. If so, selecting among roughly 30 checkpoints could introduce optimistic bias into the reported gains. Moreover, the three reported runs vary only inference seeds for a fixed trained checkpoint; they do not establish robustness across training runs. This matters particularly for relatively small margins, such as the 0.71-point improvement over the strongest listed baseline on MuSciClaims with InternVL. An explicit split definition and results from checkpoints selected independently of the test data are necessary to resolve the central concern.
-
W2 — The additional value of the proposed policy is not sufficiently separated from the stronger evidence-extraction backend. Appendix A.1 states that Chart Parse uses GPT-6-Astra with maximum reasoning effort, while Appendix B.4 states that VTool-R1 and OpenThinkIMG retain their original tool suites. Their comparison therefore changes both the learning procedure and the evidence available through tools. The results demonstrate the performance of the combined system, but do not cleanly establish the advantage of the proposed policy over alternatives with equivalent extraction capabilities. The shared-tool ablations in Table 3 partially address this issue and support a warm-up benefit. However, a task-trained verifier supplied with the same pre-extracted evidence would be a particularly informative control for establishing the value of adaptive tool acquisition. Likewise, fewer verifier tokens and tool calls in Table 4 do not establish lower end-to-end cost when a call can invoke a much stronger model. These gaps limit the demonstrated significance and practical efficiency of the proposed design.
Questions for authors
-
Clarification concerning W1: Which exact split was used for checkpoint selection? Was it disjoint from the 1,000 SciVer and 505 MuSciClaims examples used for the reported results? If those examples influenced selection, can you provide evaluation on untouched data using a selection rule fixed beforehand?
-
Additional evidence concerning W2: How does ToolSciVer compare with a task-trained verifier that receives the same OCR and chart-parser outputs directly, without learning when to request them? Alternatively, do the improvements over trained tool-use baselines persist when they receive the same SciEviKit backend and interaction budget?
-
Clarification concerning W2: Are chart-parser outputs precomputed or generated during each interaction, and what backend computation is included in the efficiency measurements? Please distinguish verifier-token savings from total system cost, including extraction or preprocessing.
7 comments
Response to Reviewer eDv6(2/3)
W2/Q2: Contributions of Policy Learning and Evidence Extraction
Thank you for suggesting these comparisons. Our additional experiments show that ToolSciVer’s benefits extend beyond access to a stronger extraction backend, highlighting the importance of its learned tool-use policy. Specifically, three complementary comparisons show that (1) ToolSciVer outperforms a direct-evidence verifier on MuSciClaims while achieving comparable accuracy on SciVer; (2) its advantage over VTool-R1 and OpenThinkIMG persists when these baselines receive the same SciEviKit backend and interaction budget; and (3) it maintains strong performance when its own backbone replaces the stronger chart parser. We present these three comparisons below.
1. Directly providing pre-extracted evidence.
To assess the value of adaptive evidence acquisition, we trained a Qwen3.5-4B direct-evidence verifier that receives all available precomputed OCR and chart-parser outputs directly in its initial prompt, alongside the original images, without learning to request tools.
| Method | SciVer Acc. (%) | MuSciClaims Acc. (%) |
|---|---|---|
| Direct-evidence verifier | 85.03 ± 0.35 | 77.43 ± 1.39 |
| ToolSciVer | 84.07 ± 1.27 | 80.79 ± 0.52 |
The direct-evidence verifier achieves 0.96 percentage points higher accuracy on SciVer, where tables and charts provide structured textual evidence that can be supplied upfront. MuSciClaims, however, consists of multi-panel scientific figures, for which this setup supplies no additional OCR or chart text. Inspecting relevant regions through zoom-in requires selecting where to look, which the direct-evidence verifier does not perform. But ToolSciVer’s learning policy supports this adaptive region inspection and achieves 3.36 percentage points higher accuracy on MuSciClaims.
These results highlight the value of adaptive evidence acquisition when verification requires inspecting claim-relevant image regions. Our goal is to learn whether additional evidence is needed, which visual tool to invoke, and which region to inspect, extending verification beyond evidence supplied upfront.
2. Equipping training-based baselines with SciEviKit.
To examine whether ToolSciVer’s advantage persists when baselines receive the same tools, we conducted a toolset replacement experiment on Gemma4-E4B. We retained the trained VTool-R1 and OpenThinkIMG checkpoints, replaced their original toolsets with SciEviKit, and updated the corresponding tool descriptions in their prompts. We then evaluated both methods on SciVer and MuSciClaims using the same SciEviKit backend and inference-time interaction budget as ToolSciVer. The results are in the following table.
| Method | Toolset | SciVer (%) | MuSciClaims (%) |
|---|---|---|---|
| VTool-R1 | Original | 74.23 ± 1.16 | 68.78 ± 0.80 |
| VTool-R1 | SciEviKit | 77.20 ± 1.00 | 71.16 ± 1.65 |
| OpenThinkIMG | Original | 72.40 ± 1.61 | 71.16 ± 0.98 |
| OpenThinkIMG | SciEviKit | 77.63 ± 0.58 | 70.76 ± 0.70 |
| ToolSciVer | SciEviKit | 81.20 ± 0.95 | 74.65 ± 0.59 |
With SciEviKit, VTool-R1 scores 2.97 and 2.38 percentage points higher than its originally reported results on SciVer and MuSciClaims. OpenThinkIMG also scores 5.23 points higher on SciVer. ToolSciVer remains ahead of both baselines when they are equipped with SciEviKit, with margins of 3.57 and 3.49 points over the stronger baseline on the two benchmarks.
Thus, ToolSciVer’s advantage persists when the extraction backend and interaction budget are shared, supporting the contribution of its learned tool-use policy beyond access to SciEviKit.
3. Chart Parser with Qwen3.5-4B.
Finally, to assess whether ToolSciVer’s gains depend on a stronger external chart parser, we replaced Astra with Qwen3.5-4B itself and retrained and evaluated ToolSciVer using the resulting self-generated chart parses.
| Setting | SciVer | MuSciClaims | Chart-type problem |
|---|---|---|---|
| ToolSciVer (Astra parser) | 84.07 ± 1.27 | 80.79 ± 0.52 | 83.31 ± 0.33 |
| ToolSciVer (self-parsing) | 83.10 ± 0.62 | 79.21 ± 0.91 | 80.59 ± 0.31 |
| VTool-R1 | 81.18 ± 0.03 | 76.78 ± 0.75 | 78.33 ± 0.45 |
| OpenThinkIMG | 75.94 ± 0.23 | 75.60 ± 0.94 | 72.92 ± 0.65 |
Compared with Astra-parsing, self-parsing reduces overall accuracy by 0.97 and 1.58 percentage points on SciVer and MuSciClaims, with a larger decrease of 2.72 points on chart-type problems. Nevertheless, it still outperforms both tool-use baselines, exceeding VTool-R1 by 1.92 and 2.43 percentage points on the two benchmarks.
Thus, ToolSciVer retains its advantage over these tool-use baselines when both training and evaluation use self-generated chart parses, demonstrating that its gains do not require a stronger external chart parser.
Response to Reviewer eDv6(3/3)
W2/Q3: Tool Execution and Efficiency
Chart-parser outputs are precomputed and retrieved on demand when the verifier invokes the tool. In ToolSciVer, Chart Parse accounts for 44.80% of all tool-call attempts, while the average number of tool-calls per example is 0.74(Table 4).
To further validate the effectiveness of on token efficiency, we extended our token-efficiency analysis. 1) First, all of our experiments(also including these two variants) use the same precomputed chart-parsing and OCR results, so offline preprocessing is a shared, one-time cost(615k tokens in total, amortized 409 tokens per example). These outputs are cached and reused across subsequent experiments on the same data, without repeating preprocessing. We therefore report this cost separately and exclude it from our comparison of verification-stage token efficiency. 2) We then calculated the average number of tool-returned text tokens per example for ToolSciVer, obtaining 843 tokens. Adding these to the tokens generated by the verifier itself yields 1362 + 843 = 2205 text tokens per example. This remains 20.11% lower than the 2760 verifier-generated tokens of the variant without , even before including that variant’s tool-returned text. Thus, the token-efficiency advantage persists when tool-returned text is included.
We also compare ToolSciVer’s 1) tool-call efficiency and 2) token efficiency with other baseline methods: VTool-R1 and OpenThinkIMG. The following table shows the tool-call efficiency; the metric in the table is average tool-call number per example.
| Method | SciVer | MuSciClaim | Overall |
|---|---|---|---|
| VTool-R1 | 1.81 | 1.84 | 1.82 |
| OpenThinkIMG | 1.93 | 1.74 | 1.87 |
| ToolSciVer w/o | 1.24 | 0.66 | 1.05 |
| ToolSciVer w/ | 0.85 | 0.54 | 0.74 |
From the table, we can see ToolSciVer use less tool-calls while achieves better performance(Table 2). Showing ToolSciVer’s good efficiency. The average text-token usage per example, including both reasoning tokens and tool-result tokens but excluding the shared, one-time preprocessing cost discussed above, is 2474 for VTool-R1 and 2324 for OpenThinkIMG, both higher than ToolSciVer’s 2205 tokens. These comparisons show that ToolSciVer achieves higher verification accuracy with fewer tool calls and lower text-token usage, highlighting its advantages in both tool-call efficiency and token efficiency.
Response to Reviewer eDv6(1/3)
Dear Reviewer eDv6,
Thank you for your thoughtful comments and constructive suggestions! We address your concerns below.
W1/Q1: Checkpoint Selection & Multi-Seed Result
Thanks for your suggestions! To address your concern on checkpoint selection, we 1) clarify the fairness of the comparisons in the original experiment, and 2) provide new results by selecting the best checkpoint on a disjoint evaluation set and reporting its performance on the test set.
First, the same checkpoint-selection and reporting protocol was applied to ToolSciVer and all training-based baselines, including VTool-R1 and OpenThinkIMG. The reported comparisons therefore use a consistent selection criterion across these methods. Similar reporting practices also appear in prior work. For example, OPSD[1] reports the best score across periodically evaluated checkpoints and reports peak performance for its GRPO baseline. We nevertheless acknowledge that a consistent protocol and prior precedent alone do not address the concern about separating checkpoint selection from final reporting.
To further address your concern about this, we partitioned each original test set into disjoint evaluation and test subsets in a 1:2 ratio, selected the best checkpoints solely based on evaluation set performance, and reported their accuracy on the test set. Under this protocol, ToolSciVer achieves the highest mean overall accuracy in all six dataset–backbone combinations. On SciVer, it outperforms the strongest baseline by 4.35, 2.19, and 3.80 percentage points for Qwen3.5-4B, InternVL3.5-4B, and Gemma4-E4B, respectively. On MuSciClaims, the corresponding improvements are 0.20, 0.99, and 1.28 percentage points. Although the margins vary across settings, ToolSciVer consistently ranks first, supporting the robustness of our comparative conclusions under evaluation-based checkpoint selection.
We additionally conducted multi-seed training experiments with Qwen3.5-4B for ToolSciVer, VTool-R1, and OpenThinkIMG, reporting mean accuracy and standard deviation across training runs. ToolSciVer achieves the highest mean accuracy on both benchmarks, outperforming the strongest baseline, VTool-R1, by 3.80 and 2.77 percentage points on SciVer and MuSciClaims. These results support the robustness of ToolSciVer’s performance gains across training seeds.
References:
[1] Zhao, S., et al. 2026. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML 2026.