- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how to verify whether a scientific image is correct, not just whether it looks plausible. It introduces SciGen-Verify, a benchmark that labels each example with a binary decision, an explanation, and, for incorrect images, an editing instruction. The authors also train SciGen-Verifier using supervised fine-tuning followed by a two-stage reinforcement-learning pipeline. On the new benchmark, the model performs close to or better than larger proprietary models.
Critical strengths
- The paper addresses an underexplored evaluation problem in image generation: how to judge whether a scientific image is correct. It provides both a benchmark and a verifier for this task.
- The proposed two-stage reinforcement-learning method improves accuracy substantially, bringing the verifier to the level of much larger proprietary models.
- The verifier does more than give a binary answer. It also produces an explanation and an editing instruction, which makes it potentially useful for agentic image generation.
Critical weaknesses
- The comparison with proprietary models is too limited. The paper only tests GPT-6-Astra. It should also include models such as GPT-5.6 (Sol, Terra, Luna), and Claude Fable and Opus.
- The paper does not test whether SciGen-Verifier can be used to evaluate existing scientific text-to-image and image-editing benchmarks, such as GenExam, ProImage-Bench, and GRADE [1]. It would be useful to show the accuracy gains and cost savings compared with the judges currently used in those benchmarks, and to discuss this application in the paper.
- The current Instruction Accuracy metric only checks whether an editing instruction looks correct. It does not show whether following the instruction actually improves the edited image. The authors should evaluate this on a real image-editing benchmark, or at least run the instructions and measure the final image quality.
[1] Mingxin Liu, et al. Benchmarking Discipline-Informed Reasoning in Image Editing. ECCV 2026.
Questions for authors
- Could the authors briefly discuss how this verifier could be used in agentic image generation?
- How much does performance drop on disciplines, sub-disciplines, or image types that were not seen during training, and on low-resolution images?
4 comments
Response to Reviewer PcCG (2/3)
Q3: The Instruction Accuracy metric only checks whether an editing instruction looks correct. It does not show whether following the instruction actually improves the edited image.
A3: Thank you for this comment. Reviewer jd1N raised the same point, and we provide our unified response below.
In Sec. 5.4 (Fig. 4(b)), the paper tests image repair end to end using SciGen-Verifier-8B in a test-time scaling loop. Our verifier flags mislabelled angles and a wrong exterior-angle equation. Guided by our instruction, the generator edits the figure and passes verification in the next round. Fig. 6 (Appendix D.2) shows a second multi-round case on a Maxwell–Boltzmann figure. In Table 9 (Appendix D.2) on GenExam with Wan2.6 as generator, the relaxed score rises from 58.7 to 63.5 over ten rounds. Gemini-3.5-Flash and Qwen3.8-Max reach 64.9 and 64.4 in the same loop. Our 8B verifier is therefore close to much larger proprietary critics at a small fraction of the cost.
We also invite domain-matched annotators to grade editing instructions on 100 negative cases by SciGen-Verifier-8B and Qwen3.8-Max. Scores span 0 (no real error), 1 (right and wrong edits), 2 (only right edits, missed errors), and 3 (all errors rightly covered). In Table R1, Qwen3.8-Max scores 2.19 on average and SciGen-Verifier-8B 2.05, with 74% and 69% of instructions rated 2 or 3. The human ranking matches the automated Acc_i on the same cases (63.0 and 58.0). This supports the reliability of the metric. We have added this study to Appendix F.5 of the revised manuscript.
Table R1. Human grading of editing instructions on 100 negative cases. Acc_i is the automated metric on the same 100 cases.
| Model | 0 (%) | 1 (%) | 2 (%) | 3 (%) | Mean score | Acc_i |
|---|---|---|---|---|---|---|
| Qwen3.8-Max | 9 | 17 | 20 | 54 | 2.19 | 63.0 |
| SciGen-Verifier-8B | 11 | 20 | 22 | 47 | 2.05 | 58.0 |
Q4: Could the authors briefly discuss how this verifier could be used in agentic image generation?
A4: Thank you for raising this question. SciGen-Verifier fits agentic image generation pipelines in three direct roles.
As shown in Appendix D.2, our model acts as a closed-loop critic in a generate, verify, and refine loop. It returns an actionable editing instruction rather than only a scalar score, so the controller routes actions from that instruction. Three or more correction items or a global-regeneration keyword trigger full regeneration, while localized corrections route to an image editor. In Table 9, this raises the Wan2.6 GenExam relaxed score from 58.7 to 63.5 over 10 steps, exceeding Seed-2.0-Pro at 62.6. Fig. 4(b) and Fig. 6 show example refinement trajectories.
Recent agents such as GenClaw [1] first plan an image as executable code like SVG, HTML, or Three.js. They render this code into a structural sketch before an image generator adds textures and realism. This workflow needs a checker to verify prompt constraints on the sketch and specify exact code edits when errors occur. That matches our verification task, and our Instruction Following and Multidisciplinary Reasoning domains directly cover such code-rendered scientific diagrams.
Both agentic workflows invoke the verifier multiple times per user query. Running an 8B open-weight model on a single GPU keeps latency and per-call cost low (see our reply to Reviewer DdY8, Q4). Querying a frontier proprietary critic at every step would otherwise dominate the pipeline's total cost.
We have added a Discussion paragraph on these uses to Sec. 5 of the revised manuscript.
[1] GenClaw: Code-Driven Agentic Image Generation, 2026.
Response to Reviewer PcCG (3/3)
Q5: How much does performance drop on disciplines, sub-disciplines, or image types that were not seen during training, and on low-resolution images?
A5: Thank you for the constructive question. We down-sample every benchmark image to 1/2 of its original resolution and re-evaluate without retraining (Table R2). The 8B baseline with no verification training degrades the most, while our same-size model stays near its full-resolution result. This shows that our training builds robust visual grounding rather than reliance on fine pixel detail.
Table R2. Overall Acc_a / Acc_e / Acc_i on SciGen-Verify at reduced input resolution (no retraining).
| Model | Original | 1/2 resolution |
|---|---|---|
| Qwen3.8-Max | 86.03 / 84.46 / 64.75 | 84.95 / 83.21 / 63.12 |
| Qwen3-VL-8B-Thinking | 61.06 / 46.51 / 23.52 | 58.12 / 43.80 / 21.37 |
| SciGen-Verifier-8B | 86.28 / 81.79 / 57.42 | 85.74 / 81.10 / 56.63 |
We run leave-one-discipline-out training by removing all physics or all biology samples from SFT and both RL stages, and evaluate each model on the full SciGen-Verify benchmark (Table R3). Removing a discipline also removes its typical image types, such as circuit diagrams and force or kinematics diagrams for physics, and food webs and cell structures for biology, so these runs also test unseen image types. Removing physics lowers overall Acc_a from 86.28 to 85.27, mostly in the Reasoning domain where most physics problems lie (89.02 to 86.93). Removing biology lowers overall Acc_a to 85.56, mostly in the Knowledge domain (94.21 to 92.99). Both models stay well above the SFT-only model trained on all disciplines (80.09 / 75.32 / 46.72), showing that the learned verification skills transfer rather than rely on discipline-specific patterns.
Table R3. Leave-one-discipline-out: a discipline is removed from all training stages and the model is evaluated on the full SciGen-Verify benchmark (Acc_a / Acc_e / Acc_i).
| Training data | Instruction | Reasoning | Knowledge | Overall |
|---|---|---|---|---|
| SFT only (all data) | 65.35 / 60.63 / 35.34 | 82.25 / 75.36 / 43.45 | 92.66 / 89.97 / 61.36 | 80.09 / 75.32 / 46.72 |
| SFT + R1 + R2 (all data, ours) | 75.62 / 70.38 / 48.93 | 89.02 / 83.47 / 57.84 | 94.21 / 91.53 / 65.48 | 86.28 / 81.79 / 57.42 |
| SFT + R1 + R2 (without physics) | 75.13 / 69.89 / 48.34 | 86.93 / 81.03 / 55.53 | 93.75 / 91.07 / 65.10 | 85.27 / 80.66 / 56.32 |
| SFT + R1 + R2 (without biology) | 75.37 / 70.13 / 48.34 | 88.32 / 82.77 / 57.07 | 92.99 / 90.31 / 63.17 | 85.56 / 81.07 / 56.19 |
Finally, our test-time scaling experiment on GenExam (Table 9, Appendix D.2) tests an unseen source with Wan2.6-generated images. GenExam is excluded from training, and Wan2.6 differs from the pipeline used for our negatives. Even so, SciGen-Verifier-8B improves the relaxed score from 58.7 to 63.5 over 10 steps. For details, please see our reply to Reviewer jd1N, Q2.
Thank you for your detailed response and additional results. My concerns have been addressed, and therefore I have raised my rating to clear acceptance.
Response to Reviewer PcCG (1/3)
Dear Reviewer PcCG,
Thank you for your careful review and for the positive assessment of the benchmark, the two-stage reinforcement learning method, and the explainable outputs. We address each point below. All changes in the revised manuscript are marked in red.
Q1: The comparison with proprietary models is limited. The paper only tests GPT-6-Astra. It should also include models such as GPT-5.6 (Sol, Terra, Luna), Claude Fable, and Claude Opus.
A1: Thank you for the suggestion. We have evaluated GPT-5.6-Sol, GPT-5.6-Terra, GPT-5.6-Luna, Claude-Fable-5.1 and Claude-Opus-5.5 under the same protocol as Table 1, with the same prompt, the same judges and the same 1,350 samples. The results are added to Table 1 of the revised manuscript. The strongest of them, Claude-Opus-5.5, reaches 84.72 overall Acc_a and is the best model on editing accuracy in the Instruction domain. SciGen-Verifier-8B still has the highest overall Acc_a (86.28) and the best results in the Knowledge domain. An 8B verifier therefore remains competitive against the latest proprietary models.
Q2: The paper does not test whether SciGen-Verifier can be used to evaluate existing scientific text-to-image and image-editing benchmarks. It would be useful to discuss its application in the paper.
A2: Thank you for this suggestion. Benchmarks such as GenExam and ProImage-Bench use rubric scoring to rank generators offline. Our work targets closed-loop verification instead. Given an instruction and a candidate image, SciGen-Verifier decides correctness, explains errors, and outputs an executable editing instruction. For verification, even a small error must be caught and explained rather than merely scored. Therefore, we did not design SciGen-Verifier as a replacement for offline leaderboard judges.
The submission applies our verifier on GenExam. As stated in Appendix D.2 (Table 9), our test-time scaling experiment runs on GenExam with GenExam's own relaxed metric. Guiding Wan2.6 with SciGen-Verifier-8B raises the score from 58.7 to 63.5 over ten iterations. This surpasses Seed-2.0-Pro (62.6) and approaches Qwen3.8-Max (64.4) as critics in the same loop. Thus, our verifier's judgements and edits match an established standard, and acting on its feedback improves that benchmark's score.