acceptodds
Under review as a conference paper at ICLR 2027

SciGen-Verifier: A Multimodal Reasoner for Explainable Verification in Scientific Image Generation

Abstract

In realistic education, a solution is often expressed not only in words but in a drawing—a circuit, a geometric construction, a function plot—and a teacher must grade the drawing as carefully as the text. Recent advances in unified multimodal models have enabled scientific image generation, yet verifying the correctness of these specialized visual outputs remains a critical bottleneck: errors often arise from intricate domain knowledge, structural reasoning, and multi-step instruction rather than surface-level artifacts. Existing verifiers mainly target natural images and compress judgement into scalar scores, leaving scientific coverage and explainable feedback for error correction underexplored. To bridge this gap, we make three main contributions. (1) We construct SciGen-Verify, a benchmark dedicated to explainable verification of scientific image generation, spanning instruction following, multidisciplinary reasoning, and world knowledge domains. It contains a three-tier hierarchical protocol over the binary judgement, supporting explanation, and corrective editing instruction. (2) We develop SciGen-Verifier, a reasoning-driven multimodal verifier trained via cold-start supervised fine-tuning followed by a curriculum-based two-stage reinforcement learning pipeline. The rubric-guided process rewards first strengthen scientific reasoning exploration and outcome rewards subsequently align output with ground-truth annotation. (3) On SciGen-Verify, SciGen-Verifier achieves competitive performance against much larger proprietary models. It further serves as a practical online critic for iterative image rectification.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal verifier, scientific image generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_PcCGOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to verify whether a scientific image is correct, not just whether it looks plausible. It introduces SciGen-Verify, a benchmark that labels each example with a binary decision, an explanation, and, for incorrect images, an editing instruction. The authors also train SciGen-Verifier using supervised fine-tuning followed by a two-stage reinforcement-learning pipeline. On the new benchmark, the model performs close to or better than larger proprietary models.

Critical strengths
  • The paper addresses an underexplored evaluation problem in image generation: how to judge whether a scientific image is correct. It provides both a benchmark and a verifier for this task.
  • The proposed two-stage reinforcement-learning method improves accuracy substantially, bringing the verifier to the level of much larger proprietary models.
  • The verifier does more than give a binary answer. It also produces an explanation and an editing instruction, which makes it potentially useful for agentic image generation.
Critical weaknesses
  • The comparison with proprietary models is too limited. The paper only tests GPT-6-Astra. It should also include models such as GPT-5.6 (Sol, Terra, Luna), and Claude Fable and Opus.
  • The paper does not test whether SciGen-Verifier can be used to evaluate existing scientific text-to-image and image-editing benchmarks, such as GenExam, ProImage-Bench, and GRADE [1]. It would be useful to show the accuracy gains and cost savings compared with the judges currently used in those benchmarks, and to discuss this application in the paper.
  • The current Instruction Accuracy metric only checks whether an editing instruction looks correct. It does not show whether following the instruction actually improves the edited image. The authors should evaluate this on a real image-editing benchmark, or at least run the instructions and measure the final image quality.

[1] Mingxin Liu, et al. Benchmarking Discipline-Informed Reasoning in Image Editing. ECCV 2026.

Questions for authors
  • Could the authors briefly discuss how this verifier could be used in agentic image generation?
  • How much does performance drop on disciplines, sub-disciplines, or image types that were not seen during training, and on low-resolution images?
4 comments
AuthorsOpenReview ↗

Response to Reviewer PcCG (1/3)

Dear Reviewer PcCG,

Thank you for your careful review and for the positive assessment of the benchmark, the two-stage reinforcement learning method, and the explainable outputs. We address each point below. All changes in the revised manuscript are marked in red.

Q1: The comparison with proprietary models is limited. The paper only tests GPT-6-Astra. It should also include models such as GPT-5.6 (Sol, Terra, Luna), Claude Fable, and Claude Opus.

A1: Thank you for the suggestion. We have evaluated GPT-5.6-Sol, GPT-5.6-Terra, GPT-5.6-Luna, Claude-Fable-5.1 and Claude-Opus-5.5 under the same protocol as Table 1, with the same prompt, the same judges and the same 1,350 samples. The results are added to Table 1 of the revised manuscript. The strongest of them, Claude-Opus-5.5, reaches 84.72 overall Acc_a and is the best model on editing accuracy in the Instruction domain. SciGen-Verifier-8B still has the highest overall Acc_a (86.28) and the best results in the Knowledge domain. An 8B verifier therefore remains competitive against the latest proprietary models.

Q2: The paper does not test whether SciGen-Verifier can be used to evaluate existing scientific text-to-image and image-editing benchmarks. It would be useful to discuss its application in the paper.

A2: Thank you for this suggestion. Benchmarks such as GenExam and ProImage-Bench use rubric scoring to rank generators offline. Our work targets closed-loop verification instead. Given an instruction and a candidate image, SciGen-Verifier decides correctness, explains errors, and outputs an executable editing instruction. For verification, even a small error must be caught and explained rather than merely scored. Therefore, we did not design SciGen-Verifier as a replacement for offline leaderboard judges.

The submission applies our verifier on GenExam. As stated in Appendix D.2 (Table 9), our test-time scaling experiment runs on GenExam with GenExam's own relaxed metric. Guiding Wan2.6 with SciGen-Verifier-8B raises the score from 58.7 to 63.5 over ten iterations. This surpasses Seed-2.0-Pro (62.6) and approaches Qwen3.8-Max (64.4) as critics in the same loop. Thus, our verifier's judgements and edits match an established standard, and acting on its feedback improves that benchmark's score.

AuthorsOpenReview ↗

Response to Reviewer PcCG (2/3)

Q3: The Instruction Accuracy metric only checks whether an editing instruction looks correct. It does not show whether following the instruction actually improves the edited image.

A3: Thank you for this comment. Reviewer jd1N raised the same point, and we provide our unified response below.

In Sec. 5.4 (Fig. 4(b)), the paper tests image repair end to end using SciGen-Verifier-8B in a test-time scaling loop. Our verifier flags mislabelled angles and a wrong exterior-angle equation. Guided by our instruction, the generator edits the figure and passes verification in the next round. Fig. 6 (Appendix D.2) shows a second multi-round case on a Maxwell–Boltzmann figure. In Table 9 (Appendix D.2) on GenExam with Wan2.6 as generator, the relaxed score rises from 58.7 to 63.5 over ten rounds. Gemini-3.5-Flash and Qwen3.8-Max reach 64.9 and 64.4 in the same loop. Our 8B verifier is therefore close to much larger proprietary critics at a small fraction of the cost.

We also invite domain-matched annotators to grade editing instructions on 100 negative cases by SciGen-Verifier-8B and Qwen3.8-Max. Scores span 0 (no real error), 1 (right and wrong edits), 2 (only right edits, missed errors), and 3 (all errors rightly covered). In Table R1, Qwen3.8-Max scores 2.19 on average and SciGen-Verifier-8B 2.05, with 74% and 69% of instructions rated 2 or 3. The human ranking matches the automated Acc_i on the same cases (63.0 and 58.0). This supports the reliability of the metric. We have added this study to Appendix F.5 of the revised manuscript.

Table R1. Human grading of editing instructions on 100 negative cases. Acc_i is the automated metric on the same 100 cases.

Model0 (%)1 (%)2 (%)3 (%)Mean scoreAcc_i
Qwen3.8-Max91720542.1963.0
SciGen-Verifier-8B112022472.0558.0

Q4: Could the authors briefly discuss how this verifier could be used in agentic image generation?

A4: Thank you for raising this question. SciGen-Verifier fits agentic image generation pipelines in three direct roles.

As shown in Appendix D.2, our model acts as a closed-loop critic in a generate, verify, and refine loop. It returns an actionable editing instruction rather than only a scalar score, so the controller routes actions from that instruction. Three or more correction items or a global-regeneration keyword trigger full regeneration, while localized corrections route to an image editor. In Table 9, this raises the Wan2.6 GenExam relaxed score from 58.7 to 63.5 over 10 steps, exceeding Seed-2.0-Pro at 62.6. Fig. 4(b) and Fig. 6 show example refinement trajectories.

Recent agents such as GenClaw [1] first plan an image as executable code like SVG, HTML, or Three.js. They render this code into a structural sketch before an image generator adds textures and realism. This workflow needs a checker to verify prompt constraints on the sketch and specify exact code edits when errors occur. That matches our verification task, and our Instruction Following and Multidisciplinary Reasoning domains directly cover such code-rendered scientific diagrams.

Both agentic workflows invoke the verifier multiple times per user query. Running an 8B open-weight model on a single GPU keeps latency and per-call cost low (see our reply to Reviewer DdY8, Q4). Querying a frontier proprietary critic at every step would otherwise dominate the pipeline's total cost.

We have added a Discussion paragraph on these uses to Sec. 5 of the revised manuscript.

[1] GenClaw: Code-Driven Agentic Image Generation, 2026.

AuthorsOpenReview ↗

Response to Reviewer PcCG (3/3)

Q5: How much does performance drop on disciplines, sub-disciplines, or image types that were not seen during training, and on low-resolution images?

A5: Thank you for the constructive question. We down-sample every benchmark image to 1/2 of its original resolution and re-evaluate without retraining (Table R2). The 8B baseline with no verification training degrades the most, while our same-size model stays near its full-resolution result. This shows that our training builds robust visual grounding rather than reliance on fine pixel detail.

Table R2. Overall Acc_a / Acc_e / Acc_i on SciGen-Verify at reduced input resolution (no retraining).

ModelOriginal1/2 resolution
Qwen3.8-Max86.03 / 84.46 / 64.7584.95 / 83.21 / 63.12
Qwen3-VL-8B-Thinking61.06 / 46.51 / 23.5258.12 / 43.80 / 21.37
SciGen-Verifier-8B86.28 / 81.79 / 57.4285.74 / 81.10 / 56.63

We run leave-one-discipline-out training by removing all physics or all biology samples from SFT and both RL stages, and evaluate each model on the full SciGen-Verify benchmark (Table R3). Removing a discipline also removes its typical image types, such as circuit diagrams and force or kinematics diagrams for physics, and food webs and cell structures for biology, so these runs also test unseen image types. Removing physics lowers overall Acc_a from 86.28 to 85.27, mostly in the Reasoning domain where most physics problems lie (89.02 to 86.93). Removing biology lowers overall Acc_a to 85.56, mostly in the Knowledge domain (94.21 to 92.99). Both models stay well above the SFT-only model trained on all disciplines (80.09 / 75.32 / 46.72), showing that the learned verification skills transfer rather than rely on discipline-specific patterns.

Table R3. Leave-one-discipline-out: a discipline is removed from all training stages and the model is evaluated on the full SciGen-Verify benchmark (Acc_a / Acc_e / Acc_i).

Training dataInstructionReasoningKnowledgeOverall
SFT only (all data)65.35 / 60.63 / 35.3482.25 / 75.36 / 43.4592.66 / 89.97 / 61.3680.09 / 75.32 / 46.72
SFT + R1 + R2 (all data, ours)75.62 / 70.38 / 48.9389.02 / 83.47 / 57.8494.21 / 91.53 / 65.4886.28 / 81.79 / 57.42
SFT + R1 + R2 (without physics)75.13 / 69.89 / 48.3486.93 / 81.03 / 55.5393.75 / 91.07 / 65.1085.27 / 80.66 / 56.32
SFT + R1 + R2 (without biology)75.37 / 70.13 / 48.3488.32 / 82.77 / 57.0792.99 / 90.31 / 63.1785.56 / 81.07 / 56.19

Finally, our test-time scaling experiment on GenExam (Table 9, Appendix D.2) tests an unseen source with Wan2.6-generated images. GenExam is excluded from training, and Wan2.6 differs from the pipeline used for our negatives. Even so, SciGen-Verifier-8B improves the relaxed score from 58.7 to 63.5 over 10 steps. For details, please see our reply to Reviewer jd1N, Q2.

Thank you for your detailed response and additional results. My concerns have been addressed, and therefore I have raised my rating to clear acceptance.

Reviewer_jd1NOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SciGen-Verify, a benchmark for evaluating multimodal models on scientific image verification, and SciGen-Verifier, an 8B-parameter model designed to determine whether a scientific image satisfies its prompt, explain identified errors, and provide corrective editing instructions. The benchmark comprises 1,350 samples across three categories: Instruction Following, Multidisciplinary Reasoning, and World Knowledge. Its construction combines examples from multiple sources with visually and textually perturbed negatives, followed by domain-expert review.

The proposed model is initialized from Qwen3-VL-8B-Instruct and trained using supervised fine-tuning followed by a two-stage reinforcement learning procedure. The first RL stage emphasizes judgment accuracy and rubric-guided reasoning, while the second further optimizes the quality of explanations and editing instructions. The experiments include comparisons with existing multimodal models, ablations of the training components, and an iterative image-correction setting.

Critical strengths
  1. Important and challenging problem setting. The paper addresses scientific diagram verification, a type of image that can appear visually plausible while containing errors in structure, scientific relationships, or consistency with the underlying prompt. The paper appropriately recognizes that verifying such images requires reasoning about their scientific elements and relations, making them more complex and more difficult to analyse than natural images.
  2. A useful formulation of verification as an explainable, actionable task. The proposed framework includes an explanation of the identified error, and a corrective editing instruction. This is a useful formulation because it aims not only to detect errors but also to make the verification process interpretable.
  3. Careful benchmark construction with expert review. The SciGen-Verify generation pipeline includes model-assisted annotation followed by review from domain experts. This expert-refinement process is a strength and provides meaningful support for the quality of the benchmark annotations.
  4. Systematic evaluation of training components. The experiments include ablations of chain-of-thought supervision and the two RL stages. The results show improvements when reasoning supervision is included and when the two RL stages are combined. The iterative correction experiment also provides an initial demonstration of how a verification model might be used in a feedback loop, rather than only as a standalone evaluator.
Critical weaknesses
  1. The editing metric does not isolate editing ability, which complicates the interpretation of the results. Acci is a cascading metric on negative examples: an edit is credited only when the model correctly identifies the image as flawed, provides a correct explanation, and produces an editing instruction judged to be corrective. Consequently, the relatively low Acci score may reflect errors at any of these stages, rather than an inability to formulate effective editing instructions specifically. This makes it difficult to interpret the gap between judgment and editing scores as evidence that correcting scientific errors is intrinsically harder than identifying them. Thus, the results do not fully substantiate the claim that “correcting scientific errors is significantly harder than identifying them,” as the authors have not reported correction accuracy given gold-standard error detections.
  2. Generalization to unseen data sources remains unclear. The paper splits data at the source-sample level, ensuring that variants derived from the same source image or problem are assigned to the same split. This is a sensible safeguard against direct overlap between variants. However, the training corpus and benchmark draw on overlapping upstream source collections. Therefore, the current evaluation does not establish how well the model generalizes to entirely unseen datasets, generation pipelines, or error distributions. Evaluating on one or more source-disjoint test sets would strengthen the evidence that the proposed approach generalizes beyond the data sources used during development.
  3. The representativeness of the benchmark’s negative examples needs further validation. The benchmark includes negatives produced through visual perturbations and textual prompt modifications. These methods offer a controllable way to construct labeled examples, but it remains unclear how closely the resulting errors reflect the mistakes made by real scientific image-generation systems in practice. Expert review helps establish annotation quality, but does not by itself establish that the error distribution is representative of real-world generation failures.
  4. Aggregate results may obscure substantial variation across sources and error types. The benchmark spans multiple scientific areas and three capability-oriented categories: Instruction Following, Multidisciplinary Reasoning, and World Knowledge. However, performance within these categories may vary substantially across source datasets, perturbation methods, and error types. The aggregate scores therefore provide limited insight into which kinds of scientific errors the model handles reliably and where it remains weak. Per-source and per-error-type results would make the claimed breadth of the approach easier to assess.
Questions for authors
  1. Do you have any quality metrics for the construction of the visual and textual perturbations?
  2. How well do the editing instructions repair images in practice? A human assessment or independent verification of corrected outputs would provide more direct evidence of the practical utility of the proposed approach.
  3. What criteria guided the selection of benchmark sources? Could you clarify the rationale for including certain datasets while excluding others? For example, related datasets such as TQA [1], MMMU [2], MathVista [3], and SciVerse [4] could potentially be used to generate training and evaluation samples, yet they are not included. Could you explain the reasons for these omissions?
  4. Can you report performance broken down by source dataset and error type? This would help clarify whether performance is consistent across the benchmark or concentrated in particular sources and categories.

[1] A. Kembhavi, M. Seo, D. Schwenk, J. Choi, A. Farhadi and H. Hajishirzi, "Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 5376-5384, doi: 10.1109/CVPR.2017.571.

[2] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, Wenhu Chen. MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 9556-9567

[3] Lu, Pan et al. “MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts.” International Conference on Learning Representations (2023).

[4] Guo, Z., Zhang, R., Chen, H., Gao, J., Jiang, D., Wang, J., & Heng, P.-A. (2025). SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problems. arXiv preprint arXiv:2503.10627.

5 comments
AuthorsOpenReview ↗

Response to Reviewer jd1N (1/4)

Dear Reviewer jd1N,

Thank you for your careful review and for recognizing the importance of the problem, the explainable task formulation, and the expert-reviewed benchmark construction. We address each point below. All changes in the revised manuscript are marked in red.

Q1: The editing metric does not isolate editing ability. Acc_i is a cascading metric on negative examples, so a low Acc_i may reflect errors at the judgement or explanation stage rather than an inability to formulate effective editing instructions. The results therefore do not fully substantiate the claim that "correcting scientific errors is significantly harder than identifying them", since correction accuracy given gold-standard error detections is not reported.

A1: Thank you for this thoughtful question. As detailed in Sec. 3.3.2, the reference annotations underlying Acc_e and Acc_i undergo strict expert verification: each sample is independently assessed by two domain-matched experts and retained only upon dual approval, following a multi-dimensional filter on visual fidelity, task difficulty, and annotation correctness alongside a manual refine-and-re-approve stage for minor defects. By design, explanations must reference concrete visual evidence and editing instructions must remain localized and executable (Appendix A.1). In addition, our automated judges achieve κ ≥ 0.76 with three human experts, approaching the 0.83 human–human ceiling, while a third-party judge maintains the exact model ranking (Appendix A.4, Tables 3–4). The cascading design follows OmniVerifier [1]: an editing instruction is practically useful only after a flaw is detected and accurately diagnosed, whereas scoring edits produced from a faulty diagnosis would reward lucky guesses.

To decouple editing capability from upstream verification, Table R1 introduces an oracle evaluation where models receive the ground-truth explanation and only generate the editing instruction. Because Acc_i is computed over negative samples (Appendix A.3), we report Acc_a and Acc_e on the exact same negative subset for direct comparison. Even when provided with the exact error location and cause, models still fail to produce valid editing instructions on 20–26% of instances. This confirms that the performance drop from error identification to correction reflects the intrinsic difficulty of formulating executable edits rather than an artifact of cascading evaluation.

Table R1. Judgement, explanation and editing accuracy on the 560 negative samples, and editing accuracy when the gold explanation is given (oracle).

ModelAcc_a (neg.)Acc_e (neg.)Acc_i (cascaded)Oracle edit
GPT-6-Astra74.670.853.5976.5
Gemini-3.5-Flash79.275.056.5476.1
Qwen3.8-Max84.582.164.7580.3
SciGen-Verifier-8B83.979.257.4274.2

[1] Generative Universal Verifier as Multimodal Meta-Reasoner, ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer jd1N (2/4)

Q2: Generalization to unseen data sources remains unclear. Evaluating on source-disjoint test sets would strengthen the evidence.

A2: Thank you for raising this. Our experiment on image generation in Appendix D.2 (Table 9) tests generalization across two dimensions. The generation prompts from GenExam is not used in training. The images come from Wan2.6 rather than the Qwen-Image-Edit pipeline used for our negative samples. Under this setting, SciGen-Verifier raises the relaxed score from 58.7 to 63.5 on GenExam over ten rounds. It trails the proprietary models Gemini-3.5-Flash (64.9) and Qwen3.8-Max (64.4) by only 1.4 and 0.9 points.

We also run a leave-one-source-out study (Table R2). In each run, we remove one entire data source from SFT and both RL stages and evaluate the resulting model on the full SciGen-Verify benchmark, so it can be compared directly with the SFT-only and full-data models. Removing AI2D lowers overall Acc_a from 86.28 to 86.11, and the drop is concentrated in the Knowledge domain that contains AI2D (94.21 to 93.90). Removing the crawled exam papers lowers overall Acc_a to 85.70 and Acc_i to 56.51, mainly in the Reasoning domain that contains them (Acc_a 89.02 to 87.63, Acc_i 57.84 to 55.52). Both models remain far above the all-source SFT-only baseline (80.09 / 75.32 / 46.72), so the gains of our recipe do not depend on in-source training data. These results are added to Appendix F.4 of the revised manuscript, and Appendix C.1 describes the train–test separation.

Table R2. Leave-one-source-out: a source is removed from all training stages and the model is evaluated on the full SciGen-Verify benchmark (Acc_a / Acc_e / Acc_i).

Training dataInstructionReasoningKnowledgeOverall
SFT only (all data)65.35 / 60.63 / 35.3482.25 / 75.36 / 43.4592.66 / 89.97 / 61.3680.09 / 75.32 / 46.72
SFT + R1 + R2 (all data, ours)75.62 / 70.38 / 48.9389.02 / 83.47 / 57.8494.21 / 91.53 / 65.4886.28 / 81.79 / 57.42
SFT + R1 + R2 (without AI2D)75.48 / 70.27 / 48.8088.95 / 83.38 / 57.7193.90 / 91.22 / 65.0886.11 / 81.62 / 57.20
SFT + R1 + R2 (without exam papers)75.41 / 70.20 / 48.7087.63 / 81.71 / 55.5294.06 / 91.40 / 65.3085.70 / 81.10 / 56.51

Q3: Visual and textual perturbations offer a controllable way to construct labeled examples, but it is unclear how closely the resulting errors reflect the mistakes made by real scientific image-generation systems in practice. Expert review establishes annotation quality, but not that the error distribution is representative of real-world generation failures.

A3: Thank you for raising this point. In preliminary trials, we collected failures directly from image generators. Most outputs were gross failures with collapsed layouts, garbled text, missing objects, or heavy blur. Every baseline verifier caught them easily. Such samples do not test scientific knowledge or structural reasoning, and they make the benchmark easier.

As stated in Sec. 3.3.1, we instead start from a correct reference image. A model acting as an experienced teacher injects a typical student misconception for that concept. The resulting image looks clean, and the error is purely scientific. Examples include series instead of parallel wiring, a shifted angle, and a swapped functional group. These are the exact subtle failures that a verifier is needed for. Knowing where and why the error was injected lets us write a reliable explanation and editing instruction, which harvested failures do not allow.

Before any human review, Gemini-3-Flash removes blurry images, unreadable text, and failed edits. Experts then apply a second Visual Fidelity check (Sec. 3.3.1–3.3.2). Two domain-matched experts must both approve every sample. Our automated judges reach κ ≥ 0.76 against human labels, close to the human-human ceiling of 0.83 (Appendix A.4). The injected errors cover structural, numerical, labelling, and factual mistakes, as shown in Table R4 of our reply to Reviewer DdY8 (Q6). Finally, the test-time scaling experiment on real Wan2.6 outputs (Table 9) shows the verifier also works on real generator errors (see our reply to Q2).

AuthorsOpenReview ↗

Response to Reviewer jd1N (3/4)

Q4: Aggregate results may obscure substantial variation across sources and error types. Can you report performance broken down by source dataset and error type?

A4: Thank you for the suggestion. Table R3 breaks down SciGen-Verifier-8B by data source, by perturbation strategy, and by error type. Performance is consistent across sources and perturbation strategies, so the gains in Table 1 are not concentrated in any particular subset. Among error types, the model is strongest on knowledge-type errors such as reversed energy-flow direction (90.0) and missing or misplaced asymptotes (87.5), and comparatively weaker on fine structural errors such as incorrect occlusion (80.6) and series/parallel topology (83.3).

Table R3. Breakdown of SciGen-Verifier-8B on SciGen-Verify (Acc_a / Acc_e / Acc_i). Error-type rows use the inventory of Table R4 in our reply to Reviewer DdY8 (Q6) and are computed on the negative samples of that type. EMMA contributes positive samples only, so its Acc_i is not defined.

DimensionGroupAcc_aAcc_eAcc_i
SourceV-Interaction76.170.949.6
SourceMathCanvas74.869.547.8
SourceCoSyn91.783.357.1
SourceDaTikZ-V389.784.158.3
SourceEMMA85.771.4—
SourceExam (crawled)88.582.855.8
SourceAI2D93.891.764.7
SourceWikipedia94.391.565.6
PerturbationVisual (code-rendered)85.180.659.3
PerturbationVisual (image-edited)82.777.954.6
PerturbationTextual84.079.256.8
PerturbationPositive (unperturbed)88.484.0—
Error typePhysics: series/parallel topology error83.379.655.6
Error typeMathematics: missing / misplaced asymptote87.582.560.0
Error typeChemistry: wrong functional group86.780.057.8
Error typeMathematics: incorrect occlusion80.675.050.0
Error typeBiology: reversed energy-flow direction90.086.763.3

Q5: Do you have any quality metrics for the construction of the visual and textual perturbations?

A5: Thank you for raising this question. Every perturbed sample passes three quality gates before entering the benchmark. First, Gemini-3-Flash screens out candidates with severe blur, illegible text, or an unapplied edit before any human sees them (Sec. 3.3.1). Second, two domain-matched experts independently check each sample for Visual Fidelity, Task Difficulty, and Annotation Correctness (Sec. 3.3.2). Positive and negative pairs go to separate expert groups to prevent prior cognitive bias. Third, the first expert rewrites minor text defects while any sample with unresolvable ambiguity or a fatal visual error is discarded (Sec. 3.3.2, Appendix A.1). A sample, including any rewrite, is kept only if both experts independently approve it.

Furthermore, the perturbations are not random corruptions. Gemini-3-Pro acts as an experienced teacher and injects a misconception that students commonly hold for that knowledge point (Sec. 3.3.1). This keeps each injected error localized, scientifically meaningful, and clean enough for the editor to apply.

Reviewer_DdY8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a practically valuable problem: how to enable MLLMs to verify the correctness of generated scientific images, explain errors, and provide guidance for image generation models to make corrections. Specifically, given a generation or editing instruction and a candidate image, a verifier predicts correctness, explains its judgment, and supplies editing instructions for incorrect images. The authors introduce SciGen-Verify, a benchmark of 1,350 examples spanning instruction following, multidisciplinary reasoning, and world knowledge, with annotations reviewed by two domain-matched annotators. The proposed SciGen-Verifier is initialized from Qwen3-VL-8B-Instruct, undergoes supervised fine-tuning, and then employs two stages of GRPO with judgment rewards and explanation/editing outcome rewards. Experimental results demonstrate that a specialized 8B verifier can achieve competitive performance compared to large general-purpose models on binary judgment tasks within a scientific image verification benchmark.

Critical strengths

The problem investigated in this paper is valuable, and the task definition is relatively clear. Rather than focusing solely on correctness judgments, the paper extends the task to combine judgment, explanation, and editing suggestions. This is more suitable than a single score for applications such as automated scientific illustration, educational systems, scientific figure review, and automatic improvement of generative models.

The benchmark construction represents a practical contribution. The authors integrate multiple scientific domains and existing resources, construct challenging negative samples through visual and textual perturbations, and have the annotations jointly verified by two reviewers with relevant domain expertise.

The authors use only an 8B model as the final verifier, yet achieve competitive results on binary judgment tasks within the specific benchmark.

The experiments cover recent state-of-the-art commercial models, open-source models, and specialized image verification models, providing valuable references for future research.

Critical weaknesses

The methodological novelty of this paper is limited, mainly consisting of a combination of existing techniques, including SFT, GRPO, rubric-guided rewards, and dynamic reward weighting. Furthermore, the current ablation experiments do not fully demonstrate the necessity of the two-stage curriculum learning strategy.

Rubric Coverage does not sufficiently demonstrate an improvement in genuine scientific reasoning ability.

For example, a model could simply write in its <think> section: "I have checked the battery connections, switch connections, bulb connections, and whether the circuit is closed." If the Coverage Checker primarily relies on textual semantics, such a response might cover multiple checkpoints without providing actual evidence of visual analysis. Even if the Checker requires more specific textual explanations, there is still a possibility that the model learns to satisfy the reward criteria rather than genuinely improving its image understanding ability.

The paper lacks critical training baselines and computational efficiency analysis. Although the authors emphasize that the model has only 8B parameters and therefore offers deployment advantages, they do not sufficiently report end-to-end inference latency, the number of generated tokens, training resources, the overhead of calling the Rubric Generator and LLM Judges, or actual cost comparisons with commercial models.

Questions for authors

Does the two-stage advantage remain when query exposure and rollout/update budgets are matched against judgment-only RL and single-stage mixed rewards? What evidence isolates the dynamic schedule or stage order? For Figure 3, can the effect of CoT be separated from the simultaneous removal of rubric supervision?

In Appendix A.4, what exactly are the 300 instances—tasks, model-output pairs, or judge decisions—and how many remain after restricting to unanimous human labels?

What are the design principles behind the error categories? Is there a more reasonable way to define Severity? Would a larger list of error categories be more beneficial for evaluating the deliverability of generated images?

6 comments
AuthorsOpenReview ↗

Response to Reviewer DdY8 (1/4)

Dear Reviewer DdY8,

Thank you for your careful review and for recognizing the value of the problem, the clarity of the task definition, and the practical contribution of the benchmark. We address each point below. All changes in the revised manuscript are marked in red.

Q1: The methodological novelty of this paper is limited, mainly consisting of a combination of existing techniques, including SFT, GRPO, rubric-guided rewards, and dynamic reward weighting.

A1: Thank you for the constructive comments and for recognizing the clarity of the task formulation and the practical value of the benchmark. Our work focuses on explainable scientific image verification.

(1) We formulate explainable scientific image verification as jointly producing a binary judgement, a fine-grained explanation, and an actionable editing instruction. We also build the expert-curated SciGen-Verify benchmark of 1,350 samples across three domains. Our post-training recipe makes an 8B open-weight model competitive with frontier proprietary systems on this task, reaching 86.28 Acc_a / 81.79 Acc_e / 57.42 Acc_i and outperforming Gemini-3.1-Pro at 81.86 / 77.11 / 54.86 and Seed-2.0-Pro at 79.47 / 72.51 / 53.95 (Table 1).

(2) To design the RL curriculum, we manually inspected 200 failure cases of the cold-start SFT checkpoint in Appendix F.1. We found that 68.5% of errors come from incomplete visual exploration where the decisive checkpoint is never examined in <think>, 22.0% come from misreading an examined checkpoint, and 9.5% come from faulty reasoning over correct observations. In the parallel-circuit problem of Fig. 1, for example, the SFT model checks that the batteries, bulb, and switch exist, but never checks how they are connected. Applying outcome-only supervision directly to the SFT model (SFT + R2 in Table 2) reaches only 82.68 Acc_a / 78.46 Acc_e / 52.78 Acc_i. In contrast, Stage 1 (R1) first expands visual evidence coverage and Stage 2 (R2) then aligns explanation and editing, raising performance to 86.28 / 81.79 / 57.42. Within Stage 1, the mastery-driven schedule (α_t, β_t) serves as a soft relaxation of outcome-gated process supervision (Appendix C.1.1) and prevents rubric reward hacking before judgement accuracy stabilizes. This analysis and the circuit case are given in Appendix F.1 of the revised manuscript.

(3) Computing the Stage-1 process reward R_rubric is non-trivial because general-purpose VLMs miss fine-grained visual details when generating checklists. We therefore built SciGen-Rubric (Appendix B: 17,677 instances, > 77k criteria, > 471k binary checking points) and trained an 8B rubric generator via SFT and GRPO using SciGen-Verifier as the reward source (Appendix D.1.3). In Table 6, our generator reaches 73.88 overall SF1, outperforming Seed-2.0-Pro (64.42), Gemini-3-Flash (58.99), and Qwen3.5-397B (58.13).

(4) The triplet output also allows SciGen-Verifier-8B to guide iterative image refinement at test time. On GenExam (Table 9, Appendix D.2), guiding Wan2.6 with our 8B verifier improves the relaxed score from 58.7 to 63.5 over 10 rounds, matching Qwen3.8-Max (64.4) and surpassing Seed-2.0-Pro (62.6).

AuthorsOpenReview ↗

Response to Reviewer DdY8 (2/4)

Q2: The current ablation experiments do not fully demonstrate the necessity of the two-stage curriculum learning strategy. Does the two-stage advantage remain when query exposure and rollout/update budgets are matched against judgment-only RL and single-stage mixed rewards? What evidence isolates the dynamic schedule or stage order? For Figure 3, can the effect of CoT be separated from the simultaneous removal of rubric supervision?

A2: Thank you for these questions, which we address in three parts below and have added to Appendix F.2 of the revised manuscript.

(1) To ensure a fair comparison, all RL variants are trained under the exact same computation budget (1.36×10⁵ rollouts over 15,685 queries, 184 policy updates) as our two-stage approach. The single-stage mixed reward trains in one stage on the same query pool with all reward terms summed (R_judge + R_rubric + R_exp-edit), where R_exp-edit is computed on the 685 queries that have reference explanations and editing instructions. As shown in Table R1, our method consistently achieves the best performance, outperforming judgement-only RL and single-stage mixed reward by 3.49 and 2.16 in overall Acc_a, with even wider margins on Acc_i (+8.55 and +3.84). Ablations further confirm our design choices. Reversing the stage order (R2 → R1) degrades performance to 83.93 / 78.66 / 52.96, verifying the need to apply process supervision before outcome alignment. Replacing our mastery-driven schedule with fixed coefficients drops Acc_a by 1.07 and Acc_i by 1.87 (to 85.21 / 80.46 / 55.55), highlighting the benefit of dynamic scheduling.

Table R1. Budget-matched RL ablations (Acc_a / Acc_e / Acc_i). Every RL row uses the same 15,685-query pool, K = 8 rollouts per prompt, 1.36×10⁵ rollouts and 184 policy updates.

SettingInstructionReasoningKnowledgeOverall
SFT only65.35 / 60.63 / 35.3482.25 / 75.36 / 43.4592.66 / 89.97 / 61.3680.09 / 75.32 / 46.72
+ judgement-only RL (matched)69.54 / 62.10 / 38.2185.37 / 76.48 / 46.3393.46 / 90.11 / 62.0782.79 / 76.23 / 48.87
+ single-stage mixed reward (matched)72.41 / 66.75 / 44.1286.41 / 80.02 / 52.6093.53 / 90.88 / 64.0284.12 / 79.22 / 53.58
+ R2 → R1 (reversed order)71.33 / 65.41 / 42.9786.76 / 79.61 / 51.7493.70 / 90.95 / 64.1883.93 / 78.66 / 52.96
+ R1 (fixed α, β) + R273.89 / 68.47 / 46.5287.80 / 81.66 / 55.2193.95 / 91.24 / 64.9185.21 / 80.46 / 55.55
+ R1 + R2 (ours)75.62 / 70.38 / 48.9389.02 / 83.47 / 57.8494.21 / 91.53 / 65.4886.28 / 81.79 / 57.42

(2) Using 685 fully annotated queries in Stage 2 is a deliberate design. Because Stage 1 gives reliable judgements and thorough evidence coverage, Stage 2 only aligns how the explanation and edit are written. As shown in Table R2, within the first 20 of 66 policy updates, overall Acc_i rises from 50.47 to 56.78 and then moves by less than 0.7 points, which matches prior findings on small RL data size [1, 2].

Table R2. Convergence of Stage 2. Overall metrics at intermediate checkpoints, where update 0 is the Stage-1 checkpoint.

Policy updates01020304466
Acc_a83.6284.9585.9086.2186.3086.28
Acc_e77.1679.8481.2281.7081.8381.79
Acc_i50.4754.6356.7857.3157.4657.42

[1] LIMR: Less is More for RL Scaling, 2025.

[2] Reinforcement Learning for Reasoning in Large Language Models with One Training Example, NeurIPS 2025.

(3) As stated in footnote 1 of Sec. 5.2, the w/o CoT model in Fig. 3 has no <think> field, so the rubric reward, which scores the <think> trace, cannot be applied there. To separate the two factors, we add a w/ CoT + judgement-only RL baseline with the same reward and budget as the w/o CoT model (Table R3). Rows 1 and 2 differ only in CoT, which adds +0.9 Acc_a, +8.9 Acc_e and +11.0 Acc_i. Rows 3 and 4 both use the Stage-2 outcome reward and differ only in whether the rubric stage R1 is applied, which adds +3.6 Acc_a, +3.3 Acc_e and +4.6 Acc_i. The full curriculum adds +3.5 Acc_a, +5.6 Acc_e and +8.5 Acc_i over row 2. Both factors contribute: CoT mainly improves explanation and editing, while the rubric-guided curriculum mainly improves judgement.

Table R3. Separating the effect of CoT from that of rubric supervision (overall metrics). Rows 1, 2 and 4 use the same matched budget (1.36×10⁵ rollouts, 184 policy updates), and row 1 uses the same reward and budget as row 2. Row 3 is SFT + R2 from Table 2 and is not budget-matched.

SettingCoTR1 (rubric)R2 (exp/edit)Acc_aAcc_eAcc_i
w/o CoT, SFT + judgement-only RL✗✗✗81.967.337.9
w/ CoT, SFT + judgement-only RL✓✗✗82.876.248.9
w/ CoT, SFT + R2✓✗✓82.778.552.8
w/ CoT, SFT + R1 + R2 (ours)✓✓✓86.381.857.4

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.