acceptodds
Under review as a conference paper at ICLR 2027

Self-Verifying Vision Language Models

Abstract

Reinforcement learning with verifiable rewards has emerged as a powerful paradigm for improving reasoning in structured domains such as mathematics and code, where deterministic checkers can directly certify generated solutions. Extending this paradigm to visual reasoning is difficult because correctness requires interpreting unstructured visual evidence, and ground-truth is scarce at scale. We introduce a training framework for vision-language models (VLMs) that makes verification a learned capability rather than relying on exact rewards to verify final answers. The model generates a candidate solution, critiques its own reasoning, and then refines its answer, while a simple proxy reward evaluates the resulting generation–verification pair. We find that jointly training for generation and verification creates a positive feedback loop during training that yields more scalable visual reasoning than optimizing answer correctness alone, even when self-verification is disabled at test time. Across diverse benchmarks, our framework improves the base VLM, outperforms answer-supervised training, and surpasses test-time scaling baselines, with the largest gains on 3D-grounded reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
self-verification, vlms, visual reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dZb6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a training framework for VLMs that trains the model with GRPO on a generate-critique-revise rollout. The reward consists of three components: final answer correctness, Gemini's score on the model's self-critique, and Gemini's rating of the feedback's usefulness, measured by comparing the before and after answers.

Critical strengths
  • Well written and easy to follow.
  • The method was validated on multiple benchmarks with solid baselines: answer-only RL with the same judge, SFT on Gemini labels, and an N=0 inference variant.
  • Seed variance is reported for the main comparison (Table 8), and the method replicates on a second model (Molmo2).
Critical weaknesses
  • The method is really an RLAIF approach with Gemini as the judge, so framing it as self-critique is a bit of an overclaim. During training, Gemini scores the initial answer, the critique, and the final answer, and even writes the critique itself in the early stages. The verifier learns to produce critiques that Gemini rewards.
  • The paper should include a true self-critique baseline so we can see how much of the gain depends on external review. Two baselines would be interesting: 1) the same base model running the same generate-critique-revise loop without training, which would separate the effect of training from the effect of running the critique loop; and 2) Self-Rewarding Language Models [1], the same generate-critique-revise loop but with the model judging its own answers and critiques instead of Gemini.
  • The self-critique contribution may not be as large as the authors claim. GenVer with N=0 basically ties with SFT distillation everywhere except RealworldQA, and GenVer with N=1 actually hurts on some tasks (CV-B -1.2 and VSR -1.9). There's no clear way to tell when to use N=0 versus N=1.

Assembled from Tables 1-3:

Omni3DRoboSp.RWQAOmniSp.CV-BBLINKVSR
Gemini-3-Flash62.374.384.363.389.891.682.0
GPT-5-mini40.976.362.254.986.081.084.4
Base (Qwen3-VL 4B)26.060.963.142.375.281.168.6
+ SFT on Gemini (T3)39.378.160.745.982.283.376.9
+ RL, answer reward, direct (T2)37.978.964.644.978.584.681.1
+ GenVer, N=0 (T2)40.479.667.545.182.883.977.5
+ GenVer, N=1 (T1)40.781.569.747.281.683.875.6

[1] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024.

Questions for authors
  • Figure 4 shows the verification reward aligns well with reasoning capability (final accuracy) on Omni3D. Could you plot the critique score and usefulness separately over training? That would show how much the critique training itself contributes.
  • The current reward design essentially treats "keeping a right answer" and "fixing a wrong answer" the same. Did you try rewarding fixing a wrong answer the most, keeping an answer neutral, and penalizing turning a right answer wrong? That would tell us whether the critique signal is being used effectively.
6 comments
AuthorsOpenReview ↗

Official Review of Submission7637 by Reviewer dZb6

We thank the reviewer for the careful analysis and are happy to hear you found the paper “well written and easy to follow.” We clarify below the meaning of self-critique, the evidence for its contribution, and the rationale behind the reward design. We will revise the camera-ready manuscript to reflect these clarifications.

The method is really an RLAIF approach with Gemini as the judge, so framing it as self-critique is a bit of an overclaim. During training, Gemini scores the initial answer, the critique, and the final answer, and even writes the critique itself in the early stages. The verifier learns to produce critiques that Gemini rewards.

Self-critique and training supervision. GenVer uses externally supplied rewards to learn self-critique. We describe this extensively in the paper (L149-152,179-203,212-213). We believe the apparent disagreement is purely terminological: RLAIF describes the source of training supervision, while self-critique describes the capability being trained. These are compatible descriptions of the same framework. Here, “self-critique” refers to the policy generating feedback on its own solution and using that feedback to revise its reasoning and answer. The source of the training reward and the source of the critique are distinct: Gemini evaluates the training outputs, while the policy learns to produce and act on its own critiques. We clarify that Gemini-generated critiques provide an initialization curriculum whose substitution probability decays to zero by the end of the first epoch (L174-178). At inference, generation, critique, and revision are performed entirely by the trained policy.

The current reward design essentially treats "keeping a right answer" and "fixing a wrong answer" the same. Did you try rewarding fixing a wrong answer the most, keeping an answer neutral, and penalizing turning a right answer wrong? That would tell us whether the critique signal is being used effectively.

Assigning equal usefulness reward to improvement and preservation of an already high-quality answer is deliberate. Both constitute successful verification: the critique should guide correction when needed and preserve a strong solution when revision is unnecessary. Giving correction strictly greater reward would rank an initially incorrect solution that is subsequently repaired above an initially correct solution that is successfully preserved, even when both reach the same final answer quality. Our objective treats these as equally desirable outcomes.

GenVer with N=0 basically ties with SFT distillation everywhere except RealworldQA, and GenVer with N=1 actually hurts on some tasks (CV-B -1.2 and VSR -1.9). There's no clear way to tell when to use N=0 versus N=1.

Quantitative results. The central question we explore in this work is whether explicitly optimizing verification improves visual reasoning beyond optimizing answer rewards alone (L237-240). Table 1 answers this question directly. In Table 1, Answer Reward uses the same generation-verification rollout structure (L252-255) and the same proxy reward model as GenVer. Addressing your concern of this baseline coming “at a lower training cost,” the Answer Reward baseline in Table 1 includes the verification rollouts and therefore includes their computational cost. This comparison isolates the benefit of explicitly rewarding verification while holding the rollout structure fixed. From Table 1, adding the verification objective improves six of seven benchmarks, including +4.6 points on Omni3D-Bench and +2.3 on RealWorldQA. These gains therefore extend beyond the benefit of including refinement steps.

Crucially, the benefit also persists when disabling verification at inference. Table 2 reports improvements over direct Answer Reward training of +2.5 points on Omni3D-Bench, +2.9 on RealWorldQA, and +4.3 on CV-Bench, with identical inference structure. Thus, joint training improves the ability to generate answers. We believe these two results to be the most important results in the paper, as it directly tackles our central research question.

Against SFT, which you reference, the gains extend beyond Omni3D-Bench: +9.0% points on RealWorldQA and +3.4% on RoboSpatial. We agree that the magnitude varies across benchmarks; our evidence supports the strongest benefits on complex grounded reasoning. We acknowledge this in our manuscript (L351-353).

AuthorsOpenReview ↗

Official Review of Submission7637 by Reviewer dZb6 continued

Could you plot the critique score and usefulness separately over training? That would show how much the critique training itself contributes.

As requested, we plot the two components separately in Appendix A.2. The critique score increases from approximately 0.65 to 0.79 over training. Usefulness remains high throughout: following an initial decline, it recovers and stabilizes around 0.79 as critique quality continues to improve. Thus, increasing critique quality is accompanied by sustained outcome-based usefulness, which rewards feedback that improves a solution or preserves an already high-quality answer. This pattern complements Table 5’s behavioral comparison, where replacing learned critiques with base-model critiques reduces performance, supporting the contribution of the learned feedback.

Two baselines would be interesting: 1) the same base model running the same generate-critique-revise loop without training, which would separate the effect of training from the effect of running the critique loop; and 2) Self-Rewarding Language Models [1], the same generate-critique-revise loop but with the model judging its own answers and critiques instead of Gemini.

We appreciate these suggestions and view them as valuable directions for future work. Our core question is whether explicitly optimizing verification improves visual reasoning beyond optimizing answer rewards alone. Our experimental setting directly addresses this question: in Table 1, Answer Reward and GenVer use the same generate–critique–revise rollout structure and the same external reward model, isolating the contribution of the verification objective. Table 2 further shows that the benefits persist when critique and revision are disabled at inference, demonstrating improvements to answer generation itself. While these are excellent suggestions for future work, we feel our suite of experiments tackles the question we pose directly.

Thanks for adding Figure 8. It's helpful, though I read it a bit differently. Usefulness ends below where it started (about 0.85 at step 20, dropping to about 0.76, and ending near 0.79), while the critique score rises steadily from 0.65 to 0.79. This fits my first concern: the critiques improve mainly in how Gemini scores them, more than in how much they improve the answers. Could you report how often revision turns a wrong answer right and a right answer wrong, measured against ground truth (Question 2)? That would show how much the learned critiques actually improve the final answers.

Reviewer_5ZcEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a VLM training framework where the model learns to self-verify for visual reasoning tasks without labels. The model generates a reasoning trace and an answer. Subsequently, it iteratively refines the solution by producing feedback to itself and using it to generate a revised solution for N rounds. Optimisation operates on both the final answer quality, which is operationalised by a VLM judge that grades the answer, and on the answer verification quality, which uses an LLM judge to score whether the feedback improved the answer in the subsequent round. The authors evaluate their method using a small (4B) Qwen model against 5 baselines across 7 tasks. The authors also provide evidence that their method outperforms direct answer supervision in some of the tasks.

Critical strengths
  • The method is intuitive and straightforward.
Critical weaknesses
  • The method depends on a plethora of modelling choices, but none of them are ablated (see questions). I have concerns regarding the robustness of the method.
  • There are leakage concerns. The method trains on SA-1B and the COCO training split. VSR draws from the COCO training and validation split. BLINK is derived from VSR.
  • The method is experimentally validated using a single, small model. This is insufficient to draw conclusions about the method.

Minor (not influencing my decision):

  • Eq (3)'s components are never defined.
  • (L131-132) ''GRPO; a policy gradient method for tuning VLMs with RL'' is imprecise
Questions for authors

How were the following choices made:

  • The sampling curriculum, including the probability p and the linear decay.
  • The 0.1 scalar in RansR_{\text{ans}}, along with the grading scale for the high-capacity VLM.
  • How were the numbers of u(t)u^{(t)} chosen? What is the noise floor of V\mathcal{V}? Can it distinguish minute differences in the model’s own feedback? Does it change its predictions when the feedback is simply rewritten?
  • How were the convex combination weights of Rver(t)R_{\text{ver}}^{(t)} chosen?

In general, how sensitive is the method to these choices?


  • Did you check for training example leakage? Even without labels, familiarisation with the images could inflate the results.
  • Were any judge biases mitigated? LLMs as judges are known to exhibit length and stylistic biases [1,2,3]. VLMs have been shown to exhibit informativeness biases [4]. Using them in the optimisation loop could reinforce these superficial patterns.
  • How much of the performance gain can be attributed to format adherence?
  • Table 4 compares to traditional test-time scaling baselines, but the method itself is much closer to test-time training regimes (TTRL [5]). Could you provide comparisons with test-time training baselines?
  • In the direct prediction VLM comparison, can you provide the full experimental setup and results? (L372-377).

[1] An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4 (Huang et al., Findings 2025)

[2] Humans or LLMs as the Judge? A Study on Judgement Bias (Chen et al., EMNLP 2024)

[3] OffsetBias: Leveraging Debiased Data for Tuning Evaluators (Park et al., Findings 2024)

[4] When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias (Zou et al., ACL 2026)

[5] TTRL: Test-Time Reinforcement Learning (Zuo et al., NeurIPS 2025)

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer 5ZcE

We thank the reviewer for their thorough and constructive review. We are glad you found the method intuitive. We address your comments below and will incorporate these clarifications in the revised manuscript.

The method is experimentally validated using a single, small model.

We point the reviewer to Table 6, which includes results for Molmo2 and we believe the reviewer has missed. Our method is experimentally validated across two model families: Qwen3-VL and Molmo2. Molmo2 trained with the same GenVer recipe, showing improvements across all seven benchmarks, including RoboSpatial from 66.3 to 79.8, RealWorldQA from 66.4 to 69.7, and VSR from 63.4 to 77.4. These results demonstrate that the gains are not confined to Qwen3-VL.

There are leakage concerns. The method trains on SA-1B and the COCO training split. VSR draws from the COCO training and validation split. BLINK is derived from VSR.

We thank the reviewer for this legitimate concern. We were aware of this potential leakage, and selected exclusively images from the COCO training split that did not overlap with VSR. We will include this detail in the updated manuscript.

The method depends on a plethora of modelling choices, but none of them are ablated (see questions).

We would like to re-iterate the role of our experiments. Our goal is to answer a fundamental question: does jointly optimizing generation and verification improve visual reasoning beyond optimizing answer generation alone? We are not proposing an exhaustively optimized model or claiming the highest achievable benchmark performance. Ablating every numerical choice would be computationally expensive and might identify configurations that gain additional percentage points, but improving absolute performance would not itself answer this scientific question. The relevant evidence is a controlled comparison that isolates the contribution of verification training.

This is precisely what our experimental setting provides. Table 1 holds the training data, backbone, rollout structure, and proxy reward model fixed while changing whether verification is explicitly rewarded. Table 2 shows that the benefits persist without verification at inference. Tables 5–7 further examine learned versus base-model feedback, a second backbone family, and different training-time verification rounds. We therefore believe these experiments directly support the central contribution; an exhaustive investigation of every hyperparameter and coefficient, while likely to lead to additional performance gain, does not further the cause of establishing that contribution.

How much of the performance gain can be attributed to format adherence?

Evidence points to the performance gain not being attributed to format adherence. We include a table below of format-adherence at each training step. Format adherence is already near the ceiling at the beginning of training: 99.6% of responses are well formatted within the first 40 steps. It remains between 98.5% and 100% throughout training, with no sustained upward trend. The observed accuracy gains therefore occur without a corresponding improvement in formatting. Moreover, the Answer Reward baseline uses the same rollout structure, formatting requirements, and format constraint as GenVer. Format supervision is therefore shared by both methods; GenVer’s additional gains arise from explicitly optimizing verification under this common protocol.

Training stepsWell-formatted responses
1–4099.6%
41–8099.8%
81–12099.0%
121–16098.5%
161–20099.2%
201–24099.2%
241–247100.0%
AuthorsOpenReview ↗

Official Response to Reviewer 5ZcE continued

Table 4 compares to traditional test-time scaling baselines, but the method itself is much closer to test-time training regimes (TTRL [5]). Could you provide comparisons with test-time training baselines?

We do not believe GenVer is closer to test-time training regimes. GenVer completes training before benchmark evaluation and keeps model parameters fixed throughout inference. Its generate–critique–revise procedure refines the solution through additional forward passes, without gradient updates or external supervision at test time. TTRL [5], in contrast, updates model parameters using unlabeled test data.

Table 4 therefore compares inference procedures under the same fixed-parameter setting: self-verification, majority voting, and LLM-as-a-judge selection. A test-time training baseline would introduce parameter adaptation on evaluation data, changing both the information available for learning and the computational protocol. We will clarify this distinction between training a model to self-verify and training the model at test time.

In the direct prediction VLM comparison, can you provide the full experimental setup and results? (L372-377).

L372–377 summarize the direct-prediction results reported in Table 1 across all seven benchmarks. These baselines generate reasoning and a final answer directly from the image–query input, without iterative verification (L249–251). Section 4 describes the evaluation framework, Appendix A.3 specifies the benchmark subsets and scoring metrics, and Appendix A.6 provides the generation prompt. Sampling settings for Qwen3-VL are documented in Appendix A.5.

I thank the authors for the detailed response.

I apologise for overlooking Table 6. However, my concern remains unresolved. Both backbones are in the same small-scale regime.

Regarding the ablations, I think my request was misread. I am not interested in an exhaustive sweep to get the best absolute performance possible. I am mainly interested in robustness. I cannot tell whether the observed effect is a property of the method or this specific configuration.

My concern regarding judge and the judge biases remains unaddressed. The model may be learning superficial patterns the judge prefers. Furthermore, the reward is only meaningful if the judge scores feedback consistently, which is why I asked about the noise floor.

Therefore, I maintain my current assessment.

Reviewer_9ixNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce a training framework that jointly optimizes both generation and self-verification for visual reasoning. The same model is used for training and verification, and Gemini-3-Flash is used as a proxy reward model for both answer quality and verification quality. The authors evaluate their trained model (based on Qwen3-VL 4B) on 7 visual reasoning benchmarks. They show some improvements on more complex benchmarks compared to SFT and a generation-only reward. However, the method seems to break down for simple benchmarks (with yes/no answers) since, as they note, the model ends up in an "over-verification" loop that can degrade initially correct responses.

Critical strengths
  • The paper addresses an important problem in vision-language reasoning, namely the lack of deterministic verifiers, which have enabled significant progress in other areas like math and code.
  • The method shows substantial improvements on Omni3D-Bench.
  • The paper reports a comprehensive set of baselines, including distillation, generation-only reward, and traditional test-time scaling baselines.
  • I appreciate the error analysis, in particular the identification of the prominent failure case of over-verification.
Critical weaknesses
  • Beyond Omni3D-Bench, the method shows modest improvement at best, and sometimes underperforms the baselines. Even on Omni3D-Bench, the improvement over SFT distillation is 1.4 points (Table 3), and the answer reward alone seems to bring most of the gains at a lower training cost (Table 1).
  • Even though the method attempts to mitigate the over-verification issue using the usefulness reward, it does not seem effective. It would be interesting to identify other ways to overcome this limitation.
  • There is little to no experimentation on how different reward models affect performance, which seems particularly important given that the current reward model appears quite noisy (Appendix A.2).
  • Previous work has tackled a similar method, focusing on using RL and a proxy VLM reward to jointly optimize answer generation and self-verification, and is not discussed in the paper:

Li, Zongxia, et al. "Vision-SR1: Self-rewarding vision-language model via reasoning decomposition and multi-reward policy optimization." International Conference on Learning Representations. 2026.

Xinyu Qiu, Heng Jia, Zhengwen Zeng, Shuheng Shen, Changhua Meng, Yi Yang, Linchao Zhu; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 36916-36925

Questions for authors
  • What is the role of the usefulness reward if it does not mitigate the over-verification issue? What happens if it is removed from the training?
  • How do different reward models affect performance?
3 comments
AuthorsOpenReview ↗

Official Response to Reviewer 9ixN

We thank the reviewer for their detailed and constructive review. We are glad you recognize the importance of learned verification and the value of our baseline comparisons and failure analysis. We address your comments below and will incorporate these clarifications in the revised manuscript.

Beyond Omni3D-Bench, the method shows modest improvement at best, and sometimes underperforms the baselines. [...] The answer reward alone seems to bring most of the gains at a lower training cost (Table 1)

Quantitative results. The central question we explore in this work is whether explicitly optimizing verification improves visual reasoning beyond optimizing answer rewards alone (L237-240). Table 1 answers this question directly. In Table 1, Answer Reward uses the same generation-verification rollout structure (L252-255) and the same proxy reward model as GenVer. Addressing your concern of this baseline coming “at a lower training cost,” the Answer Reward baseline in Table 1 includes the verification rollouts and therefore includes their computational cost. This comparison isolates the benefit of explicitly rewarding verification while holding the rollout structure fixed. From Table 1, adding the verification objective improves six of seven benchmarks, including +4.6 points on Omni3D-Bench and +2.3 on RealWorldQA. These gains therefore extend beyond the benefit of including refinement steps.

Crucially, the benefit also persists when disabling verification at inference. Table 2 reports improvements over direct Answer Reward training of +2.5 points on Omni3D-Bench, +2.9 on RealWorldQA, and +4.3 on CV-Bench, with identical inference structure. Thus, joint training improves the ability to generate answers. We believe these two results to be the most important results in the paper, as it directly tackles our central research question.

Even on Omni3D-Bench, the improvement over SFT distillation is 1.4 points (Table 3), and the answer reward alone seems to bring most of the gains at a lower training cost (Table 1).

Against SFT, which you reference, the gains extend beyond Omni3D-Bench: +9.0% points on RealWorldQA and +3.4% on RoboSpatial. We agree that the magnitude varies across benchmarks; our evidence supports the strongest benefits on complex grounded reasoning. We acknowledge this in our manuscript (L351-353).

Even though the method attempts to mitigate the over-verification issue using the usefulness reward, it does not seem effective. It would be interesting to identify other ways to overcome this limitation.

We first clarify the purpose of the usefulness reward. The feedback-quality score evaluates a critique given the current solution, before observing the revised solution. Usefulness applies a different criterion: does applying that critique improve the answer or preserve an already high-quality answer? We do not view usefulness as exclusively a term to limit oververification. The primary purpose of usefulness is to make the verification reward sensitive to the consequences of the proposed feedback.

We disagree that the limitation of oververification implies this reward is not effective. A reward that encourages critiques to be useful is unlikely to make all feedback useful. As stated in the manuscript, we view this as a limitation (L472), but do not feel it establishes that the reward provides no benefit. Our comparisons establish the benefit of optimizing for verification as a whole. Table 5 provides additional evidence that learned feedback is useful: replacing it with base-model feedback reduces Omni3D-Bench performance from 40.7 to 36.3.

AuthorsOpenReview ↗

Official Response to Reviewer 9ixN continued

There is little to no experimentation on how different reward models affect performance, which seems particularly important given that the current reward model appears quite noisy (Appendix A.2).

Our key comparisons keep the reward model fixed: both Answer Reward and GenVer use Gemini-3-Flash. All benchmarks are evaluated against ground truth, not the reward model scores. Thus, the observed gains reflect improvements on the evaluated tasks despite an imperfect proxy. We remind the reviewer of our central research question: does jointly optimizing generation and verification improve visual reasoning beyond optimizing answer generation alone? Holding the reward model fixed deliberately isolates this question under a common source of reward signal.

Previous work has tackled a similar method, focusing on using RL and a proxy VLM reward to jointly optimize answer generation and self-verification, and is not discussed in the paper:

We thank the reviewers for the references and we will incorporate them explicitly in our Related Works for the camera-ready manuscript. Both share our motivation of jointly optimizing complementary capabilities, with two important distinctions.

First, the role of verification differs. Vision-SR1 learns visual descriptions that contain sufficient information to recover the answer without revisiting the image. ADPO learns verification scores used to rank candidate answers and select among them. Both methods use auxiliary assessments to improve answer generation, but neither explicitly trains the model to critique an existing solution and use that critique to revise its reasoning and answer. On the other hand, GenVer trains a model to produce textual critiques of an existing solution, which the model then uses to revise its reasoning and answer. Its verification objective therefore rewards actionable feedback that directly guides refinement.

Second, both prior works use ground-truth answers to compute training rewards, whereas GenVer trains on image–question pairs without reference answers. We will revise the related work discussion to acknowledge these connections and clarify the differences in verification mechanism and supervision

Thank you for your detailed response. I am inclined to maintain my score since I am still not persuaded about the benefit of the method over SFT due to the variability of the results. On lines 69-71, the claim is that the largest gains are in 3D-grounded reasoning, but the SFT comparisons do not seem to support this. While the method is worth pursuing, there may be additional work that needs to be done to highlight its benefits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.