- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a training framework for VLMs that trains the model with GRPO on a generate-critique-revise rollout. The reward consists of three components: final answer correctness, Gemini's score on the model's self-critique, and Gemini's rating of the feedback's usefulness, measured by comparing the before and after answers.
Critical strengths
- Well written and easy to follow.
- The method was validated on multiple benchmarks with solid baselines: answer-only RL with the same judge, SFT on Gemini labels, and an N=0 inference variant.
- Seed variance is reported for the main comparison (Table 8), and the method replicates on a second model (Molmo2).
Critical weaknesses
- The method is really an RLAIF approach with Gemini as the judge, so framing it as self-critique is a bit of an overclaim. During training, Gemini scores the initial answer, the critique, and the final answer, and even writes the critique itself in the early stages. The verifier learns to produce critiques that Gemini rewards.
- The paper should include a true self-critique baseline so we can see how much of the gain depends on external review. Two baselines would be interesting: 1) the same base model running the same generate-critique-revise loop without training, which would separate the effect of training from the effect of running the critique loop; and 2) Self-Rewarding Language Models [1], the same generate-critique-revise loop but with the model judging its own answers and critiques instead of Gemini.
- The self-critique contribution may not be as large as the authors claim. GenVer with N=0 basically ties with SFT distillation everywhere except RealworldQA, and GenVer with N=1 actually hurts on some tasks (CV-B -1.2 and VSR -1.9). There's no clear way to tell when to use N=0 versus N=1.
Assembled from Tables 1-3:
| Omni3D | RoboSp. | RWQA | OmniSp. | CV-B | BLINK | VSR | |
|---|---|---|---|---|---|---|---|
| Gemini-3-Flash | 62.3 | 74.3 | 84.3 | 63.3 | 89.8 | 91.6 | 82.0 |
| GPT-5-mini | 40.9 | 76.3 | 62.2 | 54.9 | 86.0 | 81.0 | 84.4 |
| Base (Qwen3-VL 4B) | 26.0 | 60.9 | 63.1 | 42.3 | 75.2 | 81.1 | 68.6 |
| + SFT on Gemini (T3) | 39.3 | 78.1 | 60.7 | 45.9 | 82.2 | 83.3 | 76.9 |
| + RL, answer reward, direct (T2) | 37.9 | 78.9 | 64.6 | 44.9 | 78.5 | 84.6 | 81.1 |
| + GenVer, N=0 (T2) | 40.4 | 79.6 | 67.5 | 45.1 | 82.8 | 83.9 | 77.5 |
| + GenVer, N=1 (T1) | 40.7 | 81.5 | 69.7 | 47.2 | 81.6 | 83.8 | 75.6 |
[1] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024.
Questions for authors
- Figure 4 shows the verification reward aligns well with reasoning capability (final accuracy) on Omni3D. Could you plot the critique score and usefulness separately over training? That would show how much the critique training itself contributes.
- The current reward design essentially treats "keeping a right answer" and "fixing a wrong answer" the same. Did you try rewarding fixing a wrong answer the most, keeping an answer neutral, and penalizing turning a right answer wrong? That would tell us whether the critique signal is being used effectively.
6 comments
Official Review of Submission7637 by Reviewer dZb6 continued
Could you plot the critique score and usefulness separately over training? That would show how much the critique training itself contributes.
As requested, we plot the two components separately in Appendix A.2. The critique score increases from approximately 0.65 to 0.79 over training. Usefulness remains high throughout: following an initial decline, it recovers and stabilizes around 0.79 as critique quality continues to improve. Thus, increasing critique quality is accompanied by sustained outcome-based usefulness, which rewards feedback that improves a solution or preserves an already high-quality answer. This pattern complements Table 5’s behavioral comparison, where replacing learned critiques with base-model critiques reduces performance, supporting the contribution of the learned feedback.
Two baselines would be interesting: 1) the same base model running the same generate-critique-revise loop without training, which would separate the effect of training from the effect of running the critique loop; and 2) Self-Rewarding Language Models [1], the same generate-critique-revise loop but with the model judging its own answers and critiques instead of Gemini.
We appreciate these suggestions and view them as valuable directions for future work. Our core question is whether explicitly optimizing verification improves visual reasoning beyond optimizing answer rewards alone. Our experimental setting directly addresses this question: in Table 1, Answer Reward and GenVer use the same generate–critique–revise rollout structure and the same external reward model, isolating the contribution of the verification objective. Table 2 further shows that the benefits persist when critique and revision are disabled at inference, demonstrating improvements to answer generation itself. While these are excellent suggestions for future work, we feel our suite of experiments tackles the question we pose directly.
Thanks for adding Figure 8. It's helpful, though I read it a bit differently. Usefulness ends below where it started (about 0.85 at step 20, dropping to about 0.76, and ending near 0.79), while the critique score rises steadily from 0.65 to 0.79. This fits my first concern: the critiques improve mainly in how Gemini scores them, more than in how much they improve the answers. Could you report how often revision turns a wrong answer right and a right answer wrong, measured against ground truth (Question 2)? That would show how much the learned critiques actually improve the final answers.
Official Review of Submission7637 by Reviewer dZb6
We thank the reviewer for the careful analysis and are happy to hear you found the paper “well written and easy to follow.” We clarify below the meaning of self-critique, the evidence for its contribution, and the rationale behind the reward design. We will revise the camera-ready manuscript to reflect these clarifications.
Self-critique and training supervision. GenVer uses externally supplied rewards to learn self-critique. We describe this extensively in the paper (L149-152,179-203,212-213). We believe the apparent disagreement is purely terminological: RLAIF describes the source of training supervision, while self-critique describes the capability being trained. These are compatible descriptions of the same framework. Here, “self-critique” refers to the policy generating feedback on its own solution and using that feedback to revise its reasoning and answer. The source of the training reward and the source of the critique are distinct: Gemini evaluates the training outputs, while the policy learns to produce and act on its own critiques. We clarify that Gemini-generated critiques provide an initialization curriculum whose substitution probability decays to zero by the end of the first epoch (L174-178). At inference, generation, critique, and revision are performed entirely by the trained policy.
Assigning equal usefulness reward to improvement and preservation of an already high-quality answer is deliberate. Both constitute successful verification: the critique should guide correction when needed and preserve a strong solution when revision is unnecessary. Giving correction strictly greater reward would rank an initially incorrect solution that is subsequently repaired above an initially correct solution that is successfully preserved, even when both reach the same final answer quality. Our objective treats these as equally desirable outcomes.
Quantitative results. The central question we explore in this work is whether explicitly optimizing verification improves visual reasoning beyond optimizing answer rewards alone (L237-240). Table 1 answers this question directly. In Table 1, Answer Reward uses the same generation-verification rollout structure (L252-255) and the same proxy reward model as GenVer. Addressing your concern of this baseline coming “at a lower training cost,” the Answer Reward baseline in Table 1 includes the verification rollouts and therefore includes their computational cost. This comparison isolates the benefit of explicitly rewarding verification while holding the rollout structure fixed. From Table 1, adding the verification objective improves six of seven benchmarks, including +4.6 points on Omni3D-Bench and +2.3 on RealWorldQA. These gains therefore extend beyond the benefit of including refinement steps.
Crucially, the benefit also persists when disabling verification at inference. Table 2 reports improvements over direct Answer Reward training of +2.5 points on Omni3D-Bench, +2.9 on RealWorldQA, and +4.3 on CV-Bench, with identical inference structure. Thus, joint training improves the ability to generate answers. We believe these two results to be the most important results in the paper, as it directly tackles our central research question.
Against SFT, which you reference, the gains extend beyond Omni3D-Bench: +9.0% points on RealWorldQA and +3.4% on RoboSpatial. We agree that the magnitude varies across benchmarks; our evidence supports the strongest benefits on complex grounded reasoning. We acknowledge this in our manuscript (L351-353).