acceptodds
Under review as a conference paper at ICLR 2027

Self-Verifiable Multimodal Reasoning via Adversarial Co-Evolution

Abstract

Reliable multimodal reasoning requires answers that are supported by the model's own reasoning. A strong external judge can supervise this consistency, but evaluating every rollout with a large model is costly. A compact policy that verifies its own output offers an attractive alternative, yet its judgments require a reliable learning signal. We propose **Self-Verifiable Policy Optimization (SVO)**, which transfers external verification supervision into a co-evolving discriminator and feeds its learned score back into policy training. The policy generates a reasoning trace, an answer, and a self-verification judgment. Periodically, a frozen judge supplies verification references for a small subset of current rollouts; the discriminator learns to prefer these references and provides both a dense reward and a gate on answer-correctness rewards. This couples learning to assess consistency with learning to generate consistent responses, while reducing repeated large-judge calls. Across four Qwen2.5-VL and Qwen3-VL backbones and seven multimodal math benchmarks, SVO improves reasoning–answer consistency and consistency-constrained accuracy over GRPO by **14.7 and 6.2 percentage points** on average. It approaches direct online judge supervision, with gaps of 1.1 and 0.6 points, respectively. Comparisons with consistency-aware baselines and additional reasoning tasks support SVO as a practical approach to learning from external verification with fewer judge queries.

Primary area:
reinforcement learning
Keywords:
multimodal reasoning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dTbQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Verifiable Policy Optimization (SVO), a framework designed to improve reasoning–answer consistency in multimodal reasoning while reducing dependence on expensive external verification. The method introduces a co-evolving discriminator trained to distinguish policy-generated verification outputs from judge-aligned references. The discriminator provides a dense consistency reward and gates answer-correctness rewards during GRPO training. Periodic updates using a frozen external judge allow the discriminator to adapt to the evolving policy. Experiments across four vision-language backbones and seven multimodal math benchmarks demonstrate improvements in reasoning–answer consistency (RAC) and consistency-constrained accuracy (CCA) over conventional outcome-based RL methods.

Critical strengths
  1. Written in a clear and accessible manner.
  2. Extensive empirical evaluation with consistent improvements.
Critical weaknesses
  1. Limited innovation
  2. Significant additional inference required
  3. Unreliable co-evolution; I suspect the two might end up "hacking" each other
Questions for authors

Please take a look at "weakness."

Reviewer_HLsSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Self-Verifiable Policy Optimization (SVO), a reinforcement learning framework designed to improve reasoning–answer consistency in multimodal reasoning. SVO employs a co-evolving discriminator trained with periodically refreshed supervision from an external judge. The discriminator provides consistency rewards and gates answer-correctness rewards during policy optimization, reducing the need for frequent external judge evaluations. Experiments across four vision-language backbones and seven multimodal math benchmarks demonstrate improvements in reasoning–answer consistency (RAC) and consistency-constrained accuracy (CCA) over GRPO and DAPO.

Critical strengths
  1. Important problem formulation. The paper addresses the discrepancy between answer correctness and reasoning–answer consistency, a meaningful limitation of outcome-based reinforcement learning.
  2. Interesting optimization framework. The proposed co-evolving discriminator enables the policy to learn from external verification feedback without requiring judge evaluations for every rollout. The combination of consistency rewards and gated correctness rewards is technically interesting.
  3. Strong empirical results. The method is evaluated across four backbones and seven benchmarks, achieving average improvements of 14.7 percentage points in RAC and 6.2 points in CCA over GRPO. The additional ablations and comparisons support the effectiveness of the proposed framework.
Critical weaknesses
  1. Potential mismatch between discriminator training and consistency optimization. The discriminator is trained to distinguish judge-aligned verification from policy-generated verification, rather than directly distinguishing consistent from inconsistent reasoning. Therefore, a high discriminator score does not necessarily imply high reasoning–answer consistency.
  2. Limited verification of reasoning quality. The evaluation mainly measures agreement between the reasoning conclusion and final answer, without systematically checking the validity of intermediate reasoning steps or visual grounding. Thus, higher RAC does not necessarily indicate more faithful or logically correct reasoning.
  3. Insufficient computational efficiency analysis. Although SVO reduces external judge calls, it requires an additional vision-language discriminator for rollout scoring and periodic updates. Without end-to-end runtime or GPU-hour comparisons, the claimed practical efficiency advantage remains unclear.
Questions for authors
  1. Discriminator objective: Since judge-aligned references may contain inconsistent reasoning with correct inconsistency judgments, how does the discriminator learn to reward actual reasoning–answer consistency rather than merely verification agreement?
  2. Evaluation reliability: Would the improvements remain significant under stricter evaluation that checks intermediate reasoning correctness and visual grounding, rather than only consistency between the reasoning conclusion and final answer?
  3. Computational efficiency: Could the authors provide end-to-end training time, GPU-hour costs, and total judge calls, including initialization, and compare SVO against OVO under matched computational budgets?
Reviewer_qGEvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studied the following problem: in multimodal math reasoning, models sometimes answer the question correctly, but their reasoning trajectory fails to back the answer. The authors think RLVR only rewards the final answer and overlooks such inconsistency problem. The authors proposed self-verifiable policy optimization (SVO), which trains a discriminator to judge the reasoning consistency and apply it to the reward. It improves models’ reasoning consistency on multimodal math reasoning tasks.

Critical strengths

SVO does not need an expensive judge model in RL.

SVO improves the model’s reasoning consistency compared to the base model and GRPO.

Critical weaknesses

I am concerned about the value of the research problem. Even for Qwen3-VL-4B, a very small model released a year ago, its inconsistency rate is only 20%. I believe the number will be much lower for the latest larger MoE models. I suspect it is never a problem that the frontier folks will talk about. The actual task accuracy receives little improvement. This means solving this issue will bring little benefits to the final task performance. I doubt the actual LLM developers will adopt the method because it adds not-that-necessary RM-reward to RLVR. The reason that people use RLVR for math RL is because they are hard to hack. But RMs and LLM-as-a-judge are easier to hack. Why risk reward hacking when there is not significant downstream improvement?

The experimental settings have many issues. I don’t understand why studying the inconsistency problem in the VL setting. I guess text-only models have fewer inconsistency issues (according to figure 5, Qwen2.5-3B-Instruct has only 15% inconsistency rate, and that’s Qwen2.5-3B). But the proposed method has nothing VL-specific. It is a text-only method, but text-only tasks don’t need this method because models perform well, which is kind of ironic. Why gave the policy model a very strict output setup? The policy is instructed to immediately output the answer right after the thinking section. Frontier models always generate a full response after thinking. If you have a full response after thinking, you don’t have to 100% finish your thinking between <think> and </think>, right? Then how are you going to check the inconsistency?

Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.