acceptodds
Under review as a conference paper at ICLR 2027

VISPO: Verification-State-Aware Policy Optimization for Reliable Multimodal Reasoning

Abstract

Reinforcement learning with verifiable rewards is crucial for enhancing the complex reasoning capabilities of multimodal large language models. However, existing paradigms primarily rely on outcome supervision, providing sparse guidance for intermediate reasoning and lacking reliable verification signals to mitigate hallucination propagation during iterative reasoning. To address these limitations, we propose VISPO, Verification-State-Aware Policy Optimization, which trains an agentic verifier to acquire evidence through tool interactions, perform evidence-grounded verification across support, insufficient, and contradict states, and generate reliable verification explanations for multimodal reasoning. Moreover, VISPO introduces region-level advantage redistribution to enable state-aware credit assignment and alleviate credit misalignment. To provide structured supervision for verifier training, we construct FGMV, a Fine-Grained Multimodal Verification dataset with process-level state-aware verification annotations and explanations. Building upon VISPO, we further propose VISPO-R, a synergistic multimodal reasoning framework that leverages iterative verifier feedback via verification-aware hierarchical advantage assignment for reasoning refinement. Extensive experiments show that VISPO-R consistently improves Qwen3-VL backbones across both model scales, achieving average relative improvements of 13.6%, 5.5%, and 10.3% on mathematical reasoning, general reasoning, and hallucination benchmarks. These results demonstrate that state-aware verification effectively improves the accuracy and reliability of multimodal reasoning.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Multimodal Reasoning, Verifiable Rewards

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2W4kOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains an agentic verifier for multimodal reasoning that uses tools to gather visual evidence and labels each reasoning step as support, insufficient, or contradict, with an explanation. It then freezes this verifier and uses its step-level feedback to train and guide a reasoner through multi-round revision. Built on Qwen3-VL-4B and 8B, the method improves results on math, general, and hallucination benchmarks.

Critical strengths

Using tools to check visual evidence, instead of letting the model re-judge its own reasoning, addresses a real weakness of self-verification.

Critical weaknesses
  • The main comparisons are not backbone-matched or compute-matched. Most multimodal reasoning baselines in Table 1 are built on Qwen2.5-VL-7B, and the untrained Qwen3-VL-8B backbone alone already outperforms all of them on average, 62.1 versus at most 59.8, so the claim of best results across benchmarks largely reflects the stronger backbone. Several appendix comparisons are copied from original papers that use older backbones and different evaluation protocols. The only same-backbone comparisons are against plain GRPO and self-verification, and at inference VISPO-R runs several rounds of reasoning and verification with three verifier samples per step, taking 15.9 seconds per example, while those baselines answer in a single pass. Other process-supervised RL methods reimplemented on Qwen3-VL, and best-of-N or majority voting under the same compute, would show how much of the gain comes from the method itself.

  • It is unclear why a trained small verifier is needed instead of simply using a stronger model as the verifier. The verifier training data is generated mainly by GPT-5.5, and the explanation reward is scored by Qwen3.7-Plus, yet the paper does not report GPT-5.5 itself as a verifier, nor larger open multimodal models used as verifiers without training. All verifiers compared in Table 3 are around 8B in size. If the main motivation is cost or deployability, the paper should show how close the trained 8B verifier comes to these stronger verifiers and at what cost. Without this, it is hard to tell whether the contribution goes beyond distilling a stronger model.

  • The paper misses several closely related works. Critic-V, https://arxiv.org/abs/2411.18203, and MMC, https://arxiv.org/abs/2504.11009, both train an external multimodal critic that gives natural-language feedback on the reasoner’s steps, and let the reasoner revise iteratively until the critic is satisfied, which is essentially the reasoning loop of VISPO-R. URSA, https://arxiv.org/abs/2501.04686, trains a multimodal process reward model with supervision on both logical correctness and perceptual consistency, and uses it in online RL through PS-GRPO, which is closely related to VISPO-R’s use of step-level verifier states for credit assignment. None of these are cited, and the last one contradicts the paper’s claim that prior work does not combine verification signals with fine-grained credit assignment. The related work should discuss them and clarify what is new in VISPO beyond tool use and the three-state labels.

  • Minor: The paper overuses bold text in the main body. Method names, result numbers in the abstract, and many phrases in the introduction are bolded, and most result paragraphs open with a full bolded sentence. Together with the small-caps state labels, this makes the text harder to read. This does not affect my rating.

Questions for authors
  • How does VISPO-R compare with a GRPO-trained reasoner using best-of-N or majority voting under the same inference time or token budget?
  • How do GPT-5.5 and larger open multimodal models, used directly as verifiers without training, compare with VISPO in accuracy and cost?
  • Why are the experiments built on Qwen3-VL rather than Qwen3.5, which was released in early 2026 with native multimodal support and is a stronger and more recent backbone? Do the gains hold on Qwen3.5?
  • Could you also report results on MMMU-Pro?
10 comments

Q1: How does VISPO-R compare with a GRPO-trained reasoner using best-of-N or majority voting under the same inference time or token budget?

A1: Thank you for suggesting this comparison. We compare the GRPO-trained and VISPO-R reasoners under the same total inference token budget using Best-of-8 selection. Each reasoner generates eight complete trajectories per question, and Qwen3.7-Plus serves as the LLM judge. As shown in Table 3-1, VISPO-R outperforms the GRPO-trained reasoner on all five benchmarks with the verifier held fixed. With the GRPO verifier, mean accuracy improves by 0.5 percentage points at both scales (59.8% to 60.3% at 4B and 64.7% to 65.2% at 8B). With the VISPO verifier, the gains are 1.3 points at 4B (60.6% to 61.9%) and 1.0 points at 8B (65.5% to 66.5%). The full VISPO-R system improves over the corresponding all-GRPO pairing by 2.1 points at 4B and 1.8 points at 8B.

Table 3-1: Best-of-8 accuracy (%) of GRPO and VISPO-R reasoners under a matched inference token budget.

VerifierReasonerMathVerseMathVistaMathVisionWeMathMMStarAvg.
Qwen3-VL-4B-GRPOQwen3-VL-4B-GRPO54.776.740.156.371.259.8
Qwen3-VL-4B-GRPOVISPO-R-4B55.176.840.756.672.560.3
VISPO-4BQwen3-VL-4B-GRPO55.377.041.456.972.660.6
VISPO-4BVISPO-R-4B56.977.642.758.373.861.9
Qwen3-VL-8B-GRPOQwen3-VL-8B-GRPO61.278.648.062.573.364.7
Qwen3-VL-8B-GRPOVISPO-R-8B62.579.048.162.773.665.2
VISPO-8BQwen3-VL-8B-GRPO62.679.248.363.174.165.5
VISPO-8BVISPO-R-8B63.680.048.765.475.066.5

Q2: How do GPT-5.5 and larger open multimodal models, used directly as verifiers without training, compare with VISPO in accuracy and cost?

A2: We thank the reviewer for suggesting this comparison. We fix the VISPO-R-8B reasoner across all runs and vary only the inference-time verifier. We use Qwen3-VL-8B, Qwen3-VL-32B, Qwen3.7-Plus, and GPT-5.5 directly, without task-specific verifier training. Table 3-2 reports downstream answer accuracy and mean inference time per example.

VISPO-8B has the lowest mean inference time among the evaluated configurations, at 16.2s per example. Compared with the directly prompted 32B verifier, it improves accuracy on all five benchmarks, raising the mean from 65.4% to 66.1% while reducing inference time by 19.4% (from 20.1s to 16.2s per example). GPT-5.5 achieves the highest mean accuracy (68.4%), gaining 2.3 percentage points over VISPO-8B at an additional 3.4s per example. Qwen3.7-Plus gains 0.6 points at an additional 5.5s per example. Thus, VISPO-8B improves both accuracy and latency over the directly prompted 32B baseline and reduces latency by 17.3% relative to GPT-5.5 and 25.3% relative to Qwen3.7-Plus.

Table 3-2: Downstream accuracy and inference latency across verifiers with a fixed VISPO-R-8B reasoner.

VerifierMathVerseMathVistaMathVisionWeMathMMStarAvg.Time Cost (s)
Qwen3-VL-8B61.178.247.461.473.064.225.3
Qwen3-VL-8B-GRPO62.779.047.861.973.565.023.8
Qwen3-VL-32B62.979.248.062.973.865.420.1
VISPO-8B63.479.648.264.874.666.116.2
Qwen3.7-Plus64.280.149.265.374.966.721.7
GPT-5.566.781.251.666.476.168.419.6

We have added this experiment in Appendix B of the updated manuscript. Thank you again for your valuable feedback.

Q3: Why are the experiments built on Qwen3-VL rather than Qwen3.5, which was released in early 2026 with native multimodal support and is a stronger and more recent backbone? Do the gains hold on Qwen3.5?

A3: Thank you for suggesting this comparison. We chose Qwen3-VL for our main experiments because several of the prior methods we compare against also use it. Matching their backbone allows us to evaluate VISPO and VISPO-R while controlling for differences in backbone capability.

To test whether the gains carry over to Qwen3.5, we evaluated VISPO and VISPO-R on Qwen3.5-4B and Qwen3.5-9B. Both methods improve over their corresponding Qwen3.5 baselines on all eight benchmarks in Table 3-3. Averaged across benchmarks, VISPO gains 2.1 and 2.2 percentage points at 4B and 9B, respectively. The full VISPO-R system improves the mean by 3.5 and 3.9 points.

Table 3-3: Performance of VISPO and VISPO-R across eight multimodal benchmarks with Qwen3.5-4B and Qwen3.5-9B backbones.

ModelMathVerseMathVistaMathVisionWeMathMMMUMMMU_ProMMStarHallusionBenchAvg.
Qwen3.5-4B73.485.174.675.477.666.378.365.074.5
Qwen3.5-9B78.185.778.975.278.470.179.769.376.9
VISPO-4B76.585.977.477.979.268.179.868.176.6
VISPO-R-4B77.686.480.978.380.669.980.769.678.0
VISPO-9B79.386.881.978.579.772.681.472.579.1
VISPO-R-9B81.288.183.580.381.974.083.873.480.8

We have added this experiment in Appendix B of the updated manuscript. Thank you again for your valuable feedback.

Q4: Could you also report results on MMMU-Pro?

A4: Thank you for suggesting this evaluation. On MMMU-Pro, VISPO raises accuracy from 53.2% to 57.4% at 4B and from 55.9% to 60.5% at 8B, yielding gains of 4.2 and 4.6 percentage points, respectively. The full VISPO-R system reaches 58.2% at 4B and 62.7% at 8B, improving over the corresponding Qwen3-VL baselines by 5.0 and 6.8 percentage points, respectively.

Table 3-4: MMMU-Pro accuracy of Qwen3-VL, VISPO, and VISPO-R at 4B and 8B.

BenchmarkQwen3-VL-4BQwen3-VL-8BVISPO-4BVISPO-R-4BVISPO-8BVISPO-R-8B
MMMU-Pro53.255.957.458.260.562.7

We have added this experiment in Appendix B of the updated manuscript. Thank you again for your valuable feedback.

Q5: Table 1 lacks backbone-matched comparisons. The base Qwen3-VL-8B already scores 62.1 on average, above the best baseline, making it difficult to isolate VISPO’s contribution from the effect of the stronger backbone.

A5: Thank you for raising this concern. We agree that differences in backbone strength make it difficult to attribute the cross-model gains in Table 1 to VISPO. To address this, we use Qwen2.5-VL-7B as the common starting backbone and compare the base model, the base reasoner paired with a VISPO verifier, and the full VISPO-R system across eight benchmarks.

With the backbone held fixed, VISPO improves over the base model on all eight benchmarks, raising the average from 48.4% to 55.1% (+6.7 percentage points). VISPO-R improves on all eight benchmarks again, reaching 58.4% (+10.0 points over the base model and +3.3 points over VISPO). The largest gains for VISPO-R over the base model are on WeMath (+24.0 points) and MathVision (+13.5 points).

Table 3-5: Backbone-matched comparison on Qwen2.5-VL-7B.

ModelMathVerseMathVistaMathVisionWeMathMMMUMMMU_ProMMStarHallusionBenchAvg.
Qwen2.5-VL-7B49.268.225.130.958.638.363.952.948.4
VISPO-7B52.775.635.349.462.543.166.755.455.1
VISPO-R-7B56.876.438.654.965.746.069.359.258.4
Reviewer_Q6S9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is targeting to improve verification and credit assignment during multimodal reasoning. To do so, the authors firstly propose VISPO which trains a verifier to decide reasoning correctness through tools and return a three-state judgment with an explanation. The authors construct FGMV dataset to provide verification trajectories for training the verifier. They then introduce regional advantage redistribution, which assigns different credit to different component based on their quality and the sign of the trajectory-level advantage. The authors further propose VISPO-R, which trains a reasoner model (a pretrained VLM in their experiment) using the trained verifier. The verifier is also used during inference to guide further reasoning and corrections.

Experiments include training VISPO-R on both Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct. The evaluations cover sufficient benchmarks across mathematical reasoning, general reasoning, and hallucination, and additional evaluations of verifier quality. The authors report both better task performance and verification results. In detail, the evaluations show higher average scores on both model scales compared to the original backbones and GRPO baselines, and the inference experiments show further gains from sampling multiple verifier judgments, with increased inference latency.

Critical strengths
  • The overall work is coherent and focuses on a clear and reasonable problem. The verifier can independently acquire evidence to check intermediate reasoning from the reasoner and provide explanations to guide subsequent steps. Building on top of this idea, the authors proposed method is unified for both training and inference. This is a practical and reasonable way to identify and correct errors during multi-step reasoning.
  • The experiments provide evidence for the effectiveness of the overall method compared to GRPO. The main experiments show substantial improvements over GRPO. The ablation studies and additional experiments in the appendix provide useful insights and further support the effectiveness of the proposed components and the overall framework.
Critical weaknesses

My major concerns are the limited novelty and insufficient comparision with similar method.

  • This paper seems more like an integration of existing ideas, with limited novelty of its own. It also lacks fair comparisons with similar methods and stronger ablation studies of its own components. An independent verifier, additional tool-based verification, and three-state rewards are all existing components. The authors should clarify which parts are the incremental contributions of VISPO. VISPO also lacks reasonable comparisons with closely related methods from the same training starting point. For example, in Table 6, Qwen3-VL-4B already outperforms SRPO-7B. I do not think that VISPO trained from this starting point performing better than SRPO can be attributed to the VISPO method itself. At least a comparison from the same starting point is needed to isolate the effect of the backbone itself. In addition, I think regional redistribution is a good innovation in this paper. But it lacks sufficient evidence showing that it is better than similar credit assignment methods. The “w/o regional redistribution” comparison in the main text should correspond to the case without credit assignment I believe. Therefore, this experiment can only show that regional redistribution is better than not assigning credit. However, credit assignment topics have already shown this conclusion. The authors should provide experiments showing that regional redistribution is better than other credit assignment methods.
  • FGMV is not significant enough to be an independent contribution. According to the description in Section 4.1, FGMV seems like packages the data required to train the VISPO. It is designed for the current framework and lacks generalizability. Therefore, I think it can be considered as a resource for ensuring the reproducibility only. If the authors claim it as an independent dataset contribution, they need to explain the independent value of FGMV beyond the current VISPO pipeline.
Questions for authors
  1. In Section 3.2 row 148, the generated content before and after a tool call will be concated into one verification step. If the tool call and its result are correct, but the following analysis is wrong, how does the method avoid penalizing the correct tool call when the whole step receives a negative advantage?
  2. Why are the state values set to 0, 0.5, and 1? In particular, why is INSUFFICIENT assigned 0.5? Could the authors explain this choice and provide sensitivity experiments with other intermediate values?
  3. Could the authors report the performance of the trained VISPO-R reasoner without the verifier at inference time? One of my concerns is whether the improvement mainly comes from the verifier providing correct explanations to the reasoner. If so, the current evaluation protocol may highlight the advantage of verifier assistance, rather than show that the proposed RL training framework actually improves the reasoner itself.
  4. What exactly does VψV_\psi represent in Section 3.1? Around line 120, it is described as returning a tuple g=(z,e)g=(z,e), but in Eq.1 and Eq.2, it is used as a conditional probability. Could the authors clarify the definition and notation?
12 comments

Q1: An independent verifier, additional tool-based verification, and three-state rewards are all existing components. The authors should clarify which parts are the incremental contributions of VISPO.

A1: We thank the reviewer for raising this point. We agree that independent verifier, tool-based verification have been explored in prior work. Our contribution is how VISPO and VISPO-R use evidence-grounded verification signals to assign credit at two levels, within the verifier's generated response and across the reasoner's steps.

  1. Verification-state-aware reward and sign-aware regional advantage redistribution (Section 4.2-4.3). The verifier assigns each reasoning step one of three states—SUPPORT, INSUFFICIENT, or CONTRADICT—so it can distinguish unresolved evidence or unfinished reasoning from an identified error. VISPO uses exact agreement with these three reference states as a direct RL reward for training a tool-augmented multimodal step verifier. The verifier’s accuracy reward combines state accuracy with explanation quality. VISPO uses these signals and tool-use quality to redistribute a fixed group-relative advantage across analysis, tool-call, state, and explanation tokens. Positive advantages reinforce higher-quality regions more strongly, while negative advantages penalize lower-quality regions more strongly. Tool observations are masked. During reasoner training, explanations guide the next step, and verdicts determine a trajectory reward that credits the terminal state and penalizes regressions. With the reasoner held fixed, regional advantage redistribution improves average accuracy over uniform allocation from 60.8% to 61.4% at 4B and from 65.7% to 66.7% at 8B across MathVerse, MathVista, and WeMath (Figure 3 and Figure 10). In a separate ablation, the binary verifier merges INSUFFICIENT and CONTRADICT into a single class. Retaining all three states improves accuracy on all four evaluated benchmarks, raising the average from 55.2% to 56.9% at 4B and from 60.6% to 62.5% at 8B (Table 11).
  2. A dual-agent data construction pipeline for step-level multimodal verification (Section 4.1 and Appendix C.1). We mine hard multimodal examples and collect trajectories through iterative interaction between a reasoner and a verifier. The reasoner proposes an intermediate step, and the verifier gathers evidence with executable tools as needed and returns a three-state judgment with an evidence-grounded explanation. Both inform the reasoner’s next step. Protocol validation and rejection sampling filter the resulting trajectories to produce FGMV, which contains 2,524 verifier SFT examples and 7,043 verifier RL records. The pipeline can be adapted by changing the source tasks, agent models, or evidence tools.
  3. Hierarchical advantage assignment for reasoner training (Section 4.3). With the learned verifier frozen, VISPO-R combines a group-normalized trajectory advantage with a within-trajectory advantage derived from consecutive verification states. Each generated token receives the local advantage of its reasoning step alongside the trajectory term. This assigns feedback about intermediate progress while retaining supervision from final-answer correctness. With the verifier retained, removing hierarchical advantages reduces average accuracy by 1.8 and 1.7 points at 4B and 8B. Removing the transition reward reduces it by 0.8 and 0.7 points, respectively (Figure 4 and Figure 11).

Q2: VISPO also lacks reasonable comparisons with closely related methods from the same training starting point.

A2: We agree that the original comparison confounded the effect of the method with that of the starting backbone. In Table 2-1, both systems use Qwen2.5-VL-7B as their starting backbone. Our configuration pairs the base reasoner with a VISPO verifier initialized from the same backbone. VISPO improves accuracy over SRPO-7B, reaching 51.9% on average, compared with 50.7% for SRPO-7B and 40.8% for the base model. The largest gains over SRPO-7B are on WeMath (+2.5 points) and MathVisionmini\text{MathVision}_{\text{mini}} (+2.3 points). These results compare the two systems at a common starting backbone, with each retaining its own training and inference protocol.

Table 2-1: Comparison of VISPO and SRPO-7B on four multimodal benchmarks with Qwen2.5-VL-7B

ModelMathVistaMathVisionmini\text{MathVision}_{\text{mini}}WeMathMathVerseVO\text{MathVerse}_{\text{VO}}Avg.
Qwen2.5-VL-7B68.225.730.938.340.8
SRPO-7B76.334.546.944.950.7
Qwen2.5-VL-7B + VISPO-7B75.636.849.445.951.9

We have added this experiment in Appendix B of the updated manuscript. Thank you again for your valuable feedback.

Q3: I think regional redistribution is a good innovation in this paper. But it lacks sufficient evidence showing that it is better than similar credit assignment methods.

A3: We thank the reviewer for suggesting these comparisons. We supplement Appendix B with additional SRPO results and report avg@8 comparisons with VPPO and PGPO. All comparisons use Qwen2.5-VL-7B as the starting backbone.

(1) Comparison with SRPO on four shared benchmarks

SRPO assigns credit to perception tokens based on visual dependency and to reasoning tokens based on their consistency with the generated perception. VISPO extends this principle of role-specific credit to the operations that produce a verification judgment. It can separately assess whether the state was correct, and whether the explanation was grounded in that evidence. VISPO improves over SRPO, raising average accuracy from 50.7% to 51.9%, with the largest gains on WeMath (+2.5 points) and MathVisionmini\text{MathVision}_{\text{mini}} (+2.3 points).

(2) Comparison with VPPO under avg@8 accuracy

VPPO scales the trajectory advantage by overall visual dependency and filters policy gradients to perceptually important tokens. VISPO allocates credit according to what each region contributes to verification. This gives state and explanation tokens a learning signal even when their own visual dependency is low. VISPO raises average accuracy from 47.6% to 51.7% (+4.1 points), with gains of 6.3 points on MathVerse, 3.8 points on MathVision, and 2.3 points on MMMU-Pro.

Table 2-2: Comparison of VISPO and VPPO-7B under avg@8 accuracy

ModelMathVerseMathVisionMMMU-ProAvg.
Qwen2.5-VL-7B49.626.438.538.2
VPPO-7B71.633.337.947.6
Qwen2.5-VL-7B + VISPO-7B77.937.140.251.7

(3) Comparison with PGPO under avg@8 accuracy

PGPO and VISPO both preserve the total advantage during redistribution. PGPO uses token-level visual dependency to determine the weights. VISPO uses region-specific quality scores and makes the weighting depend on the sign of the rollout advantage. This lets the verifier distinguish a region that uses visual information from one that uses the evidence correctly. VISPO raises average accuracy from 46.5% to 51.7% (+5.2 points), with gains of 6.4 points on MathVerse, 8.1 points on MathVision, and 1.2 points on MMMU-Pro.

Table 2-3: Comparison of VISPO and PGPO-7B under avg@8 accuracy

ModelMathVerseMathVisionMMMU-ProAvg.
Qwen2.5-VL-7B49.626.438.538.2
PGPO-7B71.529.039.046.5
Qwen2.5-VL-7B + VISPO-7B77.937.140.251.7

These comparisons measure system-level performance under the reported protocols. Within VISPO, the controlled ablation in Figure 3 and Figure 10 holds the reasoner and reward definition fixed. Replacing uniform allocation with regional redistribution improves average accuracy by 0.6 points at 4B and 1.0 points at 8B.

We have added this experiment in Appendix B of the updated manuscript. Thank you again for your valuable feedback.

Reference:

[1] VPPO: Spotlight on Token Perception for Multimodal Reinforcement Learning. ICLR, 2026.

[2] PGPO: Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models. Findings of EMNLP, 2026.

Reviewer_NqBfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A two-stage framework for multimodal reasoning: VISPO fine-tunes a Qwen3-VL-4B/8B verifier (via SFT + PPO-style RL on the FGMV dataset, whose three-state labels and explanations are generated by GPT-5.5 + Qwen3-VL-8B dual agents) to judge reasoning steps as SUPPORT/INSUFFICIENT/CONTRADICT with tool-acquired evidence, using region-level advantage redistribution; VISPO-R then freezes this verifier and trains the reasoner with hierarchical (trajectory + step-local) advantages from verifier state transitions. On seven benchmarks, VISPO-R reports +5.8/+6.1 points over native Qwen3-VL backbones at 4B/8B.

Critical strengths
  1. The design is effective. The two-stage recipe — train a tool-using three-state verifier, then freeze it and train the reasoner against its feedback — delivers consistent improvements across all seven benchmarks at both scales, with no failure cases; the verifier-synergistic inference alone (+3.4/3.6) and VISPO-R training on top of it both contribute measurably.

  2. The results are solid and reasonably attributed. Beyond the headline table, Table 2 decomposes the gains within fixed backbones, component ablations (three-state vs. binary, regional redistribution, hierarchical advantage, transition reward) all point in the intended direction, and the trained verifier also stands alone (VL-RewardBench 65.6, VisualProcessBench 69.6), showing the artifact is useful beyond this paper's loop.

  3. Careful evaluation hygiene. Training data are deduplicated against evaluation sets at the problem level, and the search tool operates under explicit anti-cheating restrictions during evaluation.

Critical weaknesses

1:The paper proves its framework is effective, but not that the proposed RL method is superior to GRPO. Every comparison in Table 1 pits a system given extra data, tools, and multi-round verification against native single-pass baselines — this shows the overall recipe works, but cannot reflect the RL contribution. The only evidence bearing on RL superiority is Table 2, where GRPO is likewise effective — establishing only that RL training helps, which is already consensus. Most critically, the experiments that could resolve this question are missing: under a unified protocol, the {GRPO, VISPO} training choices are never fully crossed over the two roles. In the paper's own notation, Table 2 contains no row of the form Qwen3-VL-8B-GRPO + Qwen3-VL-8B-GRPO or Qwen3-VL-8B + Qwen3-VL-8B-GRPO. Without this diagonal control, one cannot determine whether VISPO is genuinely better than GRPO overall.

2:Even as the sole retrained RL baseline, GRPO is outdated — no newer RL algorithm is compared. GRPO dates to early 2024, and the field has since produced a series of refined verifiable-reward RL algorithms (e.g., Dr. GRPO, DAPO, GSPO); none is retrained under matched settings, so the proposed objectives are never benchmarked against the current state of the art. The comparisons with SRPO, SRPO†, and ADPO are cited published numbers obtained under different backbones, data scales, and inference budgets — not controlled experiments. The method is further validated only on Qwen3-VL (4B/8B), with no evidence of transfer to any other VLM family.

3:The supervision is sourced from stronger external models, so the "RL-trained system vs. native backbone" comparison carries no persuasive force — and this design itself dilutes the contribution of the proposed RL method. FGMV labels and explanations are generated by GPT-5.5, reasoner trajectories by Qwen3-VL-8B, and the explanation reward is scored by Qwen3.7-Plus; the model is then RL-trained on this distilled signal and compared against native Qwen3-VL in Table 1. That RL training on such data outperforms an untuned backbone is well known, so the comparison supports no method claim. By construction, moreover, this setup attributes the observed gains to the distilled data and the verification protocol, leaving the proposed RL objectives no identifiable share of the improvement — the experimental design weakens, rather than isolates, the very contribution it is meant to demonstrate.

Questions for authors

1:Missing controls for the GRPO comparison. Table 1 only shows the full recipe (extra data, tools, multi-round verification) beating native single-pass models, and in Table 2 GRPO is likewise effective — so neither establishes VISPO's superiority. Please complete the {GRPO, VISPO} × {reasoner, verifier} cross under one protocol: in the paper's own notation, Table 2 lacks the rows Qwen3-VL-8B-GRPO + GRPO-verifier and Qwen3-VL-8B + GRPO-verifier. Without these cells, can the authors demonstrate that VISPO training is genuinely better than GRPO overall?

2:Outdated RL baseline coverage. GRPO (early 2024) is the only retrained RL baseline; the SRPO/SRPO†/ADPO comparisons are cited numbers under different backbones, data scales, and inference budgets.

3:Gains attributable to distilled supervision, not the method. All supervision is sourced from stronger external models (GPT-5.5 labels, Qwen3-VL-8B trajectories, Qwen3.7-Plus judge), so the "RL-trained vs. native backbone" comparison in Table 1 is uninformative by design and leaves the RL objectives no identifiable share of the gains.

5 comments

Q1: Missing controls for the GRPO comparison.

A1:Thank you for the careful review and for identifying the missing GRPO-verifier controls. To test whether VISPO improves over a trained verifier, we added two controls at each scale, pairing the GRPO verifier with either the base Qwen3-VL reasoner or the GRPO-trained reasoner. Replacing the GRPO verifier with VISPO improves the five-benchmark average for both reasoners at both scales.

(1) End-to-end results

Pairing the base Qwen3-VL reasoner with the size-matched VISPO verifier raises the seven-benchmark average from 57.8 to 61.4 at 4B (+3.6 points) and from 62.1 to 66.1 at 8B (+4.0 points). These gains quantify what the VISPO verifier and its iterative feedback contribute to the base reasoner at inference.

Table 1-1: End-to-end results

ModelMathVerseMathVistaMathVisionWeMathMMMUMMStarHallusionBenchAvg.
Qwen3-VL-4B46.873.736.549.467.468.262.557.8
Qwen3-VL-8B56.476.043.752.969.670.965.162.1
VISPO-4B53.275.839.855.168.571.265.961.4
VISPO-R-4B56.777.442.157.869.573.567.963.6
VISPO-8B60.578.547.061.172.673.469.766.1
VISPO-R-8B63.479.648.264.873.874.672.968.2

(2) GRPO-verifier controls

  • Base reasoner held fixed. Switching from the GRPO verifier to VISPO raises the five-benchmark average from 57.6 to 59.0 at 4B (+1.4 points) and from 62.1 to 64.1 at 8B (+2.0 points).
  • GRPO reasoner held fixed. Switching the verifier to VISPO raises the average from 59.4 to 60.4 at 4B (+1.0 points) and from 64.3 to 65.2 at 8B (+0.9 points).
  • VISPO verifier held fixed. Using the VISPO-R reasoner in place of the GRPO reasoner raises the average from 60.4 to 61.5 at 4B (+1.1 points) and from 65.2 to 66.1 at 8B (+0.9 points).
  • All-GRPO versus full VISPO-R. The full system raises the average from 59.4 to 61.5 at 4B (+2.1 points) and from 64.3 to 66.1 at 8B (+1.8 points), with improvements on all five benchmarks at both scales. This diagonal comparison captures the combined effect of changing both roles.

Table 1-2: Verifier comparison with the reasoner held fixed

Reasoner + VerifierMathVerseMathVistaMathVisionWeMathMMStarAvg.
Qwen3-VL-4B + Qwen3-VL-4B47.373.937.250.068.555.4
Qwen3-VL-4B + Qwen3-VL-4B-GRPO50.974.539.153.270.457.6
Qwen3-VL-4B + VISPO-4B53.275.839.855.171.259.0
Qwen3-VL-8B + Qwen3-VL-8B57.176.644.054.071.760.7
Qwen3-VL-8B + Qwen3-VL-8B-GRPO58.077.146.855.872.862.1
Qwen3-VL-8B + VISPO-8B60.578.547.061.173.464.1

Table 1-3: Crossed comparison of GRPO and VISPO training for the reasoner and verifier

Reasoner + VerifierMathVerseMathVistaMathVisionWeMathMMStarAvg.
Qwen3-VL-4B-GRPO + Qwen3-VL-4B-GRPO54.076.739.955.570.959.4
Qwen3-VL-8B-GRPO + Qwen3-VL-8B-GRPO61.078.447.961.273.064.3
Qwen3-VL-4B-GRPO + VISPO-4B55.176.741.156.672.560.4
Qwen3-VL-8B-GRPO + VISPO-8B62.379.147.962.773.965.2
VISPO-R-4B + Qwen3-VL-4B-GRPO54.876.941.256.772.960.5
VISPO-R-8B + Qwen3-VL-8B-GRPO62.779.047.861.973.565.0
VISPO-R-4B + VISPO-4B56.777.442.157.873.561.5
VISPO-R-8B + VISPO-8B63.479.648.264.874.666.1

Q2: Outdated RL baseline coverage. GRPO (early 2024) is the only retrained RL baseline.

A2: We thank the reviewer for the careful assessment of our experimental design and for pointing out where the evidence could be strengthened. We appreciate the specific suggestions to broaden the set of RL baselines, and test a VLM from another family. To address baseline coverage and transfer, we add verifier-side comparisons with DAPO and GSPO on Qwen3-VL-4B, Qwen3-VL-8B, and InternVL3-8B.

To address the reviewer’s concern about relying on GRPO alone, we add DAPO and GSPO as retrained verifier baselines on Qwen3-VL-4B, Qwen3-VL-8B, and InternVL3-8B. Within each backbone, we fix the reasoner and vary the verifier training method. The comparison is designed to hold verifier initialization, training supervision, the verification protocol, and the inference budget fixed across the trained variants.

VISPO achieves the highest five-benchmark average on all three backbones. It scores 59.0% on Qwen3-VL-4B, 64.1% on Qwen3-VL-8B, and 56.5% on InternVL3-8B, exceeding DAPO by 0.7, 1.1, and 3.7 percentage points and GSPO by 0.5, 0.7, and 1.6 points, respectively. On InternVL3-8B, VISPO also improves over GSPO on all five benchmarks, providing evidence that the verifier training method transfers beyond Qwen3-VL.

Table 1-4: Verifier-side RL baselines with a fixed reasoner within each backbone

Fixed ReasonerVerifierMathVerseMathVistaMathVisionWeMathMMStarAvg.
InternVL3-8BInternVL3-8B40.872.030.239.869.250.4
InternVL3-8BInternVL3-8B-GRPO42.372.430.541.669.751.3
InternVL3-8BInternVL3-8B-DAPO44.073.632.244.170.152.8
InternVL3-8BInternVL3-8B-GSPO47.074.134.947.970.654.9
InternVL3-8BInternVL3-8B-VISPO49.375.036.750.271.556.5
Qwen3-VL-4BQwen3-VL-4B47.373.937.250.068.555.4
Qwen3-VL-4BQwen3-VL-4B-GRPO50.974.539.153.270.457.6
Qwen3-VL-4BQwen3-VL-4B-DAPO51.875.439.354.170.958.3
Qwen3-VL-4BQwen3-VL-4B-GSPO52.675.539.654.570.358.5
Qwen3-VL-4BQwen3-VL-4B-VISPO53.275.839.855.171.259.0
Qwen3-VL-8BQwen3-VL-8B57.176.644.054.071.760.7
Qwen3-VL-8BQwen3-VL-8B-GRPO58.077.146.855.872.862.1
Qwen3-VL-8BQwen3-VL-8B-DAPO59.478.246.557.873.163.0
Qwen3-VL-8BQwen3-VL-8B-GSPO60.178.147.258.573.163.4
Qwen3-VL-8BQwen3-VL-8B-VISPO60.578.547.061.173.464.1

Q3: The SRPO/SRPO†\text{SRPO}^{\dagger}/ADPO comparisons are cited numbers under different backbones, data scales, and inference budgets.

A3: We thank the reviewer for highlighting these differences. With Qwen2.5-VL-7B as the shared backbone, VISPO outperforms Perceval-7B and SRPO-7B, with average gains of 2.9 and 1.2 percentage points, respectively. We remove the ADPO comparison because of differences in backbone and inference strategy, and withdraw the SRPO†\text{SRPO}^{\dagger} comparison and sample-efficiency claim because the training data supervise different tasks.

With Qwen2.5-VL-7B as the common starting backbone, VISPO outperforms Perceval-7B on all four benchmarks, raising the average from 63.3% to 66.2% (+2.9 percentage points). The largest gains are on MathVision (+4.4 points) and MathVista (+3.6 points). VISPO also exceeds Perceval-7B on V*, reaching 87.2% versus 86.4%.

Table 1-5: Comparison of VISPO and Perceval-7B on four multimodal benchmarks with Qwen2.5-VL-7B

ModelMathVisionMathVistaMMStarV*Avg.
Qwen2.5-VL-7B25.168.263.962.154.8
Perceval-7B30.972.063.886.463.3
Qwen2.5-VL-7B + VISPO-7B35.375.666.787.266.2

With Qwen2.5-VL-7B as the common starting backbone, VISPO outperforms structured role-aware SRPO-7B, raising the average from 50.7% to 51.9% (+1.2 percentage points). The largest gains are on WeMath (46.9% to 49.4%, +2.5 points) and MathVisionmini\text{MathVision}_{\text{mini}} (34.5% to 36.8%, +2.3 points).

Table 1-6: Comparison of VISPO and SRPO-7B on four multimodal benchmarks with Qwen2.5-VL-7B

ModelMathVistaMathVisionmini\text{MathVision}_{\text{mini}}WeMathMathVerseVO\text{MathVerse}_{\text{VO}}Avg.
Qwen2.5-VL-7B68.225.730.938.340.8
SRPO-7B76.334.546.944.950.7
Qwen2.5-VL-7B + VISPO-7B75.636.849.445.951.9

We remove the ADPO comparison from the revised appendix. The published ADPO results use Qwen2-VL-7B and parallel selection among complete reasoning trajectories. The submitted VISPO comparison uses Qwen3-VL and samples verifier judgments at each step of a single evolving trajectory. These settings differ in both the starting model and the allocation of inference compute.

We remove the quantitative comparison with reflection-aware SRPO†\text{SRPO}^{\dagger} and the associated sample-efficiency claim because the two datasets supervise different tasks. SRPO†\text{SRPO}^{\dagger} trains a reasoner on 37K problem-level RL examples, whereas VISPO trains a verifier on 7K step-level FGMV records. Adapting the SRPO†\text{SRPO}^{\dagger} data for VISPO would require annotating verification states, explanations, and tool evidence. Conversely, adapting FGMV for SRPO†\text{SRPO}^{\dagger} would require constructing complete reasoning and reflection trajectories with final-answer targets. A controlled comparison would start from the same set of problems and account for the cost of constructing each type of supervision.

We have added this experiment in Appendix B of the updated manuscript. Thank you again for your valuable feedback.

Reference:

[1] InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv, 2025.

[2] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv, 2025.

[3] GSPO: Group Sequence Policy Optimization. arXiv, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.