- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper trains an agentic verifier for multimodal reasoning that uses tools to gather visual evidence and labels each reasoning step as support, insufficient, or contradict, with an explanation. It then freezes this verifier and uses its step-level feedback to train and guide a reasoner through multi-round revision. Built on Qwen3-VL-4B and 8B, the method improves results on math, general, and hallucination benchmarks.
Critical strengths
Using tools to check visual evidence, instead of letting the model re-judge its own reasoning, addresses a real weakness of self-verification.
Critical weaknesses
-
The main comparisons are not backbone-matched or compute-matched. Most multimodal reasoning baselines in Table 1 are built on Qwen2.5-VL-7B, and the untrained Qwen3-VL-8B backbone alone already outperforms all of them on average, 62.1 versus at most 59.8, so the claim of best results across benchmarks largely reflects the stronger backbone. Several appendix comparisons are copied from original papers that use older backbones and different evaluation protocols. The only same-backbone comparisons are against plain GRPO and self-verification, and at inference VISPO-R runs several rounds of reasoning and verification with three verifier samples per step, taking 15.9 seconds per example, while those baselines answer in a single pass. Other process-supervised RL methods reimplemented on Qwen3-VL, and best-of-N or majority voting under the same compute, would show how much of the gain comes from the method itself.
-
It is unclear why a trained small verifier is needed instead of simply using a stronger model as the verifier. The verifier training data is generated mainly by GPT-5.5, and the explanation reward is scored by Qwen3.7-Plus, yet the paper does not report GPT-5.5 itself as a verifier, nor larger open multimodal models used as verifiers without training. All verifiers compared in Table 3 are around 8B in size. If the main motivation is cost or deployability, the paper should show how close the trained 8B verifier comes to these stronger verifiers and at what cost. Without this, it is hard to tell whether the contribution goes beyond distilling a stronger model.
-
The paper misses several closely related works. Critic-V, https://arxiv.org/abs/2411.18203, and MMC, https://arxiv.org/abs/2504.11009, both train an external multimodal critic that gives natural-language feedback on the reasoner’s steps, and let the reasoner revise iteratively until the critic is satisfied, which is essentially the reasoning loop of VISPO-R. URSA, https://arxiv.org/abs/2501.04686, trains a multimodal process reward model with supervision on both logical correctness and perceptual consistency, and uses it in online RL through PS-GRPO, which is closely related to VISPO-R’s use of step-level verifier states for credit assignment. None of these are cited, and the last one contradicts the paper’s claim that prior work does not combine verification signals with fine-grained credit assignment. The related work should discuss them and clarify what is new in VISPO beyond tool use and the three-state labels.
-
Minor: The paper overuses bold text in the main body. Method names, result numbers in the abstract, and many phrases in the introduction are bolded, and most result paragraphs open with a full bolded sentence. Together with the small-caps state labels, this makes the text harder to read. This does not affect my rating.
Questions for authors
- How does VISPO-R compare with a GRPO-trained reasoner using best-of-N or majority voting under the same inference time or token budget?
- How do GPT-5.5 and larger open multimodal models, used directly as verifiers without training, compare with VISPO in accuracy and cost?
- Why are the experiments built on Qwen3-VL rather than Qwen3.5, which was released in early 2026 with native multimodal support and is a stronger and more recent backbone? Do the gains hold on Qwen3.5?
- Could you also report results on MMMU-Pro?
10 comments
Q3: Why are the experiments built on Qwen3-VL rather than Qwen3.5, which was released in early 2026 with native multimodal support and is a stronger and more recent backbone? Do the gains hold on Qwen3.5?
A3: Thank you for suggesting this comparison. We chose Qwen3-VL for our main experiments because several of the prior methods we compare against also use it. Matching their backbone allows us to evaluate VISPO and VISPO-R while controlling for differences in backbone capability.
To test whether the gains carry over to Qwen3.5, we evaluated VISPO and VISPO-R on Qwen3.5-4B and Qwen3.5-9B. Both methods improve over their corresponding Qwen3.5 baselines on all eight benchmarks in Table 3-3. Averaged across benchmarks, VISPO gains 2.1 and 2.2 percentage points at 4B and 9B, respectively. The full VISPO-R system improves the mean by 3.5 and 3.9 points.
Table 3-3: Performance of VISPO and VISPO-R across eight multimodal benchmarks with Qwen3.5-4B and Qwen3.5-9B backbones.
| Model | MathVerse | MathVista | MathVision | WeMath | MMMU | MMMU_Pro | MMStar | HallusionBench | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B | 73.4 | 85.1 | 74.6 | 75.4 | 77.6 | 66.3 | 78.3 | 65.0 | 74.5 |
| Qwen3.5-9B | 78.1 | 85.7 | 78.9 | 75.2 | 78.4 | 70.1 | 79.7 | 69.3 | 76.9 |
| VISPO-4B | 76.5 | 85.9 | 77.4 | 77.9 | 79.2 | 68.1 | 79.8 | 68.1 | 76.6 |
| VISPO-R-4B | 77.6 | 86.4 | 80.9 | 78.3 | 80.6 | 69.9 | 80.7 | 69.6 | 78.0 |
| VISPO-9B | 79.3 | 86.8 | 81.9 | 78.5 | 79.7 | 72.6 | 81.4 | 72.5 | 79.1 |
| VISPO-R-9B | 81.2 | 88.1 | 83.5 | 80.3 | 81.9 | 74.0 | 83.8 | 73.4 | 80.8 |
We have added this experiment in Appendix B of the updated manuscript. Thank you again for your valuable feedback.
Q4: Could you also report results on MMMU-Pro?
A4: Thank you for suggesting this evaluation. On MMMU-Pro, VISPO raises accuracy from 53.2% to 57.4% at 4B and from 55.9% to 60.5% at 8B, yielding gains of 4.2 and 4.6 percentage points, respectively. The full VISPO-R system reaches 58.2% at 4B and 62.7% at 8B, improving over the corresponding Qwen3-VL baselines by 5.0 and 6.8 percentage points, respectively.
Table 3-4: MMMU-Pro accuracy of Qwen3-VL, VISPO, and VISPO-R at 4B and 8B.
| Benchmark | Qwen3-VL-4B | Qwen3-VL-8B | VISPO-4B | VISPO-R-4B | VISPO-8B | VISPO-R-8B |
|---|---|---|---|---|---|---|
| MMMU-Pro | 53.2 | 55.9 | 57.4 | 58.2 | 60.5 | 62.7 |
We have added this experiment in Appendix B of the updated manuscript. Thank you again for your valuable feedback.
Q5: Table 1 lacks backbone-matched comparisons. The base Qwen3-VL-8B already scores 62.1 on average, above the best baseline, making it difficult to isolate VISPO’s contribution from the effect of the stronger backbone.
A5: Thank you for raising this concern. We agree that differences in backbone strength make it difficult to attribute the cross-model gains in Table 1 to VISPO. To address this, we use Qwen2.5-VL-7B as the common starting backbone and compare the base model, the base reasoner paired with a VISPO verifier, and the full VISPO-R system across eight benchmarks.
With the backbone held fixed, VISPO improves over the base model on all eight benchmarks, raising the average from 48.4% to 55.1% (+6.7 percentage points). VISPO-R improves on all eight benchmarks again, reaching 58.4% (+10.0 points over the base model and +3.3 points over VISPO). The largest gains for VISPO-R over the base model are on WeMath (+24.0 points) and MathVision (+13.5 points).
Table 3-5: Backbone-matched comparison on Qwen2.5-VL-7B.
| Model | MathVerse | MathVista | MathVision | WeMath | MMMU | MMMU_Pro | MMStar | HallusionBench | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-VL-7B | 49.2 | 68.2 | 25.1 | 30.9 | 58.6 | 38.3 | 63.9 | 52.9 | 48.4 |
| VISPO-7B | 52.7 | 75.6 | 35.3 | 49.4 | 62.5 | 43.1 | 66.7 | 55.4 | 55.1 |
| VISPO-R-7B | 56.8 | 76.4 | 38.6 | 54.9 | 65.7 | 46.0 | 69.3 | 59.2 | 58.4 |
A1: Thank you for suggesting this comparison. We compare the GRPO-trained and VISPO-R reasoners under the same total inference token budget using Best-of-8 selection. Each reasoner generates eight complete trajectories per question, and Qwen3.7-Plus serves as the LLM judge. As shown in Table 3-1, VISPO-R outperforms the GRPO-trained reasoner on all five benchmarks with the verifier held fixed. With the GRPO verifier, mean accuracy improves by 0.5 percentage points at both scales (59.8% to 60.3% at 4B and 64.7% to 65.2% at 8B). With the VISPO verifier, the gains are 1.3 points at 4B (60.6% to 61.9%) and 1.0 points at 8B (65.5% to 66.5%). The full VISPO-R system improves over the corresponding all-GRPO pairing by 2.1 points at 4B and 1.8 points at 8B.
Table 3-1: Best-of-8 accuracy (%) of GRPO and VISPO-R reasoners under a matched inference token budget.
A2: We thank the reviewer for suggesting this comparison. We fix the VISPO-R-8B reasoner across all runs and vary only the inference-time verifier. We use Qwen3-VL-8B, Qwen3-VL-32B, Qwen3.7-Plus, and GPT-5.5 directly, without task-specific verifier training. Table 3-2 reports downstream answer accuracy and mean inference time per example.
VISPO-8B has the lowest mean inference time among the evaluated configurations, at 16.2s per example. Compared with the directly prompted 32B verifier, it improves accuracy on all five benchmarks, raising the mean from 65.4% to 66.1% while reducing inference time by 19.4% (from 20.1s to 16.2s per example). GPT-5.5 achieves the highest mean accuracy (68.4%), gaining 2.3 percentage points over VISPO-8B at an additional 3.4s per example. Qwen3.7-Plus gains 0.6 points at an additional 5.5s per example. Thus, VISPO-8B improves both accuracy and latency over the directly prompted 32B baseline and reduces latency by 17.3% relative to GPT-5.5 and 25.3% relative to Qwen3.7-Plus.
Table 3-2: Downstream accuracy and inference latency across verifiers with a fixed VISPO-R-8B reasoner.
We have added this experiment in Appendix B of the updated manuscript. Thank you again for your valuable feedback.