- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper is a careful negative-results audit of how multimodal process reward models are evaluated, rather than a proposal of a new verifier. Its central claim is that the field’s headline instrument—the aggregate Best-of-N improvement over maj@N or pass@1—is non-identifiable: when within-problem verifier scores have low dispersion, PRM-weighted majority collapses to plain majority vote and PRM-min collapses to uniform selection, so the reported delta tracks score dispersion rather than ranking quality. The authors prove a simple degeneracy lemma, reproduce nine released scored cells with within-problem AUC, tie rate, and parse rate, and show that several published gains are arithmetically indistinguishable from majority vote or parser fallback. They then trace one mechanism to a three-valued argmax readout, show that a continuous verdict-probability readout helps only after the training objective is fixed, and demonstrate that the obvious replacement metric: step-level within-problem AUC. The only admissible level they identify is trajectory-level ranking among exchangeable completions, where their repaired verifier remains near chance and is matched by a length reranker, while an external VisualPRM-8B clears chance on the same cache; the result is therefore scoped as evidence about this checkpoint, protocol, and measurement stack, not as a refutation of multimodal process supervision in general.
Critical strengths
- The paper identifies an important and under-reported failure mode in PRM evaluation: aggregate BoN deltas can be dominated by score dispersion, majority vote, parser fallback, and the mixed-problem denominator rather than by ranking quality.
- The analysis is unusually forensic and self-critical: it audits the authors’ own released artifact, recomputes AUC/tie/parse diagnostics from logged scores, decomposes headline gains, and reports negative or inconvenient cells rather than silently dropping them.
- The measurement chain is methodologically strong: a formal degeneracy lemma motivates within-problem metrics; nuisance baselines test position and length confounds on the same pairs; trajectory-level AUC is justified as the admissible exchangeable comparison; and alternative explanations for the null are measured rather than argued away.
- Reproducibility practices are exemplary in intent: machine-generated numbers, macro provenance, problem-grouped splits, bootstrap at the problem level, release manifests, and explicit statements of which pipeline stages did or did not run.
Critical weaknesses
The evidentiary base remains narrow: one main generator/cache family, one trained in-house checkpoint at one seed, one external verifier control, no human audit of MC step labels, and an unmeasured merged-checkpoint AUC limit how far the negative conclusion can travel.
Questions for authors
See weakness
4 comments
3. One external verifier. Five released PRMs from three groups, 7B to 72B (VisualPRM-8B and -v1_1; Qwen-VL-PRM-7B, with a readout we reconstructed from its released training data and could not check against the authors' code, App. C; text-only Qwen2.5-Math-PRM-7B and -72B), a zero-shot Qwen2.5-VL-7B judge and degenerate scorers (a constant, a length reranker); within each cache all scorers see identical completions (PDF Table 1; App. H). Over 22 image contrasts on the three multimodal PRMs and 60 comparisons between released PRMs, AUC registers 19 and 46, top-1 10 and 40, and the weighted vote 3 and 25 (Secs. 5, 6). At the best threshold on the "+" probability, chosen on the benchmark, our harness reaches VisualPRM-8B's published VisualProcessBench macro-F1 (62.5 against 62.0; 55.4 at threshold 0.5; App. C).
4. No human audit of MC step labels. We did not relabel our corpus by hand; we applied our Wilson labelling rule (App. E) to 700 VisualProcessBench solutions that carry human labels (App. I). It calls 49.8% of human-correct steps wrong with our corpus recipe (question blank); with the question shown, 20.7% (Qwen2.5-VL-7B) and 15.9% (InternVL3-8B), and 35-40% on the hardest third of problems. This fits Prop. 4 (Sec. 7): an MC label records whether this generator can finish from a step, not whether the step is valid. The position check (Sec. 4) runs on both label sources, which tilt in opposite directions:
| scorer | human labels, within solution (19,054 pairs) | human, pooled (step AUC) | our MC labels, soft pairs (3,367) | rising (2,082) | falling (1,285) |
|---|---|---|---|---|---|
| position only | 0.872 (earlier higher) | 0.626 | 0.618 (later higher) | 1 | 0 |
| VisualPRM-8B | 0.696 | 0.695 | 0.432 | 0.223 | 0.770 |
| VisualPRM-8B-v1_1 | 0.611 | 0.642 | - | - | - |
| Qwen2.5-Math-PRM-7B | 0.586 | 0.646 | 0.519 | 0.496 | 0.555 |
| Qwen2.5-Math-PRM-72B | 0.543 | 0.646 | - | - | - |
| Qwen-VL-PRM-7B (our readout) | 0.319 | 0.538 | - | - | - |
| our verifier | - | - | 0.653 | 0.744 | 0.506 |
Human labels: VisualProcessBench, where 87% of within-solution pairs have the correct step first; within solutions every released PRM is below position (each paired interval excludes 0), while pooled across solutions four of the five score above it. MC labels: our held-out steps, one chain per problem, soft pairs as in point 2, split by whether the value rises or falls along the chain (Sec. 4, Apps. D and H).
5. Unmeasured merged-checkpoint AUC. Our verifier's trajectory AUCs (Part 1) come from its merged form, the LoRA adapter merged into the bf16 base model (Sec. 2). Its held-out AUC as a merged checkpoint cannot be measured: the original weights were not retained, and every number for it comes from stored scores. On the same-seed rerun, merging moves held-out within-problem step AUC from 0.774 to 0.758, inside the unmerged adapter's interval, pooled step AUC from 0.868 to 0.867 and soft-pair AUC from 0.651 to 0.654 (App. D).
How far the conclusion travels: that a Best-of-N gain does not identify what a verifier does is measured on five released PRMs, two generators and two label sources (Secs. 3 to 6), and the Scope paragraph (Sec. 8) states what it does not cover.
Thanks again. Your review named one weakness with five specific gaps -- we provided an answer for each with measurements above. If those address your concern, would appreciate if you consider raising your score. We are glad to answer anything further before the discussion closes
Thank you again for recognizing the paper’s methodological contribution and soundness. Your main reservation concerned the breadth of the evidence. We have substantially expanded it to five released PRMs from three groups, two generators, and comparisons against human labels on 700 VisualProcessBench solutions. We also measured the effect of checkpoint merging on a reproduced training run, while explicitly retaining the limitation concerning seed variation.
These additions provide evidence that the central evaluation findings extend beyond our original checkpoint and cache. Given your positive assessment of the contribution, would this broader validation support a stronger acceptance recommendation? We would appreciate knowing whether any specific concern still limits your assessment.
Thank you for a careful summary and review. You are right that a base of one main generator, one in-house checkpoint at one seed, one external verifier, no human audit of the labels and an unmeasured merged checkpoint would limit how far a negative conclusion can travel. Below is the evidence on each of your five points and where it sits in the PDF. Conventions for both parts: unless stated, numbers are on MathVista, MathVision and MathVerse, on mixed problems (at least one correct and one wrong completion), under softmin aggregation of step scores (beta = 4), with 95% bootstrap intervals over problems (over solutions on VisualProcessBench); AUC is within-problem AUC, random = pass@1, top-1 = picking the top-scored completion, wmaj = PRM-weighted majority, * = interval excludes 0. The three instruments are AUC, the top-1 gain over random and the weighted-majority gain over maj@16 (Sec. 1); an instrument registers a contrast when its paired interval excludes 0. Section, table and appendix numbers refer to the PDF on OpenReview unless marked as the submission's.
A correction first, on our verifier (the in-house checkpoint): its trajectory numbers in the submission (Tables 1 and 10, and our verifier's column of Table 11) came from the served path described in App. C, which did not deliver the image. Scored through the corrected path (image sent, native template, verdict read by token id; App. C), its within-problem AUC on the original cache of Qwen2.5-VL-7B completions is 0.545 to 0.661 (App. G, Table 3), mostly on problems that contributed a training chain; on held-out MathVista and MathVision problems it is 0.575 [0.526, 0.623] and 0.542 [0.491, 0.592] (App. D, Table 2; the MathVerse held-out row is not a clean test). Even through the served path, on the full caches of 64 completions, its scores beat the length baseline on MathVista, MathVision and MathVerse (App. C), so the submission's statement that its ranking was matched or beaten by a length reranker does not hold. The comparisons of instruments below are measured on released PRMs.
1. One main generator, one cache family. You are right that Qwen2.5-VL-7B is the main generator: our verifier, the zero-shot judge and the original cache use only its completions, and so do most trajectory comparisons (Sec. 8, Scope). A second generator, InternVL3-8B, gives 16 completions per problem, and its cache, like the fresh Qwen2.5-VL-7B cache, is scored by all five released PRMs (Sec. 2, App. H). For VisualPRM-8B with its released code (mixed problems in brackets; AUC and wmaj - maj@16 with intervals in Tables 5 and 6, shares and signs in App. H):
With either generator, majority vote supplies 78% to 102% of VisualPRM-8B's weighted-majority gain over random (above 100%, the weighted vote fell below majority vote), and its top choice never significantly beats maj@16, under any of six aggregators (App. H). Per generator, AUC, top-1 and the weighted vote register 9, 4 and 2 of 12 image contrasts (Qwen2.5-VL-7B) and 10, 6 and 1 of 10 (InternVL3-8B; Table 8). The generator changes how much a PRM needs the image: a blank canvas costs VisualPRM-8B 0.006 to 0.041 AUC on Qwen2.5-VL-7B completions and 0.012 to 0.137 on InternVL3-8B ones, so the image check must run on the completions being reranked (Secs. 5, 8).
2. One in-house checkpoint at one seed. Beyond our checkpoint, two released versions of one PRM show the gap between the instruments directly: on Qwen2.5-VL-7B MathVista completions VisualPRM-8B-v1_1 ranks above VisualPRM-8B (AUC +0.071 [+0.040, +0.104]) and selects better (top-1 +0.080 [+0.029, +0.130]), yet their weighted votes differ by +0.006 [-0.012, +0.023]; elsewhere the two rank alike or v1_1 lower (Sec. 6; intervals in App. H). Our verifier has one seed: a rerun of our recipe with the same seed reproduces its held-out within-problem step AUC (0.651 vs 0.653 on the 3367 soft pairs, the within-problem pairs of steps whose MC success counts differ; 0.774 [0.595, 0.894] vs 0.790 on the 124 hard pairs of a positive and a negative step; Sec. 4, App. D), which checks reproducibility, not seed variance.