- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the practice of process reward model evaluation through pooled AUROC. It argues that the pooled AUROC can be high even if verifier never reads the reasoning. dataset source leakage and position leakage are enough to get a high score. To address this, the authors propose A_hazard which only looks at step up to the first error and only compares steps from the same source at the same index, then validated this method on 5 models, 5 fine-tuned verifiers and 3 public PRMs.
Critical strengths
- The pair decomposition and text-blind baseline results in Eq.7 and Table 15 provide a simple, cost-effective sanity check to audit position and source shortcuts for PRMs.
- The estimand ablation study in table 3 and table 14 is solid. The ablation applies each restriction separately and each shortcut is attributed to one restriction that removes it.
Critical weaknesses
- The paper lacks more direct and convincing evidence that A_hazard is a better evaluation metric than pooled AUROC, from the results shown in section 5.4 and table 17, the advantage of A_hazard over pooled raw AUROC is marginal.
- The core critique of multi-source pooled AUROC relies on a non-standard setup, it would be better if there's a more comprehensive comparison with more published PRM evaluation baselines, the only citation for this setup is Xia et al. (2025)
Questions for authors
- Could you give one concrete scenario where A_hazard leads to a better choice than pooled AUROC, supported by downstream outcomes such as checkpoint selection, or best-of-N? Neither section 5.4 nor Table 17 shows it.
- Could you provide more evidence of published papers that pool heterogeneous sources into unstratified AUROC? If this practice is rare, the contribution of the main finding in this paper drops significantly.