- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies the plan-step conflation problem in existing process reward models (PRMs): two distinct failure modes—unreliable overall plans and incorrect execution of sound plans—are collapsed into a single step-level score. To address this, the authors propose DUET-PRM, a generative PRM fine-tuned from Qwen2.5-Math-7B-Instruct that jointly outputs two three-way scores per step: one for plan reliability and one for step correctness.
Supervision labels are constructed via a fully automated rule-based pipeline: for each problem, plans are sampled, each with independent executions (yielding ~520K step examples from ~6K problems). Plan labels are derived from empirical execution success rates; step labels combine AST-based action-plan faithfulness checks and cross-execution corroboration, with an explicit “uncertain” third class retained during training.
Evaluated on four mathematical reasoning benchmarks (MATH500, GSM8K, OlympiadBench, Gaokao2023EN) with Qwen3-4B as the policy model and Best-of-, the combined score achieves the best average AUC and the best average BoN accuracy (63.4 vs. 62.6 for the strongest baseline). Ablation studies show the combined score consistently outperforms either component alone. Further analysis demonstrates that higher plan scores monotonically predict reasoning success and can correct selection errors made by step-score-only signals.
Critical strengths
- Well-motivated problem framing with clear conceptual grounding. The plan-step conflation issue is clearly articulated, and the qualitative case study in Appendix F concretely illustrates the two distinct failure modes, making the factorization argument intuitive and principled.
- Fully automated supervision pipeline without auxiliary LLMs or human annotation. The conservative “uncertain” third-class design is a principled choice to avoid labeling noise, and the pipeline is described with sufficient detail (prompts in Appendix A, labeling rules in Appendix D) for reproducibility.
- Controlled baseline comparisons. All compared PRMs share the same backbone model, so observed performance differences can be attributed to differences in reward modeling and supervision rather than model capacity.
- Consistent and comprehensive ablation evidence. Table 3 confirms the full combined score outperforms both individual components across all four benchmarks; Table 4 supports the choice of mean aggregation and weighted-sum combination. Additional analyses in Section 4.4 and Appendix G (especially Table 8, showing plan scores improve accuracy within every step-score regime) directly support the complementary nature of the two factorized signals.
Critical weaknesses
- Unclear baseline input format comparability. The policy model produces plan-annotated, tool-integrated reasoning chains, but it is not specified whether the three public PRM baselines (Qwen2.5-PRM-7B, Skywork, PRM800K) were evaluated on this same plan+code format or adapted from their native natural-language CoT format. If the latter, the large performance gap on OlympiadBench (AUC 0.66 for Qwen2.5-PRM vs. 0.78 for DUET-PRM) may partly reflect input format mismatch rather than superior reward modeling. (Section 3.1, Section 4.1)
- Modest downstream gains with partially overlapping confidence intervals. The average BoN improvement over the strongest baseline is only +0.8 percentage points (63.4 vs. 62.6), with no reported significance tests. On GSM8K, the AUC confidence intervals of DUET-PRM [0.7772, 0.8704] and Qwen2.5-PRM [0.8372, 0.8956] overlap, and Qwen2.5-PRM achieves higher BoN accuracy. The claim of “strongest average performance” holds, but the margin is thin. (Table 2, Table 6)
- No human validation of label quality. The automated labels (empirical success rate, AST action matching, cross-execution consensus) have not been spot-checked against human judgment. With 30.4% of step labels marked “uncertain” and only ~4 executions per plan, the magnitude of label noise is unquantified. (Figure 5, Appendix D)
Questions for authors
- Exactly what input format was fed to the three public PRM baselines — the same plan-annotated tool-integrated trajectories used for DUET-PRM, or their native trajectory format? If possible, please report baseline performance under both conditions.
- Could you provide a small human audit of the automated step labels (e.g., 200 samples) to estimate label noise? Additionally, how does DUET-PRM accuracy change when uncertain labels are discarded versus retained during training?
- Are the BoN accuracy differences (e.g., 63.4 vs. 62.6) statistically significant under problem-level bootstrap, as computed for AUC?
- What is the compute cost of the (M * N=16) executions per problem, and how does it compare to bootstrapping-style PRM supervision methods such as Math-Shepherd or OmegaPRM?
Dear Reviewer EMYe,
Thank you for your careful review. We have addressed the main concerns through a controlled native-versus-plan-augmented input comparison, a 300-instance label audit and Uncertain-label ablation, problem-level bootstrap analysis, and a detailed report of data-construction cost. The overall BoN improvement is statistically significant, and retaining Uncertain improves average accuracy by 1.8 points. These additions have been incorporated into the revised PDF.
We would appreciate your assessment of whether these results address the concerns and would be grateful if you could reconsider the rating. We remain happy to clarify any outstanding issues.
Thank you for your time.