acceptodds
Under review as a conference paper at ICLR 2027

Duet-PRM: Factorizing Plan Reliability and Step Correctness in Process Reward Models

Abstract

Process reward models (PRMs) provide fine-grained supervision for multi-step reasoning, but most existing approaches collapse different sources of reasoning failure into a single step-level reward. We identify this issue as plan-step conflation: an incorrect trajectory may arise either from an unreliable plan or from incorrect execution of an otherwise reasonable plan. To address this, we propose Duet-PRM, a generative PRM that separately predicts plan-level reliability and step-level correctness. We construct the required factorized supervision through a fully automated pipeline based on multiple independent executions, without human annotation or auxiliary LLMs. Across four mathematical reasoning benchmarks, the two reward signals provide complementary information, and their combination consistently improves chain discrimination and downstream Best-of- selection over either component alone. Duet-PRM also achieves the strongest average candidate-selection performance among the evaluated PRMs. Further analysis shows that higher predicted plan scores consistently correspond to higher reasoning success rates, and that plan-level signals can correct selection errors when step-level evidence alone favors an incorrect answer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Process Reward Models, Mathematical Reasoning, Reward Factorization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_EMYeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies the plan-step conflation problem in existing process reward models (PRMs): two distinct failure modes—unreliable overall plans and incorrect execution of sound plans—are collapsed into a single step-level score. To address this, the authors propose DUET-PRM, a generative PRM fine-tuned from Qwen2.5-Math-7B-Instruct that jointly outputs two three-way scores per step: one for plan reliability and one for step correctness.

Supervision labels are constructed via a fully automated rule-based pipeline: for each problem, M=4M=4 plans are sampled, each with N=4N=4 independent executions (yielding ~520K step examples from ~6K problems). Plan labels are derived from empirical execution success rates; step labels combine AST-based action-plan faithfulness checks and cross-execution corroboration, with an explicit “uncertain” third class retained during training.

Evaluated on four mathematical reasoning benchmarks (MATH500, GSM8K, OlympiadBench, Gaokao2023EN) with Qwen3-4B as the policy model and Best-of-N=8N=8, the combined score achieves the best average AUC and the best average BoN accuracy (63.4 vs. 62.6 for the strongest baseline). Ablation studies show the combined score consistently outperforms either component alone. Further analysis demonstrates that higher plan scores monotonically predict reasoning success and can correct selection errors made by step-score-only signals.

Critical strengths
  • Well-motivated problem framing with clear conceptual grounding. The plan-step conflation issue is clearly articulated, and the qualitative case study in Appendix F concretely illustrates the two distinct failure modes, making the factorization argument intuitive and principled.
  • Fully automated supervision pipeline without auxiliary LLMs or human annotation. The conservative “uncertain” third-class design is a principled choice to avoid labeling noise, and the pipeline is described with sufficient detail (prompts in Appendix A, labeling rules in Appendix D) for reproducibility.
  • Controlled baseline comparisons. All compared PRMs share the same backbone model, so observed performance differences can be attributed to differences in reward modeling and supervision rather than model capacity.
  • Consistent and comprehensive ablation evidence. Table 3 confirms the full combined score outperforms both individual components across all four benchmarks; Table 4 supports the choice of mean aggregation and weighted-sum combination. Additional analyses in Section 4.4 and Appendix G (especially Table 8, showing plan scores improve accuracy within every step-score regime) directly support the complementary nature of the two factorized signals.
Critical weaknesses
  1. Unclear baseline input format comparability. The policy model produces plan-annotated, tool-integrated reasoning chains, but it is not specified whether the three public PRM baselines (Qwen2.5-PRM-7B, Skywork, PRM800K) were evaluated on this same plan+code format or adapted from their native natural-language CoT format. If the latter, the large performance gap on OlympiadBench (AUC 0.66 for Qwen2.5-PRM vs. 0.78 for DUET-PRM) may partly reflect input format mismatch rather than superior reward modeling. (Section 3.1, Section 4.1)
  2. Modest downstream gains with partially overlapping confidence intervals. The average BoN improvement over the strongest baseline is only +0.8 percentage points (63.4 vs. 62.6), with no reported significance tests. On GSM8K, the AUC confidence intervals of DUET-PRM [0.7772, 0.8704] and Qwen2.5-PRM [0.8372, 0.8956] overlap, and Qwen2.5-PRM achieves higher BoN accuracy. The claim of “strongest average performance” holds, but the margin is thin. (Table 2, Table 6)
  3. No human validation of label quality. The automated labels (empirical success rate, AST action matching, cross-execution consensus) have not been spot-checked against human judgment. With 30.4% of step labels marked “uncertain” and only ~4 executions per plan, the magnitude of label noise is unquantified. (Figure 5, Appendix D)
Questions for authors
  1. Exactly what input format was fed to the three public PRM baselines — the same plan-annotated tool-integrated trajectories used for DUET-PRM, or their native trajectory format? If possible, please report baseline performance under both conditions.
  2. Could you provide a small human audit of the automated step labels (e.g., 200 samples) to estimate label noise? Additionally, how does DUET-PRM accuracy change when uncertain labels are discarded versus retained during training?
  3. Are the BoN accuracy differences (e.g., 63.4 vs. 62.6) statistically significant under problem-level bootstrap, as computed for AUC?
  4. What is the compute cost of the (M * N=16) executions per problem, and how does it compare to bootstrapping-style PRM supervision methods such as Math-Shepherd or OmegaPRM?
1 comment

Dear Reviewer EMYe,

Thank you for your careful review. We have addressed the main concerns through a controlled native-versus-plan-augmented input comparison, a 300-instance label audit and Uncertain-label ablation, problem-level bootstrap analysis, and a detailed report of data-construction cost. The overall BoN improvement is statistically significant, and retaining Uncertain improves average accuracy by 1.8 points. These additions have been incorporated into the revised PDF.

We would appreciate your assessment of whether these results address the concerns and would be grateful if you could reconsider the rating. We remain happy to clarify any outstanding issues.

Thank you for your time.

Reviewer_hDH1OpenReview ↗
Rating
3(+2)
Confidence
3(+1)
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Duet-PRM, which separately models plan reliability and step correctness through automatically constructed supervision. The method uses multiple executions of the same plan to generate reward labels. Experiments on four mathematical reasoning benchmarks show that combining the two reward signals improves candidate selection performance.

Critical strengths
  1. The paper explores an interesting direction for fine-grained process reward modeling by considering plan reliability and step correctness separately.

  2. The automated supervision pipeline reduces reliance on human annotations and auxiliary LLMs. The experimental results also suggest that combining the two reward signals can improve candidate selection performance.

Critical weaknesses

1. Insufficient motivation and unclear benefits of plan and step factorization.

My main concern is that the paper does not clearly justify why reasoning quality should be decomposed into plan reliability and step correctness. It remains unclear whether these two aspects can be meaningfully separated and what unique advantages this decomposition provides. In particular, the two scores are eventually combined into a single reward. The experiments do not convincingly demonstrate why such factorization is necessary or preferable to directly learning a unified reward model. This significantly weakens the paper's core contribution.

2. Limited generalization experiments.

The evaluation relies on a single reward model backbone and uses Qwen3-4B to generate evaluation trajectories. Experiments with different model scales and families are missing, making it difficult to assess whether the proposed approach generalizes beyond the specific model configuration. This limits the strength of the empirical evidence supporting the method.

Questions for authors
  1. What are the fundamental advantages of separately modeling plan reliability and step correctness? Can the authors provide evidence that this factorization offers benefits beyond a unified reward model trained under comparable conditions?

  2. Does Duet-PRM remain effective when evaluated on reasoning trajectories generated by different model families or larger models?

5 comments

Dear Reviewer hDH1,

Thank you for your careful review. We have strengthened the motivation for separating planning and execution, added a matched unified-score baseline that DUET-PRM outperforms on all four benchmarks, and verified cross-family transfer on DeepSeek-R1-Distill-Llama-8B trajectories. These additions are included in the revised PDF.

We would appreciate your assessment of whether these results address the concerns and would be grateful if you could reconsider the rating. We remain happy to clarify any outstanding issues.

Thank you for your time.

The authors have addressed most of my concerns, and I will raise my score accordingly.

For future work, I would encourage a greater focus on efficiency and broader applicability. On mathematics tasks with verifiable answers, small gains over majority voting may offer limited practical value. More promising directions include achieving comparable or better results with lower computational cost and faster scoring, and extending to tasks such as writing, where clear reference answers are unavailable and majority voting is less directly applicable.

Thank you for recognizing our contributions and for suggesting these directions. Best-of-N gains are difficult to realize because reranking improves accuracy only when it changes the winning answer group. Although the absolute gain appears modest, DUET-PRM’s improvement over majority voting is nearly twice that of the next-best public PRM.

The efficiency of DUET lies primarily in supervision construction rather than scoring latency. The same fixed executions are reused to generate both plan labels and multiple step labels, producing approximately 520K training instances in about two days on eight A100 GPUs. The subsequent rule-based labeling requires no human annotation, auxiliary-LLM inference, or additional prefix rollouts. By contrast, commonly used public PRMs are trained on larger supervision corpora or rely on heavier human-, model-, or rollout-based annotation pipelines.

We focus on mathematics because plan–execution separation is most established in structured reasoning and agentic tasks. Its suitability for open-ended tasks such as writing remains an important question for future study. We appreciate your recognition and helpful suggestions.

Reviewer_iLzmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Duet-PRM, a process reward model that separates plan reliability from step correctness in mathematical reasoning. The authors argue that existing PRMs often conflate failures caused by poor planning with those caused by incorrect execution. To address this, they introduce an automated labeling pipeline based on multiple independent executions of the same plan, without relying on human annotations or auxiliary LLMs. The resulting labels are used to train a generative PRM that predicts both scores. Experiments on four mathematical reasoning benchmarks show that combining the two scores improves chain discrimination and Best-of-N selection, achieving the best average selection accuracy among the evaluated PRMs.

Critical strengths
  • I think separating plan reliability from step correctness is a meaningful direction for improving PRMs. The motivation is intuitive, and the proposed factorization offers a useful way to distinguish different sources of reasoning failures.

  • The proposed labeling pipeline does not require human annotations or auxiliary LLMs. The ablation results also show that combining the two reward signals performs better than using either one alone.

Critical weaknesses
  1. The plan labels are based on the success rate of only four executions from a fixed policy model. However, a reasonable plan may fail simply because the model cannot execute it correctly. I am not fully convinced that these labels reflect the quality of the plan itself rather than the execution capability of the policy model. The small number of executions also raises concerns about label stability.

  2. The reliability of the step-level labels is not sufficiently validated. The authors determine step correctness through plan-stage consistency and agreement across executions. However, different valid executions may produce different intermediate results, even under the same plan. It is unclear how often this leads to incorrect labels. Some direct validation of the automatically generated labels would make the proposed supervision more convincing.

Questions for authors
  1. Plan-level supervision (Section 3.4 and Appendix D). The plan labels are determined by the outcomes of only four executions, with plans labeled as bad when at most one execution succeeds and good when at least three succeed. How sensitive are these labels to the number of sampled executions? More importantly, how can the authors distinguish an unreliable plan from a reasonable plan that the policy model fails to execute correctly?

  2. Step-level label reliability (Section 3.4 and Appendices C–D). While Figure 5 reports the overall label distribution, it remains unclear how reliable the automatically generated step-level labels are. Could the authors report the proportion of incorrect trajectories for which the proposed procedure can identify a failure step? More importantly, have the generated labels been validated against human annotations or other independently verified references?

  3. Effectiveness of reward factorization (Section 4.3). Table 3 shows that combining plan and step scores improves performance, but it is less clear whether the gains come specifically from separating planning and execution errors. Could the authors clarify what evidence supports this distinction beyond the improved combined scores? For example, how would Duet-PRM compare with a unified PRM trained using the same automatically generated supervision?

Minor issue: In Table 4, the AUC values for the Min and Last aggregation methods are identical under α = 0.1 and α = 0.3, while the corresponding Cohen's d values differ. Please check whether these results are reported correctly.

Reviewer_9iM1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses one problem in process reward models (PRMs) for tool-integrated mathematical reasoning, which means that plan quality and step execution quality are combined into a single score. The authors build one model named Duet-PRM, which will predict overall plan reliability and current-step correct rate separately. Specifically, they develop one new pipeline that doesn’t require either human annotation or additional LLMs. At first they play the same plan for several times and then they use the final-answer success rate to label the plan. And in each step they checked if the code run as planned and if the results in this process align with the final answers. During the training process, they fine-tune the model to output both scores at each step, which are combined through a weighted sum for candidate-answer selection. Then they make some experiments to evaluate chain-level discrimination and reward-weighted voting. Based on these experiments they use some tables and graphs to show some highlight of their model, which means that Duet-PRM has the highest AUC on three benchmarks and has an average answer-selection accuracy of 63.4%. Also, Duet-PRM exceeds the SOTA PRM by 1.7 and 0.8 percentage points. The ablation results also show that combining the two scores is better than using either score alone.

Critical strengths

The two reward signals provide different information and consistently work better together. They test the method on four benchmarks. On all four, using the plan and step scores together gives better chain-level AUC and answer-selection accuracy than using only one of the scores. Appendix G shows that plan scores still provide useful information at different levels of step scores. These results tell us that using both scores will be helpful to evaluate reasoning trajectories.

Critical weaknesses
  1. The paper does not clearly show what it adds to closely related work. The paper cites ReasonFlux-PRM, which already evaluates a reasoning strategy by running it several times with the same template. It uses the average correctness in these runs as the ideal score for the whole trajectory. It also trains with step-level labels and combines the two reward scores. Duet-PRM differs through its explicit pre-generated plans, rule-based labeling, and per-prefix dual-score prediction, but the paper does not adequately explain or experimentally establish the significance of these differences. The paper does not directly compare its method with this the related approach. Without this comparison, it is hard to say whether the improvement or changes how the training labels are produced comes from the method
  2. The experiments do not directly show if the main parts of the labeling process work well. The authors do not independently check the quality of the labels. Each part should be tested separately. For example, does AST-based action matching help? Does checking whether different runs give the same intermediate results help? Does adding an “uncertain” label help? The experiments do not answer these questions clearly. Table 3 compares the results in different situations, including using the plan score, the step score, and both from the same jointly trained model. But it lacks the comparison between two training configurations, it means that training with separate plan and step labels and training with one combined label under the same conditions. The higher plan scores are, more likely it will output the correct answer. However, this does not show what the plan score measures. I am not clear if it measures the quality of the plan or it is used to evaluate the performance of the model. The results show that the combination of two scores together will be helpful. It is not so solid that the labeling process is reliable.
Questions for authors
  1. Relationship to prior work: ReasonFlux-PRM already uses templates to score the whole reasoning process. How is your plan-reliability score different with it? Beyond differences in trajectory format, can you show some measurable advantage of your method? A controlled comparison would make it clearer to understand.
  2. Validation of the proposed labeling mechanisms: The experiments show that using both scores together helps. It is great experiments. But it is not clear whether action matching based on AST is necessary or works well. Besides, they do not test whether checking intermediate results or adding an “uncertain” label is necessary. So could you provide an experiment to test the label quality and some ablations that remove or replace each component?
  3. Evidence for factorization: Are the “Only plan score” and “Only step score” results obtained by selecting outputs from the same jointly trained model, or by training separate models? Is there a matched single-score training baseline?
1 comment

Dear Reviewer 9iM1,

Thank you for your careful review. We have addressed the main concerns through label-quality validation, component ablations, a matched unified-score baseline, and further clarification of the distinction from ReasonFlux-PRM. These changes have also been incorporated into the revised PDF.

We would appreciate your assessment of whether our response addresses the concerns and would be grateful if you could reconsider the rating. We remain happy to clarify any outstanding issues.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.