acceptodds
Under review as a conference paper at ICLR 2027

When Step Boundaries Change the Reward: Auditing Process Reward Model Specifications

Abstract

Process reward models (PRMs) score reasoning steps, but completed solutions need not have a unique step partition. We formalize partition well-definedness and introduce PRM-PART to audit candidate selection while fixing the normalized ordered payload and moving only contiguous boundaries. Across three open PRMs, boundaries alter decisions beyond mechanical aggregation. In two separately frozen 1,000-problem cohorts, source-to-canonical parsing and frozen multi-view scoring change 13.2–14.7% and 25.4–43.9% of tolerance-robust winners for two PRMs; the multi-view result also transfers across two rollout generators. A selected Qwen audit retains 25/420 pools after text and AI semantic screens. Under a subsequent model-card direct-join control, boundary changes alter 13/25 winners, 11 robustly; the equal-step/equal-token subset has 9/19 changes, 7 robust. Fresh score-only search produces an actual wrong selection in one of 38 eligible eight-candidate Qwen pools (128 screened) in a separately frozen Level-5 stratum, with CPU confirmation. This stratum was motivated by the original sample's answer composition. The original 33/128 eligible pools show no harm; Level 5 also contains a repair, leaving Qwen accuracy unchanged. Equal-budget random search finds the same harmful case. Together, these results establish partition-dependent selection and an attainable correctness failure under fixed content. Reproducible scoring therefore requires a partition contract specifying normalization, atomization, ownership, serialization, readout, and aggregation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Process Reward Models, Step Partitioning, Reward Specification, Reasoning Evaluation, Test-Time Scaling, Robustness, Reproducibility, Best-of-N Selection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7r27OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how sensitive process reward models (PRMs) are to different step partitions of the same reasoning trajectory. The authors introduce PRM-PART and evaluate several PRMs across different datasets and partitioning strategies. Their results suggest that simply changing where the steps are divided can lead to noticeable differences in reward scores, and sometimes even change which candidate is selected.

Critical strengths
  1. The paper raises a relevant issue in PRM evaluation. Step boundaries are often treated as a given, so it is useful to examine how much the scoring results depend on them. This could have practical implications for how PRMs are evaluated and compared.
Critical weaknesses
  1. The novelty seems somewhat limited. PRMs are designed to evaluate individual reasoning steps, so it is not very surprising that changing the step boundaries would affect their outputs.

  2. I am not fully convinced by the practical significance of the results. The paper reports quite a few cases where the selected candidate changes under different partitions. However, most of these changes do not actually affect whether the final answer is correct.

  3. The paper is difficult to follow and feels unnecessarily complicated. There are 10 sections in the main paper. Important details about the experimental setup, evaluation metrics, and conclusions are spread across different places. As a result, it is not always easy to understand what the main findings are and which experiments directly support them. I also have some concerns about the overall research workflow. The paper gives the impression of an auto-research style process, with a large number of incremental experiments and follow-up analyses.

Questions for authors

See weaknesses above.

Reviewer_ewRxOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how different step boundaries affect the evaluation of completed reasoning traces by PRMs. The authors introduce PRM-PART, a framework that changes step partitions while preserving the original reasoning content and order. Experiments with Qwen, Math-Shepherd, and Skywork show that partition choices can change reward scores and candidate rankings.

Critical strengths

The paper identifies an important issue in PRM evaluation. Different step partitions can produce different scores even when the reasoning content remains unchanged, which is an important issue for PRM.

The experimental analysis is extensive, covering multiple PRMs, synthetic and natural reasoning traces, which strengthens the claim of the paper.

Critical weaknesses

The paper assumes that different partitions of the same reasoning should be evaluated consistently. However, changing step boundaries can change the meaning of each step. Therefore, different scores may be reasonable. More experiments are needed to show that the compared partitions are semantically equivalent.

The paper shows that changing step boundaries often changes the selected candidate. However, this does not necessarily lead to incorrect answers. The experiments find only one case where a correct answer becomes incorrect. More evidence is needed to show that this problem significantly affects accuracy.

Finally, the practical benefit of the proposed solutions is unclear. It makes scoring consistent, but it does not guarantee better evaluation. The multi-view approach also requires about 7.67 times more computation without clear accuracy improvements.

Questions for authors

Q1. How can the authors distinguish between undesirable partition sensitivity and legitimate differences in evaluating reasoning steps at different granularities?

Q2. Can the authors provide stronger evidence that partition changes systematically affect answer correctness, rather than mainly changing the identity of similarly correct candidates?

Q3. Can the authors show that using a fixed step-partition rule maintains the PRM's evaluation performance, such as its ability to detect reasoning errors, compared with the original step partitions?

Reviewer_YivbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how different step partitions affect the scoring behavior of Process Reward Models (PRMs). The authors introduce PRM-PART, a framework for evaluating changes in reward scores and candidate selection while preserving the original reasoning content and order. Experiments across multiple PRMs and datasets demonstrate that modifying step boundaries can substantially change candidate rankings. The authors also present a case where boundary manipulation causes an incorrect answer to be selected and discuss scoring specifications for improving reproducibility.

Critical strengths
  1. Meaningful research problem. The paper investigates an overlooked aspect of PRM evaluation, showing that step partitioning can influence reward scores and candidate selection even when reasoning content remains unchanged.

  2. Extensive experimental analysis. The authors evaluate multiple PRMs, datasets, and partitioning strategies. The controlled experiments provide useful evidence of partition-dependent scoring behavior.

Critical weaknesses
  1. Limited novelty and practical contribution. The observation that PRM scores may change under different step partitions is relatively intuitive, given that PRMs explicitly operate on reasoning steps. Although the paper provides extensive experiments to quantify this phenomenon, its proposed solutions remain relatively limited. Deterministic canonicalization mainly standardizes the input format rather than improving the inherent robustness of PRMs, while multi-view scoring introduces substantial computational overhead without consistent accuracy improvements. Furthermore, the experiments provide limited evidence that partition sensitivity causes meaningful degradation in downstream reasoning accuracy. Therefore, the paper does not offer a sufficiently compelling methodological contribution or practical solution beyond identifying and measuring the problem.

  2. Presentation and organization issues. The paper contains numerous experimental settings, evaluation protocols, and supplementary analyses, making the main contributions difficult to follow. A clear overview figure illustrating the overall evaluation framework is missing from the main text. Additionally, the presentation is overly dense, and the relationships among different experiments are not sufficiently clear. The overall organization and readability need improvement.

Questions for authors
  1. Given that PRM sensitivity to step partitioning is relatively intuitive, what is the main methodological contribution beyond empirically quantifying this phenomenon? Can the authors provide a more effective solution that improves partition robustness without substantial computational overhead?

  2. Can the authors provide stronger evidence that partition sensitivity has a meaningful impact on downstream reasoning performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.