- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how sensitive process reward models (PRMs) are to different step partitions of the same reasoning trajectory. The authors introduce PRM-PART and evaluate several PRMs across different datasets and partitioning strategies. Their results suggest that simply changing where the steps are divided can lead to noticeable differences in reward scores, and sometimes even change which candidate is selected.
Critical strengths
- The paper raises a relevant issue in PRM evaluation. Step boundaries are often treated as a given, so it is useful to examine how much the scoring results depend on them. This could have practical implications for how PRMs are evaluated and compared.
Critical weaknesses
-
The novelty seems somewhat limited. PRMs are designed to evaluate individual reasoning steps, so it is not very surprising that changing the step boundaries would affect their outputs.
-
I am not fully convinced by the practical significance of the results. The paper reports quite a few cases where the selected candidate changes under different partitions. However, most of these changes do not actually affect whether the final answer is correct.
-
The paper is difficult to follow and feels unnecessarily complicated. There are 10 sections in the main paper. Important details about the experimental setup, evaluation metrics, and conclusions are spread across different places. As a result, it is not always easy to understand what the main findings are and which experiments directly support them. I also have some concerns about the overall research workflow. The paper gives the impression of an auto-research style process, with a large number of incremental experiments and follow-up analyses.
Questions for authors
See weaknesses above.