- Rating
- 1
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates the distribution shift between the data used to calibrate Process Reward Models (PRMs) and the reasoning prefixes encountered during PRM-guided beam search. The authors argue that repeated pruning introduces a depth-dependent distribution shift, leading to increasingly overestimated PRM scores.
To address this issue, the authors propose an on-policy recalibration method that collects beam-search-selected prefixes and uses Monte Carlo rollout labels to further calibrate existing PRMs. Experiments involving two PRMs, two generator models, and multiple mathematical reasoning benchmarks demonstrate reduced overestimation and accuracy improvements in several configurations.
Critical strengths
The paper identifies a potentially important mismatch between PRM calibration data and the distributions encountered during beam search. This problem is relevant to understanding the reliability of PRMs in test-time reasoning.
The proposed recalibration method leverages search-selected reasoning prefixes without requiring substantial modifications to existing PRM architectures. The authors also conduct experiments across different models and mathematical reasoning benchmarks, demonstrating improvements in several configurations.
Critical weaknesses
1. Insufficient experimental validation.
The paper lacks a key ablation comparing on-policy recalibration with additional fine-tuning on unconditionally sampled prefixes under identical training settings. Therefore, it is unclear whether the improvements come from the proposed on-policy recalibration or simply from additional training. Moreover, the experiments are limited to relatively small 1B and 1.5B generator models, without validation on stronger reasoning models.
2. Presentation and formatting issues.
The paper lacks an overall framework figure that clearly illustrates the proposed method and its main components. This makes the methodology less intuitive to understand. Additionally, Figure 4 has noticeable formatting issues, with numerical annotations placed too close to the plot boundaries, reducing readability.
Questions for authors
Can the authors provide a controlled ablation study to demonstrate the effectiveness of on-policy recalibration compared with standard fine-tuning?
Can the authors improve the presentation by adding a method overview figure and fixing the formatting issues in Figure 4?
2 comments
Thank you for the clarification. The experiments are still limited to relatively weak models, with Qwen2.5-7B being the strongest model evaluated. I also did not see any corresponding changes in the revised manuscript. Given these remaining concerns, I will keep my original score.
We thank the reviewer for the comments.
1. Experimental validation
Unconditional-finetuning ablation. We trained a matched unconditional model, with the same starting adapter, labels, loss and pool size (500 MATH problems, Qwen2.5-Math-PRM-7B):
Both finetuned models improve over the baseline, and survivor data is the best in both cases. Much of the improvement comes from recalibrating on search-visited data, with survivor data adding a smaller further gain.
Larger generators. We extended the evaluation to 7B and 8B generators. Survivor finetuning improves accuracy for all four generators on MATH500:
The gains are smaller for the larger generators (+1.2 and +1.8 points) than for the smaller ones (+4.4 and +3.0).
2. Presentation and formatting
We agree. We will add an overall framework figure showing the method and its components, and fix the placement of the numerical annotations in Figure 4 in the final version.