acceptodds
Under review as a conference paper at ICLR 2027

Know What You’ve Selected: On-Policy Recalibration of Process Reward Model Drift

Abstract

Process reward models (PRMs) guide reasoning search by scoring intermediate steps, offering signal on solution quality before completion—particularly useful for weaker models on complex tasks, where methods like Best-of-N and self-consistency only evaluate complete solutions. Standard beam search is expensive, generating candidates per depth, so recent work uses calibrated PRMs to adaptively shrink search width. We show this assumption fails: PRM calibration is typically done on prefixes sampled independently of search, but deployment evaluates prefixes that survived search-induced pruning at every prior step. This creates a self-referential covariate shift that compounds with depth, causing PRMs to grow increasingly overconfident on deeper, survivorship-selected prefixes—even after calibration—across both Qwen-PRM and Math-Shepherd. We propose an on-policy correction that recalibrates PRMs using their own search-induced prefixes, substantially reducing this depth-dependent overestimation for both PRMs. Downstream, this translates into consistent beam-search accuracy gains with Qwen-PRM across generators and benchmarks, with gains concentrated at moderate self-consistency uncertainty (up to 10% relative to extremes) and additional search budget allocated preferentially to lower-certainty questions; gains with Math-Shepherd are more mixed, improving in most but not all configurations. These results suggest the covariate-shift diagnosis is general, while its downstream accuracy impact depends on the underlying PRM.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Calibration, Out of Distribution, Process Reward Models, LLM reasoning, Beam search

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NutuOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the distribution shift between the data used to calibrate Process Reward Models (PRMs) and the reasoning prefixes encountered during PRM-guided beam search. The authors argue that repeated pruning introduces a depth-dependent distribution shift, leading to increasingly overestimated PRM scores.

To address this issue, the authors propose an on-policy recalibration method that collects beam-search-selected prefixes and uses Monte Carlo rollout labels to further calibrate existing PRMs. Experiments involving two PRMs, two generator models, and multiple mathematical reasoning benchmarks demonstrate reduced overestimation and accuracy improvements in several configurations.

Critical strengths

The paper identifies a potentially important mismatch between PRM calibration data and the distributions encountered during beam search. This problem is relevant to understanding the reliability of PRMs in test-time reasoning.

The proposed recalibration method leverages search-selected reasoning prefixes without requiring substantial modifications to existing PRM architectures. The authors also conduct experiments across different models and mathematical reasoning benchmarks, demonstrating improvements in several configurations.

Critical weaknesses

1. Insufficient experimental validation.

The paper lacks a key ablation comparing on-policy recalibration with additional fine-tuning on unconditionally sampled prefixes under identical training settings. Therefore, it is unclear whether the improvements come from the proposed on-policy recalibration or simply from additional training. Moreover, the experiments are limited to relatively small 1B and 1.5B generator models, without validation on stronger reasoning models.

2. Presentation and formatting issues.

The paper lacks an overall framework figure that clearly illustrates the proposed method and its main components. This makes the methodology less intuitive to understand. Additionally, Figure 4 has noticeable formatting issues, with numerical annotations placed too close to the plot boundaries, reducing readability.

Questions for authors
  1. Can the authors provide a controlled ablation study to demonstrate the effectiveness of on-policy recalibration compared with standard fine-tuning?

  2. Can the authors improve the presentation by adding a method overview figure and fixing the formatting issues in Figure 4?

2 comments

We thank the reviewer for the comments.

1. Experimental validation

Unconditional-finetuning ablation. We trained a matched unconditional model, with the same starting adapter, labels, loss and pool size (500 MATH problems, Qwen2.5-Math-PRM-7B):

GeneratorBaselineUnconditionalSurvivor
Llama-1B60.8%63.2%65.2%
Qwen-1.5B78.4%81.2%81.4%

Both finetuned models improve over the baseline, and survivor data is the best in both cases. Much of the improvement comes from recalibrating on search-visited data, with survivor data adding a smaller further gain.

Larger generators. We extended the evaluation to 7B and 8B generators. Survivor finetuning improves accuracy for all four generators on MATH500:

GeneratorBaselineSurvivor-FT
Llama-3.2-1B60.8%65.2%
Qwen2.5-Math-1.5B78.4%81.4%
Llama-3.1-8B69.4%70.6%
Qwen2.5-7B-Math-Instruct80.2%82.0%

The gains are smaller for the larger generators (+1.2 and +1.8 points) than for the smaller ones (+4.4 and +3.0).

2. Presentation and formatting

We agree. We will add an overall framework figure showing the method and its components, and fix the placement of the numerical annotations in Figure 4 in the final version.

Thank you for the clarification. The experiments are still limited to relatively weak models, with Qwen2.5-7B being the strongest model evaluated. I also did not see any corresponding changes in the revised manuscript. Given these remaining concerns, I will keep my original score.

Reviewer_PEzpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates survivorship-induced distribution shift in PRM-guided beam search, arguing that repeated pruning leads to increasingly overconfident PRM predictions at greater search depths. To address this issue, the paper proposes on-policy recalibration by collecting beam-surviving prefixes and fine-tuning calibrated PRMs using quantile regression. Experiments show modest improvements in downstream accuracy across several model configurations.

Critical strengths
  1. The paper identifies a practically important and underexplored issue: the calibration mismatch between unconditionally sampled prefixes and beam-search survivors.
  2. The proposed on-policy recalibration method is simple and practical, requiring only lightweight LoRA fine-tuning without modifying the existing PRM architecture.
  3. The experiments cover multiple PRMs, policy models, and search configurations, demonstrating downstream accuracy improvements in most settings. The entropy-based analysis also provides useful insights into when recalibration is most beneficial.
Critical weaknesses
  1. Clarity and Writing Quality. The paper is difficult to follow due to excessive use of long sentences, awkward phrasing, and unnatural transitions. The writing often feels formulaic and potentially AI-generated, which undermines readability and the clarity of the technical arguments.

  2. Insufficient Validation of the Motivation. (1) Figure 1 lacks a clear definition and computation procedure for the PRM-implied density. (2) Figure 2 only reports results using Qwen-PRM, without evaluating Math-Shepherd. More importantly, the paper does not establish a direct relationship between the observed distribution shift and increasing overestimation.

  3. Missing Baseline. The experiments lack a controlled comparison with LoRA fine-tuning on unconditionally sampled prefixes under the same data and training budget. Consequently, it remains unclear whether the improvements come from survivor-based data collection or simply from additional fine-tuning.

  4. Limited Generalization Evidence. The AIME evaluation contains only 60 questions, with improvements corresponding to merely 0–2 additional correct answers across configurations. Such small differences are insufficient to support strong generalization claims.

  5. Limited Comparisons with Existing Methods. Although the paper compares against self-consistency and Best-of-N, these are inference-time scaling baselines rather than alternative calibration approaches. Comparisons with existing PRM calibration or distribution-shift mitigation methods would better establish the effectiveness and novelty of the proposed approach.

Questions for authors
  1. In Figure 1, how is its median density ratio computed? It does not seem to be a standard measure of distribution divergence (e.g., KL divergence).
  2. Proposition 1: In the inductive proof (Appendix A.1), why does the recursive relation (\pi_t = \pi_{t-1} \cdot P(s_t \mid x,s_{<t}) \cdot P(\mathrm{surv}(s_t) \mid x,s_{1})) hold? The authors introduce it as a "recursive definition" without further justification. Could the authors provide a rigorous derivation, particularly explaining how the conditional survival probability accounts for pruning at previous depths?
2 comments

We thank the reviewer for the detailed comments.

W1. Clarity and writing

We will make sure the writing is clear in the final version. We will also include the definitions and methods for the domain classifier.

W2. Validation of the motivation

  • Figure 1. The PRM-implied density is estimated with a domain classifier c(h) trained to separate PRM hidden states of survivor prefixes from those of unconditionally sampled prefixes at matched depth. The density ratio is w(h) = c(h) / (1 − c(h)), approximating p_survivor(h) / p_unconditional(h). We report the median per pool and per depth, because the ratios are heavy-tailed. A value of 1 means the two pools are indistinguishable.
  • Shift and overestimation. Figures 1 and 2 use the same depth axis and the same generators. Llama-1B shows both the larger divergence (Figure 1) and the larger rise in overestimation on survivor prefixes (0.22 to 0.55, against 0.22 to 0.12 for the unconditional pool, Figure 2). Qwen-1.5B shows a smaller divergence and a smaller rise (0.22 to 0.38 against 0.22 to 0.27).
  • Math-Shepherd. Figure 2 uses Qwen-PRM only. We will add the Math-Shepherd version in the final version.

W3. Missing baseline: unconditional finetuning

We trained a matched unconditional model, with the same starting adapter, labels, loss and pool size (500 MATH problems, Qwen2.5-Math-PRM-7B):

GeneratorBaselineUnconditionalSurvivor
Llama-1B60.8%63.2%65.2%
Qwen-1.5B78.4%81.2%81.4%

Both finetuned models improve over the baseline, and survivor data is the best in both cases. Much of the improvement comes from recalibrating on search-visited data, with survivor data adding a smaller further gain.

W4. Generalization evidence

AIME is a very hard dataset, so even one or two more correct answers is hard to get. We present it as additional evidence that the method works on hard problems, not as a generalization claim. The generalization evidence is MATH500, where finetuning improves accuracy for four generators:

GeneratorBaselineSurvivor-FT
Llama-3.2-1B60.8%65.2%
Qwen2.5-Math-1.5B78.4%81.4%
Llama-3.1-8B69.4%70.6%
Qwen2.5-7B-Math-Instruct80.2%82.0%

W5. Comparison with calibration methods

Our baseline is the PRM calibration of Park et al. [1], a recent and the closest prior work on PRM calibration, and our method improves on it in every table.

[Q1] Refer W2

[Q2] The recursion follows from the chain rule. For a given step i, along with the language model's likelihood for a stepistep_i, it also depends on the PRM's selection of the preceding step. We can provide a detailed chain-rule derivation for this.

[1] Park, Young-Jin, et al. "Know What You Don't Know: Uncertainty Calibration of Process Reward Models." Advances in Neural Information Processing Systems 38 (2026): 38852–38895.

Thank you for the clarifications and additional experiments, which address some of my concerns, particularly the unconditional fine-tuning baseline. However, several concerns remain:

  • The additional gains from survivor-based fine-tuning over unconditional fine-tuning are relatively modest, leaving the effectiveness of survivor-specific data selection insufficiently established.
  • The relationship between distribution shift and overestimation is still supported mainly by indirect evidence.

Additionally,

  • Given the additional experiments with larger models (e.g., Qwen2.5-7B) reported in responses to other reviewers, extending the motivation analysis to models of this scale would provide more convincing evidence.
  • I also encourage the authors to pay greater attention to the presentation quality. Adding an overview figure, providing clearer formal definitions and explanations, and simplifying overly long sentences would substantially improve the paper's readability and clarity.

Overall, while I appreciate the authors' efforts, I will maintain my original score.

Reviewer_dvuqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds upon the literature on process reward model-guided large language model mathematical reasoning. The authors argue that calibration on independently generated reasoning trajectories can fail during beam search because the search repeatedly selects prefixes using the same PRM. They instead propose collecting these surviving prefixes and estimating their continuation success rates through Monte Carlo rollouts and further fine-tuning an already calibrated PRM using quantile regression. Experiments cover two 1B generators and two 7B PRMs on MATH500 and AIME datasets.

Critical strengths

The paper studies a mismatch that appears after calibrating based on the state-of-the-art methods and proposes an intervention strategy by comparing survivor and unconditional pools that show promise.

Critical weaknesses

I believe the experiment in many aspects is limited, most importantly, the size of both generator models is 1B, and it is not obvious if the observation would extend to more capable models. Further, the experiments do not isolate the benefit of survivor-based recalibration from additional fine-tuning, changes in label construction, and increased inference compute. Matched unconditional-data training controls and accuracy comparisons at equal compute are needed to establish the specific contribution of survivor-based calibration and its efficiency.

Proposition 1 conflates single-trajectory survival with inclusion in a beam, and a shrinking survival factor alone does not imply increasing distributional divergence or PRM overestimation. The argument requires a precise definition of the search-induced distribution and assumptions connecting selection to prediction error.

Questions for authors

Please refer to the weakness section

1 comment
AuthorsOpenReview ↗

Model Scale, Unconditional Control and Compute

We thank the reviewer for the review and have run the requested experiments.

1. Model scale

We extended the study to larger generators, all with Qwen2.5-Math-PRM-7B on 500 MATH problems. Survivor finetuning improves accuracy for every generator:

GeneratorBaselineSurvivor-FT
Llama-3.2-1B60.8%65.2%
Qwen2.5-Math-1.5B78.4%81.4%
Llama-3.1-8B69.4%70.6%
Qwen2.5-7B-Math-Instruct80.2%82.0%

The gains are smaller for the 7–8B generators (+1.2 and +1.8 points) than for the 1B–1.5B generators (+4.4 and +3.0). Qwen-1.5B shows less out-of-distribution shift under search than Llama-1B, which is consistent with its smaller gain.

2. Unconditional-data control

We trained a matched unconditional model for the two smaller generators, with the same starting adapter, labels, loss and pool size:

GeneratorBaselineUnconditionalSurvivor
Llama-1B60.8%63.2%65.2%
Qwen-1.5B78.4%81.2%81.4%

Both finetuned models improve over the baseline, and survivor data is the best in both cases. Most of the improvement comes from recalibrating on search-visited data, and survivor data adds a smaller further gain. We will update the claims in the paper to say this.

3. Compute

We report the average generation calls with each result. The finetuned PRMs sometimes use more calls (for example +89% on Qwen-1.5B and +88% on Llama-8B), and slightly fewer for Qwen-7B (−7%). We will include accuracy and calls together in the revised tables.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.