acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Edit: Improving Process Verification through Calibration

Abstract

Process reward models (PRMs) assess intermediate reasoning steps but may miss errors or reject valid steps. Existing PRMs and generative critics offer alternative judgments, but deciding which alternatives improve a verifier’s current predictions remains challenging. We study calibrated step-label editing, which learns to select corrections while keeping the reasoning text and language-model unchanged. Proposed error locations are converted into candidate label edits, and a lightweight selector trained on annotated reasoning traces determines which edits to apply. Experiments on PRMBench and an adapted DeltaBench task show gains across multiple PRMs. Comparisons of candidate sources show that PRM scores provide useful corrections, while the additional value of critics varies across models. On PRMBench, learning an edit’s effect on the verification metric yields greater gains than predicting changes in first-error correctness; finer first-error categories offer little additional benefit. These findings support improving process verification through learned selection of existing model judgments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
process reward models, reasoning verification, calibrated label editing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4CydOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes calibrated a step-label editing for improving the predictions of an existing process reward model without changing the reasoning trace or model weights. An Anchor PRM first produces the original step labels. Candidate first-error edits are then supplied by a shared reference PRM or a generative Critic. A lightweight selector decides which candidate edit to apply while retaining the Anchor prediction when no candidate is predicted to be sufficiently beneficial. The paper primarily compares three selection targets: paired logistic models that predict whether an edit fixes or breaks first-error correctness; PAIR-EF, which further subdivides first-error events; and metric-linear regression, which directly predicts the marginal effect of an edit on the native step-level evaluation metric. Experiments cover PRMBench and an adapted DeltaBench reasoning-correctness task. The main results show that learned edit selection consistently improves Anchor labels, while unconditionally applying Critic edits can reduce performance. Learning the native utility of the complete edit generally outperforms increasingly fine-grained modeling of first-error events. However, the additional value of generative Critics relative to a reference PRM varies across models and datasets.

Critical strengths
  1. The experiments are thorough and clear, also covering different models.
  2. The writting is clear.
  3. The paper shows both formally and empirically that identical first-error outcomes lead to different consequences for the complete step-label vector.
Critical weaknesses
  1. The main setting uses approximately 1,000 annotated traces, which might be metric- and distribution-specific. It is unclear whether the learned policy remains effective under class or evaluation-metric shift.
  2. The benefit of generative Critics is inconsistent. Reference-only candidates outperform the combined Critic pool on both PRMBench and DeltaBench.
  3. The method primarily edits the first-error location, making it difficult to handle multiple independent errors. The paper does not evaluate whether this restriction is sufficient for traces containing multiple independent errors.
  4. It is unclear whether the proposed method outperforms the LoRA finetuned Anchor given the same labelling budget. It might be benficial to discuss this.
Questions for authors

Please refer to the Weaknesses.

1 comment

We sincerely thank you for your time, effort, and support for our work. We hope that our responses have addressed your concerns.

Reviewer_Co6aOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper seeks to improve process reward models by using a selector model to review the step-level label proposed edits and deciding which to keep and which to modify.

Critical strengths
  1. Well designed, well thought through
  2. Experiments are clean and isolate effect
  3. Demonstrates improvements over alternative methods
Critical weaknesses

Perhaps this is only true for someone who isn't steeped in process reward models but the challenge for a reader like myself is keeping track of all the moving pieces. I think this is more of a clarity point than a validity point. We have the reasoning model, we have the process reward model, we have the critic model, and now we have a selector model. Each of these are being trained semi-independently and with varying data and training strategies. I appreciate that the authors tried to reduce the complexity of this whole system by fixing as much as they could constant and focusing on improving the labels assigned by the process reward model. But even here I believe we have two more models. To whatever extent possible helping to reduce the confusion of having the reader trying to keep track of all these pieces and which is responsible for what will be very helpful to the overall impact of the paper.

Questions for authors
  1. Are there meaningful differences between EF, Paired Logistic and Metric-Linear? What are the drivers of the difference and if there are no differences how should readers think about which to use?

  2. with so many choices and moving parts how can you be confident that the results generalize? Are there other experiments you would run to ensure more general improvements?

  3. In Table 6 why does combined critic selection underperform the reference-only selection

1 comment

We sincerely thank you for your time, effort, and support for our work. We hope that our responses have addressed your concerns.

Reviewer_p8NZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes calibrated step-label editing, a post-hoc method that improves frozen PRMs by selecting useful label corrections from existing PRMs or LLM critics. A lightweight selector learns when to apply an edit, improving performance by about 4 points on both PRMBench and DeltaBench.

Critical strengths
  1. Experiments are carefully controlled, with strong baselines, grouped splits, multiple seeds, matched ablations, and replication checks, which makes the comparisons convincing.

  2. Paper shows that first-error correctness alone can miss whether an edit actually improves step-level verification, and that metric-aware selection gives more consistent gains.

Critical weaknesses
  1. Most of the gains seem to come from using a stronger reference PRM rather than the LLM critics, and the paper is missing simple baselines like using or ensembling that PRM directly.

  2. Claimed advantage of metric-aware selection is quite small once learner differences are controlled for, and it does not clearly hold on DeltaBench or translate to downstream reasoning performance.

Questions for authors
  1. How much of the gain comes from the stronger reference PRM rather than the editing framework or generative Critics?

  2. Does the metric-aware target still help under a matched learner, and do these gains hold on correct traces, longer DeltaBench examples, or downstream tasks such as best-of-N?

  3. Why were the original correct PRMBench traces not passed through the full editing pipeline, and what is the false-alarm rate when the method is applied to fully correct reasoning?

1 comment

We sincerely thank you for your time, effort, and support for our work. We hope that our responses have addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.