- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes calibrated a step-label editing for improving the predictions of an existing process reward model without changing the reasoning trace or model weights. An Anchor PRM first produces the original step labels. Candidate first-error edits are then supplied by a shared reference PRM or a generative Critic. A lightweight selector decides which candidate edit to apply while retaining the Anchor prediction when no candidate is predicted to be sufficiently beneficial. The paper primarily compares three selection targets: paired logistic models that predict whether an edit fixes or breaks first-error correctness; PAIR-EF, which further subdivides first-error events; and metric-linear regression, which directly predicts the marginal effect of an edit on the native step-level evaluation metric. Experiments cover PRMBench and an adapted DeltaBench reasoning-correctness task. The main results show that learned edit selection consistently improves Anchor labels, while unconditionally applying Critic edits can reduce performance. Learning the native utility of the complete edit generally outperforms increasingly fine-grained modeling of first-error events. However, the additional value of generative Critics relative to a reference PRM varies across models and datasets.
Critical strengths
- The experiments are thorough and clear, also covering different models.
- The writting is clear.
- The paper shows both formally and empirically that identical first-error outcomes lead to different consequences for the complete step-label vector.
Critical weaknesses
- The main setting uses approximately 1,000 annotated traces, which might be metric- and distribution-specific. It is unclear whether the learned policy remains effective under class or evaluation-metric shift.
- The benefit of generative Critics is inconsistent. Reference-only candidates outperform the combined Critic pool on both PRMBench and DeltaBench.
- The method primarily edits the first-error location, making it difficult to handle multiple independent errors. The paper does not evaluate whether this restriction is sufficient for traces containing multiple independent errors.
- It is unclear whether the proposed method outperforms the LoRA finetuned Anchor given the same labelling budget. It might be benficial to discuss this.
Questions for authors
Please refer to the Weaknesses.
We sincerely thank you for your time, effort, and support for our work. We hope that our responses have addressed your concerns.