- Rating
- 2
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates the limitations of outcome-based supervision for identifying reasoning errors and proposes OPSD-PRM, a reference-guided process verifier trained without human step-level labels. It uses on-policy self-distillation to transfer evaluations from a reference-conditioned teacher to a student that operates without references at inference time. Experiments on ProcessBench demonstrate that complete worked solutions improve first-error localization compared with answer-only supervision, with particularly promising results when successful training candidates are scarce.
Critical strengths
- The paper addresses an important limitation of outcome-based supervision by highlighting the distinction between final-answer correctness and local reasoning validity, supported by theoretical analysis and empirical diagnostics.
- The proposed reference-guided distillation framework effectively leverages worked solutions to train process verifiers without requiring human step-level annotations or references at inference time.
- The paper provides meaningful ablation studies and statistical analyses, demonstrating that complete reference solutions improve error localization, particularly for solutions with correct final answers but flawed reasoning.
Critical weaknesses
- The methodological novelty is limited, as the proposed approach mainly applies existing on-policy self-distillation to reference-guided process verification. Moreover, OPSD provides only a small improvement over Teacher-SFT (1.59 points), making its specific advantage unclear.
- The baseline comparisons are not fully convincing. OPSD performs retrospective verification using complete candidate solutions, while Implicit CE and MC-HE use causal prefix-based scoring. These differences make it difficult to isolate the benefits of the proposed supervision. Comparisons with stronger reference-guided methods are also lacking.
- The absolute error localization performance remains limited, and improvements come with a trade-off in accepting fully correct solutions. Furthermore, experiments are restricted to a single 7B backbone, leaving the generalizability unclear.
Questions for authors
- Could the authors provide stronger evidence that OPSD offers meaningful advantages over simpler Teacher-SFT?
- Could the authors conduct more controlled comparisons using matched verification interfaces and supervision resources, and include relevant reference-guided methods such as SPARE?
- Could the authors further investigate the trade-off between error localization and correct-solution acceptance, and evaluate whether the improvements generalize to other model backbones or reasoning tasks?
4 comments
Updated Manuscript Title
Thank you for reviewing our paper. We have updated the manuscript title to:
Fitting Outcomes, Missing Process Errors: Rethinking Implicit PRMs through Reference Supervision
The new title more directly reflects our analysis of the gap between outcome fitting and process-error localization, together with our experiments on what reference derivations contribute beyond final-answer supervision. The revised PDF is now available on the submission page.
Controlled reference-content comparison at 3B
To address your question about generalization beyond the original 7B setting, we added a matched Full/Answer-only comparison using Qwen2.5-3B-Instruct for both the frozen teacher and the student. Both conditions use the same 1,024 candidates, 64 updates, and reference-free student interface.
Full achieved 22.80 Test-3 F1 versus 18.89 with Answer-only (+3.91 points, paired 95% CI [1.44, 6.41]). The comparison uses one training seed, and the interval reflects evaluation-sample uncertainty. Averaged equally across the three benchmark subsets, exact localization increased from 10.74% to 13.40%, while correct-solution acceptance decreased from 81.84% to 77.56%.
Together with the three-seed 7B comparisons under OPSD and Teacher-SFT, this result extends the controlled reference-content finding to a second model size within the Qwen family. The F1 improvement and localization–acceptance trade-off both appear at 3B. We would appreciate your considering this comparison alongside the additional backbone and interface evaluations in the revised paper.
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts in addressing my concerns, particularly the stronger evidence for reference supervision and the additional generalization studies. While some concerns regarding methodological novelty and verification reliability remain, I believe the rebuttal has meaningfully strengthened the paper. I would be willing to increase my score to somewhere between 2 and 3, as my assessment has become more positive, although I do not feel it quite reaches a 3.
Response to Reviewer 6TYJ
Thank you for the simpler-baseline and input-control suggestions! We completed the paired Teacher-SFT, prefix, SPARE, and transfer experiments. The revised contribution centers outcome-driven credit assignment and reference-information findings; the title is now When Process Rewards Follow Outcomes: Understanding Credit Assignment in Implicit PRMs. Teacher-SFT provides the simpler control and repeats the reference-content contrast.
1. Teacher-SFT versus OPSD: what does the new evidence establish?
We trained both routes with Full and Answer-only references, using the same Qwen backbone, frozen teacher, 1,024 candidates, public student interface, 64 updates, and three paired training seeds.
These are Test-3 means and sample SDs across training seeds. Each route benefits from full derivations in every seed. OPSD's greedy mean is 1.70 points higher, but each seed's between-route interval includes zero. With four votes, Full Teacher-SFT scores 38.64 and Full OPSD 37.98; both retain positive Full–Answer gains. In the new first-output-target comparison, OPSD generates fewer invalid greedy verdicts in seed 20260914 (112/3,400 versus 490/3,400).
The repeated finding is the additional supervision supplied by the reference derivation. Teacher-SFT reproduces that benefit with offline evaluation text; it is an alternative implementation for this targeted contrast. The revised §2 makes the shared response-level CE coefficient and its REINFORCE-like form explicit, then links the analysis to the outcome/localization diagnostic. The reference-content evidence tests a way of supplying additional information (§§3 and 5.3). The original format-valid Teacher-SFT result remains in Appendix D.3; the new three-seed experiment uses first raw teacher outputs without filtering, and is reported separately in Appendix F.
2. Candidate visibility and reference-guided comparisons.
We trained a numeric-prefix verifier that sees only the candidate through the current step. The local systems share the Qwen backbone, 1,024 candidates, and 64 updates. All trained rows below use the common learning rate 1e-6:
The common-rate OPSD–CE difference is 13.50 [11.24, 15.81]. Separately, CE's development-selected 5e-5 result is 18.92; MC-HE selects the common 1e-6 rate. Appendix I reports both comparisons explicitly.
SPARE uses the same frozen Qwen teacher, references, and candidates to generate reference-aligned step labels, followed by step-classifier training. The released UKPLab/Llama-3-8b-spare-prm-math scores 22.84 after GSM8K calibration (14.88 at threshold 0.5).
The prefix comparison aligns candidate visibility and training visits, while output encoding, supervision density, and numerical scoring still differ. The numeric interface already starts at 29.71; the local SPARE adaptation is below its own initialization and represents a fixed-budget adaptation. Thus we distinguish system comparisons from isolated supervision effects (§6.2; Appendix I).
3. Localization/acceptance trade-offs and transfer.
We retain the original full-reference versus answer-only acceptance trade-off and add component-level results for the new interface. Numeric-prefix OPSD has mean Test-3 E/C of 22.98/47.26, versus 20.63/53.94 at initialization. Selected-rate CE has 28.17/14.32: it localizes more erroneous candidates at that operating point but accepts fewer correct solutions. Accordingly, our between-system performance claim is about the benchmark's joint F1 criterion; the evidence for reference content comes from the within-route interventions. Appendix I also distinguishes development-set acceptance matching from test-set acceptance, which is not automatically matched by threshold transfer.
For generalization, an original-backbone Llama-3.1-8B-Instruct implementation improves from 6.16 to 17.49 Test-3 F1, with a gain of 11.33 [9.26, 13.40]. Without further training, the mathematical verifier improves first-error localization on Logical Deduction from 14.97% to 38.44%, and on Word Sorting from 1.13% to 19.55%. Appendix J reports F1, correct acceptance, uncertainty, and earlier backbone settings with no improvement. These extensions expand the demonstrated settings; the replicated reference-content finding remains specifically supported by the paired Qwen experiments.
The revision distinguishes the outcome-driven diagnostic, reference-content effects, and implementation-dependent results.