acceptodds
Under review as a conference paper at ICLR 2027

The Solution Is the Supervision: Learning to Verify Reasoning without Human Step Labels

Abstract

Outcome reward models assess final-answer correctness in mathematical reasoning, while process reward models (PRMs) assess intermediate steps. Implicit PRMs learn local scores from outcome labels, avoiding costly human step annotations. But fitting a response's outcome does not directly supervise where its derivation first becomes invalid. We investigate this distinction between outcome fitting and local judgment, and ask what worked reference solutions add to verification supervision. An aggregate-score analysis and a trained-model diagnostic show that small outcome residuals can coexist with unresolved first-error judgments. To study reference information, we use on-policy self-distillation to transfer a same-backbone frozen teacher's reference-conditioned evaluations to a reference-free student. On ProcessBench, complete references improve macro F1 by 3.58 points over answer-only supervision and exact localization by 5.57 points on correct-answer solutions with process errors. Content and alignment interventions connect these improvements to information in the worked derivation. A separate candidate-composition study finds a larger advantage over Implicit CE when successful candidates are scarce, with the endpoint effect reproduced in a second training seed. Together, the results characterize a local-supervision gap and show how reference-informed evaluation can improve judgments without human candidate-step labels.

Primary area:
foundation or frontier models, including LLMs
Keywords:
process reward models, mathematical reasoning, outcome supervision, first-error localization, privileged information, on-policy self-distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6TYJOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates the limitations of outcome-based supervision for identifying reasoning errors and proposes OPSD-PRM, a reference-guided process verifier trained without human step-level labels. It uses on-policy self-distillation to transfer evaluations from a reference-conditioned teacher to a student that operates without references at inference time. Experiments on ProcessBench demonstrate that complete worked solutions improve first-error localization compared with answer-only supervision, with particularly promising results when successful training candidates are scarce.

Critical strengths
  1. The paper addresses an important limitation of outcome-based supervision by highlighting the distinction between final-answer correctness and local reasoning validity, supported by theoretical analysis and empirical diagnostics.
  2. The proposed reference-guided distillation framework effectively leverages worked solutions to train process verifiers without requiring human step-level annotations or references at inference time.
  3. The paper provides meaningful ablation studies and statistical analyses, demonstrating that complete reference solutions improve error localization, particularly for solutions with correct final answers but flawed reasoning.
Critical weaknesses
  1. The methodological novelty is limited, as the proposed approach mainly applies existing on-policy self-distillation to reference-guided process verification. Moreover, OPSD provides only a small improvement over Teacher-SFT (1.59 points), making its specific advantage unclear.
  2. The baseline comparisons are not fully convincing. OPSD performs retrospective verification using complete candidate solutions, while Implicit CE and MC-HE use causal prefix-based scoring. These differences make it difficult to isolate the benefits of the proposed supervision. Comparisons with stronger reference-guided methods are also lacking.
  3. The absolute error localization performance remains limited, and improvements come with a trade-off in accepting fully correct solutions. Furthermore, experiments are restricted to a single 7B backbone, leaving the generalizability unclear.
Questions for authors
  1. Could the authors provide stronger evidence that OPSD offers meaningful advantages over simpler Teacher-SFT?
  2. Could the authors conduct more controlled comparisons using matched verification interfaces and supervision resources, and include relevant reference-guided methods such as SPARE?
  3. Could the authors further investigate the trade-off between error localization and correct-solution acceptance, and evaluate whether the improvements generalize to other model backbones or reasoning tasks?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 6TYJ

Thank you for the simpler-baseline and input-control suggestions! We completed the paired Teacher-SFT, prefix, SPARE, and transfer experiments. The revised contribution centers outcome-driven credit assignment and reference-information findings; the title is now When Process Rewards Follow Outcomes: Understanding Credit Assignment in Implicit PRMs. Teacher-SFT provides the simpler control and repeats the reference-content contrast.

1. Teacher-SFT versus OPSD: what does the new evidence establish?

We trained both routes with Full and Answer-only references, using the same Qwen backbone, frozen teacher, 1,024 candidates, public student interface, 64 updates, and three paired training seeds.

Transfer routeAnswer-only F1Full F1Paired reference gain
OPSD29.98 ± 0.7435.68 ± 0.71+5.70 ± 1.36
Teacher-SFT29.93 ± 0.9133.98 ± 0.13+4.05 ± 1.03

These are Test-3 means and sample SDs across training seeds. Each route benefits from full derivations in every seed. OPSD's greedy mean is 1.70 points higher, but each seed's between-route interval includes zero. With four votes, Full Teacher-SFT scores 38.64 and Full OPSD 37.98; both retain positive Full–Answer gains. In the new first-output-target comparison, OPSD generates fewer invalid greedy verdicts in seed 20260914 (112/3,400 versus 490/3,400).

The repeated finding is the additional supervision supplied by the reference derivation. Teacher-SFT reproduces that benefit with offline evaluation text; it is an alternative implementation for this targeted contrast. The revised §2 makes the shared response-level CE coefficient and its REINFORCE-like form explicit, then links the analysis to the outcome/localization diagnostic. The reference-content evidence tests a way of supplying additional information (§§3 and 5.3). The original format-valid Teacher-SFT result remains in Appendix D.3; the new three-seed experiment uses first raw teacher outputs without filtering, and is reported separately in Appendix F.

2. Candidate visibility and reference-guided comparisons.

We trained a numeric-prefix verifier that sees only the candidate through the current step. The local systems share the Qwen backbone, 1,024 candidates, and 64 updates. All trained rows below use the common learning rate 1e-6:

Prefix systemTest-3 F1
Numeric-interface Initial29.71
Numeric-prefix OPSD, ordinary KL30.73
Implicit CE, common rate17.23
MC-HE, common rate16.21
SPARE-interface Initial16.69
SPARE same-teacher adaptation13.70

The common-rate OPSD–CE difference is 13.50 [11.24, 15.81]. Separately, CE's development-selected 5e-5 result is 18.92; MC-HE selects the common 1e-6 rate. Appendix I reports both comparisons explicitly.

SPARE uses the same frozen Qwen teacher, references, and candidates to generate reference-aligned step labels, followed by step-classifier training. The released UKPLab/Llama-3-8b-spare-prm-math scores 22.84 after GSM8K calibration (14.88 at threshold 0.5).

The prefix comparison aligns candidate visibility and training visits, while output encoding, supervision density, and numerical scoring still differ. The numeric interface already starts at 29.71; the local SPARE adaptation is below its own initialization and represents a fixed-budget adaptation. Thus we distinguish system comparisons from isolated supervision effects (§6.2; Appendix I).

3. Localization/acceptance trade-offs and transfer.

We retain the original full-reference versus answer-only acceptance trade-off and add component-level results for the new interface. Numeric-prefix OPSD has mean Test-3 E/C of 22.98/47.26, versus 20.63/53.94 at initialization. Selected-rate CE has 28.17/14.32: it localizes more erroneous candidates at that operating point but accepts fewer correct solutions. Accordingly, our between-system performance claim is about the benchmark's joint F1 criterion; the evidence for reference content comes from the within-route interventions. Appendix I also distinguishes development-set acceptance matching from test-set acceptance, which is not automatically matched by threshold transfer.

For generalization, an original-backbone Llama-3.1-8B-Instruct implementation improves from 6.16 to 17.49 Test-3 F1, with a gain of 11.33 [9.26, 13.40]. Without further training, the mathematical verifier improves first-error localization on Logical Deduction from 14.97% to 38.44%, and on Word Sorting from 1.13% to 19.55%. Appendix J reports F1, correct acceptance, uncertainty, and earlier backbone settings with no improvement. These extensions expand the demonstrated settings; the replicated reference-content finding remains specifically supported by the paired Qwen experiments.

The revision distinguishes the outcome-driven diagnostic, reference-content effects, and implementation-dependent results.

AuthorsOpenReview ↗

Updated Manuscript Title

Thank you for reviewing our paper. We have updated the manuscript title to:

Fitting Outcomes, Missing Process Errors: Rethinking Implicit PRMs through Reference Supervision

The new title more directly reflects our analysis of the gap between outcome fitting and process-error localization, together with our experiments on what reference derivations contribute beyond final-answer supervision. The revised PDF is now available on the submission page.

AuthorsOpenReview ↗

Controlled reference-content comparison at 3B

To address your question about generalization beyond the original 7B setting, we added a matched Full/Answer-only comparison using Qwen2.5-3B-Instruct for both the frozen teacher and the student. Both conditions use the same 1,024 candidates, 64 updates, and reference-free student interface.

Full achieved 22.80 Test-3 F1 versus 18.89 with Answer-only (+3.91 points, paired 95% CI [1.44, 6.41]). The comparison uses one training seed, and the interval reflects evaluation-sample uncertainty. Averaged equally across the three benchmark subsets, exact localization increased from 10.74% to 13.40%, while correct-solution acceptance decreased from 81.84% to 77.56%.

Together with the three-seed 7B comparisons under OPSD and Teacher-SFT, this result extends the controlled reference-content finding to a second model size within the Qwen family. The F1 improvement and localization–acceptance trade-off both appear at 3B. We would appreciate your considering this comparison alongside the additional backbone and interface evaluations in the revised paper.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts in addressing my concerns, particularly the stronger evidence for reference supervision and the additional generalization studies. While some concerns regarding methodological novelty and verification reliability remain, I believe the rebuttal has meaningfully strengthened the paper. I would be willing to increase my score to somewhere between 2 and 3, as my assessment has become more positive, although I do not feel it quite reaches a 3.

Reviewer_URQFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies first-error verification for mathematical reasoning, where a verifier must localize the earliest invalid step in a candidate solution. The authors first argue that outcome-level supervision can fit final-answer correctness while leaving localization errors unresolved. They then propose to use worked references as privileged training information: a frozen teacher reads the reference and supplies distributions at the student’s own evaluation prefixes, and an on-policy self-distillation (OPSD) student learns to generate reference-free evaluations. At deployment, the student returns a first-error index or accepts the solution without access to a reference.

Critical strengths
  1. The paper addresses a meaningful and well-motivated problem: outcome correctness does not specify process validity, and a correct final answer can conceal an invalid intermediate step.
  2. The reference interventions are informative. Full references improve both frozen-teacher judgments and trained-student verification, and the shuffled-reference and answer-only controls help isolate the contribution of derivation content and problem alignment.
Critical weaknesses
  1. The method is an application of an existing OPSD procedure. The paper is explicit that OPSD is from Zhao et al. (2026b) and that the authors use its released frozen-teacher option and pointwise cap. The novel contribution is therefore the diagnostic, the reference interventions, and the fixed-candidate verification setup, not the distillation objective itself. This is acceptable, but the contribution claim should be calibrated accordingly.
  2. Single backbone and single domain. All experiments use Qwen2.5-7B-Instruct and mathematical first-error verification. The paper acknowledges this, but the generality of the conclusions is limited. It is unclear whether the reference-content effects and the low-success advantage would hold for other backbones, other reasoning domains, or other verifier interfaces.
  3. The composition study changes two factors at once. Lowering the success proportion while fixing the total number of problems also increases the number of failed candidates. The paper notes this, but the causal interpretation of the “low-success advantage” is therefore ambiguous.
  4. Presentation and notation. The paper is dense, and several details are deferred to appendices. Equation 4 uses a pointwise cap and signed contributions; the main text would benefit from a clearer explanation of what the cap does in this task.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer URQF

Thank you for recognizing the diagnostic and reference interventions! The revision centers these findings and adds scope and composition experiments.

1. Contribution: diagnosis and reference-information findings.

The revised title is When Process Rewards Follow Outcomes: Understanding Credit Assignment in Implicit PRMs. The contribution centers the outcome-driven formation of local credit and its empirical implications. Revised §2 now presents the CE gradient alongside terminal-reward REINFORCE, with the shared response-level coefficient explicit; it also explains the prior implicit-PRM prefix-value identity and distinguishes that statistical value from step validity. The existing score-allocation example and 924-example low-residual diagnostic remain in the main text. Preference-objective cases and curvature details remain in Appendix A.

OPSD is the main implementation for testing supplementary reference information; Teacher-SFT repeats the Full/Answer-only contrast with offline evaluation text. The objective and cap retain their attribution to OPSD.

A new paired three-seed study shows that Full references improve Test-3 F1 over Answer-only by 4.05 ± 1.03 points with Teacher-SFT and 5.70 ± 1.36 with OPSD. The Teacher-SFT control therefore reproduces the reference-content benefit under offline evaluation-text supervision (§5.3, Table 2). These findings complement the supervision analysis in §2.

2. Additional backbones, tasks, and verifier interfaces.

We completed three extensions, each with an explicit comparison to its own initial model:

ExtensionInitialReference-trained verifier
Llama-3.1-8B-Instruct, Test-3 F16.1617.49
Causal numeric-prefix interface, Test-3 F129.7130.73
Logical Deduction, exact first-error rate14.97%38.44%
Word Sorting, exact first-error rate1.13%19.55%

The Llama experiment starts from the original backbone with a same-backbone frozen teacher and 32 updates. Its F1 gain is 11.33 [9.26, 13.40]. The prefix experiment removes access to all later candidate steps; its training increment is 1.02 [−0.55, 2.57], so it establishes an implemented prefix verifier without establishing a reliable training gain over that interface's strong initialization.

The two non-mathematical tasks use the existing mathematical checkpoint with no additional training. Their F1 values change from 24.44 to 43.46 and from 2.22 to 16.79, respectively. The Logical Deduction F1 interval is wide because only six examples have fully correct processes; Appendix J reports all E/C components. Earlier Llama and Mistral settings that did not improve are also retained there. These extensions demonstrate working settings beyond the original backbone/domain. The replicated Full/Answer-only and composition findings are specifically established in the Qwen mathematical setting (§6.2; Appendices I–J).

3. Success scarcity versus additional failed candidates.

We added a bridge and now display it in §5.5, Table 4: A contains 512 successful/512 failed candidates, B contains 128/512, and C contains 128/896. All receive 2,048 visits and 64 updates at the original common learning rate. OPSD scores 32.89 → 34.88 → 37.73, while CE scores 25.65 → 20.79 → 21.53.

B and C hold the successful candidate set fixed. Expanding the failed pool raises OPSD by 2.85 [0.79, 4.89] points; CE changes by 0.74 [−1.02, 2.42]. The difference in gains is 2.11 [−0.51, 4.78]. These observations connect the supervision question to the use of additional failed trajectories. The fixed visit budget also changes repetition and successful-example exposure, so we report a candidate-pool intervention, with the full protocol in Appendix G.1.

Separately, the endpoint interaction survives CE's selected 5e-5 learning rate: +5.13 [2.10, 8.17] and +4.66 [1.67, 7.62] across two seeds. Selected-rate CE is nearly stable between endpoints, while OPSD improves in the failure-heavy pool. The main text summarizes this check, with its full figure in Appendix G; Figure 3 retains the original common-rate experiment.

4. Pointwise-cap intuition and presentation.

Section 3.2 now explains the cap immediately after Equation 4. Each signed vocabulary contribution is capped before summation. Large positive terms arise when a teacher-favored token is strongly underestimated by the student; the cap limits their domination while retaining negative terms. This follows OPSD's stylistic-divergence motivation and is a signed surrogate, not a new divergence. Our full-evaluation control scores 35.43 with the cap versus 31.50 with ordinary KL; the numeric-prefix configuration instead selects ordinary KL. The revision therefore presents the cap as an implementation choice whose observed effect depends on the verification setting, with its gradient in Appendix A.3 and the original control in Appendix D.3.

AuthorsOpenReview ↗

Updated Manuscript Title

Thank you for reviewing our paper. We have updated the manuscript title to:

Fitting Outcomes, Missing Process Errors: Rethinking Implicit PRMs through Reference Supervision

The new title more directly reflects our analysis of the gap between outcome fitting and process-error localization, together with our experiments on what reference derivations contribute beyond final-answer supervision. The revised PDF is now available on the submission page.

Reviewer_Zbb6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the gap between outcome fitting and first-error judgment through a score-space argument and an Implicit CE diagnostic. It applies OPSD to verification: a frozen, reference-conditioned Qwen2.5-7B-Instruct teacher supervises a LoRA student's evaluation prefixes. The reference-free student reaches 40.38 Macro-4 F1 on ProcessBench. Reference interventions examine local judgments, with further studies of candidate budgets and composition.

Critical strengths
  • Correct-answer solutions with process errors expose a failure that answer-level evaluation can miss. Studying this gap helps clarify what evidence is needed to assess a process verifier.
  • Keeping the student interface fixed while varying reference content gives a credible basis for attributing part of the gain to aligned derivations. This provides useful guidance on the supervision needed to learn local judgments
Critical weaknesses
  • Student-prefix distillation and the signed pointwise cap come from OPSD (§3.2, Appendix B.3); SPARE already uses references for process supervision. The contribution rests on applying this recipe to verification and the empirical findings discussed below. Recent work on privileged-information forms in OPSD (arXiv:2609.20612) would provide context, though its timing makes the omission understandable.
  • Capped OPSD gains only 1.59 Test-3 F1 [−0.82, 3.97] over SFT (Table 3). Among 514 Test-3 correct-answer/process-error solutions, the localization difference is +0.58 [−3.75, 4.89], and SFT falsely accepts fewer errors (20.82% vs. 34.24%; Tables 19–20). These contrasts, including cap and reference interventions, use one training seed. Matched Full/Answer-only comparisons under both routes would test whether the reference effect survives training variation, which fixed-checkpoint bootstrap intervals leave unmeasured.
Questions for authors
  • How often do core-group candidates explicitly repair an earlier error? Does manual checking of generated verifications support their predicted indices?
  • Do gains persist with sampled majority voting on existing checkpoints? How many invalid verdicts hit the 1,024-token limit, and does the §5.5 low-success advantage survive CE's selected learning rate?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Zbb6

Thank you for identifying the paired reference study and decoding checks that would most directly test our conclusions! We completed these experiments and revised the title to When Process Rewards Follow Outcomes: Understanding Credit Assignment in Implicit PRMs to make the supervision analysis and findings explicit. OPSD remains our main experimental implementation; a matched Teacher-SFT control tests whether the reference-content finding also holds with offline evaluation-text supervision.

1. Contribution and replication of the reference-content finding.

Our starting observation is that implicit process rewards are trained through response-level outcome coefficients, even though they can be read locally. Revised §2 explicitly compares the CE update with terminal-reward REINFORCE, explains the distinction between completion-dependent prefix value and step validity, and connects aggregate-score analysis to the low-residual diagnostic. Section 5 tests what worked-reference evaluations add within a fixed student interface. The new comparison uses the same Qwen backbone, frozen teacher, 1,024 candidates, 64 updates, and public student interface in a two-route × two-reference × three-seed design.

Transfer routeAnswer-onlyFull referencePaired Full − Answer
OPSD29.98 ± 0.7435.68 ± 0.71+5.70 ± 1.36
Teacher-SFT29.93 ± 0.9133.98 ± 0.13+4.05 ± 1.03

Values are Test-3 F1 means and sample SDs over three training seeds. Every within-route reference contrast is positive in every seed, with its paired evaluation-sample interval above zero. This reproduces the reference-content benefit under a second training procedure. The OPSD–SFT intervals include zero; the repeated Full/Answer-only effect is the finding supported by this control (§5.3, Table 2; Appendix F).

The new SFT experiment uses each teacher's first raw evaluation, without output filtering or resampling. The original format-valid SFT result remains separately reported in Appendix D.3. We also added the suggested privileged-information study (arXiv:2609.20612) to the related work, distinguishing its reasoning-generation setting from our outcome/local-verification diagnostic and fixed-interface reference interventions.

2. Explicit repair and manual checking.

Two authors manually reviewed 150 core-group candidates, stratified across the four subsets, and 450 corresponding evaluations. Fourteen candidates explicitly recognized, retracted, or attempted to repair an earlier error: the stratified estimate is 9.34% [4.72, 14.07]. This includes partial and unsuccessful repairs. Overt repair is therefore present but is not the predominant explanation for answer–process disagreement in this sample.

Manual checking also separates index correctness from the mathematical support provided by the explanation:

ModelExact index / 150Exact index with supporting explanation
OPSD Full359/35
OPSD Answer-only297/29
Teacher-SFT Full3616/36

The generated explanations do not consistently substantiate correct indices. We therefore define internalization through reference-free verification performance and report explanation reliability separately, with the rubric and complete counts in Appendix H. The manual labels are used only for this evaluation.

3. Sampled voting and the token limit.

With four sampled evaluations from the existing seed-20260914 checkpoints, Full/Answer-only F1 is 37.98/30.79 for OPSD and 38.64/33.70 for Teacher-SFT. The reference gains remain positive: +7.18 [4.65, 9.70] and +4.94 [2.82, 7.02], respectively. Teacher-SFT benefits from voting as well; OPSD's greedy lead is not preserved as a universal route ranking. Invalid verdicts participate in the vote, and ties use the earliest draw.

For the original 200-update model discussed in the paper, 254 of 263 invalid verdicts (96.58%) reach the 1,024-token limit. Appendix F.3 distinguishes these original counts from the new 64-update checkpoints.

4. Does the composition effect survive CE's selected learning rate? Yes.

We used CE's GSM8K-selected rate, 5e-5, at both composition endpoints. The OPSD–CE gaps at 12.5% versus 50% success are +3.13 versus −2.00 in seed 20260914 and +0.51 versus −4.15 in seed 20260915. The changes in these gaps are +5.13 [2.10, 8.17] and +4.66 [1.67, 7.62]. Thus the relative advantage toward the failure-heavy condition persists after learning-rate selection. CE leads at 50%, and the revised interpretation is a composition-dependent gap rather than uniform OPSD superiority (§5.5; Appendix G). Figure 3 presents the original common-rate experiment; the selected-rate check has its own figure and table in Appendix G.

Thank you for the detailed response and the new experiments. You addressed each of my questions, and I appreciate that you reported the less favorable results openly. Table and section numbers refer to the original submission.

My original points

  • Replication (your point 1). Full references beat answer-only references in every seed under both OPSD and teacher SFT (+5.70 and +4.05 Test-3 F1). I now consider this effect well replicated, and I agree that it, rather than a difference between the two routes, is what the new control shows.
  • Repair and explanations (your point 2). Explicit repair (9.34%) is not the main source of answer–process disagreement. The explanation check matters most to me: in your 150-candidate check, only 9 of 35 correct OPSD indices are supported by their explanations (16 of 36 for SFT).
  • Voting, truncation, and CE (your points 3 and 4). The reference gain holds under voting, nearly all invalid outputs are truncations, and I agree with your revised reading of the CE results.

Why I am keeping my score. The replication concern is resolved. As I noted in my first weakness, however, the contribution rests on the empirical findings rather than on the training method, and the paper itself notes that teacher SFT also trains an effective verifier (§6.2). Two things keep me from considering these findings sufficient yet. First, GenRM (Zhang et al., ICLR 2025; arXiv:2408.15240), which the paper cites, already shows that reference-guided explanations can train a verifier that works without references; the new part, that the derivation helps beyond the final answer in a same-model, first-error setting, is useful but incremental. Second, most correct indices are not supported by valid reasoning, so the F1 gain does not yet show that the model has learned to check reasoning.

One question I did not raise before. I raise it now so that you can respond, and I will weigh your answer. As §5.4 notes, full references raise localization but lower the acceptance of fully correct solutions (72.77% to 59.03%). Because localization is far below acceptance, F1 rewards this trade even without better judgment. By my rough calculation from Tables 11 and 12, the MCC for detecting errors among correct-answer solutions is about 0.34 with both reference types. Strictness-robust metrics for the new runs, such as MCC (MR-GSM8K, ICLR 2025; arXiv:2312.17080) or balanced accuracy (Hard2Verify, arXiv:2510.13744), would show whether the verifier truly improves; if so, I would gladly reconsider my score.

Suggestions

  • The shuffled prompt appears to call the donor solution "a correct solution to this problem" (Appendix B.8), so it tests misleading rather than irrelevant references, which may explain the contrast with OP²SD (arXiv:2608.09228). STV (arXiv:2605.30290) finds SFT unhelpful in a different setting (verify–refine loops with RL). Both merit a brief note.
  • The title could emphasize the reference-supervision findings and relate the implicit-PRM framing to Zhang et al. (arXiv:2501.07301), Setlur et al. (arXiv:2410.08146), and recent work (arXiv:2604.13197; arXiv:2606.27739).
AuthorsOpenReview ↗

Follow-up: Judgment Metrics and Manuscript Revisions

Thank you for recognizing the replicated reference-content effect! We added the requested judgment metrics and completed the literature and framing revisions.

1. Judgment metrics and their interpretation.

For the new three-seed, 64-update paired runs, we pool 1,500 correct-answer Test-3 candidates: 514 erroneous and 986 fully correct. Values average checkpoint metrics on this cohort, rather than benchmark macro scores. MCC is unscaled; BA and differences are percentages and percentage points.

Binary detection distinguishes erroneous from fully correct solutions. ErrorID BA averages exact first-error localization and correct-solution acceptance, following Hard2Verify.

MetricRouteAnswer-onlyFullDifference [95% CI]
Binary MCCOPSD0.41690.4151-0.0018 [-0.0370, 0.0344]
Binary MCCSFT0.38080.3984+0.0177 [-0.0178, 0.0524]
Binary BAOPSD68.9471.35+2.41 [0.67, 4.19]
Binary BASFT69.6870.98+1.30 [-0.56, 3.12]
ErrorID BAOPSD53.2149.53-3.68 [-5.18, -2.16]
ErrorID BASFT46.6943.55-3.14 [-4.90, -1.37]

Intervals use 10,000 paired question-cluster bootstrap resamples stratified by benchmark subset, conditional on the trained checkpoints. Binary metrics map invalid verdicts to rejection; ErrorID gives them no credit. OPSD's binary-BA gain is positive in every seed. Mapping invalids to acceptance also retains it: +2.45 [0.71, 4.21] points. SFT's binary comparisons remain inconclusive.

For OPSD, exact localization rises by 5.51 points while correct acceptance falls by 12.88, yielding -3.68 ErrorID BA. Binary error detection rises by 17.70 points, yielding +2.41 binary BA. Both BA measures weight the classes equally. Thus OPSD improves binary detection balance at this operating point, while MCC shows no clear gain and ErrorID BA decreases. The replicated ProcessBench F1 gains reflect a different, harmonic balance of localization and acceptance.

Appendix F.4 (Tables 33-34) reports these results; Section 5.4 summarizes the trade-off. The abstract, contributions, and conclusion specify the supported verification gains without claiming uniformly improved judgment across metrics.

2. Contribution, related work, and title.

GenRM-CoT already trains reference-free mathematical verifiers using reference-guided rationales. Our contribution concerns outcome-driven credit formation and the information added by reference derivations.

Section 2 characterizes response-level weighting of context-dependent token gradients and documents confident outcome classification with incorrect localization. Section 5.5 tests additional failed candidates with the successful set fixed. Within a fixed student interface, Full/Answer-only interventions test the contribution of worked derivations: Test-3 F1 gains repeat under OPSD (+5.70) and Teacher-SFT (+4.05). The contribution is this supervision analysis and controlled evidence; OPSD is an implementation.

Sections 2 and 7 now discuss IPVRM and LCA, alongside the existing discussion of Zhang et al. and Setlur et al.. IPVRM learns prefix success scores; LCA learns credit assignment through a multiple-instance objective. We relate these approaches to our diagnostics and reference-content interventions. The revised title is Fitting Outcomes, Missing Process Errors: Rethinking Implicit PRMs through Reference Supervision, foregrounding the gap between outcome fitting and process-error identification and the role of reference supervision.

3. What the explanation audit establishes.

We agree that an exact index alone does not demonstrate sound reasoning. Supporting explanations accompany 9/35 exact OPSD indices and 16/36 SFT indices in the audit. Section 5.4 now states these counts directly; Appendix H gives the rubric and full results. Our internalization claim concerns transferred reference-free verification performance. Reliable explanation generation remains unestablished.

4. Shuffled references and the SFT comparison.

Section 5.3 now explicitly contrasts our misleading-reference prompt with OP2SD, which identifies its example as belonging to another problem. This is a difference in interventions, not an experimentally established cause of their differing results. The Full/Answer-only finding is independent of the shuffled condition. Section 7 also notes STV's unsuccessful teacher-SFT comparison in its tested verification-refinement loops and distinguishes that endpoint from our first-error judgments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.