acceptodds
Under review as a conference paper at ICLR 2027

Learning Reusable Reasoning Critics from Expert-Referenced Supervision

Abstract

Process-level reward models can score intermediate reasoning, but they typically require step-level annotations, preference data, or external verifiers. We study whether paired expert demonstrations can instead train a reusable reasoning critic without any of these. A naive expert-versus-policy classifier risks learning source or formatting cues instead of reasoning quality. We introduce the **Prefix-level Expert-Referenced Critic (PERC)**, which avoids this by using each expert trace three ways: as a positive example, as the reference answer for weakly labelling policy rollouts by agreement, and as the basis for targeted corruptions used as hard negatives, so that policy traces appear on both sides of the classifier and provenance alone cannot separate positives from negatives. Across GSM8K, MMLU-Pro, and MedReason, this critic captures a reusable reasoning-quality signal. As an **inference-time ranker**, it improves Best-of-16 pass@1 in all main reranking settings, transfers positively across all 27 source–target pairs with gains up to 12.7 points, complements consistency-based voting, and continues to improve pass@1 even for policies already trained with it. As a **process-level evaluator**, prefix-value changes localise human-labelled MedReason error units better than chance and entropy baselines and identify controlled GSM8K perturbations with up to Hit@1. It is also a viable **training signal** for GRPO and remains competitive with imitation baselines without task-specific reward tuning. Overall, PERC shows that weak expert-referenced supervision yields reusable critics for inference-time selection and error diagnosis, while remaining feasible as a policy-training reward.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, LLM, reasoning, process reward models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YxCwOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose the Prefix-level Expert-Referenced Critic (PERC), which trains a prefix classifier using expert traces and answer-agreeing policy rollouts as positives, and answer-mismatched rollouts plus targeted corrupted expert traces as negatives. Experiments on GSM8K, MMLU-Pro, and MedReason demonstrate that the frozen critic improves inference-time reranking, localises reasoning errors, and serves as a viable reward signal for GRPO policy optimisation.

Critical strengths
  1. The training setup reduces source and formatting shortcuts by putting policy traces on both sides of the classifier and using corrupted expert traces as hard negatives, so the critic tracks reasoning quality rather than style.

  2. The same frozen critic is tested for reranking, error localisation, and policy training. Reranking transfers across all 27 source–target pairs, and prefix-value drops localise both synthetic math errors and clinician-labelled medical errors better than chance and entropy.

Critical weaknesses
  1. The motivation does not match the method or the experiments. The paper argues that external verifiers are hard to obtain, but the critic is trained with exact final-answer matching, which is itself a rule-based verifier. All experiments are on GSM8K and multiple-choice tasks, where this reward can be rule-base verified. On these tasks, simple answer-agreement GRPO matches or beats PERC for policy training at much lower cost (Table 6).

  2. Training the critic does not scale across tasks. Negative examples depend on task-specific corruptions: hand-written arithmetic edits for GSM8K, and an LLM prompted to write a wrong reasoning trace for every distractor on MMLU-Pro and MedReason (Appendix B.1).

  3. The critic is not reusable for policy training. Cross-task and cross-model transfer is only shown for reranking a fixed candidate pool (Table 2). Each GRPO run still trains a new critic on that backbone and dataset. On Llama3.1-8B, this training signal falls below SFT on both MMLU-Pro and MedReason (Table 5).

  4. Appendix D.4 is unreliable as written. Figures 12–14 (MedReason) are identical to Figures 9–11 (MMLU-Pro), including the reported t-statistics.

Questions for authors
  1. Practical Justification Over Simple RLVR and Applicability to Non-Verifiable Tasks: Since exact final-answer matching (O(y)=O(yE)O(y) = O(y^E)) already serves as a rule-based outcome verifier on GSM8K, MMLU-Pro, and MedReason, and direct terminal Answer-Agreement GRPO (Table 6) achieves superior or equal policy pass@1 without the 3–4x training cost and synthetic corruption pipeline of PERC, why should practitioners train a PERC critic for policy optimisation on verifiable tasks instead of running standard RLVR? Conversely, how would PERC work on a genuinely open-ended task where rule-based answer extraction (O(y)=O(yE)O(y) = O(y^E)) and multiple-choice distractors are unavailable?

  2. Hidden Cost and Necessity of Task-Specific Corruptions C(yE)C(y^E): In Appendix B.1 ("System Prompt for Corrupted Dataset"), an LLM is prompted to write plausible wrong reasoning traces for every incorrect option in MedReason and MMLU-Pro. Which model was used to generate these synthetic corruptions, and is the compute cost of generating this dataset included in the GPU-hour comparison in Table 7? Furthermore, if you remove the task-specific corruptions C(yE)C(y^E) and train the prefix critic using only answer-agreement rollouts (similar to outcome-labelled implicit PRMs like PRIME/IPVRM), how much performance is lost in D1 (Reranking) and D2 (Error Localisation, Table 4)?

  3. True Reusability in Policy Optimisation (D3) and Cause of Llama3.1-8B Collapse: Since the critic is claimed to be reusable across backbones and datasets (Table 2), can a critic trained on one backbone/dataset (e.g., Qwen2.5-7B or GSM8K) be directly reused as the reward model to train a different policy backbone or on a different dataset via GRPO without retraining the critic on the new π0\pi_0? Additionally, why does PERC cause a >10 point drop below SFT for Llama3.1-8B on MMLU-Pro (36.6% vs. 47.2%), and why did you average prefix values into a sequence-level scalar (Eq. 5) rather than using token-level prefix-value differences (δϕ(t)\delta_\phi(t)) as step-level advantages?

  4. Duplicate Figures in Appendix D.4: Figures 12–14 (labelled as MedReason) are identical to Figures 9–11 (MMLU-Pro) in both plots and tt-test values. Please clarify and provide the correct distributions for MedReason.

8 comments
AuthorsOpenReview ↗

Response to Reviewer YxCw #1

Thank you for the careful review. We address each weakness and its related questions below.


Weakness 1: The motivation conflicts with answer-based supervision

We target a setting where reference answers are available for training questions but unavailable when choosing between solutions to new questions. This motivates learning a critic that can select candidates and identify likely reasoning errors at deployment. Exact answer agreement is indeed rule-based outcome supervision during training; PERC does not remove that requirement. It removes the need for step annotations and separately collected preference labels.

Question 1: Why use PERC instead of answer-agreement GRPO, and how would it apply to open-ended tasks?

Policy optimisation. Table 6 supports direct answer-agreement GRPO when the objective is policy accuracy on these verifiable tasks. The practical reason to train PERC is to obtain a scorer that also supports selection and diagnosis. Table 2 reports positive selection gains in all 27 source-target settings on fixed candidates, and Table 4 evaluates both natural medical errors and controlled arithmetic errors. These inference uses do not require policy optimisation or reference answers for new questions.

The 3-4x cost multiplier in Table 7 compares the full critic-plus-policy pipeline with SFT. It is not a measured multiplier over answer-agreement GRPO. In our runs, local critic preparation is estimated at 3-4 GPU-hours; policy optimisation is an additional cost. We report the separate API data-generation cost under Question 2 below.

Open-ended tasks. Applying the current training recipe would require reliable answer-equivalence labels and suitable negative examples. Our benchmarks provide extractable answers to construct training labels and measure selection accuracy.

Action taken. We revised the abstract and introduction (pp. 1-2, marked in blue) to distinguish training supervision from scoring without reference answers at deployment. Additionally, we further acknowledge the open-ended-task limitation in Section 6.


Weakness 2: Corruption construction is task-dependent

GSM8K uses programmatic arithmetic edits. MedReason and MMLU-Pro share an automated prompt that asks an LLM to produce plausible reasoning towards distractor answers. Individual traces are not manually edited, but the corruption procedure remains task-dependent. These experiments demonstrate use across the three domains, rather than unrestricted scalability.

Question 2a: Which model generates the corruptions, and what does generation cost?

MedReason and MMLU-Pro corruptions were generated offline through the GPT-5.2 API, using the question, expert reasoning, correct answer and distractors. The estimated API cost was a one-time $447:

DatasetQuestionsCorrupted tracesEstimated corruption cost
MedReason14,21142,633$205
MMLU-Pro12,011101,809$242
Total26,222144,442$447

Question 2b: What is lost if the critic uses only answer-labelled policy rollouts?

We trained six Qwen2.5-7B-Instruct critics: three mixtures under interval (stride 15) and dense (stride 1) supervision. We match the architecture, initial weights, seed, 250-update budget, and frozen Qwen2.5-3B-Instruct training rollouts. Policy Only uses eight answer-matching and eight answer-mismatching traces per batch. Adding expert positives replaces four positive slots; adding expert corruptions then replaces four negative slots.

SettingTraining mixtureBest-of-16 (%)Hit@1 (%)Hit@7 (%)
IntervalPolicy only86.4 [84.4, 88.2]12.3 [10.5, 14.2]34.4 [31.7, 37.0]
Interval+ Expert positives82.3 [80.1, 84.4]11.5 [9.8, 13.4]32.9 [30.4, 35.5]
Interval+ Expert corruptions86.4 [84.5, 88.2]27.5 [25.0, 29.8]69.1 [66.4, 71.6]
DensePolicy only86.4 [84.5, 88.3]24.0 [21.6, 26.3]43.9 [41.1, 46.6]
Dense+ Expert positives86.6 [84.6, 88.5]5.9 [4.7, 7.2]20.7 [18.6, 23.0]
Dense+ Expert corruptions85.7 [83.7, 87.5]76.1 [73.7, 78.4]90.5 [88.9, 92.2]

The clearest benefit is synthetic error localisation: dense Hit@1 rises from 24.0% to 76.1%, and Hit@7 from 43.9% to 90.5%, relative to Policy Only. Interval Hit@7 rises from 34.4% to 69.1%. Adding corruptions also improves localisation with expert positives held fixed. Reranking point estimates remain similar to Policy Only; expert positives alone do not provide a reliable selection improvement. The incremental benefit on natural errors remains untested.

Action taken. We added the matched comparison in Appendix C.4, Table 11. Additionally, we added the generation model, API cost breakdown and accounting scope to Appendix B.

AuthorsOpenReview ↗

Response to Reviewer YxCw #2

Weakness 3: Scoring transfer does not establish policy-training transfer

Table 2 demonstrates reuse for scoring: critics trained on different tasks and backbones rank a fixed Qwen2.5-7B SFT candidate pool without retraining. This is separate from training a new policy with a transferred critic.

Question 3a: Can a critic train a different policy or a policy on a different task without retraining?

A frozen critic can score another policy's rollouts, but we have not evaluated whether this gives successful transferred GRPO optimisation, due to limited time and compute during this rebuttal period. Apologies for this.

Question 3b: Why does Llama3.1-8B fall below SFT?

The MMLU-Pro result is 36.6% versus 47.2% for SFT, and the experiments do not isolate its cause. Policy-training results vary by backbone and task. Separately, Table 2 shows that a Llama MMLU-Pro critic improves selection from 48.1% to 53.8% on the fixed Qwen2.5-7B SFT candidates. These results distinguish useful scoring from reliable policy improvement.

Question 3c: Why average prefix values instead of using token-level differences?

Our GRPO implementation uses a length-normalised sequence score and one group-normalised advantage per sequence, following Cetin et al. (2025) [1] and Cui et al. (2025) [2]. Prefix-value differences are used for localisation. Turning them into token-level advantages would require a separately defined and evaluated credit-assignment method; we have not tested that alternative.

[1] Edoardo Cetin, Tianyu Zhao, and Yujin Tang. Reinforcement learning teachers of test time scaling. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=tebG8q5EeK. [2] Ganqu Cui, et al. Process Reinforcement through Implicit Rewards, February 2025. URL http://arxiv.org/abs/2502.01456. arXiv:2502.01456 [cs].


Weakness 4: The appendix repeats the MMLU-Pro plots under MedReason

Question 4: What caused the duplicate figures, and does the correction change score separation?

Apologies, a copy-pasted LaTeX figure reference pointed both dataset sections to the same image files, including their embedded t-statistics. We corrected those references and replaced the MedReason plots. The corrected plots retain statistically significant mean-score separation between correct and incorrect traces.

Action taken. Appendix D.4 now contains the dataset-specific plots, with the corrected MedReason panels in Figures 13-15 (p. 32), marked in blue.


Again, we thank the reviewer for their time and feedback, and we hope we have addressed at least some of their concerns.

Thank you for the clarification. My concern was that the paper motivates PERC by the difficulty of obtaining an external verifier, while training still depends on exact answer matching and the experiments stay on tasks where that match is already a rule-based reward. The response accepts this dependence and says that answer-agreement GRPO is preferable when the goal is policy accuracy. It then relocates the contribution to deployment-time selection and diagnosis. That is a narrower claim, and the setting in which no rule-based answer check exists remains untested.

The same narrowing applies to scalability and reuse. The response acknowledges that the negative-example construction is task-dependent, and it does not evaluate whether a critic trained on one backbone or dataset can optimise a different policy. Cross-setting evidence remains limited to scoring a fixed candidate pool. The original questions about verifier-free training, open-ended tasks, and reuse as a training reward are therefore still open, so I am maintaining my rating.

Reviewer_pv8POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PERC trains a prefix classifier from expert demonstrations, policy traces labelled by final-answer agreement with those demonstrations, and corrupted expert traces. Trace labels supervise selected prefixes; classifier logits provide sequence-ranking scores, adjacent value changes provide error-localisation signals, and aggregated scores supply a frozen reward for GRPO. Evaluation covers GSM8K, MMLU-PRO, and MEDREASON with three main backbone families. The strongest result is fixed-candidate cross-critic reranking: all 27 reported source-target combinations improve over random selection. The paper also reports natural medical-error localisation, controlled arithmetic perturbations, and policy-training feasibility, while acknowledging that training does not consistently improve upon SFT.

Critical strengths
  1. The fixed-candidate transfer design is valuable. Table 2 holds the Qwen2.5-7B SFT generator and candidate pool fixed while changing the critic, separating selection quality from generator improvement. All 27 point estimates improve, by 0.8-12.7 percentage points. Table 3 separately tests selection after policy optimisation and reports positive gains in all 18 configurations. These experiments support practical reuse of a frozen critic, even though they do not establish significance for every small gain.

  2. The evidence construction addresses a concrete failure mode of source-only discrimination. Correct policy traces enter the positive class and corrupted expert traces enter the negative class. Appendix C.3 actually tests these choices: answer-agreement supervision improves correctness discrimination, and arithmetic corruptions further improve reranking and AUROC. The medical study also uses natural errors and hides critic scores from annotators, providing a useful complement to synthetic perturbations. The explicit distinction between weak trace supervision and observed process labels is appropriate.

Critical weaknesses
  1. The central process-quality evidence needs correction and a reproducible measurement definition. Section 5.2 defines synthetic Hit@1/Hit@7 as literal ±1/±7-token windows, but p. 8 attributes their identical interval results to k=15 backfilling. That equality does not follow: with supervised values +1 at token 15 and -1 at token 30, an error at token 20 yields a backfilled drop at token 16, which hits ±7 but misses ±1. This challenges the interpretation of the headline 86.1% localisation result, rather than proving that its numerical value is wrong. Appendix D.5 provides dense examples but does not resolve the interval alignment. Separately, all nine plot panels in Figures 9-11 (MMLU-PRO, p. 26) are duplicated in Figures 12-14 (MEDREASON, p. 27), including curves and t statistics; the paired PDF drawing streams are byte-identical. Correct dataset-specific plots and their provenance are needed. This could be a figure-assembly error; it is not evidence sufficient to infer fabrication.

  2. The incremental benefit of the expert-referenced construction remains insufficiently isolated. Token-level verifiers trained from final-answer labels already appear in Cobbe et al., Section 4.3 and Appendix E. Thus, the relevant contribution is the specific expert/policy/corruption mixture and its demonstrated reuse. Appendix C.3 compares against source-only labels and tests corruptions, but never removes expert positives while retaining outcome-labelled policy-prefix supervision under the same critic architecture and budget. It also measures reranking gains without clearly documenting a shared candidate pool across these policy-training ablations. The terminal answer-agreement GRPO baseline in Table 6 cannot isolate that critic contribution. A focused ablation within the authors' existing pipeline would be more decisive than additional comparisons with named recent methods. The natural-error study is encouraging but small (110 retained cases, separate annotator batches); it does not independently settle the broad process-quality claim.

Questions for authors
  1. For Table 4, are localisation hits measured by literal token distance or by membership in the same interval? How does backfilling at k=15 lead to identical Hit@1 and Hit@7 values? If both metrics are evaluated using the stated token-distance windows, does the reported advantage of interval critics remain?

  2. Figures 9–11 and 12–14 show identical corresponding distributions under MMLU-PRO and MEDREASON captions. Could you explain this repetition and clarify whether correcting the dataset-specific plots changes the evidence for score separation across tasks?

  3. Under the same prefix-classifier architecture, training budget, held-out prompts, and fixed reranking candidates, what changes when expert positives and then corrupted expert negatives are added to an outcome-labelled-policy-only critic? Evidence that the proposed mixture improves selection and natural-error localisation would materially strengthen the contribution.

6 comments
AuthorsOpenReview ↗

Response to Reviewer pv8P #1

We thank you for your time in reviewing and identifying the measurement and figure issues, and for suggesting a controlled comparison.


Weakness 1a: The localisation measurement needs correction and a reproducible definition

You are correct: the original interval evaluation used bucket-based tolerances, whereas the manuscript described literal token windows. We have recomputed both variants using literal ±1/±7-token windows and corrected Table 4 and the associated claims. The former interval advantage does not persist; dense critics retain higher localisation point estimates than entropy and random selection across all three architectures.

Question 1: Are hits measured by token distance or interval membership, and does the interval advantage remain?

The updated evaluation uses literal token distance for both critic variants. At each token transition, we clip downward score changes at zero and select the earliest largest drop. The prediction is the token immediately after that drop; with zero-based indexing, position 1 is selected if all drops are zero. A hit requires the prediction to be within ±1 token for Hit@1 or ±7 tokens for Hit@7 of any in-range changed-token position.

The corrected GSM8K results are below.

ModelSignalHit@1 (%)Hit@7 (%)
Qwen2.5-7BInterval reward27.3 ± 1.869.1 ± 2.7
Dense reward76.1 ± 2.890.5 ± 2.6
Entropy15.3 ± 2.040.0 ± 2.8
Random11.6 ± 0.631.2 ± 1.0
Llama3.1-8BInterval reward27.3 ± 2.567.4 ± 2.6
Dense reward65.8 ± 2.880.6 ± 2.2
Entropy28.1 ± 2.554.9 ± 2.7
Random12.7 ± 0.633.3 ± 1.0
Qwen3-4BInterval reward14.3 ± 2.050.7 ± 2.8
Dense reward62.0 ± 2.770.3 ± 2.5
Entropy0.2 ± 0.21.9 ± 0.9
Random11.8 ± 0.631.5 ± 1.0

The former interval-over-dense advantage does not persist. Dense critics have higher Hit@1 and Hit@7 point estimates than entropy and the random reference across all three backbones. The MedReason results are unchanged: they measure exact or adjacent reasoning units, which can span many tokens, rather than narrow token windows.

Action taken. We recomputed both critic variants with literal token windows, corrected Table 4 and the associated claims, and revised Section 5.2 to define the prediction, targets, tie handling and tolerances (pp. 7-8, marked in blue).

Weakness 1b: The appendix contains duplicate dataset plots

Question 2: Why do the MMLU-Pro and MedReason figures repeat, and does correcting them change the evidence?

Copy-pasted LaTeX figure references pointed the two dataset sections to the same image files, including their embedded t-statistics. We corrected the references and inserted the MedReason plots. Their correct-versus-incorrect mean-score separation remains statistically significant.

Action taken. Appendix D.4 now contains the dataset-specific distributions. The corrected MedReason plots are Figures 13-15 (p. 32), with the changes marked in blue.

AuthorsOpenReview ↗

Response to Reviewer pv8P #2

Weakness 2: The contribution of the expert mixture was not isolated

We agree that outcome-supervised prefix verification predates PERC, including Cobbe et al. The contribution tested here is the expert-referenced training construction: policy traces appear in both classes, and answer-preserving expert corruptions supply negative reasoning examples that final-answer agreement alone would label positive.

Question 3: What changes when expert positives and then corrupted negatives are added under matched conditions?

The clearest gain is synthetic error localisation: dense Hit@1 rises from 24.0% to 76.1% relative to Policy Only, while reranking point estimates remain similar.

We trained six Qwen2.5-7B-Instruct critics: three mixtures under interval and dense supervision, with matched architecture, initial weights, seed, and 250-update budget. Policy Only uses eight answer-matching positives and eight answer-mismatching negatives per batch. Expert positives replace four positive slots; answer-preserving expert corruptions then replace four negative slots. Frozen Qwen2.5-3B-Instruct training rollouts are shared across mixtures.

All critics rank the same 16 frozen Qwen2.5-7B SFT candidates on GSM8K

SettingTraining mixtureBest-of-16 (%)Δ Random (pp)Hit@1 (%)Hit@7 (%)
IntervalPolicy Only86.4 [84.4, 88.2]+15.3 [13.7, 16.8]12.3 [10.5, 14.2]34.4 [31.7, 37.0]
Interval+ Expert Positives82.3 [80.1, 84.4]+11.2 [9.5, 12.8]11.5 [9.8, 13.4]32.9 [30.4, 35.5]
Interval+ Expert Corruptions (ours)86.4 [84.5, 88.2]+15.3 [13.7, 16.8]27.5 [25.0, 29.8]69.1 [66.4, 71.6]
DensePolicy Only86.4 [84.5, 88.3]+15.3 [13.6, 16.9]24.0 [21.6, 26.3]43.9 [41.1, 46.6]
Dense+ Expert Positives86.6 [84.6, 88.5]+15.5 [14.0, 17.1]5.9 [4.7, 7.2]20.7 [18.6, 23.0]
Dense+ Expert Corruptions (ours)85.7 [83.7, 87.5]+14.6 [13.0, 16.1]76.1 [73.7, 78.4]90.5 [88.9, 92.2]

Adding corruptions also improves localisation with expert positives held fixed. Expert positives alone do not reliably improve selection. The ablation supports synthetic-localisation gains, but not improved reranking over Policy Only.

Action taken. We added the matched protocol and results in Appendix C.4, Table 11, including the shared candidate pool and uncertainty intervals. We updated the related work to acknowledge outcome-supervised prefix verification, and will state the synthetic-versus-natural scope of this ablation explicitly in the main discussion.


Once again, we wish to thank the reviewer for their feedback. We hope to have clarified some of their points.

The revised figures and token-window evaluation address my reporting concerns. Recomputing localisation removes the original advantage of Interval over Dense, though Dense still performs well on synthetic errors. The matched ablation is helpful here. Adding expert data raises Dense Hit@1 from 24.0% to 76.1%, while reranking on the same candidate pool stays close to the policy-only baseline.

My main remaining concern is what the expert data add on natural errors. The MedReason experiment compares PERC with entropy and random baselines. Would a comparison with the policy-only critic on the same examples show a similar gain? At present, the strongest evidence for the expert data concerns synthetic localisation, with a more limited case for improved reranking or natural-error diagnosis. I am keeping my rating at 2.

Reviewer_ipDZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PERC, a prefix-level reasoning critic trained from expert demonstrations without requiring explicit process-level annotations. The authors construct positive and negative reasoning trajectories using expert demonstrations, answer agreement between policy rollouts and expert answers, and synthetically corrupted expert trajectories. A binary classifier is then trained to produce prefix-level scores, which are reused for three purposes: inference-time reranking, reasoning error localization, and policy optimization with GRPO. Experiments on GSM8K, MMLU-Pro, and MedReason show that the learned critic can improve Best-of-N reranking, generalize across different models and tasks, and provide some ability to identify reasoning errors.

Critical strengths

The motivation and methodology are clearly explained, and the overall framework is straightforward to understand.

The authors evaluate the critic across three benchmarks and multiple model backbones, covering reranking, error localization, and policy optimization. The cross-model and cross-task transfer experiments are particularly useful.

Rather than training separate reward models for different purposes, the authors demonstrate that a single learned critic can provide useful signals for both training and inference.

Critical weaknesses
  1. My main concern is that the proposed method seems quite similar to existing work. For example, Learning Reasoning Rewards from Expert Demonstrations with Inverse Reinforcement Learning (R-AIRL) has already explored learning reasoning rewards from expert demonstrations and using them for policy optimization, inference-time reranking, and error localization. Meanwhile, IPVRM has explored learning prefix-level values from outcome supervision and deriving process-level signals from differences between consecutive prefix values. PERC appears to share many of these ideas, with its main difference being the construction of training samples and the use of a frozen critic. The authors should better clarify the differences from these closely related works and provide more direct comparisons.

  2. Although the authors aim to learn a process-level critic, their supervision is essentially derived from final-answer agreement. All supervised prefixes within the same trajectory are assigned the same label, regardless of whether the intermediate reasoning is actually correct. This can be problematic for complex reasoning trajectories. For instance, a model may make several mistakes and successfully correct them before reaching the right answer, or perform most intermediate steps correctly but make a mistake at the final step. In these cases, the trajectory-level label does not necessarily reflect the quality of individual reasoning steps. I think the paper needs more analysis of whether the learned prefix scores genuinely capture reasoning correctness rather than just being correlated with the final outcome.

  3. For reranking, the authors simply average the prefix-level critic scores across the trajectory. However, this may introduce issues for long CoTs with repeated mistakes and self-corrections, or trajectories that are mostly correct but fail near the end. For error localization, the authors identify the largest drop in prefix value as the first reasoning failure, however the model can backtrack and correct earlier mistakes like i mentioned above in W2. The current experiments do not sufficiently investigate these cases. I would like to see more analysis across different reasoning lengths, error positions, and self-correction patterns, as well as comparisons with alternative aggregation strategies.

  4. Most of the impressive localization results come from synthetically corrupted GSM8K trajectories, which may be relatively easy to detect because the perturbations follow patterns similar to those used in training. The proposed method does not consistently outperform SFT or direct answer-agreement GRPO. In some settings, the performance is even worse. Since the critic requires additional training and inference computation, it would be useful to provide a clearer analysis of its practical advantages and cost-effectiveness over simpler outcome-based approaches.

Questions for authors

See the waekness part above, I am extremely interested in that if the author tested the reranking in below scene:

  1. In a rollouts group, there is a long CoT with repeated mistakes and successful corrections vs the shorter with many success intermediate reasoning steps but overall incorrect trajectory.

  2. in a rollouts group, a traj with mostly correct CoT with a wrong final step vs the rollouts correctly solves the problem but has an average critic value due to the length normalization.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ipDZ #1

We wish to thank the reviewer for their time to assess our paper and give us feedback


Weakness 1: Similarity to R-AIRL and IPVRM

We kindly ask the reviewer to wait on the R-AIRL clarification, as it is pending Area Chair guidance.

IPVRM learns outcome-supervised prefix scores through language-model/reference log-probability ratios, which can be seen as the DPO extension over traditional sequence reward models, adapted to our case. Thank you for pointing us towards it. To this end, we trained IPVRM on the three datasets and the three models, and directly compared it in our experiments:

D1) Reranking:

Train = eval datasetModelPaper baseline (%)PERC (%)PERC ΔIPVRM (%)IPVRM Δ
GSM8KQwen2.5-7B71.1 ± 1.786.6+15.570.2−0.9
Llama3.1-8B71.1 ± 1.782.5+11.480.2+9.1
Qwen3-4B71.1 ± 1.783.8+12.779.8+8.7
MMLU-ProQwen2.5-7B48.1 ± 1.951.3+3.252.0+3.9
Llama3.1-8B48.1 ± 1.953.8+5.753.3+5.2
Qwen3-4B48.1 ± 1.955.3+7.253.6+5.5
MedReasonQwen2.5-7B66.3 ± 2.168.5+2.264.6−1.7
Llama3.1-8B66.3 ± 2.178.0+11.772.0+5.7
Qwen3-4B66.3 ± 2.171.9+5.668.6+2.3

D2) Localisation

DatasetModelIPVRM Hit@1Interval (ours) Hit@1Dense (ours) Hit@1IPVRM Hit@7 / ±1Interval (ours) Hit@7 / ±1Dense (ours) Hit@7 / ±1
MedReason (human)Qwen2.5-7B12.7 ± 6.412.7 ± 5.914.5 ± 6.421.8 ± 7.727.3 ± 8.224.5 ± 7.7
Llama3.1-8B12.7 ± 6.430.0 ± 8.29.1 ± 5.024.5 ± 7.740.9 ± 9.127.3 ± 8.2
Qwen3-4B12.7 ± 5.918.2 ± 6.916.4 ± 6.822.7 ± 7.731.8 ± 8.728.2 ± 8.2
GSM8K (synthetic)Qwen2.5-7B60.0 ± 2.727.5 ± 1.876.1 ± 2.875.4 ± 2.469.1 ± 2.790.5 ± 2.6
Llama3.1-8B53.2 ± 2.827.3 ± 2.565.8 ± 2.867.4 ± 2.667.4 ± 2.680.6 ± 2.2
Qwen3-4B4.5 ± 1.114.3 ± 2.062.0 ± 2.731.6 ± 2.650.6 ± 2.870.3 ± 2.5
D3) Policy Training
DatasetModelIPVRMSFTInterval (ours)Dense (ours)
GSM8KQwen2.5-7B83.5 ± 2.071.1 ± 1.690.2 ± 1.389.7 ± 1.3
Llama3.1-8B79.5 ± 2.167.5 ± 2.177.9 ± 1.765.7 ± 2.0
Qwen3-4B92.4 ± 1.477.5 ± 1.690.0 ± 1.492.6 ± 1.2
MMLU-ProQwen2.5-7B50.9 ± 2.648.1 ± 1.950.6 ± 1.943.8 ± 2.0
Llama3.1-8B40.5 ± 2.547.2 ± 1.936.6 ± 1.837.9 ± 1.8
Qwen3-4B53.3 ± 2.553.9 ± 2.053.5 ± 2.455.1 ± 2.3
MedReasonQwen2.5-7B70.9 ± 2.966.3 ± 2.267.2 ± 2.568.7 ± 2.4
Llama3.1-8B77.7 ± 2.679.5 ± 2.076.3 ± 2.175.1 ± 2.1
Qwen3-4B69.6 ± 2.867.4 ± 2.369.1 ± 2.669.2 ± 2.5

Takeaway: For reranking (D1), we can see that in almost all cases, except ' Qwen2.5-7 B ' on MMLU-Pro, PERC outperforms IPVRM when reranking on the same set of generated responses. For localisation, PERC still outperforms IPVRM in all cases, both on the human-labelled dataset and the synthetic dataset. For policy training (D3), IPVRM provides more competitive results than PERC. However, we again note that policy training is mainly a feasibility experiment, and we are more interested in (D1) and (D2), as they demonstrate the usefulness of a reusable reward model for quantitatively scoring reasoning.

Action taken. We updated Section 3 to explain the supervision and classifier differences, and added the IVPRM baseline to our experiments.

AuthorsOpenReview ↗

Response to Reviewer ipDZ #2

Weakness 3: Aggregation, length and self-correction need direct tests

Test construction. We start from generated short and long correct solutions to the same question, requiring at least 40% more reasoning tokens in the long version. We create a short incorrect variant with one numerical mistake in the final reasoning step and a matching wrong answer, and a long repaired variant with two numerical mistakes, each immediately followed by an explicit correction, retaining the correct final answer. The same GSM8K-trained Qwen2.5-7B critics score all four variants.

ComparisonInterval (%)Dense (%)
A: (Short + correct) vs (Short + wrong)99.1 [95.0, 99.8]95.4 [89.7, 98.0]
B: (Long + correct) vs (Short + wrong)99.1 [95.0, 99.8]96.3 [90.9, 98.6]
C: (Long + self-corrected) vs (Short + wrong)19.3 [13.0, 27.7]10.1 [5.7, 17.2]
D: (Long + correct) vs (Long + self-corrected)100.0 [96.6, 100.0]100.0 [96.6, 100.0]

Comparison D shows sensitivity to reasoning history with the final answer held fixed; it does not adequately credit repairs. These constructed cases complement the positive fixed-candidate reranking results in Table 2. The frequency and impact of this repair failure in natural reasoning traces remain unmeasured.

Question 1: Does a long correct solution with repeated mistakes and repairs outrank a shorter incorrect solution?

In our constructed test, it usually does not. On 109 GSM8K question pairs, PERC selects the repaired, ultimately correct long solution in only 19.3% / 10.1% of pairs (Interval / Dense). This identifies insufficient credit for explicit repairs in this setting.

Question 2: Does a clean correct solution outrank a solution that fails at the final step?

Yes, in this controlled comparison: clean long correct solutions win 99.1% / 96.3% of pairs against the short solution with an unrepaired final error. Correct short solutions also win 99.1% / 95.4% against their final-error variants. The long-versus-short comparison shows that length does not overwhelm correctness here; it does not rule out length bias.

Action taken. We added the construction and results in Appendix C.7, Table 13 (p. 27, marked in blue), and stated the repair limitation directly in Section 6.

Aggregation:

Would another scoring rule improve selection?

We compared the implemented late-weighted mean (γ = 0.95) with uniform averaging, the last score and the mean of the last 10% of scores. For each critic, all four rules use the same 16 saved SFT candidates on 1,319 GSM8K questions. Values are selected-answer accuracy with 95% confidence intervals.

CriticScoringUniform mean (%)Late-weighted mean (%)Last score (%)Last 10% mean (%)
Llama3.1-8B-itInterval72.3 [69.8, 74.6]72.6 [70.1, 74.9]72.6 [70.1, 74.9]71.5 [69.0, 73.9]
Dense72.4 [69.9, 74.7]73.2 [70.7, 75.5]68.9 [66.4, 71.4]71.9 [69.4, 74.2]
Qwen2.5-7B-itInterval78.1 [75.8, 80.2]78.4 [76.1, 80.5]72.9 [70.5, 75.3]75.4 [73.0, 77.7]
Dense80.7 [78.4, 82.7]81.1 [78.9, 83.1]70.1 [67.5, 72.5]81.1 [78.9, 83.1]
Qwen3-4B-itInterval85.2 [83.2, 87.0]84.7 [82.6, 86.5]82.6 [80.5, 84.6]83.2 [81.1, 85.1]
Dense85.1 [83.1, 87.0]85.8 [83.8, 87.6]84.3 [82.2, 86.2]84.2 [82.1, 86.0]

Takeaway. Uniform and late-weighted means differ by at most 0.8 percentage points. The final score alone can reduce accuracy; the last-10% mean gives no consistent improvement. These comparisons support the current aggregate, but do not resolve the repair failure.

Action taken. We added the comparison to Appendix C.

Localisation:

Does the largest drop remain useful around errors?

We compared the largest adjacent drop with a fixed sustained-drop rule on the same synthetic GSM8K edits. It compares the mean of 15 saved token positions before and after each boundary, selecting the largest positive decrease; edge windows are shortened. The untuned rule uses future scores, so it is offline. Hit@7 uses literal ±7-token windows around edits.

CriticScoringLargest drop: Hit@7 (%)Sustained drop: Hit@7 (%)
Llama3.1-8B-itInterval67.4 [64.8, 70.0]67.4 [64.8, 70.0]
Dense80.6 [78.3, 82.7]89.5 [87.7, 91.1]
Qwen2.5-7B-itInterval69.1 [66.4, 71.6]69.1 [66.4, 71.6]
Dense90.5 [88.9, 92.2]97.6 [96.8, 98.5]
Qwen3-4B-itInterval50.6 [47.9, 53.4]50.6 [47.9, 53.4]
Dense70.3 [67.6, 72.7]86.8 [84.8, 88.6]

Takeaway. Sustained drops improve dense localisation across all three critics and the reported length/edit-position bands; interval results are unchanged.

Action taken. We added the rule, results and length/position analysis to Appendix C, and state its offline scope in the localisation discussion.

AuthorsOpenReview ↗

Response to Reviewer ipDZ #3

Weakness 2: Trace labels may not reflect individual reasoning steps

Does the critic respond to intermediate reasoning, or only to the final outcome?

Each prefix inherits its trace label, so its score is a classifier signal rather than an observed step label. Figure 6 compares original and edited reasoning with identical correct final answers and shows different prefix-score trajectories.

The added analysis in Appendix C.5 tracks how this score difference changes as reasoning continues. A decreasing signed difference describes relative score change; it does not establish recovery or repair.

Takeaway. The critic can respond to intermediate content even when final answers match. Reliable credit for explicit repairs is a separate property, tested under Question 1 below.

Action taken. We added Appendix C.5 and will connect it more clearly to Section 4.2's interpretation of prefix scores as classifier-derived signals rather than direct step labels.


Weakness 4: Synthetic errors and additional cost limit the practical case

Is there evidence on natural errors, and what gains justify training a critic?

Natural errors. Table 4 includes 110 incorrect MedReason traces annotated by doctors and final-year medical students with critic scores hidden. Llama interval achieves reasoning-unit Hit@±1 of 40.9%, versus 20.0% for chance and 16.4% for entropy.

Practical gains. One trained critic supplies selection, local error signals and an aggregate training reward. Selection improves in all 27 fixed-candidate settings in Table 2 and all 18 settings after policy optimisation in Table 3. This shows that the scorer remains useful after policy training. For policy accuracy alone on these tasks, Table 6 supports simpler answer-agreement GRPO.

Cost. Scoring and localisation require no GRPO phase. Local critic preparation is estimated at 3-4 A100 GPU-hours, mostly for rollout generation; fitting takes about one hour with existing traces. Table 7's full pipeline totals are shown below.

Method / use caseCritic preparation (GPU-h)Policy training (GPU-h)Total (GPU-h)
SFT—4.54.5
PERC critic only (D1/D2)≈3–4Not required≈3–4
PERC interval + policy optimisation≈3–4≈13.5–14.517.5
PERC dense + policy optimisation≈3–4≈12.5–13.516.5

Action taken. The natural-error protocol and results are reported in Appendix A.4 and Table 4.


Once again, we thank the reviewer for their valuable feedback, and we hope our answers addressed some of their concerns.

Thx for the response, I’ve updated the review

Comments on the paper

2 comments

Dear Reviewers,

We thank you for your time and effort in reviewing our paper. In the meantime, we have added follow-up experiments and clarified your questions. We would appreciate it if you could let us know whether these address your concerns or if there are additional points you would like us to address while the rebuttal period is still open.

Thank you again, The authors.

AuthorsOpenReview ↗

Summary of Rebuttals

We thank the reviewers for their time and valuable feedback. Our revisions and follow-up experiments address the main concerns and can be summarised as follows:

  • Contribution and baselines: Clarified the reward-modelling focus and supervision requirements, and added direct IPVRM and answer-agreement GRPO comparisons.
  • Expert-data and corruption contribution: Matched ablations increase dense synthetic Hit@1 from 24.0% to 76.1% when using PERC. The follow-up study on 110 human-labelled MedReason traces shows +10.0 pp exact-unit localisation gains over policy-only training.
  • Reuse and practical value: A frozen GSM8K-trained critic not only is useful for cross-task reranking, but now also supports policy-training transfer across backbones to MMLU-Pro (+2.1 pp pass@1). It also benefits the matched Answer-GRPO policy through reranking (+9.3 pp).
  • Reasoning behaviour: Added fixed-answer, length, self-correction and aggregation tests, plus sustained-drop localisation. These analyses identify both useful reasoning sensitivity and potentially insufficient credit for explicit repairs.
  • Reporting and scope: Corrected localisation measurements and duplicated figures; expanded confidence-interval and cost reporting; and clarified the remaining open-ended-task limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.