- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose the Prefix-level Expert-Referenced Critic (PERC), which trains a prefix classifier using expert traces and answer-agreeing policy rollouts as positives, and answer-mismatched rollouts plus targeted corrupted expert traces as negatives. Experiments on GSM8K, MMLU-Pro, and MedReason demonstrate that the frozen critic improves inference-time reranking, localises reasoning errors, and serves as a viable reward signal for GRPO policy optimisation.
Critical strengths
-
The training setup reduces source and formatting shortcuts by putting policy traces on both sides of the classifier and using corrupted expert traces as hard negatives, so the critic tracks reasoning quality rather than style.
-
The same frozen critic is tested for reranking, error localisation, and policy training. Reranking transfers across all 27 source–target pairs, and prefix-value drops localise both synthetic math errors and clinician-labelled medical errors better than chance and entropy.
Critical weaknesses
-
The motivation does not match the method or the experiments. The paper argues that external verifiers are hard to obtain, but the critic is trained with exact final-answer matching, which is itself a rule-based verifier. All experiments are on GSM8K and multiple-choice tasks, where this reward can be rule-base verified. On these tasks, simple answer-agreement GRPO matches or beats PERC for policy training at much lower cost (Table 6).
-
Training the critic does not scale across tasks. Negative examples depend on task-specific corruptions: hand-written arithmetic edits for GSM8K, and an LLM prompted to write a wrong reasoning trace for every distractor on MMLU-Pro and MedReason (Appendix B.1).
-
The critic is not reusable for policy training. Cross-task and cross-model transfer is only shown for reranking a fixed candidate pool (Table 2). Each GRPO run still trains a new critic on that backbone and dataset. On Llama3.1-8B, this training signal falls below SFT on both MMLU-Pro and MedReason (Table 5).
-
Appendix D.4 is unreliable as written. Figures 12–14 (MedReason) are identical to Figures 9–11 (MMLU-Pro), including the reported t-statistics.
Questions for authors
-
Practical Justification Over Simple RLVR and Applicability to Non-Verifiable Tasks: Since exact final-answer matching () already serves as a rule-based outcome verifier on GSM8K, MMLU-Pro, and MedReason, and direct terminal Answer-Agreement GRPO (Table 6) achieves superior or equal policy pass@1 without the 3–4x training cost and synthetic corruption pipeline of PERC, why should practitioners train a PERC critic for policy optimisation on verifiable tasks instead of running standard RLVR? Conversely, how would PERC work on a genuinely open-ended task where rule-based answer extraction () and multiple-choice distractors are unavailable?
-
Hidden Cost and Necessity of Task-Specific Corruptions : In Appendix B.1 ("System Prompt for Corrupted Dataset"), an LLM is prompted to write plausible wrong reasoning traces for every incorrect option in MedReason and MMLU-Pro. Which model was used to generate these synthetic corruptions, and is the compute cost of generating this dataset included in the GPU-hour comparison in Table 7? Furthermore, if you remove the task-specific corruptions and train the prefix critic using only answer-agreement rollouts (similar to outcome-labelled implicit PRMs like PRIME/IPVRM), how much performance is lost in D1 (Reranking) and D2 (Error Localisation, Table 4)?
-
True Reusability in Policy Optimisation (D3) and Cause of Llama3.1-8B Collapse: Since the critic is claimed to be reusable across backbones and datasets (Table 2), can a critic trained on one backbone/dataset (e.g., Qwen2.5-7B or GSM8K) be directly reused as the reward model to train a different policy backbone or on a different dataset via GRPO without retraining the critic on the new ? Additionally, why does PERC cause a >10 point drop below SFT for Llama3.1-8B on MMLU-Pro (36.6% vs. 47.2%), and why did you average prefix values into a sequence-level scalar (Eq. 5) rather than using token-level prefix-value differences () as step-level advantages?
-
Duplicate Figures in Appendix D.4: Figures 12–14 (labelled as MedReason) are identical to Figures 9–11 (MMLU-Pro) in both plots and -test values. Please clarify and provide the correct distributions for MedReason.
8 comments
Response to Reviewer YxCw #2
Weakness 3: Scoring transfer does not establish policy-training transfer
Table 2 demonstrates reuse for scoring: critics trained on different tasks and backbones rank a fixed Qwen2.5-7B SFT candidate pool without retraining. This is separate from training a new policy with a transferred critic.
Question 3a: Can a critic train a different policy or a policy on a different task without retraining?
A frozen critic can score another policy's rollouts, but we have not evaluated whether this gives successful transferred GRPO optimisation, due to limited time and compute during this rebuttal period. Apologies for this.
Question 3b: Why does Llama3.1-8B fall below SFT?
The MMLU-Pro result is 36.6% versus 47.2% for SFT, and the experiments do not isolate its cause. Policy-training results vary by backbone and task. Separately, Table 2 shows that a Llama MMLU-Pro critic improves selection from 48.1% to 53.8% on the fixed Qwen2.5-7B SFT candidates. These results distinguish useful scoring from reliable policy improvement.
Question 3c: Why average prefix values instead of using token-level differences?
Our GRPO implementation uses a length-normalised sequence score and one group-normalised advantage per sequence, following Cetin et al. (2025) [1] and Cui et al. (2025) [2]. Prefix-value differences are used for localisation. Turning them into token-level advantages would require a separately defined and evaluated credit-assignment method; we have not tested that alternative.
[1] Edoardo Cetin, Tianyu Zhao, and Yujin Tang. Reinforcement learning teachers of test time scaling. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=tebG8q5EeK. [2] Ganqu Cui, et al. Process Reinforcement through Implicit Rewards, February 2025. URL http://arxiv.org/abs/2502.01456. arXiv:2502.01456 [cs].
Weakness 4: The appendix repeats the MMLU-Pro plots under MedReason
Question 4: What caused the duplicate figures, and does the correction change score separation?
Apologies, a copy-pasted LaTeX figure reference pointed both dataset sections to the same image files, including their embedded t-statistics. We corrected those references and replaced the MedReason plots. The corrected plots retain statistically significant mean-score separation between correct and incorrect traces.
Action taken. Appendix D.4 now contains the dataset-specific plots, with the corrected MedReason panels in Figures 13-15 (p. 32), marked in blue.
Again, we thank the reviewer for their time and feedback, and we hope we have addressed at least some of their concerns.
Thank you for the clarification. My concern was that the paper motivates PERC by the difficulty of obtaining an external verifier, while training still depends on exact answer matching and the experiments stay on tasks where that match is already a rule-based reward. The response accepts this dependence and says that answer-agreement GRPO is preferable when the goal is policy accuracy. It then relocates the contribution to deployment-time selection and diagnosis. That is a narrower claim, and the setting in which no rule-based answer check exists remains untested.
The same narrowing applies to scalability and reuse. The response acknowledges that the negative-example construction is task-dependent, and it does not evaluate whether a critic trained on one backbone or dataset can optimise a different policy. Cross-setting evidence remains limited to scoring a fixed candidate pool. The original questions about verifier-free training, open-ended tasks, and reuse as a training reward are therefore still open, so I am maintaining my rating.
Response to Reviewer YxCw #1
Thank you for the careful review. We address each weakness and its related questions below.
Weakness 1: The motivation conflicts with answer-based supervision
We target a setting where reference answers are available for training questions but unavailable when choosing between solutions to new questions. This motivates learning a critic that can select candidates and identify likely reasoning errors at deployment. Exact answer agreement is indeed rule-based outcome supervision during training; PERC does not remove that requirement. It removes the need for step annotations and separately collected preference labels.
Policy optimisation. Table 6 supports direct answer-agreement GRPO when the objective is policy accuracy on these verifiable tasks. The practical reason to train PERC is to obtain a scorer that also supports selection and diagnosis. Table 2 reports positive selection gains in all 27 source-target settings on fixed candidates, and Table 4 evaluates both natural medical errors and controlled arithmetic errors. These inference uses do not require policy optimisation or reference answers for new questions.
The 3-4x cost multiplier in Table 7 compares the full critic-plus-policy pipeline with SFT. It is not a measured multiplier over answer-agreement GRPO. In our runs, local critic preparation is estimated at 3-4 GPU-hours; policy optimisation is an additional cost. We report the separate API data-generation cost under Question 2 below.
Open-ended tasks. Applying the current training recipe would require reliable answer-equivalence labels and suitable negative examples. Our benchmarks provide extractable answers to construct training labels and measure selection accuracy.
Action taken. We revised the abstract and introduction (pp. 1-2, marked in blue) to distinguish training supervision from scoring without reference answers at deployment. Additionally, we further acknowledge the open-ended-task limitation in Section 6.
Weakness 2: Corruption construction is task-dependent
GSM8K uses programmatic arithmetic edits. MedReason and MMLU-Pro share an automated prompt that asks an LLM to produce plausible reasoning towards distractor answers. Individual traces are not manually edited, but the corruption procedure remains task-dependent. These experiments demonstrate use across the three domains, rather than unrestricted scalability.
MedReason and MMLU-Pro corruptions were generated offline through the GPT-5.2 API, using the question, expert reasoning, correct answer and distractors. The estimated API cost was a one-time $447:
We trained six Qwen2.5-7B-Instruct critics: three mixtures under interval (stride 15) and dense (stride 1) supervision. We match the architecture, initial weights, seed, 250-update budget, and frozen Qwen2.5-3B-Instruct training rollouts. Policy Only uses eight answer-matching and eight answer-mismatching traces per batch. Adding expert positives replaces four positive slots; adding expert corruptions then replaces four negative slots.
The clearest benefit is synthetic error localisation: dense Hit@1 rises from 24.0% to 76.1%, and Hit@7 from 43.9% to 90.5%, relative to Policy Only. Interval Hit@7 rises from 34.4% to 69.1%. Adding corruptions also improves localisation with expert positives held fixed. Reranking point estimates remain similar to Policy Only; expert positives alone do not provide a reliable selection improvement. The incremental benefit on natural errors remains untested.
Action taken. We added the matched comparison in Appendix C.4, Table 11. Additionally, we added the generation model, API cost breakdown and accounting scope to Appendix B.