- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies what happens when a checker for one requirement is evaluated against overall task success as the policy improves. It finds that changes in the remaining failures can make even a perfect checker appear worse. In an ECG study, refreshing the checker using this misleading audit damages its performance, while replaying old failure examples prevents the damage.
Critical strengths
- The paper explains why an unchanged checker can get a worse audit score. A perfect checker's audit AUC falls from 0.833 to 0.515 (engine task) and from 0.922 to 0.519 (GSM8K) from the case mix alone (Sec. 4). The derivations are short and, as far as I can tell, correct.
- The audit error leads to worse downstream decisions. The ECG study uses subject-held-out splits, an unseen generator, and hashed predictions, and it reports a failed prediction. It shows that the aggregate-AUC trigger degrades the verifier: F-AUC drops from 1.000 to 0.719 (Table 37).
Critical weaknesses
- It is unclear how often systems use this audit-and-refresh loop. No cited system implements the exact loop (lines 79–81, Table 34). Identify one or two common practices that do, such as step-level verifiers validated against final-answer correctness, and explain where the trusted F-labels needed for the fix would come from.
- Two claims need narrower wording. Correct labels fail to prevent substantial damage in only one of four ECG settings: Llama-GBM loses 0.235 F-AUC, while the others change by −0.000 to −0.006 (Table 36), so say they "can be insufficient." The GoEmotions pair was selected through a post-hoc scan, which the paper discloses but the abstract omits. Lead the inversion evidence with the trained ECG heads (Table 35).
Questions for authors
See weaknesses
2 comments
Thank you. I keep my score in accept range
Thank you for the review. We agree that the practical consequences of our results deserve greater prominence. We intend to use the additional rebuttal page to address this directly.
Two existing practices are particularly relevant. Lightman et al. (Let's Verify Step by Step, ICLR 2024, §2.4) use final-answer incorrectness and current PRM scores to select examples for human annotation and iterative verifier retraining. The Lessons of Developing Process Reward Models in Mathematical Reasoning (Findings of ACL 2025) demonstrates that outcome-based Best-of-N evaluation can reward models that shift from process verification towards outcome prediction.
These are concrete examples of using global task outcomes to evaluate or guide the maintenance of local verifiers.
Our contribution goes beyond establishing that these objectives can disagree. We show that improving the policy itself can invalidate the audit of an unchanged, even perfect, component verifier. This creates a maintenance problem: a declining score no longer tells the practitioner whether the verifier has deteriorated or the policy has successfully eliminated the failures it was designed to detect.
The original submission already demonstrates the consequence in Table 37, which we will bring into the main text. At an illustrative audit-drop threshold of 0.05, the aggregate-based controller triggers refresh in all four ECG settings, although the deployed specialists retain F-AUC of 1.000. Acting on this signal reduces mean F-AUC to 0.719, increases constraint violations from 0.148 to 0.179, and reduces Answer@16 from 0.530 to 0.491.
Our additional post-review controls identify a further difficulty: on the improved Llama population, component AUC is undefined in all 12 held-out-patient folds because the relevant failures have disappeared. In Phi-MLP, even a computable component-specific audit on the improved population selects harmful updates (+1.44 percentage points in held-out constraint violations).
Replacing outcome labels with component labels is not sufficient. The reference population matters.
We will use the additional page to formulate practical guidance grounded in these results and Propositions 1–3: distinguish loss of audit support from loss of competence, recover independently labelled reference examples when necessary, and evaluate candidate refreshes against the no-update alternative on an appropriate reference distribution.
Such labels can come from human process annotations, as in PRM800K and ProcessBench, or from simulations and execution-based measurements in engineering optimization and pipeline search. We will explicitly discuss their costs, availability and applicability.
We agree with your interpretation of Table 36: substantial correct-label degradation occurs in one of four settings (Llama-GBM: −0.235; others: −0.000 to −0.006 F-AUC). We will narrow the claim accordingly.
We have also completed a matched-budget sensitivity analysis using reconstructed specialists. In Llama-GBM, eight retained F-failure examples recover mean F-AUC from 0.719 to 1.000. Random replay reaches 0.858 with eight examples and 1.000 with 32.
This is a concrete, setting-specific benefit, not a universal advantage of targeted replay. Under outcome-labelled refresh, budgets of up to 64 retained examples remain insufficient in three of four settings. We will report the complete results, including these negative findings.
We will explicitly disclose the post-hoc GoEmotions selection in the abstract and lead with the trained ECG heads in Table 35. The latter provide stronger evidence: the pure component head ranks last under the post-DPO audit in all four settings despite achieving base-distribution F-AUC of 1.000.
The revised manuscript will make the central distinction explicit: a worse audit is not necessarily evidence of a worse verifier, and treating it as such can actively damage the system.
We believe these changes would substantially strengthen the paper's practical contribution while preserving its theoretical and empirical integrity. Would this revision address your remaining concerns?