acceptodds
Under review as a conference paper at ICLR 2027

Better Policies, Worse Verifier Audits

Abstract

Verifier stacks increasingly sit inside post-training and test-time compute loops. We study a specific control loop: a component verifier is evaluated on fresh policy outputs against the full success label and refreshed when that aggregate score falls. For a conjunctive target , its on-policy AUC is , where . As the policy masters , can rise toward one, removing weight from -failures and rewarding information about . We call this on-policy audit collapse. We derive a sharp identified set, a complete-erasure threshold, and the exact point at which the audit can prefer a worse component verifier; its population optimum is the full-outcome predictor, not the component oracle. An exact oracle exhibits the collapse, a natural multi-label text control shows inversion can select a head with lower canonical -AUC at the observed failure mixture, and a prediction-registered ECG study shows that acting on the audit can reduce off-policy -AUC, transfer the damage to an unseen generator, and increase downstream constraint violations. Preserving own-failure support prevents the damage. Matching label-complexity bounds and a controlled-mixture audit give the corresponding repair: audit the component on its own failures, and keep those failures in the refresh distribution.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
verifiers, reward models, on-policy evaluation, audit collapse, distribution shift, partial identification, post-training, alignmen

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BZAWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what happens when a checker for one requirement is evaluated against overall task success as the policy improves. It finds that changes in the remaining failures can make even a perfect checker appear worse. In an ECG study, refreshing the checker using this misleading audit damages its performance, while replaying old failure examples prevents the damage.

Critical strengths
  • The paper explains why an unchanged checker can get a worse audit score. A perfect checker's audit AUC falls from 0.833 to 0.515 (engine task) and from 0.922 to 0.519 (GSM8K) from the case mix alone (Sec. 4). The derivations are short and, as far as I can tell, correct.
  • The audit error leads to worse downstream decisions. The ECG study uses subject-held-out splits, an unseen generator, and hashed predictions, and it reports a failed prediction. It shows that the aggregate-AUC trigger degrades the verifier: F-AUC drops from 1.000 to 0.719 (Table 37).
Critical weaknesses
  1. It is unclear how often systems use this audit-and-refresh loop. No cited system implements the exact loop (lines 79–81, Table 34). Identify one or two common practices that do, such as step-level verifiers validated against final-answer correctness, and explain where the trusted F-labels needed for the fix would come from.
  2. Two claims need narrower wording. Correct labels fail to prevent substantial damage in only one of four ECG settings: Llama-GBM loses 0.235 F-AUC, while the others change by −0.000 to −0.006 (Table 36), so say they "can be insufficient." The GoEmotions pair was selected through a post-hoc scan, which the paper discloses but the abstract omits. Lead the inversion evidence with the trained ECG heads (Table 35).
Questions for authors

See weaknesses

2 comments

Thank you for the review. We agree that the practical consequences of our results deserve greater prominence. We intend to use the additional rebuttal page to address this directly.

  1. Where does this problem arise in practice?

Two existing practices are particularly relevant. Lightman et al. (Let's Verify Step by Step, ICLR 2024, §2.4) use final-answer incorrectness and current PRM scores to select examples for human annotation and iterative verifier retraining. The Lessons of Developing Process Reward Models in Mathematical Reasoning (Findings of ACL 2025) demonstrates that outcome-based Best-of-N evaluation can reward models that shift from process verification towards outcome prediction.

These are concrete examples of using global task outcomes to evaluate or guide the maintenance of local verifiers.

Our contribution goes beyond establishing that these objectives can disagree. We show that improving the policy itself can invalidate the audit of an unchanged, even perfect, component verifier. This creates a maintenance problem: a declining score no longer tells the practitioner whether the verifier has deteriorated or the policy has successfully eliminated the failures it was designed to detect.

  1. What does this mean for an actual maintenance decision?

The original submission already demonstrates the consequence in Table 37, which we will bring into the main text. At an illustrative audit-drop threshold of 0.05, the aggregate-based controller triggers refresh in all four ECG settings, although the deployed specialists retain F-AUC of 1.000. Acting on this signal reduces mean F-AUC to 0.719, increases constraint violations from 0.148 to 0.179, and reduces Answer@16 from 0.530 to 0.491.

Our additional post-review controls identify a further difficulty: on the improved Llama population, component AUC is undefined in all 12 held-out-patient folds because the relevant failures have disappeared. In Phi-MLP, even a computable component-specific audit on the improved population selects harmful updates (+1.44 percentage points in held-out constraint violations).

Replacing outcome labels with component labels is not sufficient. The reference population matters.

We will use the additional page to formulate practical guidance grounded in these results and Propositions 1–3: distinguish loss of audit support from loss of competence, recover independently labelled reference examples when necessary, and evaluate candidate refreshes against the no-update alternative on an appropriate reference distribution.

Such labels can come from human process annotations, as in PRM800K and ProcessBench, or from simulations and execution-based measurements in engineering optimization and pipeline search. We will explicitly discuss their costs, availability and applicability.

  1. Correct-label refresh and replay.

We agree with your interpretation of Table 36: substantial correct-label degradation occurs in one of four settings (Llama-GBM: −0.235; others: −0.000 to −0.006 F-AUC). We will narrow the claim accordingly.

We have also completed a matched-budget sensitivity analysis using reconstructed specialists. In Llama-GBM, eight retained F-failure examples recover mean F-AUC from 0.719 to 1.000. Random replay reaches 0.858 with eight examples and 1.000 with 32.

This is a concrete, setting-specific benefit, not a universal advantage of targeted replay. Under outcome-labelled refresh, budgets of up to 64 retained examples remain insufficient in three of four settings. We will report the complete results, including these negative findings.

  1. Presentation of the inversion evidence.

We will explicitly disclose the post-hoc GoEmotions selection in the abstract and lead with the trained ECG heads in Table 35. The latter provide stronger evidence: the pure component head ranks last under the post-DPO audit in all four settings despite achieving base-distribution F-AUC of 1.000.

The revised manuscript will make the central distinction explicit: a worse audit is not necessarily evidence of a worse verifier, and treating it as such can actively damage the system.

We believe these changes would substantially strengthen the paper's practical contribution while preserving its theoretical and empirical integrity. Would this revision address your remaining concerns?

Thank you. I keep my score in accept range

Reviewer_WpZiOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies a problem that arises when a component verifier is evaluated against the overall success criterion (y), rather than the specific criterion (F) it is designed to assess. The authors refer to this evaluation protocol as an on-policy audit. They show that, as the policy improves, the structure of unsuccessful examples used to audit the verifier can change. So, the audit tends to assess the verifier's ability to distinguish successful from unsuccessful examples with respect to other factor associated that is the correctness of the solution to the main task (M), rather than with respect to its ability to detect violations of F. This can lead to misleading assessments of verifier quality, trigger inappropriate verifier updates, and potentially deteriorate previously acquired knowledge about detecting violations of F.

To address this problem, the authors propose avoiding the use of the aggregate success metric as the sole criterion for verifier’s assessment and preserving examples that violate F in the dataset for the verifier's training, even when the policy rarely generates such examples.

The paper identifies a methodological problem that can arise under a particular verifier’s audit protocol. Modern AI systems, including those based on LLMs, increasingly use verifiers whose evaluation results inform policy optimization, candidate selection, and updates to the evaluators themselves. This makes the problem considered relevant to the design and evaluation of such systems.

Overall, the paper reveals an interesting and non-obvious failure mode in verifier auditing, provides a formal analysis of the underlying mechanism, and derives practical recommendations for specific auditing settings. Its findings can help researchers develop more reliable procedures for verifier building, evaluation and updating.

Critical strengths
  1. A clear formalization of a specific and non-obvious problem in verifier audit that arises from changes in the structure of failures during policy optimization, leading to a paradoxical theoretical result.

  2. A derivation of the conditions under which an audit loses information about component quality and begins to favor an worse verifier (audit inversion).

3.A strong experimental evaluation across 3 datasets that provides empirical evidence supporting the theoretical claims.

  1. An idea to preserve examples violating the component F in the verifier's training data, supported by theoretical analysis and experimental results.
Critical weaknesses
  1. The authors should provide stronger evidence that auditing AI models, particularly LLMs, against the overall success criterion y rather than the specific criterion F is a practically common scenario. They should also provide concrete examples of such auditing practices and a more comprehensive review of the relevant literature. The Related Work section is placed at the end of the paper, and the motivations for this choice are not clear.

  2. The authors propose preserving examples that violate the target component F in the verifier's training data. This seems to presuppose that there is a way to identify whether an example violates F. If so, why not use the F-labels to audit the verifier directly, rather than evaluating it against the aggregate outcome y?

  3. The authors consider a setting in which the policy's satisfaction of F improves during optimization, leading to a deterioration in the verifier‘s audit. However, what happens if optimization does not improve satisfaction of F faster than satisfaction of M? The paper does not explain how to determine which component (F or M) is being satisfied more rapidly when F cannot be evaluated directly, or how to estimate the resulting changes in q (Equation (1)) and the audit degradation level.

Questions for authors
  1. How common in practice is the considered auditing scenario, in which a component verifier is evaluated against the overall success criterion y rather than the component-specific criterion F? In what practical settings are direct evaluation or reliable F-labels unavailable or prohibitively expensive?

  2. How can researchers detect and quantify audit collapse when component-specific labels for F are unavailable? In particular, how can be determined whether policy optimization increases satisfaction of F faster than satisfaction of M, and estimate the resulting changes in q and the severity of audit degradation? For which applications or models is this pattern expected to occur?

  3. How can the proposed solution that is preserving examples that violate F in the verifier's training dataset be implemented when component-level labels are expensive or unavailable for some examples? What assumptions or additional mechanisms are required to identify and retain such examples?

See Critical Weaknesses for details.

Reviewer_GewUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies a verifier maintenance loop: a verifier for one component F of a task (success y = F ∧ M) is scored on fresh policy outputs against the full label y and refreshed when that score drops. AUC against y splits exactly into a weighted mix of how well the verifier separates successes from F-failures and from M-only failures, with weight q on the latter. As the policy masters F, q goes toward 1, so the audit stops measuring F and can even prefer a worse F-verifier that leaks M information. The authors argue that acting on this falling audit by refreshing on depleted on-policy data destroys the verifier’s F competence and raises downstream violations (ECG study). Their fix is to audit the component on its own failures and keep those failures in the refresh data.

Critical strengths
  • The paper keeps the audit’s estimand (A_own) separate from canonical F-AUC (Eq. 2), which many papers would have conflated.
  • The inversion threshold is tested on a case where it predicts no inversion (GSM8K) and is confirmed, not only on favorable cases.
  • The ECG study uses subject-held-out splits, an unseen generator, and a downstream gate metric, and it reports the failed prediction (P2).
Critical weaknesses
  • The "refresh trap" is mostly label substitution. Ry trains an F-head on y labels, so losing F-AUC is expected by construction. With correct F labels (RF), Table 36 shows essentially no loss in 3 of 4 settings, so "correct labels are insufficient" rests on one setting (Llama-GBM).
  • The refresh design is not factorial. Without an Ry-plus-replay arm, the claim that the damage comes from missing support rather than wrong labels is not identified. The λ=1 heads (y labels on base data, F-AUC 0.94 to 0.97) suggest the harm is an interaction of the two.
  • Rmix is confounded with ordinary replay and doubled data size. It re-injects the very distribution S was trained and evaluated on, so preserving F-AUC is standard rehearsal, not evidence that own-failure support specifically matters.
  • The size of the harm depends on refresh settings no practitioner would use unvalidated: 100 GBM trees with no early stopping, 20 MLP epochs, and no sweep over refresh strength. The GBM drop to chance (4× the MLP drop) looks like tree extrapolation off support, not a general effect.
  • The repair presupposes the component labels whose absence is the only reason to use the y-based audit. The paper never addresses the motivating case where only y is observed.
  • The theory restates known facts (subgroup additivity of AUC, bounds on a weighted average, the classical optimal ranker), and the designed two-mixture audit is essentially label-shift correction, a literature (e.g., BBSE) the paper does not cite.
  • Fig. 1 (right) is near-tautological: plugging A_cross ≈ 1/2 into an exact identity when A_cross is measured near 0.5 will track by construction.
  • Per-pool AUC uses tiny pools (often 1 to 2 positives), and the set of "evaluable" pools changes as the policy improves. That selection effect is mixed into the collapse trajectories and never reported or controlled for.
  • GoEmotions involves no policy or optimization, and the entangled score contains M by construction, so it only shows that Eq. 1 holds on static text labels.
  • The inversion examples are tiny (0.515 vs 0.532 on engine), and no experiment shows that acting on an inversion causes material harm.
Questions for authors

Covered in weaknesses.

2 comments

Thank you for the detailed review. We address the audit estimand, consequences of acting on it, and refresh attribution. New controls and negative results will be documented in the revised appendix.

  1. Audit mechanism and contribution. With y=F∧My=F\land M, aggregate AUC compares successes with both F-failures and candidates that satisfy F but fail M. When F-failures become rare relative to M-only failures, the M-only mixture weight qq increases, reducing the audit's sensitivity to the specialist's original task. Aggregate AUC can then deteriorate for an unchanged specialist or prefer a worse verifier. Proposition 1 partially identifies AownA_{\mathrm{own}} conditional on mixture information, distinct from canonical F-AUC (Eq. 2). Proposition 2 characterizes ranking inversion, while Proposition 3 quantifies the increasing cost of obtaining informative F-failure comparisons. The novelty lies in the policy-dependent audit-to-action consequence. We will clarify the relationship to xAUC and label-shift methods such as BBSE.

  2. Consequences for decisions. The engine example (0.515 versus 0.532) tests the predicted ranking threshold; GSM8K examines a regime where inversion is not predicted. Figure 1(right) verifies the mixture decomposition. Appendix AD.5 (Table 37) already evaluates a frozen audit-triggered controller. At a 0.05 audit-drop threshold, it chooses Ry in all four settings although S retains F-AUC=1.000. Mean F-AUC drops to 0.719, F-gate violations rise from 0.148 to 0.179, and Answer@16 falls from 0.530 to 0.491. This post-hoc counterfactual connects the audit to a maintenance decision, complementing the inversion example.

  3. Labels, support, and replay. Your point about RF/Rmix is well taken: the original comparison also changes dataset size, and correct-label RF degradation is concentrated in Llama–GBM. We will refine the support-specific interpretation in Sections 5–6. New E1 controls cross F/y refresh labels with no replay, random historical replay, F-failure replay, and M-only replay, using equal sample budgets and matched feature rows for paired labels. In independently reconstructed MLP specialists, F-labelled versus y-labelled no-replay F-AUC is 0.9995 versus 0.9083 (Llama) and 0.9912 versus 0.9196 (Phi). The label-target contrast is consistent across the MLP settings; benefits from targeted replay depend on the setting. In the original GoEmotions support test, targeted replay improves F-AUC to 0.606 versus 0.550 for random replay. GoEmotions also tests semantic leakage using separately annotated F and M criteria: specialist versus entangled F-AUC is 0.679 versus 0.570, while aggregate AUC is 0.442 versus 0.681.

  4. Refresh strength and verifier selection. New E2 controls test 1, 2, 5, 10, and 20 MLP epochs, including step 0 (retaining S). In exploratory 12-subject leave-one-subject-out checks, aggregate-y validation selects a nonzero Ry update in 12/12 Llama–MLP and 12/12 Phi–MLP folds. Relative to S, patient-macro F-gate violations rise by 1.26 and 2.29 percentage points. Thus the monitored objective matters even with mild updates and optional stopping. The F-specific reference requires additional component labels. E4 also finds worse held-out gating under post-policy y-auditing versus a base-policy F reference (+0.99 pp Llama–MLP; +0.59 pp Phi–MLP). E4 compares procedures differing in target, candidate population, and AUC aggregation, so it supports the operational comparison.

  5. Measurement and scope. E3 quantifies the changing audit population: 292/500 baseline Llama and 283/500 baseline Phi pools are AUC-evaluable, and some optimized Llama policies have no evaluable pools. The appendix will report undefined AUCs, positive-negative pair counts, macro/pair-weighted results, and matched-pool analyses where feasible. The new E1/E2/E4 controls use original frozen ECG pools with independently reconstructed MLP specialists. The 500 requirements represent 12 test subjects; LOSO training folds overlap. We will include protocols, null results, and the original unsuccessful P2, distinguishing new sensitivity analyses from registered results.

With sufficient component labels, direct F auditing is preferable. Where such failures are scarce, partial identification describes what aggregate labels leave unresolved; designed-mixture recovery requires known weights and stable conditional distributions. This distinction will be clearer in the revision.

The original controller and these controls jointly support the paper's audit-to-action claim. We will add verified E1–E4 results to the appendix and make focused main-text corrections. We would welcome your assessment of any remaining concern decisive for your recommendation.

Following our previous response, we completed three further analyses, now included in Appendix AG.

E5 — Matched audit selection. On the same validation population, switching from F-AUC to outcome AUC increases held-out gate violations by 0.38/1.05 pp for Llama/Phi-MLP, with no difference for either GBM. On the improved population, the differences are +0.99/+1.56/+0.59/0 pp. However, F-AUC is undefined in all Llama folds, making these comparisons against keeping the original specialist. Under common pool eligibility, the Phi-MLP difference becomes −0.01 pp. Matched E2 also shows that component-specific validation on the improved population can select harmful updates (+1.44 pp for Phi-MLP).

E6 — Pool sensitivity. Post-DPO ranking inversions account for at least 51% of candidate-pair comparisons in both MLP settings across the defined sensitivity analyses. The frozen critic's audit decline is robust for Llama but depends on the evaluation bank for Phi.

B1 — Replay budgets. With correct labels, substantial degradation occurs only in Llama-GBM. Eight targeted historical failures restore F-AUC from 0.719 to 1.000, while random replay requires 32 among the tested budgets. Under outcome-labelled refresh, budgets up to 64 remain insufficient in three settings, and full targeted replay produces more violations than random replay in all four. We have narrowed our replay claims accordingly.

These analyses use reimplemented specialists and 12 unique ECG test patients.

We have also revised the discussion of BBSE, Figure 1, GoEmotions, and the practical implications of the controller experiment.

The results support our central claim that aggregate outcome auditing can lead to harmful maintenance decisions despite validation-based selection and early stopping.

Thank you for the detailed review. We would appreciate your assessment of whether any of your concerns remain unresolved.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript

We would like to thank all reviewers for the discussion. We have now uploaded the revised paper and incorporated the changes discussed in our individual responses.

We have expanded the motivation and discussion of practical verifier maintenance, including examples from existing work and sources of component-specific supervision. We also clarify when direct component auditing is preferable and what Proposition 1 tells us when such supervision is limited.

The new experiments in Appendix AG address the concerns about replay composition, refresh strength, early stopping, evaluation-pool selection and downstream consequences. Some of these results led us to narrow our original claims about correct-label refresh and targeted replay. We have also moved the controller results into the main text, given greater prominence to the trained ECG heads, and made the exploratory nature of the GoEmotions experiment explicit in the abstract.

We have clarified the relationship to existing work on AUC decomposition, xAUC and label shift, and revised several parts of the paper that were open to misinterpretation.

We believe the revised manuscript makes both the contribution and its limitations clearer. Thank you again for taking the time to discuss these points with us.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.