acceptodds
Under review as a conference paper at ICLR 2027

Audit Certainty Is a Signed Lever: Why a Verifier Calibrated on a Proxy Can Backfire in Deployment

Abstract

We calibrated a verifier on a proxy task, deployed it unchanged, and its strongest setting became its worst: raising audit certainty — the lever that cut violations on the proxy — raised them in deployment. Our claim — proxy calibration of audit certainty need not transfer across temporal structures for learning agents — rests on two findings of different standing. The first is derived. Classical deterrence says only the expected penalty, probability times severity, matters; for an agent that learns from the signal it does not, because rare severe penalties inflate the variance of the learned violation value. We prove an exact stationary-variance result and, under a Gaussian threshold approximation, a signed certainty law: raising certainty deters when the violation is value-suboptimal and backfires when it is value-optimal. The law predicted the inversion before the experiment; a deep-RL phase diagram confirms it, and the exact law bounds the approximation's error. The second finding is causally isolated but not derived, and the headline rests on it: the location of the sign boundary moves with task timing. With timing equalised it sits where the law predicts; delaying honest reward alone pushes it past every value gap we can measure, so a timing-mismatched proxy certifies the wrong policy. That displacement is orders of magnitude beyond the bounded approximation error, so the theory we prove does not explain it; we isolate the channel, rule out four candidate derivations and one whole route, and leave the closed form open. Under GRPO on an open LLM, over a pre-registered orthogonal grid of several hundred training runs, the certainty-over-severity ordering reproduces, and the logged advantages show why: group normalisation caps the advantage any single penalty can carry, discarding severity but not certainty, and a group-size sweep orders severity's weight as the bound predicts. Design rule: calibrate on matched timing, or measure the value gap under deployment timing before trusting a proxy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, reward hacking, scalable oversight, verifier design, deterrence, reinforcement learning theory, GRPO, RLVR, temporal credit assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EJ2pOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is studying something related to the cost of running a verifier in reinforcement learning.

Critical strengths

None

Critical weaknesses

The manuscript as presented does not satisfy basic scientific standards of communication.

  • It is incredibly dense writing.
  • The text does not clearly establish what it attempts to do or why.
  • The main claim according to the text is that L015 "proxy calibration of audit certainty need not transfer across temporal structures for learning agents". After reading the text, it is still not clear to me what that means.
  • Mathematical notation is not defined.
  • Uncontextualized numbers are scattered everywhere. These may be referring to experiments that are not described or referenced.
  • The experiments are missing proper descriptions as to what the experiments are trying to do and how.

The manuscript also seems that the paper is simply not finished since it references "earlier drafts" as if the reader should be familiar with them: L086 "Three negative results against our own earlier draft are stated where a reader meets the claim each retracts". Sentences like these are scattered throughout the text.

My review is not particularly detailed as the text is not readable. I believe extensive revisions are needed in terms of writing before it can be properly reviewed

Questions for authors

None

Reviewer_sok8OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether increasing audit probability improves behavior when expected punishment is held constant by reducing penalty magnitude correspondingly. It first derives the stationary variance of a simplified stochastic value update and obtains a sign-dependent behavioral prediction under a Gaussian threshold approximation. Controlled reinforcement-learning experiments then show that temporal structure can shift the relationship between audit certainty and undesirable behavior, producing a reversal between a proxy task and a delayed deployment task even under a matched expected-return gap. Additional experiments examine alternative learning algorithms, timing conditions, and model choices. A separate GRPO study investigates audit probability and penalty magnitude in a language-model choice task. The principal contribution is evidence that the behavioral effect of audit certainty depends on the learning process and task structure, limiting transfer from proxy calibration.

Critical strengths
  1. The paper demonstrates a consequential failure of proxy-based calibration through controlled interventions. The experiments go beyond observing a correlation: they manipulate timing, match the expected-return gap, and show opposing responses to audit certainty across conditions. These results directly challenge the assumption that a successful oversight setting can be transferred based on expected punishment alone.

  2. The manuscript contains unusually substantial robustness analysis and reports unsuccessful explanations. Equalized-timing experiments, alternative learning algorithms, exact-law checks, and additional controls help delimit the simplified theory. The authors also report nonreplicating scaling behavior and unsuccessful mechanistic models. This strengthens the credibility of the empirical contribution and makes the unresolved mechanism explicit.

Critical weaknesses
  1. The GRPO analysis overstates what its regression contrast establishes about expected-value-only explanations. Appendix J argues that any response of the form P=f(pc)P=f(pc) must enter a regression on log⁡p\log p and log⁡c\log c symmetrically because the regressors span nearly equal ranges. Similar ranges alone do not establish this property for an arbitrary nonlinear response projected onto an additive regression. The reported polynomial comparisons and robustness checks are useful evidence against the particular expected-value models tested, but do not justify excluding every expected-value-only account. This affects the interpretation of the LLM extension rather than the main controlled reinforcement-learning result.

  2. The connection to deployed language-model oversight remains indirect. The strongest transfer reversal is established in controlled reinforcement-learning environments, whereas the language-model experiment uses an audit-framed, priced-option task. It does not demonstrate the same proxy-to-deployment timing reversal in an interactive LLM agent. The manuscript acknowledges important differences between these settings, but those differences constrain the practical reach of its recommendations.

Questions for authors
  1. Clarification: Can the symmetry claim in Appendix J be established under explicit assumptions on the response function and experimental design? Otherwise, should the conclusion be narrowed to rejection of the specific expected-value models tested?

  2. Clarification: What deployment criterion follows from the experiments when a negative expected-return gap is itself insufficient to guarantee deterrence under delayed timing? Please distinguish a measured empirical response boundary from the approximate theoretical sign condition.

Reviewer_S1H7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the different effects of audit probability p and penalty magnitude c on learning agents. It derives the exact stationary variance of a constant-step-size single-action value estimator and a signed certainty law under a Gaussian threshold approximation at fixed E=pc. Changing the timing of honest rewards can reverse a proxy-calibrated behavioral response, even with the violation–honesty value gap matched. It also shows a standardized-advantage bound under GRPO and investigates its incomplete role in behavioral sensitivity to penalty severity.

Critical strengths
  1. If delay merely makes violation value-optimal, reversal is unsurprising. Section 4.4 goes further by fixing the deployment gap to the proxy's −0.070 and still observing opposite responses, ruling out that simple explanation.
  2. The stationary-variance calculation is transparent, and the exact-law analysis checks the Gaussian approximation and its sparse-audit failure. 3.Joint logging of rewards, advantages, entropy, and behavior helps establish whether a larger penalty actually reaches optimization.
Critical weaknesses
  1. The single-point bound (K−1)/K(K-1)/\sqrt K under sample-standard-deviation normalization is correct and directly checkable. However, the cap-free controls cover only p=0.5, with a pooled behavioral permutation p-value of 0.11, so they do not identify the change in the full certainty-minus-severity contrast.
  2. The matched-timing and matched-gap experiments support the nontransfer phenomenon, but the visit-rate predictor in Appendix R is not independent mechanistic validation.
Questions for authors
  1. Which direct timing-interaction results support the main claim without any extrapolated boundary estimates or visit-rate correlations?
  2. Please report gradeability by cell and the principal rates/regressions both over all outputs and over gradeable outputs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.