acceptodds
Withdrawn from ICLR 2027

Access and Representation Barriers to Auditing under Proxy-Score Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language models are optimized against cheap proxy scores, but how costly is it to audit the resulting target change? We identify access and representation barriers linked by likelihood-ratio geometry. Baseline archives face a no-hit lower bound, while active auditors trade observable-transcript KL against sample count. An exact duality characterizes worst-case target change hidden from a fixed feature span. For exact single-prompt one-sided exponential-power tilts, minimax log archive complexity and half-mass likelihood-ratio-tail rarity asymptotically match optimization KL. Using exact conditional reweighting on a finite pool of judge-scored Qwen2.5-7B math traces, we find a -order increase in the tested archive requirement, seven practical samplers above the movement–sample benchmark, and substantial held-out error in fitted score-blind monitors. At 18 score-matchable Group Relative Policy Optimization (GRPO) checkpoints, the plug-in KL residual from a score-matched one-parameter tilt accounts for – of measured KL, indicating substantial departure from that tilt path. Reported LR-tail estimates differ substantially from score-tail rarity, although differing likelihood-evaluation paths and a finite reference pool limit their interpretation. This motivates measuring LR-tail overlap under a common likelihood convention before using proxy-score tails as overlap diagnostics.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
language model auditing, proxy-score optimization, model monitorability, distribution shift, scalable oversight, likelihood-ratio geometry

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eNybOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how difficult it is to measure changes in true performance after optimizing a proxy score. It identifies two obstacles: old data may rarely contain the outputs that become common after optimization, and monitoring features may miss changes relevant to the target. It formalizes these obstacles, derives sample and representation requirements under specified assumptions, and studies them using reweighted language-model outputs and GRPO checkpoints.

Critical strengths
  1. The paper distinguishes limited access to new labels from limitations of the monitoring features. This helps clarify two different reasons why an audit can fail.

  2. The experiments include controlled distribution shifts and actual GRPO checkpoints. The authors also report several limitations and negative results, including substantial differences between the controlled setting and trained policies.

Critical weaknesses
  1. The likelihood-ratio measurements are inconsistent. New-policy and baseline samples use different tokenization and scoring procedures. This weakens the quantitative overlap conclusions in Figure 4.

  2. The main auditing experiments are disconnected from actual trained policies. They evaluate an estimator on artificially reweighted distributions, while the GRPO results show substantial departures from this distribution family. The estimator is not directly evaluated on those checkpoints.

  3. E1 does not separate estimator limitations from intrinsic difficulty. It tests one binning estimator without comparisons against standard importance-sampling alternatives. Its empirical sample requirements therefore do not establish that other estimators would struggle similarly, even though the paper provides separate lower bounds under specific assumptions.

  4. The monitor experiments do not directly establish failure to estimate correctness. They primarily measure reconstruction of the distribution shift, which is a different objective. The stronger linear certificates also use weights that are not renormalized on the evaluated subset, limiting their interpretation as hidden changes in target performance.

  5. The practical importance of the restricted-label setting is insufficiently demonstrated. The mathematical benchmarks allow direct correctness checks. They provide a controlled test, but do not show when obtaining new labels becomes the main obstacle in a realistic application.

  6. The added value over existing overlap and importance-sampling results needs clearer justification. The paper establishes additional results under specific assumptions, but the experiments do not clearly demonstrate what these results explain beyond established coverage difficulties.

Questions for authors

Please see weaknesses above.

Reviewer_P1jGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is interested in determining, after a model has been optimized, how costly it is to guess what changed. I stopped after abstract/intro, as these are simply unreadable.

Critical strengths

None

Critical weaknesses

I consider myself an expert in AI auditing, yet I simply cannot read the abstract. It is a stack of undefined jargon. Abstracts are supposed to provide a quick overview of the field, the problem(s) with it, the proposed approach, and results; we have here an overly technical listing of what the paper is supposed to show, that I cannot see who it can benefit as a reader.

The introduction reflects this, plus the fact that it is not even possible to get things out of it, even with the most careful reading. The whole problem of the field and related works are evacuated in a single paragraph, then claims start to stack, as if we were reading a paper Rationale section, for instance:

  • Precise example of an unreadable sentence: the first one of the second paragraph in the intro "Consider estimating accuracy change after optimization from a labeled baseline archive, baseline-model probes, or unlabeled optimized outputs. Fresh gold labels from both policies would make this routine." What are "gold labels"? (undefined). "both"? I see three elements if the authors refer to the previous sentence. "policies" ? (undefined). make "this routine": I cannot get what it refers to. "two barriers arise (Figure 1)": the Figure does not depict any barrier or two separated concepts that would match a notion of barriers.
  • line 42: "Access asks whether data reach behavior made common by optimization (Song et al., 2024; Tajwar et al., 2024)." Is not a proper sentence.
  • line 44: "Representation asks whether monitor features express the induced shift": -> the features that are monitored?
  • line 46: "episode laws" What is an episode law ? (undefined)
  • line 50: "active" or "passive" accesses are undefined, yet the authors follow with bounds, as if this is sufficient to get anything from that sentence.
  • line 52: "no-hit bound" and "tilt" are undefined concepts yet.
  • line 72: "rarity matches optimization KL at leading order". Assuming I am correct on what is KL, the sentence is not proper.
  • line 82: "closed monitor subspace": undefined, not understandable at that stage.

At that stage, I claim this paper is unsuitable for submission to any conference, regardless of the potential of the research idea trying to be conveyed.

Questions for authors

None

Reviewer_BsguOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the problem of auditing the true target objective change when optimizing cheap proxy objectives. The paper argues that it gets difficult as the model changes more. Theoretically, they show that samples from the original model rarely cover the new behavior, so the amount of labeled data needed grows quickly, and that label-free monitors can miss any change they are unable to represent. They support these claims with experiments on MATH reasoning traces from Qwen2.5-7B, including GRPO-trained checkpoints.

Critical strengths
  • The proofs are complete and seem correct as far as I could check.
  • It is a clean and unified framing that the likelihood ratio W and the centered shift g = W - 1 tie together coverage (access) and monitor expressivity (representation) for post-optimization auditing.
  • The paper is candid about prior work and scope, e.g. it explicitly does not claim the exp(KL) importance-sampling threshold as new.
Critical weaknesses
  1. The theory seems to be mostly a repackaging of standard tools. For example, the no-hit bound is Le Cam's two-point method. The active law is a data-processing/change-of-measure argument (the paper itself cites Kaufmann et al. 2016, Lemma 1). The representation 'law' is textbook L1/L∞ best-approximation duality via Hahn–Banach. Statistic sufficiency iff g is σ(T)-measurable is nearly immediate. The exp(KL) threshold is also known (Chatterjee–Diaconis).

  2. The practical relevance of the access barrier is under-motivated. Prop. 1 shows that fresh labeled samples from the optimized policy solve the problem with O(ε^-2 log 1/δ) labels regardless of r, and the optimized model is normally queryable. The barrier therefore applies only to auditors restricted to old baseline archives or baseline probes, and the paper never gives a concrete deployment scenario where that restriction holds. The near-attaining active oracle itself needs direct q-sampling plus likelihood-ratio access.

  3. The GRPO analysis largely restates one fact (GRPO spends far more KL than an exact tilt) and compares quantities that are not commensurate. The trained policy's whole-sequence KL includes reward-irrelevant drift (style, length, formatting) and has a 0.16-nat estimator offset at the base model. The tilt KL is pool-restricted and bounded by log N. A 96.7–99.8% 'residual' is therefore expected. It amounts to 1 - d(t*)/d̂, i.e. the same efficiency gap, and does not by itself show that (X,S) is insufficient (the authors concede this).

Questions for authors

See weakness section for my main concerns. Other questions are:

  • In what concrete auditing scenario can the auditor not draw and label fresh samples from the optimized model? Without one, does the access barrier reduce to the known OPE overlap problem?
  • Can the GRPO residual be decomposed further, e.g. a tilt on (X, S, length) or other reward-irrelevant statistics, to show how much of the 97–99% is score-orthogonal drift versus genuine departure in score-relevant geometry?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.