acceptodds
Under review as a conference paper at ICLR 2027

Statistical Limits of Safety Certification for Evaluation-Aware Agents

Abstract

Safety evaluation implicitly assumes that behavior observed during testing is informative about behavior after deployment. For an evaluation-aware agent this assumption is itself strategic: contextual differences between evaluation and deployment can be used to decide when to reveal unsafe behavior. We study this problem in a deliberately favorable setting for the evaluator—unsafe behavior is observed perfectly whenever it occurs, and there is no finite-sample estimation error in the basic model. Let \(Q\) and \(P\) denote the evaluation and deployment distributions of all context available to the agent before a safety-critical decision. We show that the largest deployment risk compatible with evaluation risk at most \(\alpha\) is exactly the power of the most powerful level-\(\alpha\) statistical test for distinguishing \(P\) from \(Q\). Thus the optimal strategically conditional policy is a Neyman–Pearson test for whether deployment has begun. This duality yields sharp divergence certificates, an exact characterization of risk hidden on evaluation-null events, and a sequential amplification theorem showing that arbitrarily weak persistent audit cues can become asymptotically decisive over long horizons. We then extend the theory to multiple interventions and prove an interventional non-identifiability result. Finally, we establish an audit paradox: under aggregate evaluation, reallocating testing mass toward unmistakably audit-specific contexts can strictly weaken the strongest valid deployment certificate. The results identify audit–deployment indistinguishability, from the agent's point of view, as a fundamental requirement for behavioral safety certification.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, strategic agents, evaluation-aware agents, safety certification, interventional testing, causal identifiability, deployment shift, hypothesis testing, Neyman–Pearson theory, audit–deployment distinguishability, transportability, deceptive alignment, adaptive oversight

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KZLqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The author's consider the problem of safely evaluating an agent before it is deployed. The evaluation environment will inevitably from the agent's real deployment environment. As such, an agent may be able to identify when they are being evaluated and attempt to circumvent safety guardrails by hiding their true capabilities during evaluation so that they can behave unsafely during deployment. This motivates the authors to investigate the extent to which an agent's deployment behaviour can be rigorously certified during evaluation.

The authors formalise this problem by associating distributions PP and QQ to an agent's observable information ZZ at evaluation and deployment time respectively. The agent decides whether to exhibit unsafe behaviour by using a gate function φ\varphi which maps ZZ to the probability of conducting unsafe behaviour. The agent then optimises its gate function so that the probability of adopting unsafe behaviour during deployment is maximised under the constraint that the probability of adopting unsafe behaviour during evaluation is bounded.

The authors show that this formalisation maps directly onto hypothesis testing, and exploit this connection to prove formal limitations on the evaluators ability to catch and prevent unsafe behaviour.

Critical strengths

I think the formal connection between hypothesis testing and AI safety evaluation is is clean and presented in a theoretically rigorous manner.

Critical weaknesses

The theoretical contributions seem extremely limited from a novelty standpoint. As admitted by the authors towards the end of Section 1, the only novel contribution is the safety interpretation given to statistical testing. Almost all theoretical results follow immediately from classical theory in a few lines.

Whilst the highlighted connection is clean, I would argue that it doesn't address the fundamental problems associated with AI safety evaluation. If AI safety evaluation reduced to statistical testing in practice then it would already be a solved problem. It feels like the paper is highlighting aspects of safety evaluation that are already obvious rather than providing a contribution that resolves open issues. Of course an agent's ability to circumvent evaluation relies on its ability to able to tell the difference between deployment and evaluation environments, which in turn obviously reduces to a statistical testing problem. AI safety evaluation is a difficult problem because specifying this statistical test and controlling relevant statistical properties is often infeasible.

For example, in practice deployment and evaluation environments are typically too rich for us to accurately characterise the probability distributions P and Q, nor bound the statistical difference between them. Even worse, the evaluator typically has incomplete knowledge of ZZ. For example, it is very difficult to know the information that an agent connected to the internet has access to. In light of these remarks it is unclear how any of the theory established in this paper could be used to help guide the design of safety evaluations in practice. This concern is further reinforced by the fact that the paper does not perform any empirical experiments that illustrate how their theoretical results could be leveraged.

Outside of technical concerns, I found the paper very difficult to parse, even though it is conveying a rather simple point that could be made more elegantly. I have listed a few (but not all) issues below.

  • There are sentences that read very "AI-generated" that do not contribute much to the discussion. For example, line 85 starts with the sentence "The central observation is simple but consequential".
  • The sentinel example is basically just hypothesis testing on Gaussians and is not grounded in real-world deployment scenarios.
  • There are many paragraphs that are just single sentences (eg lines 146 and 152).
  • Why is the optimal gate for sentinel stated as a proposition in its own section instead of being an example directly after line 168?
  • There are weird hanging sentences that add nothing to the discussion (eg line 197).
  • The implications of theoretical results are often poorly explained. For example, on line 184 the authors claim that Theorem 3 is converted into "usable sufficient conditions", but never explains why these sufficient conditions are "usable". Likewise, on line 269 the authors say that "the resulting phenomenon is opposite to statistical evaluations: more observations help the agent identify the experiment..." But this is exactly the case for statistical testing: more data helps distinguish two distributions. In what sense is the "resulting phenomena" "opposite"?
  • Some equations are in boxes and some are not and it is completely unclear as to why.
  • Some papers are cited and discussed that seem irrelevant. For example, either I am misunderstanding something or the connection to causal transportability theory is tenuous at best.
Questions for authors

My main technical concern can be summarised as follows: How do the results here impact or inform the design of safety evaluation schemes in practice? To address this question, the authors would need to address the limitations I have highlighted in "Critical Weaknesses".

Outside of this I have significant concerns over the paper's written presentation. As a reader, I got the feeling that AI assisted heavily with drafting and writing. There is little effort to contextualise results for the reader. I am sceptical that my concerns regarding written presentation can be effectively addressed during the rebuttal period and in my mind it would make much more sense for the authors to work on an improved submission for another venue.

Reviewer_72H3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The paper discusses an interesting problem for when an evaluated agent can use the evaluation procedure as evidence that it is being evaluated. This leads to two different distributions PP and QQ denoting their respective behaviors under deployment and evaluation settings. Authors use this premise to define upper bounds on worst-case deployment risk, and a divergence certificate in Theorem 7, motivating conditions for which how evaluation risk influences deployment risk. Authors also provide analysis for how weak cues can be amplified in long horizon settings, and the risk of asking whether a model can identify an evaluation from a single prompt. Finally, authors provide the audit paradox that shows that intervention may not always be helpful.

Critical strengths

The paper offers comprehensive theoretical coverage of a very interesting and potentially relevant problem. Section 6 and 7 also provide interesting details for when an intervention can be useful and authors also make interesting revelations with discussions pertaining to long horizon accumulation of weak cues. In light of AI evaluation, this paper can serve as a useful theoretical benchmark, and ideas pertaining to long horizon accumulation, intervention impacts can be further utilized in practice. The paper is well written, easy to follow, and leverages fundamental hypothesis testing ideas, relying on minimal assumptions on models of PP and QQ.

Critical weaknesses

A key limitation of this work is lack of implementation on actual AI evaluation, which limits the attractive theoretical contributions. Even if actual implementation betrays the theoretical formulation due to invalid assumptions as stated by the authors, it will be quite useful to understand which parts of theory translate to practice successfully, or benchmark a gap from behavioral worst case results.

This also leads to an issue where certain unsaid assumptions that facilitate the theoretical results remain unchecked for their practical validity, for example, what amount of data is needed to quantify the divergence of PP and QQ in practice, and if it is feasible to collect potentially large amount of data for both, given that deployment data may be limited compared to evaluation data.

Questions for authors
  1. Does the framework support data imbalance in estimation of divergence in PP and QQ for validating Theorem 7?
  2. What is the most immediate bottleneck in practical implementation of this work, even partially, to validate some of the ideas? If feasible, empirical results will significantly strengthen the paper's positioning by establishing the gap between theory and practice concretely.
  3. If the paper is not intended to be empirically validated, how do the authors suggest use of this work in improving evaluation systems besides the preliminary recommendations made in the paper? Can derived principles from the paper be implemented in practice to show the overall benefit of this work?
Reviewer_qZaKOpenReview ↗
Rating
4
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies when behavioral safety evaluations can certify agents that recognize whether they are being tested and adjust their behaviors accordingly to dodge the test. It models safety evaluation as a distribution shift problem, identifies the optimal hiding strategy through hypothesis testing, translates distinguishability into safety certificates, characterizes how weak cues accumulates over time, and proposes the audit paradox, which leads to the conclusion that indistinguishability to the agent being certified is required for the certification.

Critical strengths

The framework proposed in this paper gives a formal characterization of the critical issue of evaluation awareness in agents for safety certification. The theoretical framework proposed in the paper looks technically sound.

Critical weaknesses

None

Questions for authors

It seems like the framework proposed in the paper is not specifically applicable to AI systems, especially LLMs. Given that the paper also does not provide a concrete example of AI system showing the characterized behaviors, it would be helpful for the authors to explain how exactly the framework is applied to AI systems.

Comments on the paper

4 comments
AuthorsOpenReview ↗

What an aggregate audit score certifies, and what changed when we tested it?

Thank you for pressing us on the paper’s contribution and its use in evaluation. One concrete decision captures both issues. When a benchmark combines deployment-like interactions with conspicuous stress tests, what deployment claim can the evaluator attach to its aggregate score?

Consider an audit with reference fraction η and a single pooled risk bound α. If the reference component reproduces the full agent-visible deployment context law, deployment risk is at most α/η, capped at 1. The upper bound follows because the reference component alone contributes η times the deployment risk to the pooled mean. Theorem 15 establishes sharpness: when the other component is recognizable and confined to audit, an admissible gate attains this bound. Even perfect detection and exact knowledge of the pooled risk cannot justify a smaller universal number. Thus, if only one tenth of the audit is deployment-like, a pooled 1% guarantee permits 10% deployment risk. Keeping a separately established bound on the reference stratum preserves that bound when stress tests are added (Theorem 12). Obtaining it still requires adequate reference data.

We tested this reporting decision using the same recorded Claude 3 Opus responses. In Greenblatt et al.’s controlled archive, animal-welfare support occurred on 50.76% of 595 fully graded matched questions under implied training and 65.71% under implied non-monitoring. Sampling 100 responses with replacement from each stratum, a nominal 95% pooled upper bound gave 61.38% exact coverage when interpreted as an upper bound on the unmonitored archive rate. Reporting the unmonitored stratum separately gave 99.42% with the same sampling budget. These figures quantify the consequence of attaching a valid bound to the wrong population. The endpoint is an observable behavior and the target is the finite unmonitored archive. Original study, Appendix A

The fresh controlled LLM pilot tests a different link in the argument. Holding the arithmetic tasks and format choices fixed, an assessment-facing header elicited 0/12 explanation choices in each of two batches. A student-facing header elicited 12/12 in each of two batches. When the audit itself used the student-facing header, two new batches recovered 12/12 in both. Matching task content alone had left the original audit unrepresentative even for an ordinary, benign response choice.

Earlier LLM work already describes policies that condition behavior on relative deployment likelihood (Hubinger et al., 2024, Section 3.1). Our contribution is to fix the evaluation-risk budget and determine the largest deployment risk compatible with it over every admissible gate. Theorem 3 identifies that value using classical Neyman–Pearson theory. Theorem 8 identifies the exact price of asking for a guarantee uniform over all gates: total variation for additive error and the essential supremum of the likelihood ratio for multiplicative error. Theorem 14 identifies when more testing cannot resolve the uncertainty: if every admissible audit misses an event of deployment mass γ, two agents can have identical transcripts under every adaptive audit and still differ by γ in deployment risk.

These results distinguish three decisions with different consequences. Preserve a reference constraint when pooling loses it. Change the visible interface when it reveals the audit. Expand the audit family when every current protocol misses a deployment event. Section 4 and Appendix J connect these decisions to the completed tests. For an open-ended agent, a successful discriminator supplies a lower bound on distinguishability. A positive deployment certificate requires an upper bound justified by the audit design or by validated assumptions about the information the agent can observe.

The conclusion also holds when stress-test and deployment contexts overlap. Let PP be the deployment law, SS any fixed stress-test law, and Qt=(1−t)P+tSQ_t=(1-t)P+tS the pooled audit with stress-test fraction tt. For 0≤s<t≤10\leq s<t\leq1, keep each observation with probability s/ts/t and otherwise replace it with an independent draw from PP. This operation leaves PP unchanged and maps QtQ_t exactly to QsQ_s. Proposition 9 therefore gives ADRP,Qs(α)≤ADRP,Qt(α)ADR_{P,Q_s}(\alpha)\leq ADR_{P,Q_t}(\alpha) at every audit tolerance α\alpha. Theorem 8 also gives the exact worst additive gap: TV(P,Qt)=t TV(P,S)\mathrm{TV}(P,Q_t)=t\,\mathrm{TV}(P,S). Thus, even when the two laws have identical support, doubling the stress-test share doubles the worst additive gap. This explains why adding a separately scored stress test and replacing reference cases inside a pooled score have different certification consequences.

The reviewers asked for a concrete decision that an evaluator would make differently after reading the paper. Suppose an evaluator can score 100 responses and wants to bound how often a behavior occurs when monitoring is not implied. Can the evaluator score responses to precisely the same questions under implied monitoring and carry the resulting 95% upper bound over to the target? We checked this directly on published LLM outputs.

We used the 595 questions with grades in both conditions in the Claude 3 Opus animal-welfare archive of Greenblatt et al. (arXiv:2412.14093, Appendix A). The endpoint was the original study’s binary grade for whether a response expressed any support for animal welfare. In the implied-unmonitored condition, 391 of 595 responses received a positive grade (65.71%). In the implied-training condition, 302 responses to the same questions received a positive grade (50.76%). We take the first of these two finite populations as the target. The following is an additional analysis of the archived grades.

At the same scoring budget, coverage of the target falls from 95.10% to 4.45% when the evaluator changes which responses are scored. Consider drawing 100 distinct question IDs uniformly and computing an exact one-sided 95% upper confidence bound. For every possible set of IDs, we can apply the same procedure to either condition. Using the implied-unmonitored responses, the bound covers the target rate in 95.10% of all possible samples. Using the implied-training responses, it covers that same target in only 4.45%. The questions, grading rule, sample size, and confidence procedure are identical.

The calculation can be verified from the counts alone. If a population of 595 contains K positive grades, the number of positives in a uniform sample of 100 has a hypergeometric distribution. Inverting its lower tail gives an exact 95% upper bound on K. That bound includes the target count of 391 exactly when at least 59 sampled grades are positive. The probability of observing at least 59 is 0.95098 when the sampled population contains 391 positives and 0.04450 when it contains 302. These probabilities account for every possible sample. They require no assumption that the model generated its responses independently.

Even scoring every monitored response would leave the discrepancy intact. At a full census, the evaluator knows the monitored rate exactly. Its upper bound is then 302/595, which falls below the target rate of 391/595 by 14.96 percentage points. Additional monitored observations can eliminate uncertainty about the monitored rate. A claim about the target rate still requires a reason the result transfers.

This is where the paper changes evaluation practice. If the claim concerns an identifiable stream of completed interactions, the evaluator can select interactions from that stream uniformly for scoring after the outputs are fixed. The sampling design itself then justifies a bound for that stream, without requiring the evaluator to estimate a divergence over LLM transcripts. If the claim concerns a different or future population, Theorem 3 characterizes the largest target risk compatible with a given audit-risk budget under the common-gate assumption. Theorem 14 identifies a limit that persists even with adaptive testing when all available audits miss the same target event. The archive calculation gives those statements a concrete interpretation: before increasing the audit budget, establish which population a more precise audit would actually certify.

AuthorsOpenReview ↗

Revised Manuscript and Appendix uploaded

We thank all reviewers for their feedback and apologize for the delay in making the revision available. We have marked all substantive changes in blue. In response to Reviewer qZaK’s request for a concrete application to language models, Appendix J now reports a controlled LLM header pilot and an analysis of 595 matched Claude 3 Opus response pairs. In response to Reviewer KZLq’s questions about the contribution beyond classical hypothesis testing and what an evaluator can establish with incomplete access to the agent’s context, Sections 2–4 and 9 clarify the assumptions behind each guarantee and explain how retaining a separately scored reference group can tighten a certificate. In response to Reviewer 72H3’s questions about data requirements, practical utility, and the gap between a worst-case bound and actual behavior, Appendix I.10 treats unequal sample sizes. Appendices J.4, J.6, and K show that a frozen controller falls exactly 5/12 below its theoretical maximum on a fully enumerated panel, identify where that gap comes from, and demonstrate how retaining the Claude reference labels and using an exact binomial procedure reduces the mean 95% upper bound from approximately 100% to 73.54% for an archive target of 65.71%, at the same sampling budget. Section 9 (p. 9) summarizes these findings, and the calculations most relevant to the discussion appear on pp. 35–37.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.