acceptodds
Under review as a conference paper at ICLR 2027

What Zero Detections Certify: Statistical Safety Cases for Model-Aware Monitoring

Abstract

Monitoring is now one of the main safeguards for deployed AI systems. A monitor reviews only a subsample of traffic, and zero or few detections are often treated as evidence that harmful behavior is rare. That reasoning is not always valid: it assumes that the monitored system cannot use features of the context to infer where oversight is weak. We study monitoring-based safety cases when this assumption fails. Under invisible per-unit randomization and audited lower bounds on recall, we give finite-sample upper confidence bounds on the number of harmful actions. The bounds make no stochastic assumption on the system's behavior, but they depend on a sensitivity parameter that limits how much harmful behavior can concentrate in weakly monitored traffic. In a simple stratified design, a 95% bound of about three harmful actions under proportional allocation becomes 244 when harms may be twice over-represented in weakly monitored traffic. Using the same budget to equalize effective coverage instead gives a bound of about eight that is robust to . When the system only partially observes the monitoring policy, the relevant quantity is not average coverage, AUC, or guessing accuracy. It is the extreme lower tail of the system's posterior detection probability. We show that standard discrimination summaries cannot control this tail. Subsampled monitoring also cannot rule out a single catastrophic action unless per-unit detection is nearly complete, but it can bound the chance that a multi-step campaign escapes interruption. These results lead to a monitoring design principle: make knowledge of the policy strategically unhelpful, by equalizing effective coverage or adding a hidden randomized coverage floor. We turn the theory into a reporting protocol and illustrate it on public descriptions of monitoring systems from Anthropic and OpenAI. On three public attack benchmarks scored by seventeen open-weight monitor configurations, the recall of a single monitor ranges from to across attack strata, and the certificate is set by the strata where recall is lowest.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, monitoring, finite-sample confidence bounds, sensitivity analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jTHJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines what AI monitoring can actually prove when the monitored system can predict where oversight is weak. The authors show that a clean monitoring record can be far less reassuring than it appears if harmful behavior concentrates in rarely checked situations, and they provide statistical bounds that account for this. The paper suggests that monitoring should be designed so that each situation has an equal and unpredictable chance of being checked. Experiments with open-source monitors on several public attack benchmarks show large variation in detection rates across attack types.

Critical strengths
  1. The paper is well-written, and it addresses an important problem.
  2. The paper identifies why zero detections can provide misleading reassurance when harmful activity is concentrated in contexts with weak oversight and makes the relevant strategic assumptions explicit.
  3. The certificate places no probabilistic model on the system’s behavior and relies on the monitor’s randomization and parameter \Gamma. The exact Poisson constant is shown to be tight (Lemma 1c), and its validity is verified on many configurations (Figure 4a).
  4. The paper provides constructive design guidance and quantifies its cost. When \Gamma is above the threshold, equal coverage is minimax-optimal, and a hidden coverage floor results in \Gamma-free certificates.
  5. Experiments across many monitor configurations and several benchmarks show substantial recall differences among attack strata, supporting the framework's practical relevance.
  6. The paper demonstrates that bulk discrimination summaries such as AUC or guessing accuracy are unable to place a bound on the relevant risk, this being due to the fact that the risk depends on an extreme lower tail, and it provides an example of this through a tail stress test carried out on real monitors.
Critical weaknesses
  1. Footnote 1 says the \Gamma limit can rule out every option when the number of harmful actions is small. With the Section 7 shares and \Gamma = 1, there is no allowed way to place 3 harmful actions. But the abstract’s main number is “about three.” The paper mentions this only in a footnote.
  2. In the experiments, each attack type is treated as a category with an equal share. But Proposition 2 says the AI can pick any attack type, so the worst type can be used. It is not clear how these fit together.
  3. The paper uses well-known mathematical tools for AI (Poisson limits, Hoeffding, CVaR, linear programming, etc.) to solve the problem.
  4. The paper does not talk about how to choose \Gamma for an existing system.
  5. The paper addresses the concern that an AI notices where monitoring is weak and puts its harmful actions there. Can the authors run a small experiment that tests this (if possible, not mandatory)? For instance, in BashArena, give an agent a task together with a monitoring policy that includes obvious signals and some setups or tools that are checked frequently, while others are checked only rarely. Then check whether the agent tends to carry out its attacks on the setups that are checked only rarely. This would show whether \Gamma is meaningfully greater than 1.
  6. The bounds treat every harmful action the same, whether it is small or serious. Also, Corollary 2 shows that if only some actions are checked, one very harmful action can always slip through. That single very harmful action is often the case people worry about most.
Questions for authors

Q1: For an existing system that cannot be redesigned right away, how should a safety team choose which \Gamma to report? Is there any kind of evidence that can show that \Gamma is small? Q2 Which results do the authors see as new, rather than direct applications of known tools from survey statistics, sensitivity analysis, or disease surveillance? Q3 The main monitor was chosen using pilot data that overlaps the test data. Can the authors repeat the main results with a monitor chosen on separate data, or report certificates for all ten default-prompt monitors? Q4 The method requires that the AI should not be able to determine whether a particular action has been sampled (Assumption 1). In practice, sampling can cause visible side effects, such as delays or interventions. How can a team check that this assumption holds, and how much does the bound change if it partly fails? Q5: When the limit is the number of false alarms or the number of human reviews rather than the fraction of traffic that is monitored, how do designs that give equal coverage and have hidden floors perform?

Reviewer_nqgdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Monitoring LLMs for harmful behavior is important for ensuring safe deployment into sensitive areas. At scale, human monitoring is insufficient since the amount of text generated by an LLM while producing a solution to a problem is enormous. Therefore, we must rely on automated systems to monitor and judge the safety of the actions being taken by an LLM. The careful design of such automated systems becomes critical in the event that the LLM being monitored has the capacity to be aware that it is being monitored. A problem with current safety reporting is observed monitoring records may have few or zero harmful actions stopped. This limits the ability with which we can judge how effective the monitoring system in place works to stop harmful action. This paper seeks to fill that gap by establishing upper bounds for harmful activity under few or zero observed harmful actions. The process for monitoring an LLM can be broken down into a two-stage process: the first stage is routing. This determines whether a subset of the LLM's generated text is considered for harmfulness by a monitor. The second stage is then the assessment of whether or not harm is detected within the subset of the LLM's generation. This creates a survey sampling setup, meaning a report which shows zero or few harmful actions can be inaccurate because harmful actions were not routed. The authors provide an analysis of the number of harmful actions that can go undetected under different assumptions about how the LLM being monitored allocates its harmful actions. If an LLM can identify 'weak-spots' in the monitoring system, it can produce harmful actions at a far greater rate than uniform allocation presumes. The authors suggest monitoring systems which do not have such weak spots, to avoid harmful actions to go undetected from LLMs which learn to allocate those actions to rarely monitored generations.

Critical strengths

The author seems to be familiar with the relevant literature and is seeking to address a critical problem in AI safety. This paper tries to address a critical gap in monitoring AI models under zero harmful detections and provides tight analytical bounds for the amount of harmful action that could have taken place under such circumstances. The main result of the paper is intuitive and suggests current state of the art monitoring reports are overly generous and may be allowing more harmful actions than previously thought.

Critical weaknesses

The paper is overall difficult to parse. Many variables are introduced and not defined. Many variables are defined, but not used until many paragraphs or pages later. There are many key words or phrases introduced which are not clearly defined either (e.g. 'system', 'units', 'traffic', 'traffic shares', 'deployment period', 'campaign', 'certificate', 'coverage scale', 'information structures', 'blind spot mass', 'hazard scale'). The introduction fails to motivate the problem and does not clearly define the gap the authors are filling. Together, these weaknesses make it nearly impossible for a non-expert to follow along with the paper. Second, no conclusions are made from the experimental results, so it unclear what questions the experiments are answering. Results are merely reported, without much detail, and the reader is left wondering what they mean. The discussion does nothing to indicate the meaning of the results. There is not a clear takeaway that is supported by evidence presented in the paper. There is an overuse of nonrestrictive clauses throughout the text which make many of the sentences confusing and hard to read.

Questions for authors

Appendix B seems to introduce the problem and illustrate the gap more clearly than anything in the main text. Why were some the points made there not in the main text? The AUC's for figure 3b are confusing because they are higher for the lower curves. Are these AUC values the AUC of the curves shown in that plot? If so, the y-axis seems to be reversed. What do the results of the experiments show? What does the high AUC for the probe indicate? How does Figure 3a support/demonstrate Theorem 3? You write, "The numbers attached to the Anthropic and OpenAI architectures are conditional illustrations built from public documents with assumed traffic shares. They are not audits of either company’s systems." What does "conditional illustrations" mean? Can you provide a demonstration of an LLM allocating its harmful actions to less-monitored units in practice? This would be substantial evidence that the claims in the paper address a real threat with current models. Without that, all the paper has is assumptions about the distribution of harmful actions with no experimental evidence demonstrating a strategic system attempting to elude a monitor. For the OpenAI and Anthropic section, where do the assumed weights come from? Are those reasonable choices?

Reviewer_wNcqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper analyzes the efficacy of monitoring-based safety cases for AI deployments in the regime where the monitored system can predict and exploit the monitor. It uses a simple model with finite strata, each with its own monitor recalls and routing rates, and a parameter Γ\Gamma that bounds the factor by which the system is able to overrepresent any stratum. It then provides some empirical results with public monitoring benchmarks and applies its framework to public descriptions of high-stakes frontier monitoring systems.

Contributions:

  • The paper provides a certificate, i.e. an upper confidence bound, on the number of undetected harmful actions given the number of detections (including zero).
  • The paper shows that the certificate depends on the tail of the posterior detection probability distribution, and shows that many summary statistics (including AUC and guessing accuracy) can't bound that tail.
  • The paper shows that setting effective coverage to be equal across strata is minimax optimal (assuming sufficiently high Γ\Gamma). It proposes a hidden randomized coverage floor across strata, which gives a certificate that holds for any Γ\Gamma.
  • The paper shows that although subsampled monitoring can't rule out a single catastrophic action, it can bound the probability that a kk-step campaign succeeds uninterrupted.
  • The paper applies its methodology to an empirical setting with 17 open-weight monitors using APPS backdoors, BashArena, and SLEIGHT-Bench.
  • The paper also applies its methodology to public descriptions of Anthropic's and OpenAI's internal monitoring systems.
Critical strengths
  1. The paper develops a statistical framework for what currently deployed AI monitoring systems can certify. It shows theoretically and empirically that commonly cited summary statistics (average coverage, and AUC or guessing accuracy for monitoring awareness) are insufficient to bound harmful actions. This matters because these are load-bearing parts of current frontier AI safety cases.

  2. The paper demonstrates the importance of accounting for monitor awareness in the monitored system. It shows how sharply certificates degrade when the monitored system can concentrate harm in weakly monitored traffic, and gives an actionable design fix (equal coverage or a hidden randomized floor) that helps mitigate this.

Critical weaknesses
  1. The certificate assumes monitor success is independent across units (Assumption 1). This is far from holding in practice: detection depends on the action's content, which the system controls, and monitor failures are often highly correlated, e.g. across actions that use similar attack pathways, or via particular idiosyncratic blind spots. This means that P(zero detections)P(\text{zero detections}) no longer decays geometrically in the number of harmful actions, which is the basis for every certificate in the paper. The hidden floor does not address this, since it only randomizes routing. Section 5's requirement that attack types be fine enough that the system can't pick hard instances within a type likely can't be met in practice. Monitor randomization (i.e. Proposition 3) helps only to the extent that the monitors have uncorrelated blind spots.

  2. Γ\Gamma is unmeasurable in practice, and for the pessimal case the paper focuses on, there's little reason to expect it to be small. So in practice the only defensible certificates are the Γ=∞\Gamma = \infty case. The finite-Γ\Gamma results can't be used in practice as an actual bound without justifying a specific Γ\Gamma, and as the paper notes (Section D), experiments can only refute a small Γ\Gamma.

Despite these weaknesses, I think that the paper provides a meaningful first step towards improving monitoring-based safety cases.

Questions for authors
  1. Assumption 1 requires monitor success to be independent across units, but this likely won't hold in practice. How should the certificate be modified when monitor errors are correlated across units? What recall guarantee would then enter the certificate?

  2. Are there settings where high Γ\Gamma could be ruled out, and what kinds of evidence could upper-bound it? If none, should the paper's practical recommendations be stated in terms of the Γ=∞\Gamma = \infty certificate alone?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.