acceptodds
Under review as a conference paper at ICLR 2027

Limits of Statistical Assurance for Embodied AI under Adaptive Adversaries

Abstract

Embodied AI increasingly places language models, vision-language-action models, and learned controllers in physical systems, and their deployment rests on finite statistical assessment, presuming that inspected behavior persists once inspection ends. We show that a statistically valid certificate can become an attack signal for an adversary with continuing access. An adversary that preserves the unattacked system's joint observation distribution until observable, irrevocable certification and attacks afterward attains expected post-certification events over trials, where is the event probability under attack and the unattacked certification time. The bound holds for adaptive and privately randomized evaluators, regardless of policy architecture, and depends on the full certification-time distribution, so equal pass rates can conceal different exposure. For the fixed likelihood-ratio test, a pathwise coupling proves switching at certification exactly optimal under memoryless interventions that raise event probability. Requiring a fixed positive certification probability for unattacked systems forces linear worst-case expected excess over unmonitored deployment, even when certification is deferred to the assessment deadline. In simulated RoCo multi-robot collaboration, a switching controller records commanded separation at or below a calibrated threshold in 34 of 36 post-certification episodes. In preregistered randomized AgentDojo trials, attacks initiated after certification cause 1.98 additional security violations per four tasks (95% interval ). A language-model controller infers whether certification has occurred from noisy signals of evaluation status. Informative signals raise observed verified violations from 1.41 to 2.07 per trajectory, a 47% increase over uninformative signals. Fresh preregistered trials independently confirm the increase, yielding 0.93 additional violations per trajectory with no missing deployment outcomes. With certificate revocation, a mixture likelihood-ratio test bounds cumulative excess violations relative to a prespecified reference rate at square-root order, up to logarithmic factors, for every adaptive adversary. This bound holds on every sample path. Monitoring with a single fixed alternative can permit linear expected excess even under stationary attacks. Against the same controller given the monitor's exact rule and current statistic, live mixture monitoring reduces verified violations from 12.0 to 3.8 per 24-task deployment, a 68% reduction.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, adversarial machine learning, sequential testing, embodied AI, evaluation gaming, adaptive adversaries

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_A5GaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the limitations of statistical safety certification for embodied AI systems operating in environments with adaptive adversaries. The authors point out that an attacker can remain inactive during the evaluation phase and trigger a system failure only after certification has occurred. The study derives a lower bound for the risk of post-certification exposure and proves the optimality of certification-triggered switching under a memoryless intervention model. Furthermore, the authors explore this behavior through experiments involving RoCo multi-robot collaboration and AgentDojo tool-use agents.

Critical strengths
  1. The authors provide a detailed theoretical analysis and proof.
  2. The author provides a detailed experimental protocol and acknowledges some limitations.
Critical weaknesses
  1. The practical significance of the theoretical results is constrained by strong assumptions.
  2. The experiment did not test the theory's distinctive predictions, and the certificates used were also very weak.
  3. The main results (Theorem 1, Corollary 1) follow almost directly from the assumption that an attacker can observe certificates and perfectly mimic a clean transcript; the proof essentially relies on a coupling argument combined with summation. Whether the conclusion is meaningful depends fundamentally on the validity of this assumption in a real-world setting, yet the paper provides insufficient justification for it.
Questions for authors
  1. Is it possible to verify the premises of Theorem 1—such as the learned conditional behavior—in a setting where the certificate is not directly observable by the attacker?
  2. How should the finding that there is no significant difference in relative fixed latency for informative receipts be understood and explained?
  3. How does certificate revocation affect task completion, system availability, and operational safety, especially when execution cannot be immediately terminated?
Reviewer_MZHwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether pre-deployment testing can provide reliable safety assurance when an attacker can adapt to the evaluation process. The key setup is that the system is tested for some time, then approved for deployment, and that approval is not revoked based on later behavior. Under this setup, an attacker can simply behave normally during testing, wait until the system is approved, and attack afterward. The paper formalizes how much failure such an attacker can cause under this deployment model.

Experiments on RoCo multi-robot collaboration and AgentDojo show this behavior in practice using several attack strategies, including manually triggered attacks and LLM-based controllers that decide when to activate attacks based on the evaluation history. The results show that delaying attacks until after approval can substantially increase post-deployment failures.

Critical strengths

I appreciate the author's contribution to the theory side, but with the current presentation, it's hard to identify critical strengths that can support acceptance. Additionally, I have several major concerns related to the current problem formulation and finding significance (discussed below).

Critical weaknesses

1. Serious writing and readability issues.

The paper suffers from substantial presentation and readability problems. Many key terms and assumptions are used extensively in the abstract and introduction before they are clearly defined, making it unnecessarily difficult to understand the basic problem setting and technical contribution.

  • "certification": this can refer to certification in "certified robustness", but in the paper, it refers to the time the evaluator thinks it's valid to deploy the system. The term is used extensively in the abstract and introduction before its formal definition in Section 2, and its intended meaning is not made sufficiently clear beforehand.
  • Similarly, terms such as "unattacked observation law" and "certification-time distribution" are largely unexplained when they are first introduced, and I don't believe these are well-established terms that the community can understand without clarification.
  • The threat model, which is the most critical for security-related research, is not clearly explained in the introduction: what's the assumed attacker capability, victim system, etc., until Section 2. Therefore, it's extremely difficult to follow the abstract and introduction without knowing the problem formulation.

I have read the same paragraph or sentence multiple times but still find it hard to follow. Therefore, the current writing seriously prevents conveying the core technical contribution, as I believe it also does for the general ICLR audience.

2. Limited informativeness due to strong and arguably unrealistic assumptions.

I found the core technical message of the paper relatively straightforward compared with the amount of theory it introduced. The main result essentially states that "if an attacker can behave identically to the clean system during evaluation, observe when an irrevocable certification decision has been made, and still retain an effective attack capability afterward, then finite pre-deployment evaluation cannot guarantee post-certification robustness."

  • I am not convinced that the assumed setting is sufficiently representative of real-world embodied AI systems. The main result relies on the assumption that "certification can not be revoked once the decision is made". However, real safety-critical systems can employ runtime monitoring, fail-safe mechanisms, or revocation (e.g., autonomous vehicles, robotics, agents, etc.). In fact, the paper itself later shows that introducing continued monitoring and revocation substantially changes the guarantee. Therefore, the claimed "limitation of statistical assurance" seems to apply mainly to one-shot, irrevocable certification rather than in general (or practice).

  • The threat model treats evaluation as passive and takes the attack’s effectiveness as given; The evaluator is modeled as observing the system as a black box and estimating an event rate. Theorem 1 then assumes that a fully inactive attacker can afterward exercise strong system privileges. In the principal RoCo experiments (interfaces F1 and F2 in Table 1), these privileges include replacing the planner’s state information and modifying its feedback and approval state. However, in practical security analysis, the attacker's capability and exposed attack surface are generally constrained. In embodied systems, an adversary typically has access to only restricted inputs, communication links (e.g., packet injection), or physical perturbations (e.g., adversarial patch). A competent evaluation actively red-teams these exposed interfaces and thereby bounds the achievable attack effect, which corresponds to v in Equation (1). The paper doesn't consider this by placing v in the formulation as an assumed quantity rather than deriving it from a constrained attack surface.

Overall, I learned relatively little from the central theoretical observation itself: "behavior observed during evaluation cannot constrain an attacker that is strong enough to (1) know the certification time and (2) direct modification access to security-critical planner interfaces.". Under the paper's assumption, the conclusion is largely expected but wrapped in unnecessarily dense theory.

I would have found the work more informative if it characterized when realistic, bounded attack surfaces still lead to a non-trivial impossibility result, rather than assuming the post-certification attack effectiveness as part of the formulation (e.g., v in Equation 1).

Questions for authors

Please see the suggestion above.

Reviewer_tEhEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the limits of finite statistical assessment when an adversary retains intervention access after certification. An adversary that reproduces the clean observation law until observable, irrevocable certification and attacks afterward achieves an expected post-certification event count of vE[(n−S0)+]v\mathbb{E}[(n-S_0)_+]. The paper establishes exact optimality for a fixed likelihood-ratio certification rule under memoryless, risk-increasing interventions and derives a deployment excess lower bound under a clean-certification requirement. Experiments in RoCo and AgentDojo demonstrate certification-dependent interventions, inference of certification from noisy signals, and reductions in violations through revocable monitoring. A mixture likelihood-ratio monitor additionally provides a pathwise cumulative-excess guarantee against adaptive adversaries.

Critical strengths
  1. The paper precisely distinguishes statistical validity from deployment assurance. Theorem 1 explains how a certificate can retain its statistical guarantee while permitting substantial subsequent exposure. Theorem 2 provides a concise coupling argument establishing exact switching optimality within the specified intervention model. These results clarify the roles of observability, continued attacker access, and irrevocable certification. The dependence on the full clean certification-time distribution, rather than terminal certification probability alone, is a useful quantitative insight.
  2. The randomized continuation experiment provides meaningful causal evidence, and the reporting is transparent. Appendix D.11 compares intervention and inactive continuations from shared assessment histories, matches subsequent tasks, randomizes branch assignment and execution order, and independently checks final-state outcomes. Even conservatively treating the missing control outcome as a violation yields 1.98 additional violations per four-task continuation, with a 95% interval of [1.67,2.29][1.67,2.29]. The explicit reporting of failed predictions, sensitivity analyses, and protocol corrections also helps readers assess the strength and scope of the empirical evidence.
Critical weaknesses
  1. The monitoring comparison does not give the baseline the same opportunity to depend on the deployment horizon. Proposition 1 uses a mixture whose grid depends on HH, whereas its linear-excess result concerns a single alternative that remains fixed as HH increases. A simple horizon-tuned monitor already provides a pathwise sublinear guarantee under the same clean conditional-mean assumption. Specifically, consider $ E_t = \exp ! { \lambda (K_t-p_0 t)- \lambda^2 t/8 }$ Under E[Yt∣Ft−1]≤p0\mathbb{E}[Y_t\mid\mathcal F_{t-1}]\le p_0, this is a nonnegative supermartingale, so stopping at Et≥1/αE_t\ge1/\alpha controls false alarms at level α\alpha. For T=τ∧HT=\tau\wedge H, every outcome path satisfies $ K_T-p_0T \le \frac{\log(1/\alpha)}{\lambda}+\frac{\lambda H}{8}+1. $ Taking λ=8log⁡(1/α)/H\lambda=\sqrt{8\log(1/\alpha)/H} gives $ K_T-p_0T\le\sqrt{\frac H2\log(1/\alpha)}+1. $ This construction requires neither a mixture nor the additional log⁡H\log H factor in the stated bound. It follows from standard exponential-supermartingale techniques; see Howard et al. (2021). This does not contradict Proposition 1(iii), but it limits what that proposition establishes about the advantage of mixing. The comparison should include a horizon-tuned single-alternative monitor and clarify what additional benefit the mixture provides.

  2. The empirical results establish controlled mechanisms more convincingly than broader deployment relevance. The strongest RoCo result measures commanded separation under feedback-modification access, rather than physical collision or harm. In the noisy-receipt experiment, the controller receives the receipt law and certification-time prior. Informative receipts outperform fair coins, but the difference from the fixed-delay baseline is 0.30 violations per trajectory, with a 95% interval of [−0.18,0.77][-0.18,0.77]. The pooled adverse completion of missing outcomes also has an interval containing zero. The fresh 46-triplet experiment provides positive evidence against the fair-coin control, but the demonstrated scope remains specific to the supplied observation model. Furthermore, the reported 68% reduction in violations includes reduced exposure from permanently terminating execution: the mixture monitor revokes 45 of 48 trajectories. Appendix D.14 does not pair this result with legitimate-task completion outcomes. The authors acknowledge several of these limitations, but they still constrain the practical significance of the findings.

Questions for authors
  1. How does the proposed monitor compare with the horizon-tuned exponential process above and with a horizon-tuned single-alternative likelihood-ratio monitor under the same false-alarm constraint and attacker objective? What additional advantage does mixing provide?
  2. Should the noisy-receipt results be interpreted as demonstrating that a controller can exploit a supplied observation model, or as establishing a broader advantage over simple timing policies? Please clarify the intended claim in light of the fixed-delay comparison and missing-outcome sensitivity analysis.
  3. What are the executed-task counts and legitimate-task success outcomes for the live monitoring arms? How is p0=0.05p_0=0.05 justified operationally? These quantities would help assess the protection achieved alongside the cost of revocation.

Comments on the paper

1 comment
Reviewer_MZHwOpenReview ↗

Reviewer's response

Thanks for the response. The clarifications on the AgentDojo attack surface, the noisy-receipt experiment, and the task-utility numbers are helpful, but my main concerns remain.

  1. The response agrees that the problem only exists when certification cannot be revoked, but the title and abstract still read as a general limit of statistical assurance.

  2. If red-teaming already shows the attack works (v > p_clean), then deploying the system is a choice to accept that risk, not a failure of the statistical test. In that case, Corollary 1 just multiplies the per-step gap by the number of unchecked steps, which follows directly from the assumptions.

  3. The LLM controllers are told their goal and told that episodes after certification are not checked (L134, L363–366), so waiting until after certification to attack follows from the prompt and is not something the model figured out by itself.

Given the above reason, I will keep my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.