1. Serious writing and readability issues.
The paper suffers from substantial presentation and readability problems. Many key terms and assumptions are used extensively in the abstract and introduction before they are clearly defined, making it unnecessarily difficult to understand the basic problem setting and technical contribution.
- "certification": this can refer to certification in "certified robustness", but in the paper, it refers to the time the evaluator thinks it's valid to deploy the system. The term is used extensively in the abstract and introduction before its formal definition in Section 2, and its intended meaning is not made sufficiently clear beforehand.
- Similarly, terms such as "unattacked observation law" and "certification-time distribution" are largely unexplained when they are first introduced, and I don't believe these are well-established terms that the community can understand without clarification.
- The threat model, which is the most critical for security-related research, is not clearly explained in the introduction: what's the assumed attacker capability, victim system, etc., until Section 2. Therefore, it's extremely difficult to follow the abstract and introduction without knowing the problem formulation.
I have read the same paragraph or sentence multiple times but still find it hard to follow. Therefore, the current writing seriously prevents conveying the core technical contribution, as I believe it also does for the general ICLR audience.
2. Limited informativeness due to strong and arguably unrealistic assumptions.
I found the core technical message of the paper relatively straightforward compared with the amount of theory it introduced. The main result essentially states that "if an attacker can behave identically to the clean system during evaluation, observe when an irrevocable certification decision has been made, and still retain an effective attack capability afterward, then finite pre-deployment evaluation cannot guarantee post-certification robustness."
-
I am not convinced that the assumed setting is sufficiently representative of real-world embodied AI systems. The main result relies on the assumption that "certification can not be revoked once the decision is made". However, real safety-critical systems can employ runtime monitoring, fail-safe mechanisms, or revocation (e.g., autonomous vehicles, robotics, agents, etc.). In fact, the paper itself later shows that introducing continued monitoring and revocation substantially changes the guarantee. Therefore, the claimed "limitation of statistical assurance" seems to apply mainly to one-shot, irrevocable certification rather than in general (or practice).
-
The threat model treats evaluation as passive and takes the attack’s effectiveness as given; The evaluator is modeled as observing the system as a black box and estimating an event rate. Theorem 1 then assumes that a fully inactive attacker can afterward exercise strong system privileges. In the principal RoCo experiments (interfaces F1 and F2 in Table 1), these privileges include replacing the planner’s state information and modifying its feedback and approval state. However, in practical security analysis, the attacker's capability and exposed attack surface are generally constrained. In embodied systems, an adversary typically has access to only restricted inputs, communication links (e.g., packet injection), or physical perturbations (e.g., adversarial patch). A competent evaluation actively red-teams these exposed interfaces and thereby bounds the achievable attack effect, which corresponds to v in Equation (1). The paper doesn't consider this by placing v in the formulation as an assumed quantity rather than deriving it from a constrained attack surface.
Overall, I learned relatively little from the central theoretical observation itself: "behavior observed during evaluation cannot constrain an attacker that is strong enough to (1) know the certification time and (2) direct modification access to security-critical planner interfaces.". Under the paper's assumption, the conclusion is largely expected but wrapped in unnecessarily dense theory.
I would have found the work more informative if it characterized when realistic, bounded attack surfaces still lead to a non-trivial impossibility result, rather than assuming the post-certification attack effectiveness as part of the formulation (e.g., v in Equation 1).
Reviewer's response
Thanks for the response. The clarifications on the AgentDojo attack surface, the noisy-receipt experiment, and the task-utility numbers are helpful, but my main concerns remain.
The response agrees that the problem only exists when certification cannot be revoked, but the title and abstract still read as a general limit of statistical assurance.
If red-teaming already shows the attack works (v > p_clean), then deploying the system is a choice to accept that risk, not a failure of the statistical test. In that case, Corollary 1 just multiplies the per-step gap by the number of unchecked steps, which follows directly from the assumptions.
The LLM controllers are told their goal and told that episodes after certification are not checked (L134, L363–366), so waiting until after certification to attack follows from the prompt and is not something the model figured out by itself.
Given the above reason, I will keep my current score.