acceptodds
Withdrawn from ICLR 2027

Adversarial Attacks on World Action Models via Manipulating Imagination–Action Causal Consistency

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World-Action Models (WAMs) guide action generation by predicting future observations, requiring causal consistency between imagined futures and the consequences of generated actions. We reveal imagination–action causal consistency as a new attack surface in WAMs, where attackers can manipulate the causal relationship between imagination and action to achieve adversarial objectives. However, directly manipulating this consistency is challenging because its evaluation requires environment rollouts outside the model's computation graph, making the causal objective non-differentiable with respect to the perturbation. To address this challenge, we propose Causal Attack via Gradient Geometry (CAGE), a gradient-geometry-based attack framework that manipulates causal consistency through differentiable surrogates of imagination and action. Specifically, we design two tailored gradient strategies: orthogonal gradient decomposition isolates the optimization directions of the two objectives to disrupt causal consistency, while equiangular descent coordinates their optimization toward a causally consistent adversarial target to hijack imagination and action. We evaluate our attacks on LingBot-VA across LIBERO, RoboTwin, and real-world settings. Future Decoupling and World–Action Hijacking reduce the average task success rate from 96.0% to 15.2% and 28.6%, respectively. These results demonstrate that manipulating imagination–action causal consistency enables effective and flexible attacks on WAMs, revealing a new attack surface in prediction-driven robot control.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
World-Action Models, Adversarial Attack, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tvRyOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper provides a method to attack world-action models via constructing a perturbation targeting the geometric relationship between the action- and state-relevant directions in the world model's representation. The approach, CAGE, constructs proxy objectives to provide a differentiable target for projected gradient descent attacks on the model. The paper conducts experiments applying CAGE attacks to simulated and real-world robots.

Critical strengths
  • Empirically, the paper is strong. The experiments on both simulated and real world robotics applications demonstrate the value and applicability of the method clearly.
  • The paper also diligently provides suggested defenses against the proposed attack with some analysis.
Critical weaknesses
  • The justification for critical parts of the method are not clearly articulated, making it difficult to understand why the proposed method is designed as such, and how it differs from prior work. For example, it is unclear why constraining the causal discrepancy in Equations 6 and 7 is necessary, and what it is meant to achieve. It is also unclear why the environment rollouts affect the differentiability of the attack; so long as the environment dynamics distribution is not changing and are not themselves dependent on the policy, this should remain tractable.
  • Prior work is not appropriately engaged with; the related works section reads like a list of papers with no verbiage distinguishing the proposed work from prior work.
  • Additionally, the methodological section is imprecise with missing definitions and inconsistent notation. For example, Obs() and Rollout() are not defined or introduced. Additionally, OO and VV both refer to observations, while VV additionally refers to representations.
Questions for authors

Could the authors expand on why the environment rollout affects differentiability, for instance what inputs to the environment transition function are that are not differentiable w.r.t the model?

Reviewer_LJCCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CAGE, an adversarial attack framework on World-Action Models (WAMs) that aims to manipulate the consistency between the imagined future and the generated actions. Since this consistency is not differentiable, the method uses the distances on the predicted future and the action as surrogates. It changes one branch while keeping the other for disruption, and pushes both branches toward a target pair for hijacking, using a semi-transparent universal watermark as the perturbation. Experiments on LingBot-VA in simulation and on a real robot show that the attacks reduce the average success rate from 96.0% to 15.2–37.0%.

Critical strengths

The paper looks at the security of world-action models from a different angle. Instead of only attacking the action output, it considers the relation between the imagined future and the generated action, and defines two types of manipulation, disruption and hijacking. The paper is well organized, the problem formulation is clearly written with formulas, and the appendix gives the derivation of the two update directions.

The paper also tests the attack with a printed patch on a real dual-arm robot, not only in simulation, which makes the threat more concrete.

Critical weaknesses

My main concerns are about whether the results support the claimed mechanism and whether the experimental settings are clear:

The core motivation is not directly supported by experiments. Hijacking is designed to keep the imagination and the action consistent (Eq. 7), so its D̂cau is expected to be small. However, it has the highest average D̂cau among the three attacks (37.6 vs. 19.1 for “Future Decoupling”, Table 1). This suggests that the success rate drop may come from strongly perturbing both branches rather than from manipulating their consistency. Could the authors explain this result and how D̂cau relates to task failure?

The setting of the hijacking target is not described. The target action chunk A⋆ only appears in Eq. 6 and Eq. 12, and the paper does not explain how it is chosen, for example a random action, a fixed offset, or a specific wrong behavior. The difficulty of hijacking depends heavily on this choice, since a target close to the clean action is much easier to reach than one that needs a very different behavior. In addition, only the success rate drop is reported, so it is not clear whether the robot actually follows the target behavior. Could the authors describe how A⋆ is chosen for each task and report how often the target behavior is achieved?

The three attack modes do not seem to be selective as designed. Since “Future Decoupling” and “Action Decoupling” aim to change opposite branches, the relation between the imagination deviation and the action deviation is expected to flip between them. However, the imagination deviation is larger in both modes (100.1 vs. 58.1 and 73.7 vs. 54.7, Table 1), which suggests that the designed attack modes do not achieve their intended effect. Could the authors explain this?

The experiments are not sufficient. First, the attacks are only compared with random and Gaussian noise, which are expected to be much weaker than any optimized patch. There is no comparison with existing patch attacks on robot policies, e.g., Wang et al. (2025a). Second, the evaluation only covers one white-box model, LingBot-VA, although the paper presents this as a new attack surface of WAMs in general. It would be better to add such comparisons and evaluate more WAMs.

Questions for authors

I would like the authors to address the following points about the claimed mechanism and the experimental settings:

Hijacking is designed to keep the imagination and the action consistent (Eq. 7), but it has the highest average D̂cau in Table 1. How do the authors explain this result? What value of ε_C is used, and how is this constraint enforced during optimization? Please also clarify how D̂cau is computed, e.g., whether D_clean and D_attack come from two independent closed-loop rollouts, and show how D̂cau relates to task failure at the episode level. Please describe in detail how the target action chunk A⋆ is chosen for each task, e.g., a random action, a fixed offset, or a specific wrong behavior. Since A⋆ depends on the current state while the patch is universal, how is a single patch optimized for different targets? Please also report how often the target behavior is achieved, and how close the outputs get to the target pair, e.g., D_V(V^δ, V⋆) and D_A(A^δ, A⋆). Given that the imagination deviation is larger than the action deviation in both “Future Decoupling” and “Action Decoupling” (Table 1), how do the authors explain that the two modes do not show the expected opposite pattern? What values of ε_A and ε_V are used? It would also help to report D_V and D_A under random and Gaussian perturbations as a reference. If possible, could the authors compare with an existing patch attack on robot policies, e.g., Wang et al. (2025a), under the same setting, and evaluate the attacks on at least one more WAM? This would show whether the gain comes from manipulating causal consistency and whether the attack surface holds for WAMs in general.

If the authors can address these points well, I would consider raising my score.

Reviewer_ZCVQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CAGE, a visual patch attack on world-action models that aims to manipulate the consistency between imagined futures and generated actions. Disruption and hijacking are implemented with orthogonal gradient decomposition and equiangular gradient descent. The attack is evaluated on LingBot-VA in simulation and on a real robot.

Critical strengths
  1. The authors clearly separate disruption and hijacking, and correctly note that rollout-based consistency lies outside the computation graph.
  2. The authors provide local first-order guarantees for both gradient updates and state the assumptions explicitly.
  3. The authors evaluate three attack variants on LIBERO and RoboTwin and also report a physical Franka deployment.
Critical weaknesses
  1. The authors do not attack the causal consistency they claim to attack. CAGE only minimizes branch-wise squared error, and the consistency constraint in Eq. 7 is never enforced. Their own proxy is largest under hijacking (37.6), the mode that is supposed to preserve consistency. On LIBERO-Long, a proxy of 152.1 still leaves 90% success, while LIBERO-Spatial success falls to 14–32% at a proxy of about 4–5. The metric does not explain the attack.
  2. The authors’ success-rate drops are ordinary action corruption. Future decoupling is supposed to keep actions fixed, but its average action deviation is higher than action decoupling’s (58.1 vs. 54.7; 95.9 vs. 29.9 on LIBERO-Object). Action decoupling changes imagination more than action on four of the five suites. No action-only control at matched action deviation is reported, so Table 1 does not support a consistency attack.
  3. The authors overstate the novelty. Eq. 11 is standard orthogonalization, and Eq. 14 is a sum of unit gradients. These are not the MGDA or PCGrad updates they cite. The only baselines are random and Gaussian noise. The claimed new surface is already the subject of BadWAM, JailWAM, and Chen et al. (2026) in their own related work. The authors present the favorable setting as the main result. Table 1 is the wrist camera; on the main camera, LIBERO-Long success stays at 58%, 55%, and 93%. Hijacking never defines the target action or measures target success. The same no-defense proxy is 43.8 in 4. Table 4 and 65.5 in Tables 1–2, and the simulation budget, trial count, and seeds for Table 1 are missing. The authors’ defense results do not support the threat they advertise. JPEG, bit reduction, median filtering, and resizing are weak generic transforms, and resizing alone drops clean LIBERO-Long success from 99% to 76%. They never test the consistency monitors that the introduction uses to justify this attack surface.
Questions for authors
  1. What are the success rate and final action deviation for patches trained on action only, imagination only, and the unprojected sum of the two gradients, under the Table 1 budget?
  2. How is the hijacking target action chosen, and how often is that target behavior actually executed?
  3. Which camera, patch size, opacity, episode count, and seeds produce Table 1, and why does the no-defense hijacking proxy disagree between Table 4 and Tables 1–2?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.