acceptodds
Under review as a conference paper at ICLR 2027

Disintegrating the Visuomotor Dependency in World-Action Models

Abstract

World Action Models (WAMs) integrate world modeling and action generation, offering a promising paradigm for robotic policy. While WAMs show strong potential, their robustness under adversarial attack remains underexplored. A defining characteristic of WAMs is the coupling between world modeling and action generation. How this coupling shapes action responses to semantic changes remains insufficiently characterized. We investigate how changes in visual observations propagate to downstream action generation. However, existing task-level metrics, e.g., success rate, primarily reflect final task outcomes, offering limited insight into the strength of this visuomotor dependency. We therefore introduce Visuomotor Sensitivity (VMS), which quantifies the magnitude of action responses to controlled semantic transformations. Using VMS, we compare attention-pathway interventions in WAMs and identify visual-to-action attention as an internal attack target. Based on these findings, we propose World-Action Disintegration (WAD), which combines visual-to-action attention suppression with a VMS-inspired surrogate targeting paired action differences across consecutive observations from the existing training dataset. Experiments show locally consistent VMS measurements and reproducibility across inference seeds. In simulation, we observe near-static behavior when minimizing the VMS-based surrogate and large-amplitude unstable motion when maximizing it; both conditions fail the task. WAD substantially degrades task performance across WAM architectures in simulation and in real-world trials. These findings characterize action responsiveness and its associated execution behaviors beyond aggregate action deviation and task success.

Primary area:
applications to robotics, autonomy, planning
Keywords:
WAM, AI Safety, Embodied AI, Adversarial Attack

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_atu4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors investigate how changes in visual observations within world-action models influence action generation, and whether this dependency between vision and action is vulnerable to adversarial attacks. To this end, they propose the VMS metric, which quantifies action responses to controlled changes in a scene—analyzing these responses through experiments involving object translation, shifts in camera perspective, and interventions in the model's attention mechanism. Building on this, the authors introduce the WAD adversarial attack, which disrupts the model's action generation by suppressing vision-to-action attention and leveraging a VMS-based surrogate loss to minimize the discrepancy between action predictions derived from consecutive observations. Finally, the authors validate the impact of this attack on task execution across various world-action models, simulated benchmarks, and real-world robotic experiments.

Critical strengths
  1. The paper addresses an important evaluation problem in robotics by introducing VMS to characterize how action predictions respond to controlled visual changes, providing a complementary diagnostic perspective beyond task success rate.

  2. The proposed WAD attack is well motivated and exposes vulnerabilities in the visuomotor dependency of the evaluated WAMs by combining visual-to-action attention suppression with a paired-action response objective.

  3. The attention interventions, component ablations, and comparison with an action-deviation baseline under matched perturbation constraints and optimization settings provide empirical support for the attack design.

Critical weaknesses
  1. The diagnostic value of VMS needs further validation. A policy should respond to changes that require action adjustments, but may appropriately remain unchanged when the required action is unaffected. The current experiments establish local stability and response consistency, but do not fully show how VMS distinguishes appropriate invariance from impaired responsiveness or consistently incorrect responses.

  2. The mechanistic interpretation of the attention-logit interventions remains limited. As the authors acknowledge, scaling AV logits also changes competing AA probabilities through softmax normalization and may alter the distribution within the AV block. The observed changes in VMS therefore cannot be attributed solely to changes in visual-to-action attention mass. Reporting the actual attention redistribution and adding controls that separate these effects would strengthen the analysis.

  3. The attack evaluation is limited to white-box, digitally injected perturbations, including in the real-robot experiment. Perturbations are optimized separately for each model and task, leaving their transferability and effectiveness under restricted model access or physical perturbations unclear. This limits the generality of the demonstrated attack.

Questions for authors

See weakness.

Reviewer_8RR2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the robustness of World-Action Models (WAMs) by examining how visual information influences action generation. It introduces Visuomotor Sensitivity (VMS) to measure action responses to visual changes and proposes World-Action Disintegration (WAD), an adversarial attack that combines visual-to-action attention suppression with minimizing action differences between consecutive observations. Experiments across multiple WAM architectures on LIBERO and RoboTwin show that WAD significantly reduces task success, outperforming existing attack baselines. Real-world robot experiments further demonstrate its effectiveness.

Critical strengths
  1. The paper studies the robustness of World-Action Models and investigates how visual information affects action generation, addressing an important problem in robotic safety.

  2. WAD consistently reduces task success across multiple WAM architectures and benchmarks, outperforming a well-matched action-deviation baseline.

  3. The paper evaluates four WAMs across LIBERO and RoboTwin, with additional ablation studies, attention analysis, and real-world robot experiments that strengthen its empirical findings.

Critical weaknesses

W1. unclear whether WAD specifically weakens the model’s use of visual information

WAD makes action predictions for consecutive observations more similar, but the robot’s state and task progress may also differ between these observations. As the authors acknowledge, this does not directly measure responses to visual changes. Smaller action differences and nearly motionless behavior could reflect either a reduced ability to use visual information or a broader failure that makes the model produce similar actions for all inputs.

W2. The novelty of combining the two objectives is not yet clear

Prior work has already used visual attention to design attacks on robotic policies. In Table 3, attention suppression alone reduces FastWAM’s success rate to 0%; in Table 4, adding the objective that reduces action differences lowers success from 68.2% to 62.6%. This gain is potentially meaningful, but the limited component analysis and single-run optimization protocol do not establish its reliability across architectures and initializations or its advantage over prior attention-guided objectives.

W3. The practical scope of the demonstrated vulnerability is not sufficiently characterized

WAD learns separate perturbations for each model, task, and camera stream, and its real-robot experiment also injects perturbations digitally rather than through a physical attack. This is a legitimate robustness evaluation, but it establishes vulnerability under a particular combination of task-specific optimization and camera-input access. The experiments do not determine how strongly effectiveness depends on access to all camera streams or optimization data from the target task.

Questions for authors

Regarding W1: Validating the mechanism

Q1. Please evaluate whether WAD prevents the model from correctly adjusting its actions to controlled object shifts, while keeping other inputs fixed

Q2. Can you show whether WAD specifically weakens the model's responses to visual changes, rather than making its actions insensitive to all inputs?

Regarding W2: Establishing the incremental contribution

Q3. Can you establish the reliability of the paired-response objective’s additional benefit over attention suppression alone through component ablations across architectures and independent attack-optimization seeds?

Q4. Does WAD outperform an appropriately adapted attention-hijacking objective from prior work, such as VLAGuard or the SARF study, under matched input-space constraints and optimization budgets?

Regarding W3: Characterizing the threat’s scope

Q5. How effective is WAD when only one camera stream is perturbed and the remaining camera inputs are clean?

Q6. How well does a fixed perturbation transfer to tasks not used during attack optimization, without further optimization?

Reviewer_wF96OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the relationship between world modeling and action generation and introduces a novel metric called Visuomotor Sensitivity (VMS) to quantify action responsiveness to visual changes. By minimizing a VMS-based surrogate and suppressing visual-to-action attention, the authors propose World-Action Disintegration (WAD), an attack that can induce near-static robot behavior. Experiments in both simulation and real-world settings demonstrate that WAD effectively degrades the performance of WAMs.

Critical strengths
  1. The authors propose an interesting perspective on the relationship between world modeling and action generation in WAMs. WAD is developed through an exploration of WAMs’ internal characteristics and is lightweight at execution time.

  2. Comprehensive experiments are conducted across multiple WAM paradigms and benchmarks, as well as on a real robot, demonstrating the effectiveness of WAD across the evaluated settings and revealing shared characteristics of WAMs. The performance degradation in both simulation and real-world experiments is substantial.

Critical weaknesses
  1. Although WAD effectively degrades the performance of WAMs, it is a white-box attack that requires access to internal parameters, gradients, and attention representations, limiting WAD’s practical applicability.

  2. The authors use consecutive observations as inputs for attack optimization and use them to construct a VMS-based attack surrogate, but the inputs also include robot states (proprioceptive states), which may differ between observations. An additional controlled post-attack experiment that holds the robot states fixed could be helpful.

Questions for authors
  1. As a core contribution of the paper, to what extent can VMS serve as a general metric beyond its use in developing WAD?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.