- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors investigate how changes in visual observations within world-action models influence action generation, and whether this dependency between vision and action is vulnerable to adversarial attacks. To this end, they propose the VMS metric, which quantifies action responses to controlled changes in a scene—analyzing these responses through experiments involving object translation, shifts in camera perspective, and interventions in the model's attention mechanism. Building on this, the authors introduce the WAD adversarial attack, which disrupts the model's action generation by suppressing vision-to-action attention and leveraging a VMS-based surrogate loss to minimize the discrepancy between action predictions derived from consecutive observations. Finally, the authors validate the impact of this attack on task execution across various world-action models, simulated benchmarks, and real-world robotic experiments.
Critical strengths
-
The paper addresses an important evaluation problem in robotics by introducing VMS to characterize how action predictions respond to controlled visual changes, providing a complementary diagnostic perspective beyond task success rate.
-
The proposed WAD attack is well motivated and exposes vulnerabilities in the visuomotor dependency of the evaluated WAMs by combining visual-to-action attention suppression with a paired-action response objective.
-
The attention interventions, component ablations, and comparison with an action-deviation baseline under matched perturbation constraints and optimization settings provide empirical support for the attack design.
Critical weaknesses
-
The diagnostic value of VMS needs further validation. A policy should respond to changes that require action adjustments, but may appropriately remain unchanged when the required action is unaffected. The current experiments establish local stability and response consistency, but do not fully show how VMS distinguishes appropriate invariance from impaired responsiveness or consistently incorrect responses.
-
The mechanistic interpretation of the attention-logit interventions remains limited. As the authors acknowledge, scaling AV logits also changes competing AA probabilities through softmax normalization and may alter the distribution within the AV block. The observed changes in VMS therefore cannot be attributed solely to changes in visual-to-action attention mass. Reporting the actual attention redistribution and adding controls that separate these effects would strengthen the analysis.
-
The attack evaluation is limited to white-box, digitally injected perturbations, including in the real-robot experiment. Perturbations are optimized separately for each model and task, leaving their transferability and effectiveness under restricted model access or physical perturbations unclear. This limits the generality of the demonstrated attack.
Questions for authors
See weakness.