acceptodds
Under review as a conference paper at ICLR 2027

TAPDreamer: Transferable Adversarial Patches on World Action Models

Abstract

World models learn to predict how their environment will evolve, making them an important foundation for general-purpose robotic control. Yet world action models depend on camera inputs whose manipulation can corrupt the visual representations used across tasks and action policies. Existing attacks on these models optimize against the victim's actions or predicted futures and therefore require access to target-model outputs. In this paper, we propose an attack, TAPDreamer, against world action models that instead uses a public encoder alone to construct a fixed local perturbation that transfers across tasks and action architectures. TAPDreamer requires no target-policy queries. Our key insight is that interactions between patch-induced changes in attention weights and value vectors broadcast a nearly identical representation shift far beyond the patch footprint, and this shift remains stable across task observations. Guided by this insight, TAPDreamer uses six frames from one source task to maximize the global L1 distance between clean and patched encoder representations. In closed-loop evaluation, one frozen patch per benchmark, covering about 6.5% of the input, reduces FastWAM's success rate from 97.7% to 0.0% across 40 LIBERO tasks and from 90.8% to 0.0% across 50 RoboTwin tasks; matched random patches retain 81.5% and 79.2% success. The same patches reduce success to 2.1% and 0.8% on two DreamWAM configurations and to 10.0% on Motus. These results show that protecting downstream action generation alone is insufficient: defenses for world action models must also secure shared visual encoders against persistent local perturbations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action models, adversarial patches, transferability, robot learning, embodied AI

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_ub2zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TAPDreamer constructs a fixed adversarial patch using six observations and a public visual encoder, without querying the target policy. The paper reports substantial degradation across WAM-controlled manipulation tasks and investigates attention-mediated perturbation propagation.

Critical strengths

1.Strong closed-loop attack results across two benchmarks and multiple downstream architectures.

2.Matched random-patch controls demonstrate that optimized patch content causes substantially greater degradation than random pixel replacement.

3.Attention-restoration experiments provide useful mechanistic evidence beyond qualitative visualizations.

Critical weaknesses

1.Sensitive to patch position limit physical deployability. The attack requires persistent patch injection at fixed image coordinates. However, a scene-placed physical patch would undergo translation, scaling, and viewpoint changes during robot execution. Table 5 shows that relocating the unchanged patch largely eliminates its effectiveness, while robustness to smaller geometric transformations is not evaluated. Consequently, the strong digital attack results do not establish practical effectiveness against physically deployed robotic systems.

2.Incomplete statistical reporting. The meaning of “±0.00,” independent repeat counts, seed configurations, and sampling structure of the 48 mechanism-analysis states are insufficiently specified. (Tab.1 / Tab. 2)

3.Unclear dependence of transferability on encoder sharing. The experiments demonstrate transfer across downstream control architectures, but do not clearly distinguish this from transfer across visual encoders. Without specifying which models share identical encoder weights, the generality of the cross-model transfer claim remains difficult to assess.

4.Incomplete causal validation of the proposed mechanism. Attention-restoration experiments support the role of attention in perturbation propagation. However, they neither isolate the contribution of the attention-weight/value interaction term nor establish whether reducing latent and action differences restores closed-loop task success. The connection between the proposed mechanism and task failure therefore remains only partially validated.

Questions for authors

1.Do the authors plan to conduct real-world experiments to evaluate whether the frozen patch remains effective under changes in position, scale, and viewpoint during robot execution?

2.What does “±0.00” represent, how many independent runs and seeds were used, and how were the 48 mechanism-analysis states sampled across tasks and episodes?

3.Which evaluated models share identical visual encoder weights, and does the patch transfer to models with different visual encoders?

4.Can additional interventions isolate the attention-weight/value interaction term’s contribution, and does restoring attention components recover closed-loop task success?

Reviewer_K8HcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TAPDreamer, a transferable adversarial patch attack method designed for "World Action Models" (WAMs). By leveraging only a publicly available visual encoder, the method constructs a fixed, localized patch that maximizes the representational discrepancy between clean and perturbed observations, all without requiring access to the target policy’s outputs or the downstream action-generation modules. The patch is optimized using just six observation samples from a single source task, yet it remains directly applicable to unseen tasks and WAMs with different architectures. Experiments on the LIBERO and RoboTwin benchmarks demonstrate the high effectiveness of this attack, reducing the task success rate of FastWAM to 0%. The attack also proves transferable to DreamWAM and Motus, leads to significant performance degradation in both. The authors investigate the underlying mechanisms through attention-recovery experiments, spatial representation analysis, and cross-observation consistency metrics, revealing how localized perturbations propagate through visual representations to ultimately disrupt downstream robotic control.

Critical strengths
  1. The paper departs from the established paradigm of attacks on world action models. As the paper notes, existing attacks optimize against actions or against predicted futures, and require access to the victim's outputs or generation modules. TAPDreamer instead needs only white-box access to a public visual encoder and six demonstration frames from a single source task, yet it still achieves very high task failure rates on WAMs with different architectures within the same benchmark. This is consequential for my assessment for two reasons: defenses deployed at the action head would not eliminate the vulnerability, and the attack is relatively cheap to mount.

  2. Strong attack effectiveness under a clearly defined threat model. The paper demonstrates that a fixed adversarial patch optimized using only a public visual encoder and a small number of observations can significantly degrade the performance of multiple WAM architectures without requiring target-policy queries. The extensive closed-loop evaluations on LIBERO and RoboTwin provide convincing evidence of the vulnerability of visual representation interfaces in robotic control systems.

  3. Beyond demonstrating attack effectiveness, the paper investigates how localized perturbations pass through attention mechanisms, and generate consistent representation shifts across different observations. The attention restoration experiments and behavioral analyses provide useful evidence linking representation disturbances to downstream action deviations.

Critical weaknesses
  1. Although TAPDreamer demonstrates strong transferability across different WAM architectures (Section 3.2, Lines 288–295; Table 2), the paper does not clearly specify whether the evaluated models share the same visual encoder or use independently trained encoders. Since the attack directly optimizes perturbations against a public encoder (Section 2.2, Lines 119–125), the reported results may primarily reflect transferability across downstream architectures rather than different visual encoders. The authors should clarify the encoder configurations and further validate the generality of the claimed transferability.

  2. The proposed L1 representation-displacement objective (Section 2.3, Eq. (4)) is closely related to existing feature-disruption attacks, as also acknowledged in the Related Work section (Lines 432–453). However, the experimental comparisons mainly involve random patches and VLA attacks transferred from different source models (Appendix A.2, Table 6). These baselines do not adequately isolate the methodological advantages of TAPDreamer. Comparisons with existing feature-level adversarial methods optimized against the same visual encoder under identical experimental settings would better substantiate the paper's contribution.

  3. The patch-position ablation in Table 5 reveals a significant limitation of TAPDreamer. Although reoptimizing the patch at different image locations consistently reduces task success rates to 0%, directly moving the originally optimized patch from the upper-left corner to other positions results in success rates of 93.50%-95.00%, indicating a substantial loss of attack effectiveness. This suggests that the optimized patch is highly sensitive to its spatial placement, potentially limiting its robustness under realistic conditions where precise patch positioning cannot be guaranteed.

Questions for authors
  1. Do FastWAM, DreamWAM, and Motus share the same visual encoder, or do they employ independently trained encoders? Could the authors clarify the differences among their encoder configurations and provide evidence that TAPDreamer remains effective when transferred across distinct visual encoders?

  2. How does TAPDreamer compare with existing feature-disruption or representation-based adversarial attacks when all methods are optimized against the same public visual encoder under identical experimental settings? Such comparisons would help clarify whether the performance improvements originate from the proposed optimization strategy rather than differences in source models.

  3. Table 5 shows that relocating an optimized patch substantially reduces its effectiveness. What causes this strong dependence on patch location? Have the authors evaluated the robustness of TAPDreamer under small positional shifts, image transformations, or changes in camera viewpoints? These results would help assess the practical applicability of the proposed attack.

Reviewer_o9jmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TAPDreamer, a transferable adversarial patch attack for World Action Models (WAMs). The method optimizes a fixed patch by maximizing the difference between clean and perturbed visual features using a public visual encoder. It only requires six images from a single task and does not need access to the target policy. The authors evaluate TAPDreamer on LIBERO and RoboTwin using several WAMs, including FastWAM, DreamWAM, and Motus. The results show large drops in task success rates across different tasks and models. The paper also analyzes how the patch affects attention, visual representations, and robot actions.

Critical strengths
  1. TAPDreamer reduces FastWAM's success rate to 0% on both LIBERO and RoboTwin. A single patch also works across many tasks without further optimization.

  2. The method only requires a public visual encoder and a few images. It does not need to query the target policy or optimize through the action generation process.

  3. The authors examine how local perturbations spread through attention layers and affect robot actions. The attention-restoration experiments and latent feature analysis help explain why the attack works.

Critical weaknesses
  1. The method mainly maximizes the difference between clean and perturbed visual features. Similar ideas have already been explored in adversarial attack research. It is unclear what the main technical contribution is beyond applying this approach to WAMs.

  2. The evaluated models appear to use similar Wan2.2 visual encoders. The reported cross-model transferability may therefore come from their shared encoder rather than generalization across different visual architectures.

  3. Table 5 shows that moving the optimized patch to another position greatly reduces its effectiveness. The experiments also rely on digitally inserted patches at fixed positions. It is unclear how well the attack would work under more realistic conditions.

  4. Most comparisons are against random patches. The authors also evaluate existing VLA attacks, but those patches are not optimized for the same WAM encoder. Comparisons with other feature-based attacks under the same settings would make the results more convincing.

Questions for authors
  1. Can the authors explain how the proposed optimization differs from existing feature-disruption attacks? Have other feature-distance objectives been tested?

  2. Have the authors tested patch transfer across WAMs with different visual encoders? How much of the observed transferability comes from sharing the same pretrained encoder?

  3. Why is the attack so sensitive to patch position? Would small translations, scaling, or other image transformations also reduce its effectiveness?

  4. Can the authors compare TAPDreamer with other feature-based attacks using the same visual encoder, patch size, and optimization budget?

Reviewer_5Y64OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TAPDREAMER, a transferable fixed patch attack for world action models. By utilizing the open-source visual encoder, the proposed method maximizes the feature distance in the latent space, and the optimized texture is able to transfer to various downstream WAMs and achieve remarkable attack performance. Meanwhile, this paper presents an in-depth analysis of the attention-restoration interventions and cross-observation latent consistency, which are convincing and good supporting points. However, the overall methodological novelty is relatively limited, as the attack objective essentially maximizes feature-space deviation, which has been extensively explored in prior adversarial attack literature. Overall, I find the empirical observations interesting and potentially impactful, while the novelty of the methodology is partially limited.

Critical strengths
  1. This paper investigates an interesting and potentially impactful problem, and the experimental results are quite remarkable.

  2. The paper is well-structured and easy to follow.

  3. In-depth analysis is provided to support and illustrate the mechanism behind the effectiveness of the adversarial patch.

Critical weaknesses
  1. My major concern is the novelty of the proposed method, as the attack objective is only to maximize the feature-space derivation, which has been extensively explored in previous work. Meanwhile, the paper lacks a sufficiently strong encoder-level attack baseline [1][2]. It is recommended to discuss and compare the performance of those methods to enhance the comprehensiveness of the experiments.

  2. In the appendix, the author provides the results on " PATCH AREA AND POSITION," which is good. However, the experimental results also indicate that the optimized patch is quite sensitive to the area and position, which may raise concerns about its actual applicability. I wonder whether incorporating random variations in patch size and position during optimization, such as through an Expectation over Transformation (EOT) strategy, could improve robustness to such changes. It would be valuable to investigate whether a single optimized patch can maintain its effectiveness across different spatial configurations without requiring position-specific reoptimization.

[1 ]Downstream-agnostic Adversarial Examples, in ICCV 2023. [2] Transferable Adversarial Attacks on SAM and Its Downstream Models, in NeurIPS 2024.

Questions for authors

Please see weaknesses 1 and 2.

Reviewer_puHgOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TAPDreamer, a transferable adversarial patch attack against World Action Models (WAMs) that requires only a public visual encoder and no access to target policies. By disrupting visual representations, a small fixed patch significantly reduces robotic task success rates across multiple models and benchmarks. The paper further analyzes how these perturbations propagate spatially and persist across observations.

Critical strengths

1.Extensive experiments across multiple models and benchmarks demonstrate the effectiveness of the proposed approach.

2.Analyses of attention weights, value vectors, and persistent representation shifts provide valuable insights into how adversarial perturbations affect global visual representations.

3.The attack requires only a small amount of training data for optimization, demonstrating its data efficiency.

Critical weaknesses

1.The position ablation study in Appendix A.1 shows that relocating an optimized patch substantially reduces its effectiveness, indicating a strong dependence on precise spatial placement and raising concerns about its robustness in real-world scenarios.

2.The proposed method is relatively straightforward, primarily relying on maximizing the L1 distance between clean and perturbed visual representations. Its methodological novelty could be further clarified.

3.The paper lacks an ablation study examining how the number and selection of frames used during optimization affect attack effectiveness and generalizability.

Questions for authors

1.Have the authors explored alternative loss functions, such as L2 distance or cosine similarity? How would these alternatives affect attack performance?

2.In real-world scenarios, the appearance and position of an adversarial patch may vary due to camera movements, changes in viewing angles, or variations in object scale. How robust is TAPDreamer to such transformations? Could the authors provide additional experiments evaluating its robustness under different viewpoint and spatial transformations?

2 comments

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts in addressing my concerns. However, some concerns remain.

Robustness to spatial transformations. The additional experiments primarily evaluate fixed pixel-level translations at a single input resolution. However, the same pixel-level shift can correspond to different relative displacements across image resolutions. Moreover, in real-world scenarios, the apparent size, position, and perspective of the patch may vary significantly due to changes in camera distance and viewpoint. Therefore, the current experiments do not fully demonstrate robustness to realistic spatial transformations.

Methodological novelty. My primary concern regarding methodological novelty remains. The proposed approach appears to be a relatively straightforward adaptation of existing feature-space adversarial attack methods to the WAM setting, without introducing substantial new technical innovations. While the empirical results and mechanistic analyses are valuable, I remain unconvinced that extending existing techniques to a new model family constitutes a sufficiently novel methodological contribution.

Overall, although the rebuttal strengthens the empirical evaluation, it does not fully address my concerns, particularly regarding methodological novelty. After reconsidering the paper's contributions, I have decided to lower my score.

AuthorsOpenReview ↗

Clarification Follow-up

Dear Reviewer puHg,

Thank you for your time and follow-up. We respectfully disagree with your assessment, as it overlooks key empirical evidence in our revision and mischaracterizes the nature of our contribution:

  1. Realistic spatial transformations: Your comment states that we primarily evaluated fixed pixel-level translations at a single resolution. This overlooks both our EOT evaluation (Exp. 2, Appendix A.1), which tests placements across the full image range (4 px to 138 px) with skew while maintaining 0% SR, and our physical robot experiment on SO-101 (Exp. 3, Section 3.2 and Figure 2). In the real-robot evaluation, the optimized patch is printed as a physical sticker and tested on the SO-101 arm, where the physical placement, viewing angle, and apparent scale in the camera view naturally deviate from exact digital pixel alignment. Under these real-world conditions, the patch still drops FastWAM success from 92% to 0% and DreamWAM from 80% to 20%.

  2. Simplicity as a strength and non-trivial WAM adaptation: In embodied AI security, recent works accepted at top-tier venues, such as UPA-RFAS (CVPR 2026) [1] and UADA (ICCV 2025) [2] for VLA models, also rely on clean and simple adversarial formulations. In security research, we believe simplicity is a strength when a minimal, assumption-light method reliably exposes a severe vulnerability. Moreover, adapting this to WAMs is non-trivial: unlike standard vision models or image-based VLAs, WAMs rely on video/world-model encoders and closed-loop replanning where evolving observations and multi-step dynamics can easily wash out naive perturbations. Our specific design choices, including joint multi-frame optimization and the L1 objective over full encoder latents, are critical. As shown in our ablations (Cosine: 56.65% SR; L2: 15.20% SR vs. L1: 0.00% SR) and matched baseline comparisons in Appendix A.5, directly adapting existing feature-space attacks (AdvEncoder: 45.85% SR; GRAT: 38.75% SR) or VLA patch attacks (UPA-RFAS: 74.50% SR; UADA: 80.00% SR) under the same budget fails to reliably break closed-loop control, whereas TAPDreamer achieves 0.00% SR. Beyond the attack itself, our core contribution also includes uncovering and causally validating why an encoder-only frozen patch breaks closed-loop WAMs via the attention-value interaction (ΔAΔV\Delta A \Delta V).

We hope these clarifications provide a clearer picture of our revision and experimental evidence.

Best regards,

The Authors

References: [1] When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Models, CVPR 2026. [2] Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics, ICCV 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.