- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates black-box adversarial patch attacks against Vision-Language-Action (VLA) models by exploiting their publicly available ancestor Vision-Language Models (VLMs), without requiring access to the target VLA. The authors propose three attack objectives, namely Vision Disruption (VIS), Instruction-Grounded Semantic Evidence Suppression (IGS), and a joint optimization strategy that combines both objectives. Extensive experiments are conducted on OpenVLA, UniVLA, and Pi0.5, using the LIBERO benchmark and BridgeData V2 dataset. The results demonstrate that the proposed joint attack reduces the average task success rates of OpenVLA and UniVLA from 68.75% and 93.08% to 33.20% and 45.95%, respectively. Furthermore, the authors investigate cross-architecture transferability and identify limitations of vulnerability inheritance, particularly for Pi0.5, where the proposed attacks exhibit limited effectiveness.
Critical strengths
- The paper investigates a realistic black-box attack scenario in which adversaries exploit publicly available ancestor VLMs without accessing the parameters, gradients, or action spaces of target VLAs. This setting has practical implications for the security of deployed robotic systems.
- The proposed attacks achieve substantial reductions in task success rates on OpenVLA and UniVLA, consistently outperforming random patches and adapted baselines, including EDPA and UPA-RFAS. These results provide empirical evidence that adversarial vulnerabilities can transfer from pretrained VLMs to their downstream VLAs.
- The paper includes evaluations of cross-domain transferability, comparisons across different ancestor models, and ablation studies of the proposed objectives and joint optimization strategy. The analysis of attack failures on Pi0.5 also provides useful insights into the limitations of vulnerability inheritance.
Critical weaknesses
- The central idea closely resembles conventional transfer-based adversarial attacks, where perturbations optimized on publicly available ImageNet-pretrained CNN backbones are transferred to fine-tuned downstream models. The paper does not sufficiently establish what fundamentally distinguishes VLM-to-VLA vulnerability inheritance from the well-studied transferability of adversarial perturbations across pretrained and fine-tuned models.
- Compared with VIS alone, the joint objective provides only approximately 2 percentage points of additional reduction in task success rates on OpenVLA and UniVLA. Moreover, the visual-token hidden states used by IGS do not directly attend to the instruction tokens, raising concerns about whether the proposed objective effectively captures instruction-conditioned semantic grounding.
- The strongest attack results are primarily obtained on OpenVLA and UniVLA, which share the same Prismatic-7B ancestor, whereas the proposed attacks perform close to random patches on Pi0.5. Although the authors attribute this discrepancy to representation shifts during fine-tuning and the perturbation-smoothing effects of iterative action generation, these explanations are not sufficiently supported by controlled experiments that isolate the contributions of the two factors.
- The evaluation is limited to simulated environments and static real-world images, without experiments on physical robotic platforms. Consequently, the effectiveness and reliability of the proposed adversarial patches under real-world physical conditions remain unclear.
Questions for authors
- How does the proposed ancestor-only attack fundamentally differ from conventional transfer-based adversarial attacks that optimize perturbations on publicly available pretrained CNN backbones and transfer them to fine-tuned downstream models? Can the authors provide empirical evidence demonstrating that VLM-to-VLA vulnerability inheritance involves mechanisms distinct from those established in prior transferability studies?
- Given the relatively modest improvement introduced by IGS and the fact that the probed visual-token hidden states cannot directly attend to instruction tokens, can the authors provide stronger evidence that IGS specifically disrupts instruction-conditioned semantic grounding?
- To what extent is the limited effectiveness of the proposed attacks on Pi0.5 attributable to representation shifts during fine-tuning versus the perturbation-smoothing properties of the flow-matching action head? Could the authors conduct controlled experiments to disentangle these two factors?
5 comments
Response to Reviewer YnwE (2/3)
Q2 & W2: Instruction guidance and the value of IGS
The reviewer is right that the visual tokens we probe cannot attend to the instruction tokens, since the image precedes the text under causal attention. We do not claim they do. The instruction acts through the choice of probes — the instruction and the frame together decide which concept words (object, attribute, relation) to probe — not through attention. Besides, VIS alone is already a strong attack that removes most of the exploitable visual signal, so IGS's contribution is necessarily an increment layered on top of an already-low success rate.
To provide direct evidence that IGS disrupts instruction-conditioned grounding, we queried OWLv2's detection confidence for the object named in the instruction, on clean and patched LIBERO frames, where the patches are optimized based on the IGS objective. OWLv2 (Minderer et al.) is an off-the-shelf open-vocabulary detector with no role in optimizing our patches or in IGS's own logit-lens scoring. A lower value means the instructed object is less detectable. The results are as follows:
| Condition | OWLv2 confidence (Spa.) | OWLv2 confidence (Obj.) |
|---|---|---|
| Clean | 0.118 | 0.073 |
| Random Patch | 0.118 | 0.045 |
| Ours-IGS | 0.084 | 0.039 |
| Ours-IGS with Shuffled probes | 0.086 | 0.045 |
| Ours-IGS with Irrelevant probes | 0.092 | 0.044 |
Ours-IGS gives the lowest confidence in both suites, below random patches and below both shuffled and unrelated probes. The margin is small, so we treat this as directional rather than quantitative evidence. Since OWLv2 never sees the instruction and is not our model, it still confirms that IGS's suppression specifically tracks the instructed concept.
Response to Reviewer YnwE (3/3)
Q3 & W3: Representation mismatch versus flow-matching smoothing
We first note that these two factors can be disentangled but the flow matching factor cannot be fully isolated since there is no clean way to replace π0.5's flow-matching head with a non-flow-matching one while holding everything else fixed. We therefore measure the representation drift's contribution directly and characterize what remains.
To remove representation drift, we attacked π0.5 itself in a white-box setting (no transfer from the ancestor, using the parameters and gradients of π0.5’s own vision–language module), using our attacks and UADA (a representative white-box baseline) — on LIBERO-Spatial:
| Setting | Attacker | No-attack SR | Attacked SR | Abs. drop (pp) |
|---|---|---|---|---|
| White-box (no drift) | UADA | 98.4% | 93.2% | 5.2 |
| White-box (no drift) | Ours-VIS | 98.4% | 88.8% | 9.6 |
| White-box (no drift) | Ours-IGS | 98.4% | 91.2% | 7.2 |
| White-box (no drift) | Ours-Joint | 98.4% | 94.8% | 3.6 |
| Ancestor-only | Ours-VIS | 98.4% | 95.2% | 3.2 |
| Ancestor-only | Ours-IGS | 98.4% | 95.8% | 2.6 |
| Ancestor-only | Ours-Joint | 98.4% | 95.8% | 2.6 |
Even with representation drift removed, these attacks remain weak, so representation drift is not the main cause of π0.5's resistance. This residual weakness is consistent with prior work showing that diffusion/flow-matching models can be more robust against adversarial attacks (DensePure, ICLR 2023; Nie W et al., DiffPure, ICML 2022; FlowPure, arXiv:2505.13280), and we attribute the residual resistance most plausibly to the flow-matching action head. To quantify the representation drift, we compare against our ancestor-only attacks. The drift cost is the abs. drop difference between the two settings. So drift accounts for only 1.0–6.4 percentage points of π0.5's resistance.
[1] Xiao C, Chen Z, Jin K, et al. DensePure: Understanding Diffusion Models for Adversarial Robustness. ICLR 2023.
[2] Nie W, Guo B, Huang Y, et al. Diffusion models for adversarial purification. ICML, 2022.
[3] "FlowPure: Continuous Normalizing Flows for Adversarial Purification." arXiv:2505.13280.
Response to Reviewer YnwE (1/3)
We thank the reviewer for recognizing the practical relevance of the threat model and the substantial attack effects on OpenVLA and UniVLA, together with the value of our cross-domain experiments and ablations. We have added new experiments and clarifications to address the concerns raised.
We agree both settings are "transfer-based" at a high level, but the classical CNN case and ours differ in exactly the respect that matters: the output modality changes between source and target. A classifier pretrained on ImageNet and a classifier fine-tuned from it still both read out a class label from essentially the same kind of feature space, so attacks optimized on a generic "push the features far from clean" objective can work on the downstream models. In VLM→VLA this alignment breaks: the ancestor's native output is next-token text, while the VLA's action head reads a task-specific slice of the representation to produce robot actions. A generic feature-distance attack spends its perturbation budget on whatever increases the aggregate feature or embedding distance, with no guarantee this impacts the action head — the modality gap is exactly what decouples "looks different to the ancestor" from "causes the actuator to fail."
Our two mechanisms are built to bridge that gap. VIS does not perturb at the level of a pooled embedding or encoder output; it operates on the post-projector token sequence — the interface that the ancestor's frozen language decoder consumes and that the derived VLA inherits (Appendix B.1). Its relative term specifically scrambles position-to-content correspondence rather than just inflating a norm, which matters because the action ultimately depends on which token encodes which spatial location. IGS goes a level deeper: instead of hoping that more feature distance happens to also remove the object/attribute/relation evidence the action head needs, it uses the ancestor's own unembedding matrix to find and directly suppress exactly those scene-relevant concept directions, selected by the instruction and the frame.
For empirical evidence that shows the mechanisms are distinct, the paper already offers two pieces of evidence, each with no counterpart in classical transfer. First, the relative-term ablation (Fig. 2): the absolute term alone is essentially a generic feature-distance attack, and adding the relative, token-position-scrambling term produces the gain. Second, the baselines: EDPA and UPA-RFAS, which reduce to generic feature/alignment terms once their VLA-specific losses are disabled in the ancestor-only setting, underperform our full method by a wide margin (Table 1: 46.2%/59.1% and 41.5%/55.85% vs. our 33.2%/45.95%). These results validate that the generic recipe does not fully cross the modality gap on its own. Furthermore, we adopt an external detector to show that IGS's semantic mechanism actually destroys grounding. OWLv2 — an off-the-shelf open-vocabulary detector that never sees the instruction and plays no role in optimization or in IGS's own scoring — finds the instructed object less confidently under IGS than under a random patch (Spatial: 0.084 vs 0.118; Object: 0.039 vs 0.045), and the correct probes suppress it more than shuffled or unrelated probes. This is an instruction-related semantic mechanism that cannot even be defined for a CNN, which has no instruction and no vocabulary.
In one sentence, what distinguishes our method from conventional transfer-based adversarial attacks that optimize perturbations on publicly available pretrained CNN backbones and transfer them to fine-tuned downstream models is: a new threat model (ancestor-only, no surrogate VLA) and two mechanisms designed specifically to cross the text-to-action modality gap.