acceptodds
Under review as a conference paper at ICLR 2027

Black-Box Adversarial Patch Attacks on VLAs via Ancestor VLM Exploitation

Abstract

Vision-Language-Action models (VLAs) are increasingly deployed in safety-critical physical environments, yet their adversarial robustness remains poorly understood. Existing attacks typically assume white-box access or rely on surrogate VLAs, which rarely holds in real-world deployments. Our key insight is that most VLAs are adapted from a publicly released pretrained vision-language model (VLM), inheriting two capabilities essential for action generation: visual perception and instruction-conditioned grounding. Therefore, this paper explores a previously unaddressed question: can an adversary attack deployed VLAs using only their ancestor VLMs? To this end, we propose three adversarial patch attacks that disrupt the inherited capabilities: a vision disruption attack that corrupts the projected visual tokens through relative and absolute terms, an instruction-grounded semantic evidence suppression attack that removes the visual evidence required for instruction-grounded concepts, and a joint attack that unifies both objectives under a two-phase curriculum. Experiments across different VLA families on both simulation and static real-world images show that patches optimized on the ancestor VLM cause substantial degradations in VLA task success rates, demonstrating that VLAs inherit adversarial vulnerabilities alongside their foundational capabilities. This effect is not uniform: it is strongest on tasks that require precise instruction-grounded localization, and nearly vanishes on policies whose adaptation rewrites the shared visual-semantic representation or whose action head iteratively smooths perturbations away. By characterizing the boundary conditions of vulnerability inheritance and providing analysis of why the inheritance effect holds or fails, we advance the understanding of safety for VLA-involved systems.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
adversarial attack, VLA

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YnwEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates black-box adversarial patch attacks against Vision-Language-Action (VLA) models by exploiting their publicly available ancestor Vision-Language Models (VLMs), without requiring access to the target VLA. The authors propose three attack objectives, namely Vision Disruption (VIS), Instruction-Grounded Semantic Evidence Suppression (IGS), and a joint optimization strategy that combines both objectives. Extensive experiments are conducted on OpenVLA, UniVLA, and Pi0.5, using the LIBERO benchmark and BridgeData V2 dataset. The results demonstrate that the proposed joint attack reduces the average task success rates of OpenVLA and UniVLA from 68.75% and 93.08% to 33.20% and 45.95%, respectively. Furthermore, the authors investigate cross-architecture transferability and identify limitations of vulnerability inheritance, particularly for Pi0.5, where the proposed attacks exhibit limited effectiveness.

Critical strengths
  1. The paper investigates a realistic black-box attack scenario in which adversaries exploit publicly available ancestor VLMs without accessing the parameters, gradients, or action spaces of target VLAs. This setting has practical implications for the security of deployed robotic systems.
  2. The proposed attacks achieve substantial reductions in task success rates on OpenVLA and UniVLA, consistently outperforming random patches and adapted baselines, including EDPA and UPA-RFAS. These results provide empirical evidence that adversarial vulnerabilities can transfer from pretrained VLMs to their downstream VLAs.
  3. The paper includes evaluations of cross-domain transferability, comparisons across different ancestor models, and ablation studies of the proposed objectives and joint optimization strategy. The analysis of attack failures on Pi0.5 also provides useful insights into the limitations of vulnerability inheritance.
Critical weaknesses
  1. The central idea closely resembles conventional transfer-based adversarial attacks, where perturbations optimized on publicly available ImageNet-pretrained CNN backbones are transferred to fine-tuned downstream models. The paper does not sufficiently establish what fundamentally distinguishes VLM-to-VLA vulnerability inheritance from the well-studied transferability of adversarial perturbations across pretrained and fine-tuned models.
  2. Compared with VIS alone, the joint objective provides only approximately 2 percentage points of additional reduction in task success rates on OpenVLA and UniVLA. Moreover, the visual-token hidden states used by IGS do not directly attend to the instruction tokens, raising concerns about whether the proposed objective effectively captures instruction-conditioned semantic grounding.
  3. The strongest attack results are primarily obtained on OpenVLA and UniVLA, which share the same Prismatic-7B ancestor, whereas the proposed attacks perform close to random patches on Pi0.5. Although the authors attribute this discrepancy to representation shifts during fine-tuning and the perturbation-smoothing effects of iterative action generation, these explanations are not sufficiently supported by controlled experiments that isolate the contributions of the two factors.
  4. The evaluation is limited to simulated environments and static real-world images, without experiments on physical robotic platforms. Consequently, the effectiveness and reliability of the proposed adversarial patches under real-world physical conditions remain unclear.
Questions for authors
  1. How does the proposed ancestor-only attack fundamentally differ from conventional transfer-based adversarial attacks that optimize perturbations on publicly available pretrained CNN backbones and transfer them to fine-tuned downstream models? Can the authors provide empirical evidence demonstrating that VLM-to-VLA vulnerability inheritance involves mechanisms distinct from those established in prior transferability studies?
  2. Given the relatively modest improvement introduced by IGS and the fact that the probed visual-token hidden states cannot directly attend to instruction tokens, can the authors provide stronger evidence that IGS specifically disrupts instruction-conditioned semantic grounding?
  3. To what extent is the limited effectiveness of the proposed attacks on Pi0.5 attributable to representation shifts during fine-tuning versus the perturbation-smoothing properties of the flow-matching action head? Could the authors conduct controlled experiments to disentangle these two factors?
5 comments
AuthorsOpenReview ↗

Response to Reviewer YnwE (1/3)

We thank the reviewer for recognizing the practical relevance of the threat model and the substantial attack effects on OpenVLA and UniVLA, together with the value of our cross-domain experiments and ablations. We have added new experiments and clarifications to address the concerns raised.

Q1 & W1: Distinction from conventional transfer attacks

We agree both settings are "transfer-based" at a high level, but the classical CNN case and ours differ in exactly the respect that matters: the output modality changes between source and target. A classifier pretrained on ImageNet and a classifier fine-tuned from it still both read out a class label from essentially the same kind of feature space, so attacks optimized on a generic "push the features far from clean" objective can work on the downstream models. In VLM→VLA this alignment breaks: the ancestor's native output is next-token text, while the VLA's action head reads a task-specific slice of the representation to produce robot actions. A generic feature-distance attack spends its perturbation budget on whatever increases the aggregate feature or embedding distance, with no guarantee this impacts the action head — the modality gap is exactly what decouples "looks different to the ancestor" from "causes the actuator to fail."

Our two mechanisms are built to bridge that gap. VIS does not perturb at the level of a pooled embedding or encoder output; it operates on the post-projector token sequence — the interface that the ancestor's frozen language decoder consumes and that the derived VLA inherits (Appendix B.1). Its relative term specifically scrambles position-to-content correspondence rather than just inflating a norm, which matters because the action ultimately depends on which token encodes which spatial location. IGS goes a level deeper: instead of hoping that more feature distance happens to also remove the object/attribute/relation evidence the action head needs, it uses the ancestor's own unembedding matrix to find and directly suppress exactly those scene-relevant concept directions, selected by the instruction and the frame.

For empirical evidence that shows the mechanisms are distinct, the paper already offers two pieces of evidence, each with no counterpart in classical transfer. First, the relative-term ablation (Fig. 2): the absolute term alone is essentially a generic feature-distance attack, and adding the relative, token-position-scrambling term produces the gain. Second, the baselines: EDPA and UPA-RFAS, which reduce to generic feature/alignment terms once their VLA-specific losses are disabled in the ancestor-only setting, underperform our full method by a wide margin (Table 1: 46.2%/59.1% and 41.5%/55.85% vs. our 33.2%/45.95%). These results validate that the generic recipe does not fully cross the modality gap on its own. Furthermore, we adopt an external detector to show that IGS's semantic mechanism actually destroys grounding. OWLv2 — an off-the-shelf open-vocabulary detector that never sees the instruction and plays no role in optimization or in IGS's own scoring — finds the instructed object less confidently under IGS than under a random patch (Spatial: 0.084 vs 0.118; Object: 0.039 vs 0.045), and the correct probes suppress it more than shuffled or unrelated probes. This is an instruction-related semantic mechanism that cannot even be defined for a CNN, which has no instruction and no vocabulary.

In one sentence, what distinguishes our method from conventional transfer-based adversarial attacks that optimize perturbations on publicly available pretrained CNN backbones and transfer them to fine-tuned downstream models is: a new threat model (ancestor-only, no surrogate VLA) and two mechanisms designed specifically to cross the text-to-action modality gap.

AuthorsOpenReview ↗

Response to Reviewer YnwE (2/3)

Q2 & W2: Instruction guidance and the value of IGS

The reviewer is right that the visual tokens we probe cannot attend to the instruction tokens, since the image precedes the text under causal attention. We do not claim they do. The instruction acts through the choice of probes — the instruction and the frame together decide which concept words (object, attribute, relation) to probe — not through attention. Besides, VIS alone is already a strong attack that removes most of the exploitable visual signal, so IGS's contribution is necessarily an increment layered on top of an already-low success rate.

To provide direct evidence that IGS disrupts instruction-conditioned grounding, we queried OWLv2's detection confidence for the object named in the instruction, on clean and patched LIBERO frames, where the patches are optimized based on the IGS objective. OWLv2 (Minderer et al.) is an off-the-shelf open-vocabulary detector with no role in optimizing our patches or in IGS's own logit-lens scoring. A lower value means the instructed object is less detectable. The results are as follows:

ConditionOWLv2 confidence (Spa.)OWLv2 confidence (Obj.)
Clean0.1180.073
Random Patch0.1180.045
Ours-IGS0.0840.039
Ours-IGS with Shuffled probes0.0860.045
Ours-IGS with Irrelevant probes0.0920.044

Ours-IGS gives the lowest confidence in both suites, below random patches and below both shuffled and unrelated probes. The margin is small, so we treat this as directional rather than quantitative evidence. Since OWLv2 never sees the instruction and is not our model, it still confirms that IGS's suppression specifically tracks the instructed concept.

AuthorsOpenReview ↗

Response to Reviewer YnwE (3/3)

Q3 & W3: Representation mismatch versus flow-matching smoothing

We first note that these two factors can be disentangled but the flow matching factor cannot be fully isolated since there is no clean way to replace π0.5's flow-matching head with a non-flow-matching one while holding everything else fixed. We therefore measure the representation drift's contribution directly and characterize what remains.

To remove representation drift, we attacked π0.5 itself in a white-box setting (no transfer from the ancestor, using the parameters and gradients of π0.5’s own vision–language module), using our attacks and UADA (a representative white-box baseline) — on LIBERO-Spatial:

SettingAttackerNo-attack SRAttacked SRAbs. drop (pp)
White-box (no drift)UADA98.4%93.2%5.2
White-box (no drift)Ours-VIS98.4%88.8%9.6
White-box (no drift)Ours-IGS98.4%91.2%7.2
White-box (no drift)Ours-Joint98.4%94.8%3.6
Ancestor-onlyOurs-VIS98.4%95.2%3.2
Ancestor-onlyOurs-IGS98.4%95.8%2.6
Ancestor-onlyOurs-Joint98.4%95.8%2.6

Even with representation drift removed, these attacks remain weak, so representation drift is not the main cause of π0.5's resistance. This residual weakness is consistent with prior work showing that diffusion/flow-matching models can be more robust against adversarial attacks (DensePure, ICLR 2023; Nie W et al., DiffPure, ICML 2022; FlowPure, arXiv:2505.13280), and we attribute the residual resistance most plausibly to the flow-matching action head. To quantify the representation drift, we compare against our ancestor-only attacks. The drift cost is the abs. drop difference between the two settings. So drift accounts for only 1.0–6.4 percentage points of π0.5's resistance.

[1] Xiao C, Chen Z, Jin K, et al. DensePure: Understanding Diffusion Models for Adversarial Robustness. ICLR 2023.

[2] Nie W, Guo B, Huang Y, et al. Diffusion models for adversarial purification. ICML, 2022.

[3] "FlowPure: Continuous Normalizing Flows for Adversarial Purification." arXiv:2505.13280.

Reviewer_Cm1kOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies black-box adversarial patch attacks on VLAs using only their public ancestor VLMs. It proposes visual disruption, semantic evidence suppression, and a staged joint objective. Experiments show strong transfer to OpenVLA and UniVLA, including across datasets, but limited effectiveness on π0.5.

Critical strengths

1.The ancestor-only setting is well motivated. Optimizing attacks without target-policy queries, parameters, or action supervision identifies a relevant attack surface for VLA systems.

2.The attacks substantially outperform random and blank patches on OpenVLA and UniVLA. For example, the joint attack reduces average success from 68.75% to 33.20% on OpenVLA and from 93.08% to 45.95% on UniVLA.

3.The experiments include useful controls and ablations, including different surrogate VLMs, cross-domain optimization data, individual loss components, and the staged optimization schedule. Reporting the weak results on π0.5 also helps clarify the method’s limitations.

Critical weaknesses

1.Semantic grounding: Lower logit-lens scores do not necessarily imply impaired grounding, so controls with shuffled or irrelevant probes are needed to validate the semantic objective.

2.π0.5 analysis: The weak transfer to π0.5 does not isolate representation drift or flow-matching as the cause, so these explanations require controlled experiments or more cautious wording.

3.Baseline fairness: Matched attack locations, optimization budgets, and tuning procedures are needed to distinguish improvements from the proposed losses versus implementation choices.

4.Statistical uncertainty: Please report variability across patch seeds and confidence intervals, particularly for the modest gains of Joint over VIS.

5.Generality: Since the strongest positive results share the Prismatic ancestor, broader claims about vulnerability inheritance require additional evidence or qualification.

Questions for authors

1.How are the probe layer, semantic-loss weight, activation threshold, and final patch checkpoint selected? Are all selections made exclusively using ancestor-side information?

2.What train/validation separation is used for patch optimization and evaluation, particularly for LIBERO demonstrations and task suites?

3.Does IGS retain its advantage when instruction-relevant probes are replaced with shuffled or unrelated concepts?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Cm1k (1/2)

We thank the reviewer for recognizing the well-motivated ancestor-only setting and substantial improvements over random and blank patches, as well as the value of our controls and ablations. We have added new experiments and clarifications to address the concerns raised.

Q1: Ancestor-only hyperparameter and checkpoint selection

These choices are all fixed by the ancestor VLM or by our fixed training schedule. None of them involves the VLA.

  • Probe layer: we use a middle layer of the ancestor VLM (decoder block 12). We chose a middle layer because its features are neither too low-level nor too abstract, which makes the instruction-relevant concepts most readable there. This choice is made on the ancestor alone.
  • Semantic-loss weights (λ_sem = 0.5, λ_sem-abs = 1.0, λ_vis-abs = 2.0): these balance the gradient sizes of the two loss terms on the ancestor, so that neither term dominates.
  • Activation threshold: this refers to the gate γ = 0.45 in our two-phase schedule, the only activation condition in the method. It decides when the semantic term is admitted, and it is read from the ancestor-side training curves (the point where visual disruption stabilizes), not from any victim signal.
  • Final patch: The patch is taken after the fixed 3500-iteration schedule.

Q2: Optimization–evaluation separation and cross-dataset transfer

We use a clean separation. Patches are optimized only on the first frame of each training demonstration (500 frames for LIBERO-Spatial). Evaluation then runs fresh rollouts (10 tasks × 50 trials) with randomized initial states; these rollouts are not the demonstration frames. So the patch never sees an evaluation rollout during optimization.

Two pieces of evidence already in the paper demonstrate that the attack breaks general perception rather than memorized frames. First, in Table 1, patches optimized on BridgeData V2 — a different dataset that contains no LIBERO frames at all — transfer to LIBERO with almost no loss. If the patch were memorizing LIBERO frames or tasks, this cross-dataset transfer would fail. Second, the patch is placed at a uniformly random location during both optimization and testing and is optimized over random geometric and photometric transforms, so it does not rely on any specific position or pose.

AuthorsOpenReview ↗

Response to Reviewer Cm1k (2/2)

Q3 : Probe relevance and semantic suppression

IGS suppresses the semantic evidence of the concepts an instruction refers to — the object, its attributes, and its spatial relations — in the ancestor VLM's intermediate visual tokens. The "probes" are exactly those concept words. To answer whether this needs the correct probes, we reran IGS with three probe sets on two victims: (1) the instruction-derived probes (Ours-IGS); (2) shuffled probes, which substitute another sample's object/attribute/relation — the same kind of words, but not the ones the current instruction refers to; and (3) irrelevant probes, which name plausible objects that are not present in the instruction.

Target VLAProbe setSpa. SRObj. SR
OpenVLAOurs-IGS49.6%43.2%
OpenVLAOurs-IGS with Shuffled probes51.2%43.4%
OpenVLAOurs-IGS with Irrelevant probes53.8%44.6%
UniVLAOurs-IGS56.8%56.2%
UniVLAOurs-IGS with Shuffled probes58.8%58.0%
UniVLAOurs-IGS with Irrelevant59.8%58.6%

Lower success rate means a stronger attack. The answer is yes: IGS with the correct probes is the strongest in all four victim–suite settings (all eight comparisons with the two probe replacements), and it keeps most of its advantage under both replacements. To further verify IGS's advantage on disrupting semantic grounding, we used OWLv2 (Minderer et al.), an off-the-shelf open-vocabulary detector with no role in optimizing our patches or in IGS's own scoring, on 200 LIBERO frames, querying its detection confidence for the object named in the instruction:

ConditionOWLv2 confidence (Spa.)OWLv2 confidence (Obj.)
Clean0.1180.073
Random Patch0.1180.045
Ours-IGS0.0840.039
Ours-IGS with Shuffled probes0.0860.045
Ours-IGS with Irrelevant probes0.0920.044

We can find that using the correct probes pushes the confidence lower than using shuffled or irrelevant probes, with the correct-probe scores below those for both clean images and random-patched images. Because OWLv2 never sees the instruction and is not our model, this is independent evidence that the instruction match is doing real, additional work, not just that any optimized perturbation on this channel would do the same thing.

Correct probes name the concept the current instruction points at and that is present in the frame, so they remove the most task-relevant evidence and give the strongest attack. Shuffled probes are not random strings: they substitute another sample's object/attribute/relation, so they are still real concepts from the same dataset, and in LIBERO's constrained scenes such a concept is often still plausible or present in the current frame. Suppressing its evidence therefore removes some but not all of the task-relevant signal, which is why shuffled probes are only slightly weaker. Irrelevant probes name objects that are absent from the scene, and suppressing evidence of an absent object removes much less that the action decision uses, which is why they weaken the attack more. They still beat a random patch because they are still optimized: a wrong probe misdirects the target, but the patch still corrupts the visual-token representation broadly, whereas a random patch is unoptimized noise. Although shuffled and irrelevant probes still beat a random patch, an attacker has no reason to use them: the correct probes come directly from the instruction and give the strongest result.

The new probe comparisons address my main concern about the semantic objective, so I’m raising my rating to weak accept. I’d still like the authors to give more insight about why transfer is weak on π0.5 in revised version.

We thank the reviewer for the positive assessment and the constructive follow-up. Regarding why π0.5 resists, our current evidence already indicates that representation drift is not the main cause, since even white-box attacks on π0.5 remain weak. The residual weakness is consistent with prior work showing that diffusion/flow-matching models can be more robust against adversarial attacks. We will analyze π0.5's resistance in depth in the revised version from both architectural factors (e.g., the flow-matching action head and the inference pipeline) and training factors (e.g., open-world co-training). We appreciate the suggestion, which will make the boundary-condition analysis more complete.

Reviewer_c2H7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a black-box adversarial attack method targeting VLA models. The paper observes that many VLA models are fine-tuned from publicly available VLMs, allowing attackers to construct adversarial patches using their publicly available ancestor VLMs without accessing the target VLA. Based on this observation, the paper proposes three attack methods, including disrupting visual features through Vision Disruption, suppressing semantic information related to task instructions through Instruction-Grounded Semantic Evidence Suppression, and combining the two attack objectives through joint optimization. The experiments are conducted on OpenVLA, UniVLA, and π0.5, and the results show that adversarial patches optimized on ancestor VLMs can significantly reduce the task success rates of OpenVLA and UniVLA, but have limited effectiveness against π0.5. The paper further analyzes the impact of model architecture and ancestor relationships on attack transferability and discusses the potential boundary conditions of vulnerability inheritance.

Critical strengths
  1. The ancestor-based threat model proposed in this paper is novel and has practical significance.

  2. The experimental results demonstrate a relationship between architectural similarity to the ancestor VLM and attack transferability. These results provide preliminary evidence for understanding how architectural differences among VLA models relate to their adversarial vulnerabilities.

Critical weaknesses
  1. The paper lacks an in-depth analysis of the relationship between the extent of fine-tuning and vulnerability inheritance. Although the authors argue that the extent to which different VLAs retain representations from their ancestor VLMs affects attack effectiveness, there are no quantitative measures of intermediate-layer feature drift or representation similarity, nor is there an analysis of how attack effectiveness varies with the extent of fine-tuning. In particular, in the analysis of π0.5, multiple factors, such as the model's ancestor architecture, training procedure, and action generation mechanism are different. The authors attribute the attack failure to representation changes and the smoothing effect of flow matching, but do not isolate the effects of these factors through controlled experiments. Therefore, the current analysis is insufficient to determine the specific contributions of different factors to attack effectiveness, making it difficult to predict the vulnerability of VLAs with different degrees of fine-tuning to ancestor-based attacks.

  2. The limited number of ancestor VLM families evaluated in the paper limits the generalizability of the conclusions. OpenVLA and UniVLA are both derived from Prismatic, so the main successful attack cases are actually concentrated within the same ancestor family, while the attack effectiveness on another family is close to that of random patches. Additional VLAs may need to be included in the experiments to determine whether the attack method can be effectively extended to other VLA models with different backbones and training procedures.

  3. The attack proposed in the paper is highly dependent on the extent to which the target VLA retains representations from its ancestor VLM, which may make its applicability more limited than that of other black-box attacks. This method not only requires attackers to know and have access to the correct ancestor VLM, but also relies on the target VLA retaining sufficiently similar representations after fine-tuning. Compared to other black-box attacks that do not rely on specific ancestor models, these additional requirements may significantly limit its applicability in practical deployment scenarios.

Questions for authors
  1. How does fine-tuning affect vulnerability inheritance, and what are the respective contributions of representation changes and flow matching to attack failure in π0.5?

  2. How well does the proposed attack generalize to VLAs derived from different ancestor VLM families?

  3. How does the applicability of this method compare to other black-box attacks when the ancestor VLM is unknown or the target VLA has undergone extensive fine-tuning?

3 comments
AuthorsOpenReview ↗

Response to Reviewer c2H7 (1/2)

We thank the reviewer for recognizing the novelty and practical significance of the ancestor-only threat model, and for highlighting the evidence connecting ancestor similarity with attack transferability. We have added new experiments and clarifications to address the concerns raised.

Q1 & W1: Representation retention and π0.5 resistance

Part 1: how does the degree of fine-tuning affect vulnerability inheritance?

The answer is straightforward, and is what one would expect: the more fine-tuning shifts a VLA's representation away from its ancestor, the weaker the inherited vulnerability. The same principle already shows up in Table 2 on the attacker's side, where optimizing on surrogate VLMs progressively farther from the victim's true ancestor yields progressively weaker attacks (Ours-joint 33.2% with Prismatic-7B, 43.8% with LLaVA-1.5-7B, 48.7% with PaliGemma-3B, versus 49.0% for a random patch). To further confirm this, we measure representation drift caused by finetuning between each ancestor VLM and its derived VLA's visual-language trunk on the same LIBERO frames, using CKA. We compare it to our attack's effect in absolute percentage points:

Task SuiteAncestor → victimCKANo-attack SROurs-VIS drop (pp)Ours-IGS drop (pp)Ours-joint drop (pp)
SpatialPrismatic-7B → OpenVLA0.5879.4%37.431.440.4
SpatialPrismatic-7B → UniVLA0.6288.8%45.030.646.8
SpatialPaliGemma-3B → π0.50.2098.4%3.22.62.6
ObjectPrismatic-7B → OpenVLA0.54268.8%33.422.634.6
ObjectPrismatic-7B → UniVLA0.56998.1%45.543.347.9
ObjectPaliGemma-3B → π0.50.1598.2%1.41.61.0

Drift is smallest for the two Prismatic-derived VLAs, which our attack degrades the most, and largest for π0.5, which is nearly unaffected. Tying drift to the attack effect, there is a clear trend that less drift goes along with a stronger attack impact, which is exactly as expected.


Part 2: disentangling the two candidate causes (i.e., representation changes and flow matching) on π0.5.

We first note that these two factors can be disentangled but the flow matching factor cannot be fully isolated since there is no clean way to replace π0.5's flow-matching head with a non-flow-matching one while holding everything else fixed. We therefore measure the representation drift's contribution directly and characterize what remains.

To remove representation drift, we attacked π0.5 itself in a white-box setting (no transfer from the ancestor), using our attacks and UADA (a representative white-box baseline) — on LIBERO-Spatial:

SettingAttackerNo-attack SRAttacked SRAbs. drop (pp)
White-box (no drift)UADA98.4%93.2%5.2
White-box (no drift)Ours-VIS98.4%88.8%9.6
White-box (no drift)Ours-IGS98.4%91.2%7.2
White-box (no drift)Ours-Joint98.4%94.8%3.6
Ancestor-onlyOurs-VIS98.4%95.2%3.2
Ancestor-onlyOurs-IGS98.4%95.8%2.6
Ancestor-onlyOurs-Joint98.4%95.8%2.6

Even with representation drift removed, these attacks remain weak, so representation drift is not the main cause of π0.5's resistance. This residual weakness is consistent with prior work showing that diffusion/flow-matching models can be more robust against adversarial attacks (DensePure, ICLR 2023; Nie W et al., DiffPure, ICML 2022; FlowPure, arXiv:2505.13280), and we attribute the residual resistance most plausibly to the flow-matching action head. To quantify the representation drift, we compare against our ancestor-only attacks. The drift cost is the abs. drop difference between the two settings. So drift accounts for only 1.0–6.4 percentage points of π0.5's resistance.

[1] Xiao C, Chen Z, Jin K, et al. DensePure: Understanding Diffusion Models for Adversarial Robustness. ICLR 2023.

[2] Nie W, Guo B, Huang Y, et al. Diffusion models for adversarial purification. ICML, 2022.

[3] "FlowPure: Continuous Normalizing Flows for Adversarial Purification." arXiv:2505.13280.

AuthorsOpenReview ↗

Response to Reviewer c2H7 (2/2)

Q2 & W2: Generalization across ancestor families

We agree our positive results concentrate on the Prismatic family, and we want to be precise about what we claim rather than overreach. The reason we could not evaluate more families is practical: there are essentially no other open-source ancestor VLMs with released LIBERO-fine-tuned VLAs. To broaden the evaluation within the PaliGemma family, we additionally tested ancestor-only patch transfer to π0, complementing the π0.5 results in the paper.

This paper explores a basic question: can an adversary attack a deployed VLA using only its public ancestor VLM? Our hypothesis is that, when a VLA retains representations supporting its ancestor's visual perception and semantic capabilities, it may also retain vulnerabilities accessible through those representations. This motivates representation retention as a factor influencing vulnerability inheritance, rather than restricting the attack to a particular ancestor family. Our new measurements provide evidence consistent with this hypothesis. Prismatic→OpenVLA and Prismatic→UniVLA exhibit higher representation similarity and substantial attack effects, whereas PaliGemma→π0.5 exhibits lower similarity and weak transfer (Part 1 of Q1&W1). The additional π0 experiment provides a comparison within the same PaliGemma family. In this supplementary evaluation, π0 has higher ancestor–policy CKA than π0.5 (0.329 versus 0.198). Applying the same ancestor-optimized Joint patches reduces π0's task success rate from 94.0% to 88.8%, compared with 98.4% to 96.2% for π0.5: absolute reductions of 5.2 and 2.2 percentage points, respectively. The surrogate-side comparison in Table 2 provides complementary evidence that the relationship between the optimization model and the victim matters: Joint achieves 33.2% SR with Prismatic, 43.8% with LLaVA, and 48.7% with PaliGemma against the same OpenVLA victim. The contribution is therefore general in the following sense: the public ancestor VLM constitutes a shared supply-chain attack surface. Our results help characterize how representation retention and downstream readout affect the exposure of derived VLAs.

Q3 & W3: Unknown ancestry and attack applicability

We'd like to reframe this comparison: relative to other black-box attacks, our approach already requires strictly less information. Prior black-box attacks on VLAs (EDPA, UPA-RFAS, VLA-Hijack) need access to the VLA encoder or a surrogate VLA, which is almost never publicly available for a deployed robot. We replace that requirement with knowledge of the public ancestor VLM, which is normally disclosed: OpenVLA, UniVLA, and π0.5 all state their base VLM in their own papers and model cards, because this is standard practice for releasing a fine-tuned foundation model. So know the ancestor is not a strict or unusual assumption, whereas obtain a surrogate VLA is the harder requirement that prior black-box attacks could not avoid. Table 2 already shows that our attacks still work even when the correct ancestor is unknown: using a related but incorrect VLM ancestor, LLaVA-1.5-7B (shares only the language backbone with OpenVLA), our attack still beats the random patch. To further demonstrate applicability compared to other black-box attacks when the ancestor is unknown, we directly compare our attacks with EDPA (a representative prior black-box attack, already a baseline in Table 1) in the harder case the reviewer raises. We optimized EDPA on the same two surrogate VLMs — LLaVA-1.5-7B and PaliGemma-3B — using EDPA's own patch-contrastive and image-text alignment losses, and evaluated the resulting patches on OpenVLA under the identical LIBERO protocol.

Surrogate → OpenVLASpa.Obj.Goa.Lon.Avg
EDPA on LLaVA-1.5-7B58.048.859.623.847.55
EDPA on PaliGemma-3B64.650.462.223.450.15
Ours-joint on LLaVA-1.5-7B54.844.556.819.043.78
Ours-joint on PaliGemma-3B62.049.961.022.048.73

Our method stays ahead of EDPA with the direct-ancestor and partially matched surrogates, and with the architecturally unrelated PaliGemma surrogate it is at least not worse (48.73% vs 50.15%), where both sit near the random-patch floor. So even in the reviewer's harder setting — attacking with an imperfect or wrong guess at the ancestor — our mechanisms could extract more damage than a representative existing black-box method does. Finally, a broader point: any attack degrades as the attacker's information shrinks. This is not specific to our method and is true of every black-box attack in this literature. Our contribution is not the strongest attack under every condition but the first attack that works with the least access of any prior method, plus boundary conditions that tell both attackers and defenders exactly when this risk is live.

Thank you for the response. Most of my concerns have been addressed, and I have raised my score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.