- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper shows that adversarial patches added directly to images can overestimate their real-world impact on VLA models. Accounting for physical geometry weakens existing attacks, but re-optimizing patches under these constraints makes them stronger again. Real-robot experiments confirm that printed patches can still disrupt robot behavior, while geometry-consistent adversarial training can improve robustness.
Critical strengths
-
A meaningful problem & better evaluation protocol. Separating patch generation from evaluation under physical constraints clarifies how deployment assumptions affect measured VLA vulnerability.
-
Consistent empirical evidence. The deployment gap appears in all 24 tested combinations, while re-optimization strengthens attacks in 23, providing strong support for the central finding across different models and tasks.
-
Practical relevance for robot robustness. Printed-patch attacks on a real robot and improved robustness through geometry-consistent adversarial training demonstrate concrete implications for both vulnerability assessment and defense.
Critical weaknesses
-
Scope for deeper methodological insight. The paper addresses a meaningful and practically relevant problem, although its approach largely builds on existing attack objectives and geometric projection. Further analysis of the specific roles of temporal and multi-view consistency would help clarify the additional insights contributed to VLA robustness.
-
Limited benchmark diversity. Although the experimental evaluation is extensive, covering 24 model–attack–suite combinations, the simulation experiments are confined to LIBERO. Evaluation on additional benchmarks would strengthen confidence in the generality of the findings across different tasks and environments.
Questions for authors
-
LIBERO-Long robustness. Could the authors report clean performance before and after AT to help explain the low libero-long success rates in Table A7?
-
LoRA-rank sensitivity. Are the large performance differences across ranks in Table A5 consistent across runs, and was Pixel AT tuned comparably?
-
Physical evaluation. Could the authors explain the difference between Tables 4–5 (16.7% vs. 10.0%), report per-scene clean performance in Table 7, and clarify whether patches were re-optimized for each defense?
3 comments
Thank you for the clarifications and extra VLABench exp. Meaningful topic, I will maintain my positive recommendation.
Response to reviewer Zzxi's reply
Dear Reviewer Zzxi,
Thank you again for your timely feedback and your efforts in reviewing our rebuttal and additional experiments. We sincerely appreciate your recognition of the meaningful topic and your continued positive recommendation. If you have any additional questions, we are glad to further answer them.
Thank you for the valuable review
Thank you for the thoughtful review. We appreciate your recognition of the evaluation protocol, consistent empirical evidence, and practical implications. Below we clarify the existing evidence on temporal and multi-view consistency, rank sensitivity, and adaptive physical evaluation.
W1: Further analysis of temporal and multi-view consistency.
Thanks for your valuable suggestion. We have examined the role of multi-view consistency in Table 2. A physical patch must use one texture across camera views, so a texture optimized for one view may be less effective when observed from another. Under the same geometry-consistent evaluation, joint optimization reduces OpenVLA-OFT task success from 31.6% for either single-view texture to 25.2%. On π₀, success decreases from 85.6–86.4% to 72.0%. These results show the benefit of optimizing the shared texture through both camera projections.
For temporal consistency, the patch stays at a fixed world location, while its appearance changes with the camera during robot execution. This matters because the robot’s actions affect subsequent observations. We examine the resulting behavior in Section 4.4 through paired clean and attacked rollouts. Geometry-consistent EDPA increases the mean end-effector deviation from 5.74 cm to 6.56 cm and causes earlier divergence from clean execution. These results show that the attack affects the robot throughout execution, beyond the final task outcome. We will make these insights clearer in the discussion of Table 2 and Figure 3.
W2: Limited benchmark diversity.
Thanks for your valuable suggestion. We have added experiments with π0 on five VLABench tasks in Appendix B (Table A1), using a VLABench-finetuned checkpoint and demonstrations from this benchmark to optimize new patches jointly across the five tasks. Each task is evaluated with 50 rollouts under clean, P/P, P/G, and G/G settings.
On average, transferring pixel-generated patches from P/P to P/G increases task success from 38.0% to 42.8%, while re-optimization under G reduces it to 32.4%. These results support the overall deployment-gap and re-optimization trends beyond LIBERO. We have also added a reference to these experiments in Section 4.1.
Q1: Could the authors report clean performance before and after AT to help explain the low LIBERO-Long success rates in Table A7?
Thanks for your valuable suggestion. We evaluated the three OpenVLA-OFT models on LIBERO-Long without patches, using 500 episodes per model. The clean success rates are reported below.
Q2: Are the large performance differences across ranks in Table A5 consistent across runs, and was Pixel AT tuned comparably?
Thanks for raising this question. We used rank 8 for both Pixel AT and Geometry AT from the outset, without a rank search for either method. The other ranks in Table A8 were evaluated afterward to examine sensitivity. Each configuration used one randomly selected training seed. We will clarify this experimental procedure.
Following your suggestion, we additionally trained Pixel AT with rank 16 for 100,000 steps and re-optimized a geometry-consistent EDPA patch against this model. We evaluated it over 500 episodes on LIBERO-Spatial, obtaining 81.0% clean success and 38.0% success under attack. The comparison with the existing results is shown below, higher success indicates stronger robustness.
Geometry AT outperforms Pixel AT at both tested ranks, by 17.4 and 8.8 percentage points, respectively. This additional comparison supports the benefit of geometry-consistent training beyond rank 8.
Q3: Physical evaluation.
Thanks for raising these questions. The 16.7% in Table 4 and 10.0% in Table 5 correspond to different patch-placement settings: Table 4 uses random placement, whereas the default condition in Table 5 fixes the patch on the right side of the workspace. We will clarify this distinction in the captions.
We also report the clean success rates (%) for both scenes below and will include them in Table 7. Each unseen-scene result is based on 30 trials.
Finally, the patches were re-optimized separately against each evaluated model. The defense results in Tables 4 and 7 therefore use adaptive attacks, rather than patches generated against the vanilla model alone. We will state this explicitly in the experimental description.