acceptodds
Under review as a conference paper at ICLR 2027

Across Viewpoints and Time: Trajectory-Generalizable Physical Attacks on World Models and Video Understanding Models

Abstract

World models and video understanding models are increasingly deployed in dynamic visual environments, yet their robustness to physical adversarial threats remains largely underexplored. We study whether a single physical adversarial texture can mislead the victim model across different real-world scenarios where observing viewpoints vary along uncontrolled object–camera trajectories. We identify that heterogeneous video semantics and trajectories induce poorly aligned gradients for optimizing the adversarial texture. To address that, we propose the Trajectory-aware adversarial Alignment and Refinement Attack (TARA), which incorporates trajectory-aware optimization to capture the evolving geometric variations of a printed texture under real-world scenarios. To improve cross-instance optimization compatibility, TARA incorporates trajectory-aware contrastive alignment to encourage attacked representations from diverse video–trajectory instances to approach a shared adversarial region and then performs trajectory-aware task refinement to translate the aligned representation into task-level prediction errors. For reproducible evaluation, we develop a geometry-grounded tracking pipeline that provides physically plausible adversarial regions with temporally consistent trajectories on EPIC-KITCHENS-100 and Something-Something V2. We further collect real-world print-and-recapture videos to validate attack effectiveness under physical deployment conditions. Together, the annotated benchmarks and real-world captured data form a reproducible testbed for future evaluation. Notably, our results reveal that a single printable adversarial texture can persistently disrupt world models and video understanding models under physical conditions, highlighting the fundamental security challenge.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
World model robustness, Physical adversarial attack

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qxAfOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TARA, a physical adversarial attack method for world models and video understanding models. TARA combines trajectory-aware optimization, contrastive alignment, and task refinement to make a single adversarial texture effective across different videos, viewpoints, and object-camera trajectories. The authors conduct experiments on EK100, SSV2, and real-world print-and-recapture data, demonstrating the effectiveness of TARA on action recognition, future action anticipation, and future frame generation tasks.

Critical strengths
  1. The paper studies physical adversarial attacks in dynamic environments, which is a meaningful and practical problem. The use of trajectory-aware optimization provides a certain degree of novelty.
  2. The experiments are comprehensive, covering multiple models, tasks, and datasets, including real-world physical attack experiments.
Critical weaknesses
  1. The texture attachment regions are manually selected. It is unclear whether different attachment locations would significantly affect the attack performance.
  2. The real-world physical evaluation is still somewhat limited. Compared with EOT, TARA shows relatively modest improvements, and the physical print-and-recapture experiments are conducted on only a subset of the evaluated models.
Questions for authors
  1. How exactly are the texture attachment regions selected? Would changing the attachment locations significantly affect the attack performance?
  2. Why do all experiments use circular adversarial textures? Would the proposed method remain effective with square or other texture shapes?
  3. Are the improvements of TARA over EOT in the real-world physical experiments statistically significant? Could the authors provide more evidence of the stability of these improvements?
4 comments
AuthorsOpenReview ↗

Response regarding to Weakness 1, Q1, and Q2

Q1. The texture attachment regions are manually selected. It is unclear whether different attachment locations would significantly affect the attack performance. How exactly are the texture attachment regions selected? Would changing the attachment locations significantly affect the attack performance?


Thank you for the constructive question. We clarify the selection criteria and provide additional experiments to examine the influence of attachment location on attack performance.

1. Texture attachment selection. The initial attachment regions are manually selected based on physical feasibility, requiring an approximately planar surface with sufficient space to accommodate the circular texture (Sec. 3.2).

2. Effect of the attachment location. To investigate the effect of attachment location on attack performance, we selected 74 annotated EK100 clips that are correctly classified by the V-JEPA2-L. In addition to their original texture locations, we manually select two alternative attachment locations for each clip, which have a mean center displacement of 32.1±30.332.1 \pm 30.3 pixels from the original locations. For each alternative location, we re-track the selected region to estimate the corresponding homographies and visibility matrices across frames. We evaluate three adversarial textures, optimized by TARA, EOT, and PhyCamo, along the new locations without location-specific re-optimization. Across the three attachment locations, TARA achieves an average Top-1 ASR of 38.7±1.638.7 \pm 1.6%, compared with 32.4±0.032.4 \pm 0.0% for EOT and 18.5±1.618.5 \pm 1.6% for PhyCamo. These results suggest that attachment location variations have limited impact on attack performance, and the optimized texture remains robust to location changes. We thank the reviewer for the constructive feedback, and we have added these results and discussion in Appendix A.6, page 18 of our uploaded revised manuscript.

Q2: Why do all experiments use circular adversarial textures? Would the proposed method remain effective with square or other texture shapes?


Thank you for your feedback; please find our response below.

1. Motivation for using circular adversarial textures. We initially adopted circular textures as an empirical design choice motivated by their rotational symmetry. As the viewing angle changes along a trajectory, the circular shape remains invariant to in-plane rotations, reducing shape-induced orientation dependence. To further investigate the influence of texture shape, we conduct additional experiments using square textures under comparable experimental conditions.

2. Performance with square textures. We replace the original 48×4848\times48 circular texture with a 42×4242\times42 square texture to maintain a comparable texture area. The square covers 1,764 pixels, closely matching the 1,810 pixels covered by the circular texture. We maintain the same texture center and reuse the recorded trajectories and homographies, keeping the same texture placement and motion consistent across both shapes. We evaluate EOT, PhyCamo, and TARA using square textures on V-JEPA2-L and AVT-B for EK100, and V-JEPA2-L and VideoMAE-B for SSV2. The results are shown in the table below.

Table. Comparison of circular and square textures on EK100 action anticipation and SSV2 action recognition. Top-1 ASR (%, ↑) is reported. Each entry denotes Train/Val.

SettingEK100 V-JEPA2-LEK100 AVT-BSSV2 V-JEPA2-LSSV2 VideoMAE-BOverall
EOT-circular30.3/25.132.9/25.025.5/30.614.9/16.925.9/24.4
EOT-square27.7/22.334.1/28.322.3/25.013.7/17.724.5/23.3
PhyCamo-circular13.9/13.742.4/30.017.9/12.515.4/16.922.4/18.3
PhyCamo-square11.7/12.840.0/30.014.1/7.614.9/17.720.2/17.0
TARA-circular37.7/26.137.6/28.338.6/34.717.1/17.732.8/26.7
TARA-square35.1/21.836.5/28.329.3/29.917.7/18.529.6/24.6

Overall, square textures achieve comparable attack performance to circular textures, with a slight average ASR drop of approximately 1–3 percentage points across the evaluated methods. These findings suggest the feasibility of exploring alternative texture shapes within the proposed attack framework.

We thank the reviewer for the constructive feedback, and we have added these results and discussion in Appendix A.5, pages 17-18 of our uploaded revised manuscript.**

AuthorsOpenReview ↗

Response regarding to Weakness 2 and Q3

Q3. The real-world physical evaluation is still somewhat limited. Compared with EOT, TARA shows relatively modest improvements, and the physical print-and-recapture experiments are conducted on only a subset of the evaluated models. Could the authors provide more evidence of the stability of these improvements?


Thank you for your feedback; please find our response below.

We conduct additional print-and-recapture experiments on V-JEPA2-L for action anticipation and recognition, and InternVideo2 for action recognition to further evaluate the effectiveness of TARA in physical settings. We also include PhyCamo as an additional competitive baseline. For each model-task setting, we select 100 action events and record four video clips per event: one reference clip with a white texture and three attacked clips with printed textures generated by TARA, EOT, and PhyCamo, respectively, yielding a total of 1,200 video clips across the three settings. For each event, all printed textures are placed at the same scene location, and the same action is repeated under consistent recording conditions to ensure a fair comparison.

Table: Additional physical print-and-recapture results. We evaluate 100 events per model-task setting and report Top-1 ASR (%), computed over events correctly classified on white reference clips.

\hline \text{Tasks} & \text{Anticipation} & \text{Recognition} & \text{Recognition} & \text{Overall} \\\\ \text{Models} & \text{V-JEPA2-L} & \text{V-JEPA2-L} & \text{InternVideo2} & \\\\ \hline \text{Physical EOT} & 35.8 & 20.8 & 32.3 & 29.6 \\\\ \text{Physical PhyCamo} & 30.2 & 25.0 & 32.3 & 29.2 \\\\ \hline \mathbf{Physical\\ TARA} & \mathbf{45.3} & \mathbf{31.3} & \mathbf{41.5} & \mathbf{39.4} \\\\ \hline \end{array}$$ As shown in the above Table. TARA consistently achieves the highest Top-1 ASR across all three settings. Overall, TARA improves averaged ASR over EOT and PhyCamo by **9.7 and 10.2 percentage points**, respectively. These results further support the effectiveness of TARA in physical settings, with consistent improvements across the evaluated models and tasks. **We thank the reviewer for the constructive feedback, and we have added those results and discussion in Appendix, A.3, page 15-16 of our uploaded revised manuscript.**

Thanks for the detailed response. The authors have addressed all my concerns. I will raise my score and recommend acceptance.

We sincerely thank the reviewer for the positive assessment and constructive feedback throughout the review process. We are especially grateful for your willingness to raise the score and recommend acceptance.

Reviewer_iiE5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies physical adversarial attacks against video understanding models and world models, focusing on generalization across heterogeneous video content and unseen object-camera trajectories. The authors propose TARA, which combines trajectory-aware contrastive alignment and task refinement to improve the effectiveness of a shared printable adversarial texture. They also construct a geometry-grounded trajectory benchmark and conduct real-world print-and-recapture experiments. Experimental results show improvements over existing baselines on action anticipation, action recognition, and future-frame generation tasks.

Critical strengths
  1. Practical problem setting. The paper investigates physical adversarial attacks under dynamic viewpoints, occlusion, and unseen trajectories, which is relevant to evaluating the robustness of video models in real-world environments.

  2. The experiments propose a new benchmark for evaluating physical robustness and adversarial methods.

Critical weaknesses
  1. The idea of improving the initialization with model-related features is not new for optimizing adversarial patches. See [1]. What we need is a more detailed study to reveal its mechanism. However, the theoretical & empirical justification of the contrastive alignment method is limited. See questions 1 - 3.

  2. Limited evaluation of physical robustness and transferability. Since the main supposed contribution of the Trajectory-Conditioned Task Refinement is in the modeling of the real-world deployment, physical-world experiments are important. However, the physical print results are mainly reported for V-JEPA2-L and compare only EOT with TARA. More concerns are in question 4.

[1] Universal Adversarial Patch Attack for Automatic Checkout Using Perceptual and Attentional Bias. TIP 2021.

Questions for authors
  1. In the ablation study, what is the performance of CA compared with other initialization methods? Could the statistical significance be reported? Since initialization matters, the randomness in the optimization should be considered for both baselines and the proposed methods. Moreover, the detailed settings in the ablation study should be clarified.

  2. The paper attributes the contribution of initialization to better optimization in the latter stage. However, in the ablation study, without CA achieves better training performance compared to with CA in attacking AVT-B.

  3. The optimization schedule uses Adam in the CA stage and sign-PGD in the latter stage for no particular reason. Do the advantages of CA still hold with other optimizers, other hyperparameters, or longer optimization steps?

  4. Since TARA eliminates the randomness in the adversarial optimization by reducing the space of physical environments to Trajectory-Conditioned, it may harm generalization and transferability, while additional randomness may improve them. In the ablation study, the val ASR is already tied for AVT-B, while the train ASR is better for TR, indicating overfitting. How does the proposed method's transferability compare to other baselines, and how well does it generalize to more diverse physical environments?

6 comments
AuthorsOpenReview ↗

Response Regarding weaknesses 1 and comparison with other initialization method in Q1.

W1: The idea of improving the initialization with model-related features is not new for optimizing adversarial patches. See [1]. What we need is a more detailed study to reveal its mechanism. However, the theoretical & empirical justification of the contrastive alignment method is limited.


We appreciate the reviewer’s constructive feedback and agree that leveraging model-related features for adversarial patch initialization is not new, as explored in [1]. However, our contrastive alignment (CA) aims to mitigate conflicting optimization gradients across heterogeneous video—trajectory when optimizing a shared texture. To verify this motivation, we compared CA with direct optimization in Figure 3 and section 3,1. Figure 3 (a) shows that CA increases the proportion of beneficial cross-instance updates from 60.8% to 72.9%, based on statistics aggregated across training clips and optimization updates. Figure 3(b) further shows improved pairwise gradient similarities across training batches and optimization steps, suggesting better gradient compatibility. These results support CA's contribution in improving cross-instance optimization compatibility.

Q1: In the ablation study, what is the performance of CA compared with other initialization methods?


Thank you for your feedback; please find our response below.

Table. Comparison of initialization strategies on EK100. Top-1 ASR (%, ↑).

\hline \text{Initialization} & \text{TR epochs} & \text{Train} & \text{Val} & \text{Avg.} \\\\ \hline \text{Gaussian random} & 5 & 14.7 & 10.9 & 12.8 \\\\ & 10 & 16.0 & 10.9 & 13.5 \\\\ \hline \text{Perceptual prior (120 updates)} & 5 & 12.6 & 9.5 & 11.0 \\\\ \text{Perceptual prior (240 updates)} & 10 & 31.2 & 21.8 & 26.5 \\\\ \hline \text{ODI }(3\times4\text{ sweeps}) & 5 & 13.9 & 12.3 & 13.1 \\\\ & 10 & 16.5 & 12.3 & 14.4 \\\\ \hline \text{CA (120 updates)} & 5 & \underline{34.6} & \underline{24.6} & \underline{29.6} \\\\ \text{CA (240 updates)} & 5 & \mathbf{37.7} & \mathbf{26.1} & \mathbf{31.9} \\\\ \hline \end{array}$$ **The performance of CA compared with other initialization methods.** We compare CA with perceptual-prior initialization [1], output-diversified initialization (ODI) [2], and Gaussian random initialization on EK100 using V-JEPA2-L. The perceptual prior constructs an informative initial texture by matching feature statistics from challenging training examples while increasing prediction uncertainty. We adapt this approach to video inputs and use Grad-CAM to identify salient regions for texture extraction. ODI diversifies initialization by optimizing a shared texture along randomly sampled directions in the model's output space. As shown in the table, CA achieves the highest ASR among all initialization strategies. With 240 initialization updates and 5 trajectory-aware refinement (TR) epochs, CA achieves 31.9\% average ASR, **surpassing the strongest baseline, perceptual prior with 240 updates and 10 TR epochs, with an average ASR improvement of 5.4 points. We thank the reviewer for the constructive feedback, and we have added the results and discussion in Section 5.4 in page 9. More implementation details for ODI and perceptual-prior initialization are provided in Appendix A.12, page 23.** [1] Universal adversarial patch attack for automatic checkout using perceptual and attentional bias, in TIP 2021. [2] Bias-based universal adversarial patch attack for automatic check-out, in NeurIPS 2020.
AuthorsOpenReview ↗

Response regarding statistical significance, and ablation details in Q1.

Q1: Could the statistical significance be reported? Since initialization matters, the randomness in the optimization should be considered for both baselines and the proposed methods. Moreover, the detailed settings in the ablation study should be clarified.


1. Statistical significance

Following your good suggestion, we evaluate the statistical stability of TARA and two competitive baselines, EOT and PhyCamo, across three random seeds (0, 1, and 2) on EK100 and SSV2. The results are shown in Tables below, with the mean and standard deviation of Top-1 ASR reported.

Table. Statistical stability evaluation on EK100 future action anticipation. Top-1 ASR (%) is reported as mean ± standard deviation over three seeds.

\text{Method} & \text{V-JEPA2-L} & & \text{V-JEPA2.1-B} & & \text{AVT-B} & & \text{Overall} & \\\\ & \text{Train} & \text{Val} & \text{Train} & \text{Val} & \text{Train} & \text{Val} & \text{Avg. Train} & \text{Avg. Val} \\\\ \hline \text{EOT} & \underline{31.3}\pm1.1 & \underline{24.5}\pm0.7 & \underline{56.0}\pm1.1 & \underline{53.3}\pm6.8 & 32.5\pm0.7 & 24.4\pm2.5 & \underline{40.0}\pm0.6 & \underline{34.1}\pm3.0 \\\\ \text{PhyCamo} & 13.9\pm0.0 & 14.2\pm0.8 & 53.5\pm4.6 & 26.9\pm2.1 & \mathbf{40.8}\pm2.7 & \underline{29.4}\pm1.0 & 36.1\pm2.3 & 23.5\pm0.7 \\\\ \hline \mathbf{TARA} & \mathbf{37.5}\pm0.6 & \mathbf{25.6}\pm0.9 & \mathbf{79.1}\pm2.2 & \mathbf{61.2}\pm5.4 & \underline{36.9}\pm0.7 & \mathbf{30.0}\pm1.7 & \mathbf{51.1}\pm1.0 & \mathbf{38.9}\pm1.5 \end{array}$$ **Table.** Statistical stability evaluation on SSV2 action recognition. Top-1 ASR (%) is reported as mean ± standard deviation over three seeds. $$\begin{array}{c|cc|cc|cc|cc} \text{Method} & \text{V-JEPA2-L} & & \text{InternVideo2} & & \text{VideoMAE-B} & & \text{Overall} & \\\\ & \text{Train} & \text{Val} & \text{Train} & \text{Val} & \text{Train} & \text{Val} & \text{Avg. Train} & \text{Avg. Val} \\\\ \hline \text{EOT} & \underline{26.6}\pm0.9 & \underline{31.7}\pm1.1 & 11.7\pm3.6 & 11.0\pm2.7 & 14.5\pm0.7 & \underline{16.9}\pm0.8 & 17.6\pm1.1 & \underline{19.9}\pm0.8 \\\\ \text{PhyCamo} & 17.4\pm0.5 & 13.2\pm0.7 & \underline{21.7}\pm2.5 & \underline{22.1}\pm1.5 & \underline{15.0}\pm0.3 & \underline{16.9}\pm0.0 & \underline{18.0}\pm1.1 & 17.4\pm0.4 \\\\ \hline \mathbf{TARA} & \mathbf{38.0}\pm0.9 & \mathbf{34.2}\pm0.4 & \mathbf{41.1}\pm0.3 & \mathbf{25.7}\pm0.0 & \mathbf{18.9}\pm1.1 & \mathbf{17.7}\pm0.8 & \mathbf{32.6}\pm0.2 & \mathbf{25.9}\pm0.3 \end{array}$$ The results show that TARA achieves the highest mean ASR in 5 out of 6 attackexd models in both training and validation sets, demonstrating a consistent overall advantage over the competing methods. Specifically, over three seeds, TARA improves the average ASR over the strongest baseline by **11.1/4.8 percentage points on EK100 and 14.6/6.0 percentage points on SSV2**, for the training/validation splits, respectively. Moreover, TARA exhibits limited variation, with standard deviations of its overall average ASR remaining within 1.5 percentage points across both datasets. **We thank the reviewer for the constructive feedback, and the corresponding results, reported as mean ± standard deviation, have been added to Section 4.4, page 9 in our uploaded revised manuscript.** **2. Ablation details.** We evaluate the contributions of TA, CA, and TR on EK100 using V-JEPA2-L, V-JEPA2.1-B, and AVT-B with random seed $0$. Each setting optimizes a shared $48\times48$ circular RGB texture, with Top-1 ASR reported on the full training and validation sets. 1. For experiments *Without TA*, the texture is placed at the image center throughout training, without using trajectory-aware geometric transformations and visibility variations to render adversarial frames. We perform CA for 10 epochs followed by TR for 5 epochs, using the same learning rate schedule as full TARA. 2. For experiments *Without CA*, we directly optimize the texture from zero initialization using trajectory-aware task refinement for 15 epochs, using sign-PGD with cosine learning rate decay from $0.03$ to $0.005$. 3. For experiments *Without TR*, we perform CA for 15 epochs and directly evaluate the resulting texture without task refinement, using sign-PGD with cosine learning rate decay from $0.03$ to $0.005$. **We thank the reviewer for the helpful suggestion, and we have added a detailed description of the ablation study setup in Appendix A.10 (page 22) of the uploaded revised manuscript.**
AuthorsOpenReview ↗

Response regarding Q2 and Q3.

Q2: The paper attributes the contribution of initialization to better optimization in the latter stage. However, in the ablation study, without CA achieves better training performance compared to with CA in attacking AVT-B.


We appreciate the reviewer for the constructive feedback. Removing CA indeed achieves slightly higher ASR on the training set of AVT-B (40.0% vs. 37.6%). However, our primary goal is to improve generalization across unseen video–trajectory instances. As shown in Table 6, incorporating CA improves the overall validation ASR across three models from 36.4% to 41.1%. These results support the contribution of CA to improved generalization and overall attack effectiveness.

Q3: The optimization schedule uses Adam in the CA stage and sign-PGD in the latter stage for no particular reason. Do the advantages of CA still hold with other optimizers, other hyperparameters, or longer optimization steps?


Thank you for your feedback; please find our response below.

Table. Hyperparameter analysis of CA on EK100. Top-1 ASR (%) is reported.

\hline & & \text{V-JEPA2-L} & & \text{AVT-B} & & \text{Average} & \\\\ \text{Ablation} & \text{Setting} & \text{Train} & \text{Val} & \text{Train} & \text{Val} & \text{Train} & \text{Val} \\\\ \hline \text{CA/TR optimizer} & \text{sign-PGD/sign-PGD} & 37.7 & 26.5 & 37.6 & 30.0 & 37.7 & 28.3 \\\\ & \text{Adam/sign-PGD (default)} & 37.7 & 26.1 & 37.6 & 30.0 & 37.7 & 28.1 \\\\ & \text{Adam/Adam} & 34.2 & 25.6 & 36.5 & 28.3 & 35.3 & 27.0 \\\\ \hline \text{CA epochs} & 5 & 34.6 & 24.6 & 35.3 & 25.0 & 35.0 & 24.8 \\\\ & 10\text{ (default)} & 37.7 & 26.1 & 37.6 & 30.0 & 37.7 & 28.1 \\\\ & 20 & 38.9 & 25.6 & 36.5 & 28.3 & 37.7 & 27.0 \\\\ \hline \text{CA learning rate} & 0.015\to0.0015 & 32.9 & 25.6 & 35.3 & 26.7 & 34.1 & 26.1 \\\\ & 0.03\to0.003\text{ (default)} & 37.7 & 26.1 & 37.6 & 30.0 & 37.7 & 28.1 \\\\ & 0.06\to0.006 & 34.6 & 26.1 & 37.6 & 30.0 & 36.1 & 28.0 \\\\ \hline \end{array}$$ We analyze the effects of the **CA/TR optimizers, the number of CA epochs, and the initial CA learning rate** on EK100. By default, we optimize CA using Adam for 10 epochs and TR using sign-PGD for 5 epochs, with the CA learning rate cosine-annealed from 0.03 to 0.003. **Each ablation varies one hyperparameter while keeping the others fixed**. As shown in the above table, CA achieves relatively stable performance across optimizer choices, with average validation ASR ranging from 27.0\% to 28.3\%. Similarly, varying the initial CA learning rate from 0.03 to 0.06 yields comparable performance (28.1\% vs. 28.0\%). For CA training epochs, 10 epochs achieve the highest average validation ASR of 28.1\%, compared with 24.8\% and 27.0\% for 5 and 20 epochs, respectively. Overall, these results indicate that CA maintains **competitive performance across different optimization settings, with limited sensitivity to optimizer and learning-rate choices.** **We thank the reviewer for the helpful suggestion, and we have added those results and discussion in Appendix, A.2, page 15 of our uploaded revised manuscript.**
Reviewer_jHy9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies physical adversarial attacks on video models: a single pre-optimized printable texture attached to an object must mislead video models across different videos, even though the viewpoint and object motion are beyond the attacker's control. To address the problem that heterogeneous videos and observation trajectories produce conflicting optimization gradients, the authors propose TARA, which renders the texture along real trajectories, aligns the attacked features of different videos with contrastive learning, and then refines the texture for the task; they also annotate frame-wise texture trajectories on public video datasets and collect real print-and-recapture videos. Experiments show that the method outperforms existing attacks on action recognition, action anticipation, and future frame generation.

Critical strengths
  1. Attacking video models with a single fixed texture under uncontrolled observation trajectories is closer to real deployment than prior physical attacks evaluated under predefined conditions.

  2. The authors annotate texture trajectories with frame-wise homographies and occlusion masks on EK100 and SSV2, and collect real captured videos, so future work can compare attacks under identical geometric conditions.

Critical weaknesses
  1. The printed evaluation compares only EOT and TARA on a single model, and the number of videos in the printed setting is not clearly reported. This is not enough to establish TARA's advantage under physical deployment, and the physical evidence for the central claim rests on a single model.
  2. All main results come from a single seed, and on some models TARA ties with the strongest baseline (e.g., AVT-B, VideoMAE-B).
  3. A separate texture is optimized for each model, with no cross-model transfer experiments and no evaluation against any defense, so it is hard to judge how serious this threat is in practice.
Questions for authors
  1. Does the texture transfer to other models in a black-box setting, and how does it perform against defenses?
6 comments
AuthorsOpenReview ↗

Response regarding to Weakness 1

W1: The printed evaluation compares only EOT and TARA on a single model, and the number of videos in the printed setting is not clearly reported. This is not enough to establish TARA's advantage under physical deployment, and the physical evidence for the central claim rests on a single model.


Thank you for your feedback; please find our response below.

1. Clarification of the number of videos in the physical evaluation. We collect 175 and 173 independent action events for action anticipation and action recognition, respectively. Each event is recorded under three texture conditions, white (reference), TARA, and EOT, with consistent recording conditions, yielding total of 525 (3x175) and 519 (3x173) video clips for the two tasks, respectively. For digital rendering evaluation, we annotate the texture trajectories in 328 (175+173) reference clips, covering a total of 8,645 frames. For the physical print-and-recapture evaluation, we exclude events with substantial variations in recording conditions or trajectory misalignment across the three texture conditions, retaining 140 well-aligned video clip pairs for evaluation. We thank the reviewer for pointing out the need to clarify the scale of our physical evaluation. We have added the following clarification in Appendix A.12, page 23 of our uploaded revised manuscript.


2. More evidence to support physical experiments. Following your good suggestion, we conducted additional print-and-recapture experiments on V-JEPA2-L for action anticipation and recognition, and InternVideo2 for action recognition to further evaluate the effectiveness of TARA in physical settings. We also include PhyCamo as an additional competitive baseline. For each model-task setting, we select 100 action events and record four video clips per event: one reference clip with a white texture and three attacked clips with printed textures generated by TARA, EOT, and PhyCamo, respectively, yielding a total of 1,200 video clips across the three settings. For each event, all printed textures are placed at the same scene location, and the same action is repeated under consistent recording conditions to ensure a fair comparison.

Table. Additional physical print-and-recapture results. We evaluate 100 events per model-task setting and report Top-1 ASR (%), computed over events correctly classified on white reference clips.

\hline \text{Tasks} & \text{Anticipation} & \text{Recognition} & \text{Recognition} & \text{Overall} \\\\ \text{Models} & \text{V-JEPA2-L} & \text{V-JEPA2-L} & \text{InternVideo2} & \\\\ \hline \text{Physical EOT} & 35.8 & 20.8 & 32.3 & 29.6 \\\\ \text{Physical PhyCamo} & 30.2 & 25.0 & 32.3 & 29.2 \\\\ \hline \mathbf{Physical\\ TARA} & \mathbf{45.3} & \mathbf{31.3} & \mathbf{41.5} & \mathbf{39.4} \\\\ \hline \end{array}$$ As shown in the above Table. TARA consistently achieves the highest Top-1 ASR across all three settings. Overall, TARA improves averaged ASR over EOT and PhyCamo by **9.7 and 10.2 percentage points**, respectively. These results further support the effectiveness of TARA in physical settings, with consistent improvements across the evaluated models and tasks. **We thank the reviewer for the constructive feedback, and we have added those results and discussion in Appendix A.3, pages 15- 16, of our uploaded revised manuscript.**
AuthorsOpenReview ↗

Response regarding to Weakness 2

W2: All main results come from a single seed, and on some models TARA ties with the strongest baseline (e.g., AVT-B, VideoMAE-B).


Thank you for your feedback; please find our response below.

To provide a more comprehensive assessment of statistical stability, we have conducted additional experiments with three random seeds (0, 1, and 2) for TARA and two competitive baselines, EOT and PhyCamo, on both EK100 and SSV2. The results are shown in the tables below:

Table 1. Statistical stability evaluation on EK100 future action anticipation. Top-1 ASR (%) is reported as mean ± standard deviation over three seeds.

\text{Method} & \text{V-JEPA2-L} & & \text{V-JEPA2.1-B} & & \text{AVT-B} & & \text{Overall} & \\\\ & \text{Train} & \text{Val} & \text{Train} & \text{Val} & \text{Train} & \text{Val} & \text{Avg. Train} & \text{Avg. Val} \\\\ \hline \text{EOT} & \underline{31.3}\pm1.1 & \underline{24.5}\pm0.7 & \underline{56.0}\pm1.1 & \underline{53.3}\pm6.8 & 32.5\pm0.7 & 24.4\pm2.5 & \underline{40.0}\pm0.6 & \underline{34.1}\pm3.0 \\\\ \text{PhyCamo} & 13.9\pm0.0 & 14.2\pm0.8 & 53.5\pm4.6 & 26.9\pm2.1 & \mathbf{40.8}\pm2.7 & \underline{29.4}\pm1.0 & 36.1\pm2.3 & 23.5\pm0.7 \\\\ \hline \mathbf{TARA} & \mathbf{37.5}\pm0.6 & \mathbf{25.6}\pm0.9 & \mathbf{79.1}\pm2.2 & \mathbf{61.2}\pm5.4 & \underline{36.9}\pm0.7 & \mathbf{30.0}\pm1.7 & \mathbf{51.1}\pm1.0 & \mathbf{38.9}\pm1.5 \end{array}$$ **Table 2**. Statistical stability evaluation on SSV2 action recognition. Top-1 ASR (%) is reported as mean ± standard deviation over three seeds. $$\begin{array}{c|cc|cc|cc|cc} \text{Method} & \text{V-JEPA2-L} & & \text{InternVideo2} & & \text{VideoMAE-B} & & \text{Overall} & \\\\ & \text{Train} & \text{Val} & \text{Train} & \text{Val} & \text{Train} & \text{Val} & \text{Avg. Train} & \text{Avg. Val} \\\\ \hline \text{EOT} & \underline{26.6}\pm0.9 & \underline{31.7}\pm1.1 & 11.7\pm3.6 & 11.0\pm2.7 & 14.5\pm0.7 & \underline{16.9}\pm0.8 & 17.6\pm1.1 & \underline{19.9}\pm0.8 \\\\ \text{PhyCamo} & 17.4\pm0.5 & 13.2\pm0.7 & \underline{21.7}\pm2.5 & \underline{22.1}\pm1.5 & \underline{15.0}\pm0.3 & \underline{16.9}\pm0.0 & \underline{18.0}\pm1.1 & 17.4\pm0.4 \\\\ \hline \mathbf{TARA} & \mathbf{38.0}\pm0.9 & \mathbf{34.2}\pm0.4 & \mathbf{41.1}\pm0.3 & \mathbf{25.7}\pm0.0 & \mathbf{18.9}\pm1.1 & \mathbf{17.7}\pm0.8 & \mathbf{32.6}\pm0.2 & \mathbf{25.9}\pm0.3 \end{array}$$ The results show that TARA achieves the highest mean ASR in 5 out of 6 attacked models in both training and validation sets, demonstrating a consistent overall advantage over the competing methods. Specifically, over three seeds, TARA improves the average ASR over the strongest baseline by **11.1/4.8 percentage points on EK100 and 14.6/6.0 percentage points on SSV2**, for the training/validation splits, respectively. Moreover, TARA exhibits limited variation, with standard deviations of its overall average ASR remaining within 1.5 percentage points across both datasets. We acknowledge that TARA’S improvement on the validation set of AVT-B (30.0% vs. 29.4%) and VideoMAE-B (17.7% vs. 16.9%) is relatively modest, and PhyCamo outperforms TARA on the AVT-B training split. Nevertheless, the overall results demonstrate that TARA achieves non-trivial performance gains over the strongest competing baseline regarding ASR across different models and tasks. **We thank the reviewer for the constructive feedback, and the corresponding results, reported as mean ± standard deviation, have been added to Section 4.4, page 9 in our uploaded revised manuscript.**
AuthorsOpenReview ↗

Response regarding to Weakness 3 and Question 1.

W3: A separate texture is optimized for each model, with no cross-model transfer experiments and no evaluation against any defense, so it is hard to judge how serious this threat is in practice. Does the texture transfer to other models in a black-box setting, and how does it perform against defenses?


Thank you for your feedback; please find our response below.

1. Black-box transferability. To assess black-box transferability, we conduct additional experiments on the EK100 validation set by directly transferring adversarial textures optimized on a source model to a different target model. We evaluate all six transfer directions among three models, comparing TARA against two competitive baselines, EOT and PhyCamo. The results are presented in the following table.

Table. Black-box transferability on the EK100 validation set. L, B, and A denote V-JEPA2-L, V-JEPA2.1-B, and AVT-B, respectively. Top-1 ASR (%) is reported.

\hline \text{Method} & L\to B & L\to A & B\to L & B\to A & A\to L & A\to B & \text{Avg.} \\\\ \hline \text{EOT} & \underline{7.9} & \mathbf{5.0} & \mathbf{9.0} & \underline{5.0} & \mathbf{6.6} & \underline{10.7} & \underline{7.4} \\\\ \text{PhyCamo} & 5.7 & \underline{1.7} & \mathbf{9.0} & 3.3 & \mathbf{6.6} & 8.6 & 5.8 \\\\ \hline \mathbf{TARA} & \mathbf{10.0} & \underline{1.7} & \underline{8.5} & \mathbf{13.3} & \mathbf{6.6} & \mathbf{11.4} & \mathbf{8.6} \\\\ \hline \end{array}$$ The results show that TARA achieves the highest average ASR of 8.6% across six transfer directions, outperforming EOT and PhyCamo by 1.2 and 2.8 points, respectively. The transferability remains relatively low across all methods, highlighting the challenges of cross-model transfer under practical conditions with unknown and varying object trajectories and scene configurations. These findings suggest that improving cross-model transferability remains an important direction for future research on physical adversarial attacks. **We thank the reviewer for the constructive question, and we have added the results and discussion in Appendix A.1 of our uploaded revised manuscript.** --- **Table**. Top-1 ASR (%) under different defenses, averaged across three models. Each entry denotes Train/Val. $$\begin{array}{cc|ccc} \hline \text{Dataset} & \text{Defense} & \text{EOT} & \text{PhyCamo} & \mathbf{TARA} \\\\ \hline \text{EK100} & \text{JPEG} & 34.9/33.8 & 28.6/19.3 & \mathbf{40.3/35.8} \\\\ & \text{Blur} & 38.0/34.1 & 32.9/20.6 & \mathbf{46.3/37.2} \\\\ & \text{Resize} & 35.1/33.6 & 28.5/20.2 & \mathbf{41.5/35.3} \\\\ & \text{DRUNet} & 41.3/39.4 & 35.1/23.7 & \mathbf{51.3/43.4} \\\\ \hline \text{SSV2} & \text{JPEG} & 18.7/21.5 & 19.3/17.9 & \mathbf{27.7/24.9} \\\\ & \text{Blur} & 19.8/21.4 & 19.5/18.4 & \mathbf{30.8/27.2} \\\\ & \text{Resize} & 20.2/22.0 & 19.2/19.8 & \mathbf{26.1/24.4} \\\\ & \text{DRUNet} & 19.3/22.1 & 17.9/18.6 & \mathbf{33.2/29.0} \\\\ \hline \end{array}$$ **2. Performance under Defenses.** We evaluate EOT, PhyCamo, and TARA against three transformation-based defenses, including JPEG compression ($Q=85$), Gaussian blur ($\sigma=0.5$), and resizing (downsampling to 0.85\$\times\$ followed by upsampling), as well as a denoising-based defense, DRUNet [1] ($\sigma=10/255$). All defenses are applied framewise to both clean and adversarial inputs. We assume the attackers have no prior knowledge of the defenses. The average results among three models for each task are reported in the above table, and the model-specific results can be found in Table 13 and Table 14 of our uploaded manuscript. Overall, **all three attacks retain considerable effectiveness under both transformation-based and denoising-based defenses, and TARA consistently achieves the highest averaged Top-1 ASR among both datasets.** DRUNet provides limited mitigation in our setting, potentially because applying it without dataset-specific fine-tuning degrades clean accuracy without sufficiently suppressing adversarial effects. **Performance under stronger defenses.** We further investigate the trade-off between attack mitigation and clean accuracy under varying defense strengths on EK100 with V-JEPA2-L. The results are shown in **Figure 9 in page 17**, indicating that stronger defenses generally reduce ASR at the cost of clean-task performance. For instance, increasing Gaussian blur from $\sigma=0.5$ to $1.0$ reduces average ASR by approximately 17 percentage points, while clean accuracy drops from 70.2% to 38.3%. These findings highlight the need for dedicated defenses that effectively suppress physical adversarial effects while preserving clean-task performance. **We thank the reviewer for the constructive question, and we have added those results and discussion in Appendix A.4 of our uploaded revised manuscript.** [1] Plug-and-play image restoration with deep denoiser prior, in IEEE T-PAMI 2021.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of our revision

We thank the reviewers for reviewing our manuscript and giving their constructive and helpful feedback. We have revised our uploaded manuscript according to their feedback. The revised parts are marked by blue color. Our revision is summarized as follows:

  1. Additional physical experiments. Following the suggestions of reviewers jHy9, iiE5, and qxAf, we conduct additional physical print-and-recapture experiments on two tasks, evaluating V-JEPA2-L and InternVideo2 using adversarial textures generated by EOT, PhyCamo, and TARA. The results are presented in Appendix A.3, pages 15–16 of the revised manuscript.
  2. Video number of physical experiments. Following the suggestions of reviewers jHy9,we clarify the details of our physical experiments Appendix A.12, page 23 of the revised manuscript.
  3. Statistical stability.: Following the suggestions of reviewers jHy9 and iiE5, we evaluate the statistical stability of TARA, EOT, and PhyCamo across three random seeds on three models in both EK100 and SSV2. The results are presented in Section 4.4, page 9 of the revised manuscript.
  4. Black-box transferability. Following the suggestions of reviewers jHy9 and iiE5, we evaluate black-box transferability on the EK100 across three models and compared performance of EOT, PhyCamo, and TARA. The results are presented in Appendix A.1, page 15, and we give the summary in Section 4.4, page 9 of the revised manuscript.
  5. Performance under defense. Following the suggestions of reviewers jHy9, we evaluate EOT, PhyCamo, and TARA against four general defense methods, including three transformation-based methods and one denoising-based method. The results are presented in Appendix A.4, page 17 of the revised manuscript.
  6. Comparison with other initialization. Following the suggestions of reviewers iiE5, we compare CA with perceptual-prior initialization [1], output-diversified initialization (ODI)~[2], and Gaussian random initialization on EK100 using V-JEPA2-L. The results are presented in Section 4.5, page 9, and implementation details are provided in Appendix A.14, pages 24-25 of the revised manuscript.
  7. Detailed settings of ablation. Following the suggestions of reviewers iiE5, we clarify the ablation details in Appendix A.12, page 22 of the revised manuscript.
  8. Analysis of the hyperparameters of CA optimization. Following the suggestions of reviewers iiE5, we analyze the effects of the CA/TR optimizers, the number of CA epochs, and the initial CA learning rate on EK100. The results are presented in Appendix A.2, page 15 of the revised manuscript.
  9. Effect of attachment location. Following the suggestions of reviewers qxAf, we investigate the effect of attachment location on attack performance in EK100. The results are presented in Appendix A.6, page 18 of the revised manuscript.
  10. Discussion on the texture shape. Following the suggestions of reviewers qxAf, we clarify the motivation of using circular texture and evaluate the performance of TARA, EOT, and Phycamo using square textures. The results are presented in Appendix A.5, pages 17-18 of the revised manuscript.
AuthorsOpenReview ↗

Update of our revision

Dear Reviewers,

We sincerely appreciate your time and effort in reviewing our manuscript and providing constructive feedback throughout the discussion, which has greatly helped us improve our work. According to the follow-up suggestions from Reviewer jHy9, we have made the following additional revisions, which are highlighted in red color in the uploaded manuscript.

  1. Analysis of transferability between V-JEPA families. Following the suggestion of reviewer jHy9, we analyzed the transferability between V-JEPA model families in Appendix A.1, page 15 of the revised manuscript.

  2. Additional patch-based defense. Following the suggestion of reviewer jHy9, we evaluated the performance of two additional patch-based defenses. The results are presented in Appendix A.4, pages 16-17

If you have any further questions or if any concerns remain unresolved, we are always happy to discuss and address them. Thank you once again for your valuable time and thoughtful feedback.

Best Regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.