acceptodds
Withdrawn from ICLR 2027

When Adaptation Becomes Vulnerability: Dynamic Patches Open the Temporal Attack Surface of Online Trackers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual object tracking has witnessed a paradigm shift from offline static matching to online adaptive temporal learning, where trackers achieve robustness by continuously learning from observations and dynamically updating their target models. However, we reveal a critical flaw hidden beneath this learning-from-observation flexibility: a static adversarial patch is progressively memorized and assimilated by the tracker's online update mechanism, so the attack signal is not defeated but absorbed, and its efficacy decays sharply as memory accumulates. Ours DyPatch is the first to characterize this categorical mismatch between static carriers and learning systems as a directly exploitable attack surface: the tracker's temporal learning process itself, rather than its perceptual output. Unlike traditional attacks that target the perceptual output, this attack surface enables simultaneous manipulation of both the model's semantic representation and its online evolution trajectory. Accordingly, we propose a dynamic adversarial patch framework, which, through temporal switching and adversarial prior inheritance, produces dynamic patches that precisely hijack the tracker, trapping its online update into a vicious cycle of the more it updates, the more it errs. Experiments across six trackers and four benchmarks demonstrate that dynamic patches sustain effective attacks where static patches invariably fail, and successfully transfer to the physical world.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Adversarial attack, Dynamic adversarial patch, Online adaptive tracker, Temporal attack

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_CFqxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DyPatch, a dynamic adversarial patch attack against online visual object trackers. The method generates a cyclic sequence of patches using a temporal switching loss and an adversarial prior inheritance framework, which combines historical adversarial features with recurrent refinement. It also introduces spatio-temporal degradation augmentation to improve robustness under physical-world capture conditions. The method is evaluated on four online trackers, two offline trackers, four single-object tracking benchmarks. The reported results suggest that DyPatch can maintain persistent tracking degradation.

Critical strengths

1.This paper proposes treating the internal temporal state/memory module of the tracker as a novel attack surface—shifting the attack objective from disrupting single-frame predictions to persistently contaminating the internal state, thereby amplifying erroneous iterations. This insight transcends the static paradigm of traditional patch attacks and offers a new research perspective for the field of tracker countermeasures.

2.The evaluation covers multiple tracker architectures and datasets, and includes digital, physical-world, offline-tracker, and multi-object-transfer experiments.

Critical weaknesses
  1. In the experiments, all dynamic patches were individually optimized for a single tracker, and their cross-tracker transfer performance was not validated. In real-world attack scenarios, however, attackers typically do not know the specific tracker model employed by the target system. If patches can only be effective against a particular model, the generalizability of this approach would be significantly compromised—a point that was not addressed in this paper.
  2. The threat model and patch training protocol are inadequately described. The paper fails to clearly specify whether the patches are cross-video, cross-target-class, and cross-sequence generalizable; whether they are individually optimized for each tracker; and whether real bounding boxes from test sequences are utilized during optimization.
  3. Physical deployment scenarios have inherent limitations; in physical experiments, wearable electronic displays are used to implement patch switching, whereas the core advantages of traditional static patches are that they can be printed, adhered, and require no power supply. Dynamic patches, however, rely on electronic display devices, which significantly restricts the applicable scenarios for physical attacks—for instance, they cannot enable long-range, low-cost attacks via stickers or spray applications.
  4. The baseline comparisons may lack fairness. DyPatch employs multiple patches, cyclic switching, and specific patch scales; however, the paper does not sufficiently specify whether the different baselines utilize the same patch size, number of switches, number of display frames, number of optimization iterations, and model access permissions. Consequently, the performance improvements may not entirely stem from the inheritance of temporal priors.
Questions for authors
  1. How exactly should the patch be trained and tested?
  2. Can DyPatch be migrated to a tracker that has not undergone optimization?
  3. How to balance patch size and visibility?
  4. What's the main difference and advantages compared to previous attack methods, such as: [1] Towards Patch-based Noise Compression for Adversarial Attack against Transformer-based Visual Tracking, TIFS26 [2] Towards Universal and Sparse Adversarial Examples for Visual Object Tracking, ASOC24 [3] Towards Universal Physical Attacks on Single Object Tracking, AAAI21
Reviewer_83jBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DyPatch, an attack strategy to online tracking models. DyPatch reveals the flaws of existing static attacks and random dynamic attacks in failing to exploit the temporal dimension. To address issue, DyPatch formulates the task as a temporal switching attack, and designs the Temporal Adversarial Cue Fusion module and the Adversarial Feature Refinement module. In addition, an augmentation is applied to improve physical attack performance.

Critical strengths
  1. A meaningful perspective on temporal adversarial attacks. The paper highlights the vulnerability of online trackers to temporally evolving adversarial patches. This perspective is reasonable and intuitive, as it considers the interaction between adversarial perturbations and the tracker's online adaptation mechanism rather than focusing solely on frame-level attacks.

  2. Promising experimental performance and practical considerations. The proposed method achieves competitive attack performance across multiple trackers and benchmarks. The inclusion of spatio-temporal degradation augmentation and physical-world experiments further demonstrates the potential practical applicability of the approach.

Critical weaknesses
  1. Limited methodological novelty and insufficient justification of the proposed temporal attack mechanism. Both the proposed DyPatch framework in Figure 1(c) and the dynamic patch strategy in Figure 1(b) fundamentally rely on switching between adversarial patches over time. Although DyPatch introduces additional optimization modules, the underlying idea is similar. More importantly, the current comparisons do not sufficiently isolate the benefits of the proposed temporal optimization strategy. For example, alternating between patches generated by different attack methods may not constitute a sufficiently strong baseline, as these patches can differ substantially in attack strength. Furthermore, the central claim that static patches are gradually assimilated by online trackers (Section 3.1) lacks convincing theoretical or empirical support. These issues make it difficult to assess whether the proposed method establishes a substantively new attack mechanism beyond existing dynamic patch combinations.

  2. Limited generalizability and insufficient experimental transparency. The proposed approach appears to rely on tracker-specific optimization, with separate adversarial patch sequences potentially trained for individual victim models. This dependence limits its practical applicability, given the diversity of tracking architectures encountered in real-world scenarios. Moreover, important implementation details, including the training data, model configurations, and evaluation protocols, are insufficiently specified. Without evidence of cross-tracker transferability and clearer experimental settings, it is difficult to assess the generalizability of the approach and the fairness of its comparisons with existing methods.

Questions for authors
  1. Evidence for the proposed vulnerability: In Section 3.1, the authors claim that static adversarial patches are progressively assimilated by the online adaptation mechanism, eventually reducing their attack effectiveness. Could the authors provide direct empirical evidence supporting this claim, such as tracking performance over time under static and dynamic patch attacks? It would also be helpful to clarify how this phenomenon is distinguished from other factors contributing to attack degradation.

  2. Comparison with stronger dynamic patch baselines: Figure 1(b) considers switching between patches generated by different attack methods. Could the authors additionally evaluate a baseline that alternates between multiple independently optimized patches generated by the same strong attack method, using the same patch count, patch size, and switching frequency as DyPatch? This comparison would help clarify whether the improvements primarily arise from temporal switching itself or from the proposed optimization and inheritance mechanisms.

  3. Tracker-specific optimization and transferability: Are the dynamic patch sequences trained separately for each victim tracker? If so, how well do patches optimized for one tracker transfer to other tracking architectures without additional optimization? Such results would be important for evaluating the practical generalizability of the approach.

  4. Training and evaluation protocols: Could the authors clarify the training datasets, model configurations, and optimization settings used for DyPatch and the compared methods? In particular, were all baselines evaluated under comparable patch sizes, training budgets, and attack settings? These details are necessary to establish the fairness and reproducibility of the reported comparisons.

Reviewer_4VUxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DyPatch, a dynamic adversarial patch framework that reveals a novel attack surface within the online update mechanisms of visual object trackers. Traditional static patches are often gradually assimilated by a tracker’s continuous learning and model update processes, leading to a decay in attack efficacy. To address this, DyPatch employs temporal switching and the inheritance of adversarial priors to generate dynamic patches. These patches continuously disrupt both the target’s semantic representation and the model’s evolution, trapping the tracker in a vicious cycle where it becomes increasingly prone to errors as it continues to update. Experiments conducted across six types of trackers and four benchmark datasets demonstrate the framework’s effectiveness, its capacity for cross-model transfer, and its feasibility in the physical world.

Critical strengths
  1. The research problem possesses a degree of novelty, and the chosen perspective on attacks holds clear research value. This study extends the scope of attacks from single-frame perception outputs to the temporal learning and state evolution processes of trackers, constructing dynamic patch sequences through "Temporal Switching" and "Adversarial Prior Inheritance." This approach is innovative and holds significant implications for algorithmic research; in particular, by identifying online adaptive mechanisms as a potential attack surface, it offers a fresh perspective for the study of adversarial attacks in the field of visual tracking.

  2. The experimental design is systematic and comprehensive. The proposed method is evaluated across six types of online and offline trackers using four mainstream tracking benchmarks. The research encompasses multi-dimensional validation, including attacks on long sequences, analysis of performance across complex scene attributes, studies on hyperparameter sensitivity, ablation experiments for individual modules, and testing in real-world physical environments. The ablation experiments, in particular, provide deep insight into the specific roles of components such as "Temporal Switching," "Cue Fusion," and "Feature Refinement." Furthermore, the appendix includes additional experiments on multi-object tracking based on ByteTrack.

  3. The study reflects a substantial experimental effort and yields compelling results, effectively demonstrating the potential of this method for application across a diverse range of tracking architectures, data scenarios, and physical environments.

Critical weaknesses

1.There are inconsistencies in the mathematical definition of the core attack loss, and the optimization objectives of the proposed method lack sufficient clarity. In the main text, Eq. (2) and Eq. (15) in the appendix define the sign of the temporal switching loss with respect to the threshold δt\delta_t in opposite ways, leading to inconsistent optimization constraints. Furthermore, the authors do not explicitly explain the relationship between the attack quality qtq_t and GIoU, nor do they clarify whether the switching loss applies to all video frames or only to frames where a patch switch occurs. Given that the "Temporal Switching Attack" is a core innovation of this paper, these issues make it impossible to accurately determine the optimization objective constrained by this loss, and they raise concerns regarding the mathematical soundness, implementation consistency, and reproducibility of the proposed method.

2.There is a lack of sufficiently direct experimental validation to support the paper’s core claims regarding the innovation of its online state-update mechanism. The authors posit that the effectiveness of DyPatch stems primarily from the continuous disruption of the online state-update process within trackers. However, the current experimental evidence is only indirect. It fails to conclusively demonstrate that the attack’s success is specifically attributable to the online update mechanism, rather than to general, time-varying dynamic perturbations. In particular, the study lacks controlled experiments that compare the performance of the proposed method with the online update mechanism both enabled and disabled, while keeping the tracker architecture constant. Furthermore, the paper provides insufficient detail regarding several critical aspects: the attack threat model, the utilization of ground-truth bounding boxes, the partitioning of videos for patch training versus testing, and whether patches are optimized independently for different trackers or specific test sequences.

Consequently, I have reservations concerning the validity of the claimed attack mechanism, the generalizability of the experimental conclusions, and the distinct innovative value of this approach compared to existing dynamic adversarial attack methods. I look forward to the authors addressing these concerns.

Questions for authors

1.There are inconsistencies in the mathematical definition of the core attack loss, and the optimization objectives of the proposed method lack sufficient clarity. In the main text, Eq. (2) and Eq. (15) in the appendix define the sign of the temporal switching loss with respect to the threshold δt\delta_t in opposite ways, leading to inconsistent optimization constraints. Furthermore, the authors do not explicitly explain the relationship between the attack quality qtq_t and GIoU, nor do they clarify whether the switching loss applies to all video frames or only to frames where a patch switch occurs. Given that the "Temporal Switching Attack" is a core innovation of this paper, these issues make it impossible to accurately determine the optimization objective constrained by this loss, and they raise concerns regarding the mathematical soundness, implementation consistency, and reproducibility of the proposed method.

2.There is a lack of sufficiently direct experimental validation to support the paper’s core claims regarding the innovation of its online state-update mechanism. The authors posit that the effectiveness of DyPatch stems primarily from the continuous disruption of the online state-update process within trackers. However, the current experimental evidence is only indirect. It fails to conclusively demonstrate that the attack’s success is specifically attributable to the online update mechanism, rather than to general, time-varying dynamic perturbations. In particular, the study lacks controlled experiments that compare the performance of the proposed method with the online update mechanism both enabled and disabled, while keeping the tracker architecture constant.

3.Eq. (1) in the paper utilizes frame-by-frame ground-truth bounding boxes as a supervisory signal for attack optimization. However, the authors do not explicitly state the extent to which an attacker can access the tracker’s model parameters or its online update mechanism. Furthermore, it remains unclear whether the dynamic patches are optimized for specific trackers or for specific test videos, and whether ground-truth trajectory data from the test videos are used during the optimization phase. Additionally, there is no clear indication as to whether the training and test datasets are entirely independent and non-overlapping.

4.The paper presents experiments involving physical-world attacks on object-tracking systems using wearable display devices. Although the text includes an "AI Use Statement," it lacks a dedicated "Ethics Statement." It is not specified whether these physical experiments were conducted with appropriate authorization, whether informed consent was obtained from participants, or whether any safeguards were implemented to mitigate the risk of potential misuse. The authors should provide additional ethical disclosures to clarify the compliance of their experimental procedures and to define the boundaries of the risks associated with their research findings.

5.This paper provides insufficient detail regarding several critical aspects: the attack threat model, the utilization of ground-truth bounding boxes, the partitioning of videos for patch training versus testing, and whether patches are optimized independently for different trackers or specific test sequences.

Consequently, I have reservations concerning the validity of the claimed attack mechanism, the generalizability of the experimental conclusions, and the distinct innovative value of this approach compared to existing dynamic adversarial attack methods. I look forward to the authors addressing these concerns.

Reviewer_1Td2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a dynamic adversarial patch framework (DyPatch) targeting online visual object trackers. Traditional static patches are progressively absorbed and memorized by trackers with online adaptive update mechanisms, leading to a sharp decay in attack efficacy over time. To address this, the proposed method introduces temporal switching attacks, adversarial prior inheritance, and spatio-temporal degradation augmentation to persistently disrupt the tracker's historical memory and semantic evolution trajectory, successfully uncovering and exploiting a new temporal attack surface where the tracker's updates lead to compounding errors.

Critical strengths

Propose a spatio-temporal degradation augmentation strategy

Through the spatio-temporal degradation augmentation strategy, it jointly models real-world physical disturbances such as frame rate jitter, motion blur, and illumination variations, ensuring that dynamic patches remain highly effective in complex physical scenarios.

Critical weaknesses

Excessive prior dependency and insufficient evidence for tracking flaws:

The framework heavily relies on preset adversarial priors and complex dynamic evolution paths. Furthermore, its discussion of the inherent "flaws" in online tracking tasks lacks rigorous theoretical support and sufficient empirical evidence.

Baseline comparisons biased toward CV and Security:

Existing experiments and baselines are predominantly confined to computer vision (CV), without adequately exploring or comparing against recent security-oriented defenses from CV and Security.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.