When Adaptation Becomes Vulnerability: Dynamic Patches Open the Temporal Attack Surface of Online Trackers

Visual object tracking has witnessed a paradigm shift from offline static matching to online adaptive temporal learning, where trackers achieve robustness by continuously learning from observations and dynamically updating their target models. However, we reveal a critical flaw hidden beneath this learning-from-observation flexibility: a static adversarial patch is progressively memorized and assimilated by the tracker's online update mechanism, so the attack signal is not defeated but absorbed, and its efficacy decays sharply as memory accumulates. Ours DyPatch is the first to characterize this categorical mismatch between static carriers and learning systems as a directly exploitable attack surface: the tracker's temporal learning process itself, rather than its perceptual output. Unlike traditional attacks that target the perceptual output, this attack surface enables simultaneous manipulation of both the model's semantic representation and its online evolution trajectory. Accordingly, we propose a dynamic adversarial patch framework, which, through temporal switching and adversarial prior inheritance, produces dynamic patches that precisely hijack the tracker, trapping its online update into a vicious cycle of the more it updates, the more it errs. Experiments across six trackers and four benchmarks demonstrate that dynamic patches sustain effective attacks where static patches invariably fail, and successfully transfer to the physical world.