PACE: Persistent Adversarial Coefficient Evolution for Recurrent Visual Agents

Recurrent visual agents maintain internal states that integrate past observations, reducing the immediate impact of isolated perturbations while introducing a new attack surface: persistent adversarial influence over their evolving internal dynamics. Existing attacks typically optimize each frame independently, ignoring the temporal dependency introduced by recurrent policies and resulting in ineffective or unstable perturbations. We propose Persistent Adversarial Coefficient Evolution (PACE), a framework that formulates adversarial manipulation as the evolution of a compact latent attack state over time. PACE first constructs a low-dimensional adversarial basis by querying the frozen agent, generating performance-degrading perturbations, and extracting dominant perturbation directions through factorization. During interaction, the previous attack coefficient initializes the current optimization, while temporal smoothing regulates abrupt changes in the adversarial state. The resulting perturbations are projected onto the valid ${\ell }_{\infty }$ constraint and image domain before being applied to the victim agent. Through warm-started coefficient updates and temporal smoothing, PACE retains attack information across frames while adapting perturbations to the current observation. Experiments on DreamerV3 across DMC Walker Walk, Atari Pong, and Crafter show that PACE achieves the largest reported mean reward reduction among the compared methods, tying with UAP-RL on Crafter, while exhibiting substantially lower temporal variation than MAD, PA-AD, Illusory, and DAPGD. These results highlight the importance of modeling adversarial attacks as evolving processes when evaluating the robustness of recurrent visual decision-making systems.