Towards Continual Robustness: Adaptive Expert Composition for Evolving Adversarial Attacks

Continual adversarial defense aims to incrementally adapt a model to newly emerging adversarial attacks while preserving robustness against previously encountered ones. Existing methods primarily focus on mitigating catastrophic forgetting but largely overlook two fundamental challenges: effectively reusing transferable knowledge across evolving attacks and improving generalization to unseen adversarial distributions. To address these limitations, we propose a parameter-efficient continual adversarial adaptation framework based on Low-Rank Expert Expansion and Residual Composition (LREERC). Specifically, we incrementally expand the model with task-specific low-rank experts and reuse historical knowledge through adaptive residual composition without modifying previously learned experts. To avoid negative transfer, we further introduce a gradient-fingerprint-based expert retrieval mechanism, which selects optimization-compatible historical experts according to adversarial gradient similarity, together with a probability-weighted orthogonality regularization that discourages alignment between expert parameters during continual adaptation. Moreover, we propose a Dual-Channel Adaptive Perturbation Augmentation (DCAPA) strategy that jointly explores attack-specific perturbation directions guided by transferable experts and more universal vulnerability directions shared by the clean and current experts, broadening local perturbation sampling and improving robustness beyond observed attacks. Extensive experiment results demonstrate that our approach achieves state-of-the-art performance.