acceptodds
Under review as a conference paper at ICLR 2027

Towards Continual Robustness: Adaptive Expert Composition for Evolving Adversarial Attacks

Abstract

Continual adversarial defense aims to incrementally adapt a model to newly emerging adversarial attacks while preserving robustness against previously encountered ones. Existing methods primarily focus on mitigating catastrophic forgetting but largely overlook two fundamental challenges: effectively reusing transferable knowledge across evolving attacks and improving generalization to unseen adversarial distributions. To address these limitations, we propose a parameter-efficient continual adversarial adaptation framework based on Low-Rank Expert Expansion and Residual Composition (LREERC). Specifically, we incrementally expand the model with task-specific low-rank experts and reuse historical knowledge through adaptive residual composition without modifying previously learned experts. To avoid negative transfer, we further introduce a gradient-fingerprint-based expert retrieval mechanism, which selects optimization-compatible historical experts according to adversarial gradient similarity, together with a probability-weighted orthogonality regularization that discourages alignment between expert parameters during continual adaptation. Moreover, we propose a Dual-Channel Adaptive Perturbation Augmentation (DCAPA) strategy that jointly explores attack-specific perturbation directions guided by transferable experts and more universal vulnerability directions shared by the clean and current experts, broadening local perturbation sampling and improving robustness beyond observed attacks. Extensive experiment results demonstrate that our approach achieves state-of-the-art performance.

Primary area:
causal reasoning
Keywords:
Continual learning, Adversarial robustness, Low-rank adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sheROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LREERC, which adds task-specific LoRA experts to a frozen ViT and reuses selected historical experts through residual composition. Historical experts are retrieved using cosine similarity between gradient fingerprints, while an orthogonality loss encourages complementary expert parameters. Experiments on four image datasets report strong performance against the evaluated attack sequence.

Critical strengths

The framework is parameter-efficient. The paper reports memory, parameter, and training-time costs. The empirical comparison covers four datasets and a broad collection of continual-learning and defense baselines. The paper evaluates a complete predictor without assuming a test-time attack label.

Critical weaknesses
  1. The main robustness results focus primarily on attacks already adopted for training. Tab. 1 lists AutoAttack as the fifth training task, achieving 84.20% final accuracy on CIFAR-10. In contrast, Tab. 9 demonstrates 0.00% accuracy for genuinely unseen AutoAttack after Clean–FGSM–PGD–CW training, for both 4/255 and 8/255 perturbation radii. This heavily weakens the generalization claim for unseen attacks. Additionally, the final predictor applies class-wise maximum over several configurations (Eq. 38), but the work does not report results for attack unions, smooth-max surrogate attacks, or stricter multi-restart assessments.

  2. The key gradient-fingerprint component has not been validated in isolation. Tab. 2 merges residual composition, fingerprint selection and orthogonality into the LREERC setup, with missing comparisons against random selection, the newest single expert, and an experiment without the orthogonality term. The fingerprints are gradients computed after training; their norms and stability across sampled subsets are relevant characteristics but go unreported. The theoretical transfer bound depends on unregulated discrepancy and curvature terms, and consequently cannot confirm cosine similarity as a trustworthy selector in real use.

Questions for authors
  1. Does the advantage remain under a per-example union of PGD and AutoAttack failures, attacks targeting each retained configuration separately, or a smooth approximation to the class-wise maximum in Eq. (38)?

  2. Can the authors provide isolated ablations for fingerprint selection and orthogonality, including random/all/latest-expert selection, and report fingerprint norms and selection stability across subsets and seeds? Please clarify how each incoming adversarial training set is generated: against which checkpoint, whether it is regenerated during adaptation.

  3. Which statement about the r=0.6r=0.6 gate is correct, Appendix A.4 or Appendix B.3.1?

Reviewer_jyurOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual adversarial defense, where a classifier sequentially adapts to different attacks while retaining performance on previously encountered attacks. It proposes LREERC, which adds attack-specific LoRA experts to a frozen pretrained vision transformer and selectively composes historical expert residuals using gradient similarity. A weighted orthogonality penalty encourages complementary adapter parameters, while dual-channel perturbation augmentation uses historical, current, and clean-model experts to generate additional training examples. The method is evaluated separately on CIFAR-10, CIFAR-100, CUB-200, and ImageNet-R using a clean, FGSM, PGD, CW, AutoAttack sequence with a fixed class set. Validation includes comparisons with continual-learning and defense baselines, component ablations, forgetting measurements, unseen-attack evaluations, and computational-cost analysis. The reported improvements vary across datasets and attacks, with the largest gains on CIFAR PGD and AutoAttack evaluations.

Critical strengths
  1. The method addresses the practically relevant problem of continual adversarial defense without requiring attack identity at inference. It reports improvements over the strongest evaluated baselines on CIFAR-10 and CIFAR-100 under PGD and AutoAttack after completing the attack sequence. Although these gains are not uniform across benchmarks, they provide evidence of effective adaptation to encountered attacks.

  2. The framework achieves these results while training relatively few parameters through LoRA experts and a frozen backbone. This offers a potentially useful efficiency–performance trade-off for continual adversarial adaptation.

Critical weaknesses
  1. Limited novelty (major). The method combines established ideas: frozen LoRA experts, similarity-based selection, residual composition, orthogonality regularization, and perturbation augmentation (Sections 3.2–3.4). Related mechanisms appear in O-LoRA, CLLoRA, GainLoRA, TreeLoRA, and ELREA. Although the combination differs, the paper does not sufficiently establish its substantive advance over these approaches. The evolving-attack problem also predates this work, as acknowledged in Section 3.1.

  2. Insufficiently clear experimental setup and limited evaluation scope (minor). Section 3.1 defines attack-dependent tasks, but crucial details are scattered across the appendix: all classes remain present at every stage (Table 3), inference does not require task identity (Appendix D.6), and attack settings and training budgets appear in Appendices D.7 and E.7. These details should be summarized in the main paper to clarify the continual-learning setting. The fixed attack order and shared underlying dataset also limit conclusions about broader evolving-attack scenarios.

  3. Uneven evidence for robustness, retention, and efficiency (major). Table 1 shows large gains in some settings, but losses or small improvements in others. More importantly, on CIFAR-10, the checkpoint trained on Clean → FGSM → PGD → CW achieves 0% accuracy against unseen standard AutoAttack at budgets of 4/255 and 8/255 (Appendix E.5, Table 9). CLLoRA also scores 0% at these budgets; this concerns limited absolute robustness rather than worse performance relative to the baseline. These results qualify the unseen-attack claim but do not contradict Table 1, which evaluates a checkpoint subsequently trained on AutoAttack. Figure 3(a) measures only clean-task forgetting, while Table 7 shows PGD accuracy declining from 86.56% to 78.99%. Parameter efficiency also does not translate into lower training time: Appendix E.8 reports 18.58 hours versus 11.79 for CLLoRA.

  4. Unclear comparison fairness and insufficient ablations (moderate). Appendix D.3 describes adaptations of AIR and other baselines, but does not sufficiently establish that these preserve their intended functionality or that each method receives appropriate tuning under the short training budget (Appendix E.7). In particular, AIR’s original task-identity assumptions differ from the proposed inference setting; the adaptation and its implications should be explained. Matching epochs alone does not establish comparable optimization or computational budgets. Furthermore, Table 2 does not separately isolate the gradient fingerprint, expert-selection rule, and orthogonality term. These gaps complicate attributing the gains to the proposed mechanisms rather than baseline adaptation, training choices, or augmentation.

  5. Limited explanatory support from the theory (moderate). In Section 4.1, Proposition 4.1 shows that frozen historical experts remain unchanged, but the final classifier can still forget because its predictions also depend on newly added experts. In Section 4.2, Proposition 4.2 connects gradient similarity to beneficial transfer only under additional conditions whose validity is not checked experimentally. Likewise, the orthogonality regularizer in Section 3.3 separates parameter directions, but this does not necessarily produce experts with complementary defenses, as Appendix A.5 acknowledges. The theory therefore does not establish that selecting similar experts and separating their parameters will improve resistance to future attacks.

  6. Inconsistent expert-selection descriptions (minor). Section 3.3, Eq. (8) permits selecting multiple historical experts using relative similarity, consistent with Appendix A.4. However, Appendix B.3.1 states that the threshold permits at most one expert and selects none when three or more candidates qualify. The authors should resolve this discrepancy and identify which rule generated the reported results. This concern would become substantive if it reflects an implementation difference rather than a textual error.

Questions for authors
  1. What specifically distinguishes your method from TreeLoRA, ELREA, and GainLoRA? Please provide controlled ablations of similarity-based selection and orthogonality, keeping capacity and augmentation fixed.

  2. Please confirm whether each dataset forms a separate attack sequence with all classes retained. Explain how AIR and other baselines were adapted, particularly regarding task identity, and tuned under the short training budget.

  3. Table 9 reports 0% unseen-AA accuracy at 4/255 and 8/255. Please evaluate the final AA-trained checkpoint using that same protocol and clarify which results establish unseen-attack generalization.

  4. Figure 3(a) measures clean-task forgetting, while Table 7 shows declining historical-attack accuracy. Please report attack-wise forgetting across baselines and seeds, and assess sensitivity to an alternative attack order.

  5. What evidence connects the theoretical conditions to the observed gains? Please show whether fingerprint similarity predicts transfer benefit and whether orthogonality improves robustness in a controlled ablation.

  6. Equation (8) and Appendix B.3.1 describe different selection behavior. Please identify the implemented rule, correct the inconsistency, and report the number of experts selected at each stage.

Reviewer_rm2WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual adversarial defense, where a classifier adapts to sequentially arriving attacks while retaining performance on previously encountered ones. The proposed framework, LREERC, maintains a frozen pretrained backbone and expands task-specific LoRA experts. Historical experts are selected using parameter-gradient fingerprints and incorporated into new configurations through weighted residual composition. A probability-weighted orthogonality penalty encourages complementary adaptation. The framework also introduces DCAPA, which augments current adversarial inputs through expert-specific perturbations and mixed gradient directions from the clean and current experts. Inference aggregates feature–prototype similarities across retained configurations without requiring attack identities. Experiments on CIFAR-10, CIFAR-100, CUB-200, and ImageNet-R compare adaptation performance, retention, and trainable parameter counts, with additional component ablations and evaluations on unseen attacks.

Critical strengths
  1. A useful approach to historical knowledge reuse under limited adaptation budgets. The framework uses historical adversarial knowledge in two ways: selected experts contribute frozen residuals to a new configuration, and expert configurations guide perturbation augmentation. This design addresses both adaptation and retention while supporting inference without attack identities. It also requires relatively few trainable parameters: Figure 3c reports approximately 376K peak trainable parameters, compared with approximately 85.8M for the full-backbone group. These properties make the approach relevant to continual defense when updating the entire model is costly.
  2. Substantial reported improvements across several benchmarks. The evaluation covers four datasets and a broad set of continual-learning and robustness baselines, with three-run statistics in Table 1. Under the reported protocol, the gains are substantial. For example, CIFAR-10 PGD accuracy reaches 75.08%, compared with 27.13% for the strongest reported baseline, while clean accuracy remains high. Tables 2 and 8 also provide evidence that historical reuse and both augmentation channels contribute to performance. The unseen-attack results in Table 8, where average accuracy increases from 40.72% for the LoRA baseline to 55.39% for the full method, provide additional evidence of useful transfer in the evaluated settings.
Critical weaknesses
  1. The experiments do not sufficiently establish the specific methodological contribution. Appendix C.2.1 appropriately narrows the contribution to gradient-compatibility-driven reuse of frozen historical experts. However, closely related methods already address important parts of this design: TreeLoRA (https://proceedings.mlr.press/v267/qian25b.html) uses gradient similarity to organize historical LoRA knowledge, while GainLoRA (https://arxiv.org/abs/2505.15424) studies weighted integration of old and new LoRA branches. The proposed fixed-expert, cross-distribution fingerprint differs from these approaches, but its additional value in continual adversarial defense is not sufficiently established. Tables 2 and 8 evaluate LREERC as a bundle, and Figure 1b varies the scope of perturbation sources. These comparisons do not isolate the fingerprint-based selector from simpler alternatives using the same number of experts and a matched augmentation budget. Although Appendix A.5 provides an analytical interpretation of the orthogonality penalty, the reported experiments do not isolate its empirical contribution or the benefit of its particular weighting. This matters because the central contribution concerns how historical experts are selected and organized, whereas the current evidence mainly establishes the usefulness of historical reuse and multi-source augmentation as a whole.
  2. The demonstrated gains on unseen attacks have a limited scope. Figure 1 and Tables 5 and 8 provide useful evidence of cross-attack transfer. However, the full-predictor standard AutoAttack evaluation in Table 9 gives a more limited picture. Before any AutoAttack adaptation, the method achieves 3.36% and 1.74% robust accuracy at perturbation budgets of 1/255 and 2/255, respectively, and 0% at 4/255 and 8/255, in the reported resized 224 × 224 input space. Appendix E.5 correctly acknowledges that these results demonstrate a modest, budget-dependent benefit. Although this limitation is clearly disclosed, it restricts the demonstrated practical benefit on this strong unseen-attack evaluation, which is relevant because generalization beyond observed attacks is a central motivation for DCAPA. The current results do not establish useful unseen AutoAttack robustness at the two larger budgets considered. These results should be distinguished from Table 1, which includes adaptation to AutoAttack and therefore evaluates a different setting.
Questions for authors
  1. Evidence for the expert-selection mechanism. With the number of retrieved experts, augmentation samples, and perturbation-generation budget held fixed, how does the proposed fingerprint-based selection compare with random selection or a simple loss- or feature-based criterion? Can the authors empirically separate the effect of the compatibility criterion on residual selection from its effect on perturbation-source prioritization and allocation, keeping the other pathway fixed? Such a comparison would help establish the value of the specific compatibility measure beyond historical expert reuse.
  2. Evidence for the weighted orthogonality penalty. With the selected experts and augmentation procedure fixed, what changes when the orthogonality penalty is removed or its similarity weights are replaced with uniform weights? These controls would clarify whether the proposed weighting contributes independently to performance.
  3. Contribution of augmentation to unseen AutoAttack robustness. Figure 1 and Tables 5 and 8 provide evidence of transfer to selected unseen attacks, and Appendix E.5 already limits the interpretation of Table 9 to modest small-budget gains. Using the same training schedule and evaluating each variant immediately after the Clean–FGSM–PGD–CW sequence under the threat model in Table 9, can the authors provide or point to a comparison of LREERC trained with and without DCAPA? This would clarify whether the observed unseen AutoAttack benefit is specifically attributable to the proposed augmentation, beyond the expert-composition framework.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.