acceptodds
Under review as a conference paper at ICLR 2027

Reliable Adversarial Continual Learning via Robust Expert Collaboration

Abstract

Adversarially robust continual learning must acquire new tasks while preserving previously learned knowledge and adversarial robustness. These objectives can conflict because adversarial updates to a shared representation may amplify forgetting, induce representation drift, and yield class-imbalanced robust solutions. We propose Robust Expert Collaboration (REC), a replay-free framework that separates task-level robust expert construction from guarded backbone integration. REC first applies Diversified Multi-Expert Learning (DMEL), which jointly optimizes temporary task experts with feature-diversity and gradient-diversity regularization, together with image- and feature-space adversarial objectives, before fusing them into a single task expert. Robust Anchoring and Stability Cooperation (RASC) then adapts the shared backbone using clean and adversarial supervision, clean–adversarial consistency, a frozen teacher feature anchor, and historical decision distillation. A Class-wise Robust Constraint (CRC) selects checkpoints on a held-out gating split using clean accuracy, adversarial accuracy, per-class adversarial recall, teacher feature similarity, and historical decision agreement; when no candidate satisfies the predefined criteria, REC restores the pre-integration checkpoint. Experimental results demonstrate that REC achieves a favorable robustness–retention trade-off while maintaining competitive clean accuracy.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
adversarial continual learning, adversarial robustness, vision transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_i2JZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Robust Expert Collaboration (REC), a replay-free framework for adversarial task-incremental continual learning. REC separates robust task-expert construction from shared-backbone integration, using Diversified Multi-Expert Learning (DMEL), Robust Anchoring and Stability Cooperation (RASC), and a Class-wise Robust Constraint (CRC) for validation-based checkpoint selection. Experiments on Split CIFAR-10 and Split CIFAR-100 with a CLIP-pretrained ViT-B/16 backbone evaluate clean and adversarial accuracy under FGSM, PGD-Linf, PGD-L2, and BIM, showing a favorable robustness–retention trade-off.

Critical strengths

A main strength is that the method addresses both adversarial robustness and catastrophic forgetting within a unified continual-learning framework, rather than optimizing only one.

The design is also supported by component-wise ablations; for example, on Split CIFAR-10, the full configuration achieves 95.05% clean accuracy and 79.32% average adversarial accuracy, demonstrating that the proposed components meaningfully contribute to robustness while preserving clean performance.

Critical weaknesses

The experimental evaluation is limited to Split CIFAR-10 and Split CIFAR-100, both image-classification benchmarks. This makes it difficult to judge how well REC generalizes to larger-scale datasets, different domains, or more challenging continual-learning settings.

The evaluation uses a task-incremental, task-aware setting with a single CLIP-pretrained ViT-B/16 backbone. It would strengthen the paper to evaluate whether the conclusions remain valid under other continual-learning settings, particularly class-incremental learning, and with different backbone architectures.

Questions for authors

Please follow the strengths and weaknesses.

The experimental evaluation is limited to Split CIFAR-10 and Split CIFAR-100, both image-classification benchmarks. This makes it difficult to judge how well REC generalizes to larger-scale datasets, different domains, or more challenging continual-learning settings.

The evaluation uses a task-incremental, task-aware setting with a single CLIP-pretrained ViT-B/16 backbone. It would strengthen the paper to evaluate whether the conclusions remain valid in other continual learning settings, particularly class-incremental learning, and across different backbone architectures.

1 comment
AuthorsOpenReview ↗

Response to Reviewer i2JZ

We thank you for suggesting broader dataset evaluation and for asking whether REC can operate without a task ID or with another backbone. We address these points with the following results.

Q1. Can REC be evaluated beyond CIFAR?

A1. Thank you for this suggestion. We chose TinyImageNet because its ImageNet-derived object categories provide a different image source and a broader class set than CIFAR-10 [1], directly addressing your request for evidence beyond CIFAR. Under the same Task-IL backbone, class split, and attack evaluation for both methods over 100 seen classes, REC and the controlled FLAIR-style adaptation obtain:

TinyImageNet (%)FLAIR-styleREC
Clean82.3283.10
FGSM45.4647.12
PGD-ℓ∞\ell_\infty-1037.7239.10
PGD-ℓ2\ell_2-1079.7480.24
BIM-1037.2038.50
RobustAvg50.0351.24

REC is higher on all six reported metrics, including +0.78 pp Clean, +1.38 pp PGD-ℓ∞\ell_\infty-10, and +1.21 pp RobustAvg. We will add this matched comparison to the main results table and briefly specify its protocol in Section 4 of the revised manuscript.

Q2. Can REC operate without a task ID or use another backbone?

A2. Thank you for this proposal. REC also performs better in a task-agnostic inference audit. We combine the Task-IL-trained experts into a ten-class predictor and attack its global output without providing a task ID:

Task-agnostic audit (%)RECUSP-FSR-AT
Joint Clean46.1744.70
Joint PGD-ℓ∞\ell_\infty-1013.6011.58
Routing Clean46.3444.82
Routing PGD-1015.5712.81

REC exceeds USP-FSR-AT by +1.47 pp Clean and +2.02 pp PGD-10 under joint prediction. Both methods score below their task-aware Task-IL results, reflecting the harder no-task-ID setting. This is an inference extension, not Class-IL-specific training; all reported runs still use CLIP ViT-B/16. We will clarify the audit protocol in Appendix F and this scope in the Limitations.

[1] H. Pouransari and S. Ghili, Tiny ImageNet Visual Recognition Challenge, Stanford CS231n report, 2015.

Reviewer_ATMBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on adversarially robust continual learning, which requires the model to protect old tasks while learning new ones and maintain robustness. This is a novel problem setting. To address this issue, the authors proposed the Robust Expert Collaboration framework (REC), which is a replay-free framework that separates the construction of task-level robust experts from the protected backbone. The authors validated the method on PGD, FGSM, and BIM, and the results showed that this method achieved an improvement in robustness on CIFAR-10 and CIFAR-100.

Critical strengths
  1. The authors focuse on a novel and important problem setting, namely adversarial robust continuous learning. This setting requires the model to learn new tasks while protecting the old ones and maintaining robustness.
  2. This work provides a new research approach for adversarial robust continuous learning, namely multi-expert collaboration.
Critical weaknesses

1、There are some issues with the presentation of the paper. From the introduction section, I am not clear about the significance of adversarial continual learning, and the overall impression gives the impression that two completely independent studies have been combined together.

2、The paper mentions that adversarial training and continual learning influence each other. Similarly, adversarial training may introduce updates that exacerbate forgetting, but there is currently no corresponding literature or experiments to support this, which makes me difficult to believe.

3、According to the problem definition, I was unable to accurately obtain the settings of adversarial continual learning and the training and testing procedures; the relevant content is more related to continual learning.

4、The validation is mainly conducted on two datasets. However, the number of included datasets is too small to provide sufficient support for this method. Additionally, there seems to be a lack of assessment of forgetting situations.

5、FLAIR seems to use ResNet as the main network, but the paper uses the pre-trained CLIP ViT-B/16 main network. I did not clearly see in the main text that the authors focused on ViT, but only noticed that this main network was adopted in the experimental setup. I am not sure why the authors made this modification to the main network.

6、I think the proposed method is more like a patchwork of a large amount of content, lacking an explanation of the theoretical mechanism. For example, when introducing adversarial gradient diversity (GradDiv), it is only stated that relying solely on the previous feature diversity is not enough, and we need this mechanism.

7、This method introduces many additional hyperparameters, such as (λc, λa, λtr, λinv, λteach, λold) = (1, 1, 0.5, 0.2, 1, 1), and T = 2. Moreover, there is a lack of ablation experiments, which makes me doubt these experimental results.

8、The author seems to have adjusted the line spacing of the title, and I am not sure if this is allowed. For example: 2 Related Works.

Questions for authors

See the Critical Weaknesses

7 comments
AuthorsOpenReview ↗

Response 1/3 to Reviewer ATMB

Thank you for your careful review. Your questions led us to expand the evaluation beyond final accuracy: we added a matched TinyImageNet comparison, a paired stage-wise retention audit, and six one-factor sensitivity runs, and clarified their protocols and manuscript locations below.

Q1. Why does adversarial continual learning matter, and how do the three stages form one solution?

A1. The central challenge is to acquire robustness on incoming tasks while preserving robust decisions on earlier tasks without replaying their training images. Retaining old experts alone does not solve this problem: updating their shared backbone changes the features they receive, while freezing the backbone throughout restricts adaptation.

We clarified how REC addresses these coupled requirements. DMEL freezes the backbone and trains three complementary current-task experts, then uniformly fuses them into one expert. RASC updates the active backbone and fused expert under frozen-teacher feature and historical-decision constraints. Although the reference modules are frozen, their losses differentiate through the active backbone. CRC checks whether candidate updates satisfy clean, robust, class-wise, and historical validation requirements, restoring the post-DMEL checkpoint if none qualifies. Thus expert construction, backbone adaptation, and checkpoint acceptance have distinct roles within one learning process.

The Introduction and Secs. 3.2-3.4 explain this mechanism; Fig. 1 summarizes it, and App. A.1 specifies the stage-wise optimization scope. The temporary branches are discarded after fusion, retaining one current-task expert at inference.

Q2. What supports the connection between adversarial training and continual-learning retention?

A2. Thank you for requesting stronger grounding. Prior studies discuss robust acquisition and retention [1,2]. We connected this motivation to three complementary analyses within REC: component removal, layer-wise feature displacement, and historical-task trajectories.

EvidenceFull RECw/o Robust Anchor
Table 2: Clean (%)95.0594.98
Table 2: PGD-ℓ∞\ell_\infty-10 (%)72.0734.82
Fig. 6: Block-12 relative feature displacement (%)28.31111.46

Removing the anchor reduces PGD-10 by 37.25 pp while Clean changes by 0.07 pp. Clean accuracy alone therefore misses a substantial loss of robustness. The separate Fig. 6 diagnostic measures clean/adversarial feature displacement on the same inputs at each fixed checkpoint and associates anchoring with less representation disruption. Fig. 5 traces historical-task PGD-10 accuracy; the paired audit in A4 further tests retention against USP-FSR-AT.

The Introduction and App. C.2 connect these evidence sources. We distinguish the feature diagnostic from the Table 2 run and interpret it as supporting association rather than a causal proof of the full mechanism.

Q3. What training, inference, and attack protocol produced the reported Task-IL results?

A3. Thank you for asking us to make the protocol reproducible. We evaluate fully supervised, replay-free Task-IL: task ID selects the expert, and white-box attacks differentiate through its complete backbone-expert path.

AspectProtocol
StreamCIFAR-10: (4,2)→(7,6)→(0,3)→(5,8)→(9,1)(4,2)\to(7,6)\to(0,3)\to(5,8)\to(9,1); CIFAR-100: ten ten-class tasks.
ModelCommon CLIP-pretrained ViT-B/16; 224×224224\times224 input.
DMELFrozen backbone; three experts; five epochs; image PGD-7 (ϵ=8/255\epsilon=8/255, step 2/2552/255), feature PGD-3 (radius 0.05∥z∥20.05\lVert z\rVert_2).
Task-0 anchorTwelve pre-DMEL epochs, including two clean warm-up epochs.
RASC/CRCBackbone and fused expert updated using PGD-10 for 8/10 epochs (Task 0/later); CRC selects or restores checkpoints.
Test attacksAfter resize, before normalization. FGSM/PGD-ℓ∞\ell_\infty-10/BIM-10: ϵ=8/255\epsilon=8/255; iterative step 2/2552/255. PGD-ℓ2\ell_2-10: radius 128/255128/255, step 15/25515/255.
AggregationTask-macro accuracy; RobustAvg averages four attacks, excluding Clean.

Epoch counts above refer to CIFAR-10; CIFAR-100 has its own schedule. App. A.1 also specifies the deterministic 10% training holdout (at most 500 samples), epoch-wise validation, selection gates, ranking, and fallback. Test data do not select checkpoints. Baselines share the evaluation protocol but retain documented method-specific schedules and selection rules. Sec. 4 summarizes these details.

[1] Mi et al., Adversarial Robust Memory-Based Continual Learner, ICCV 2025. [2] Cho et al., Enhancing Robustness in Incremental Learning with Adversarial Training, AAAI 2025.

AuthorsOpenReview ↗

Response 2/3 to Reviewer ATMB

Q4. Do two datasets and final accuracies adequately establish breadth and historical retention?

A4. Thank you for suggesting broader evaluation and explicit forgetting measures. We addressed these as two experimental questions: whether the gains extend beyond CIFAR, and whether final accuracy conceals different retention behavior.

First, we added a matched TinyImageNet comparison covering 100 seen classes. REC and the FLAIR-style adaptation use the same backbone, Task-IL split, and post-resize white-box evaluation.

TinyImageNet (%)FLAIR-styleRECGain (pp)
Clean82.3283.10+0.78
FGSM45.4647.12+1.66
PGD-ℓ∞\ell_\infty-1037.7239.10+1.38
PGD-ℓ2\ell_2-1079.7480.24+0.50
BIM-1037.2038.50+1.30
RobustAvg50.0351.24+1.21

REC is higher on all six reported metrics in this matched stream, adding evidence from a different image source and a broader class set than CIFAR-10. Table 1 and App. B.3 report the comparison and its scope.

Second, the paired E1 audit evaluates all 15 valid stage-task cells per method, with both Clean and random-start PGD-10 on the full test sets, under the same order and seed 1993. We report the complete matrices and derive retention metrics from them, rather than inferring forgetting from final accuracy.

E1 PGD-10 metricRECUSP-FSR-AT
FAA (%)71.9469.93
AIA (%)67.5463.94
BWT (pp)+0.79-2.50
Forgetting F (pp)0.514.11

REC improves FAA by 2.01 pp, AIA by 3.59 pp (before rounding), and BWT by 3.29 pp, while reducing forgetting by 3.60 pp. The final historical-task PGD-10 average is 75.04% versus 70.74%; the newest-task accuracy is lower (59.55% versus 66.70%). The measured advantage is stronger historical robust retention, alongside a remaining newest-task plasticity gap.

App. C.1 provides the matrices, definitions, and task-level analysis. E1 uses saved REC stage checkpoints and dedicated USP-FSR-AT E1 checkpoints; it is separate from the original final benchmark. We also relate retention to the measured main-run training times, 25.72 h versus 13.11 h (1.96x), in App. E. This is evidence from one paired stream, rather than a multi-seed retention claim.

Q5. Why use CLIP-pretrained ViT-B/16 when the original FLAIR uses ResNet?

A5. Thank you for highlighting the need to explain this adaptation. Using a common ViT backbone aligns capacity and pretraining across REC and the adapted baselines, making the comparison more informative about their learning mechanisms under the same stream and threat model.

App. A.1 now separates the retained FLAIR components from the changed experimental setting. FLAIR-AT retains the adversarial-distribution stability objective, frozen previous-task teacher, and feature/prediction distillation. We adapt its ResNet/Class-IL implementation to CLIP ViT-B/16, task-local experts, and fully supervised replay-free Task-IL. The Table 1 caption and Sec. 4 identify it as our FLAIR-style adaptation. Method-specific optimization schedules and checkpoint gates are documented separately; a shared backbone does not imply identical training budgets.

Q6. How do the components form one mechanism, and what does GradDiv add beyond feature diversity?

A6. Thank you for asking us to connect the mechanism to controlled evidence. Within the process explained in A1, HSIC addresses dependence between minibatch representations, while GradDiv addresses alignment of per-sample loss sensitivities. For each temporary expert, GradDiv differentiates cross-entropy with respect to the detached CLS feature, normalizes each sample's gradient, and averages squared cosines over unordered expert pairs and samples.

We distinguish the full expert-stage ablation from the independent GradDiv ablation:

Split CIFAR-10 (%)CleanPGD-10RobustAvg
Full REC95.0572.0779.32
w/o DMEL94.3872.2578.90
w/o GradDiv94.8271.8979.05

The full expert stage improves Clean/RobustAvg by 0.67/0.42 pp, with PGD-10 0.18 pp lower. Holding HSIC and other components fixed, GradDiv adds 0.23 pp Clean, 0.18 pp PGD-10, and 0.27 pp RobustAvg. These results support a complementary, modest contribution from GradDiv and a better measured clean/multi-attack balance from DMEL. Eq. (3), App. A.1, and Sec. 4.3 specify the operation and connect it to Table 2. This experiment does not isolate uniform fusion alone or directly measure attack transfer.

AuthorsOpenReview ↗

The response

Dear Reviewer ATMB,

We truly appreciate your constructive feedback on our submission. We sincerely hope you can read our rebuttal, and we would like to know whether our responses have addressed your questions and concerns. We warmly welcome any new questions or feedback, and thank you again for your time and help in improving the quality of this paper.

Best regards,

The Authors

Reviewer_CVwxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies adversarial task-incremental learning, where a model must learn new tasks while retaining clean accuracy and adversarial robustness on previous tasks without storing historical training examples. It proposes Robust Expert Collaboration (REC), which consists of three components. Diversified Multi-Expert Learning trains temporary task experts using feature and gradient diversity regularization, together with image- and feature-space adversarial training, before merging their parameters into one expert. Robust Anchoring and Stability Cooperation uses teacher-feature anchoring, historical-decision distillation, and clean–adversarial consistency to constrain model adaptation. A Class-wise Robust Constraint selects checkpoints on held-out current-task data and restores the pre-integration checkpoint when no candidate meets the criteria. Experiments on Split CIFAR-10 and Split CIFAR-100 evaluate robustness and retention, with additional strong-attack, matched-seed, ablation, and resource-cost analyses.

Critical strengths
  1. The framework considers both historical-task retention and class-wise adversarial failures. Its checkpoint selection includes per-class recall, which can reveal failures hidden by average accuracy.

  2. The evaluation includes stronger PGD attacks, task-local AutoAttack, and matched-seed comparisons. These analyses expose both robustness and limitations, including weaker performance on the newest task.

Critical weaknesses
  1. The backbone update policy is inconsistent: Section 3.3 and Appendix A describe backbone adaptation, whereas Appendix E states that only the current expert is optimized after Task 0. This makes it unclear whether historical retention results from anchoring or simply freezing the backbone.

  2. The matched-seed results show almost no aggregate advantage over USP-FSR-AT: RobustAvg is 78.69% versus 78.65%, while PGD accuracy is lower (Table 6). REC also requires approximately twice the training time (Table 9), leaving the benefit of its added complexity unclear.

Questions for authors
  1. Is the backbone updated after Task 0? If it is frozen, how do the teacher-feature and historical-decision losses affect optimization?

  2. What are the numerical CRC thresholds and exact checkpoint-ranking rule, and do the baselines receive comparable validation-based selection?

  3. Removing DMEL slightly improves PGD accuracy in Table 2; what evidence shows that expert diversity and parameter averaging improve the final model?

  4. Is GradDiv computed with respect to input pixels, as in Equation 3, or backbone features, as stated in the introduction?

  5. Do the retention and robustness trade-offs persist across different task orders and matched-seed strong-attack evaluations on Split CIFAR-100?

6 comments
AuthorsOpenReview ↗

Response 1/2 to Reviewer CVwx

We appreciate your careful questions about the backbone update, checkpoint selection, and the contribution of DMEL.

Q1. Is the backbone updated after Task 0, and how do the frozen teacher and historical experts constrain it?

A1. Thank you for identifying where the update policy needs to be clearer. Yes, RASC updates the shared backbone after Task 0 as well as on Task 0. The freeze applies to DMEL, which trains three current-task experts (1.169M trainable parameters); RASC then updates the active backbone and fused current expert (85.807M + 0.390M = 86.197M). The teacher snapshot and old experts remain frozen, but their feature and historical-decision losses send gradients through the active path, constraining its update. CRC selects or restores a checkpoint without training parameters. Thus, 0.390M denotes one expert, not the full RASC update. We will align the update-policy wording in Secs. 3.2–3.3 and App. E, and place a stage-wise freeze/update table beside the method overview.

Q2. What are CRC's exact selection rules, and how is baseline checkpoint selection compared?

A2. We appreciate the request for a reproducible comparison. CRC evaluates each RASC epoch on a deterministic 10% training holdout (at most 500 images) using random-start, post-resize PGD-ℓ∞\ell_\infty-10 (ϵ=8/255\epsilon=8/255, step 2/2552/255). Relative to the post-DMEL fusion checkpoint A0A_0, its gates are:

GateRASC candidate requirement
CleanTask 0: ≥max⁡(92%,A0clean−4 pp)\geq\max(92\%,A_0^{\rm clean}-4\,\mathrm{pp}); later: ≥max⁡(88%,A0clean−4 pp)\geq\max(88\%,A_0^{\rm clean}-4\,\mathrm{pp})
PGD-10≥A0PGD+0.5 pp\geq A_0^{\rm PGD}+0.5\,\mathrm{pp}
Class retentionMinimum clean class recall ≥80%\geq80\%
Historical retentionTeacher-feature cosine ≥0.88\geq0.88; old-decision agreement ≥85%\geq85\% when applicable

Feasible checkpoints rank by PGD-10, then Clean, then agreement; ties favor the earlier epoch. If none passes, the active backbone and expert revert to A0A_0. The Task-0 anchor has a separate pre-DMEL gate. Baselines share the holdout scale, attack, and per-epoch checking, but keep their own selection rules. REC evaluates 8 Task-0 RASC epochs and 10 later epochs; USP-FSR-AT evaluates 20 and 10. No test examples select checkpoints. We will juxtapose the shared inputs, method-specific gates, and candidate budgets in Sec. 3.4 and App. A.1/C.3.

Q3. What evidence shows that expert diversity and parameter averaging improve the final model?

A3. Thank you for asking what the expert stage contributes in the full model. DMEL trains complementary candidates, then uses uniform parameter averaging to form one current-task expert. HSIC reduces dependence among intermediate representations, while GradDiv discourages aligned per-sample loss gradients with respect to the frozen backbone's CLS feature. The controlled Table 2 comparisons are:

Split CIFAR-10 (%)CleanPGD-10RobustAvg
Full REC95.0572.0779.32
w/o DMEL94.3872.2578.90
w/o GradDiv94.8271.8979.05

With DMEL, Clean increases by 0.67 pp and RobustAvg by 0.42 pp. The full Table 2 also shows gains on FGSM (+0.53 pp), PGD-ℓ2\ell_2 (+0.74 pp), and BIM (+0.59 pp); PGD-ℓ∞\ell_\infty has a different, small ordering, visible in the table above. This pattern supports DMEL's contribution to the overall clean/multi-attack balance. Keeping HSIC and the other components, GradDiv adds a further 0.23 pp Clean, 0.18 pp PGD-10, and 0.27 pp RobustAvg. We will explain how complementary experts are combined by uniform averaging and present these metric-specific comparisons beside Table 2 in Sec. 4.3.

AuthorsOpenReview ↗

Response 2/2 to Reviewer CVwx

We next address the feature derivative, the measured cost–retention trade-off, and the CIFAR-100 evidence.

Q4. What does GradDiv differentiate, and how does this differ from PGD?

A4. Thank you for asking us to clarify the derivative. GradDiv differentiates expert loss with respect to a detached backbone feature, not input pixels. For sample bb and temporary expert kk, Eq. (3) defines zb=fθ(xb)z_b=f_\theta(x_b) and gk,b=∇zbLCE(Ek(zb),yb)g_{k,b}=\nabla_{z_b}\mathcal L_{\rm CE}(E_k(z_b),y_b). The penalty ℓ2\ell_2-normalizes these per-sample gradients, squares their pairwise cosines, and averages over unordered expert pairs and the minibatch. DMEL's frozen backbone is not updated; its CLS feature is the gradient probe. Pixel- and feature-space PGD are separate adversarial-example objectives, not the derivative used by GradDiv. Eq. (3) and App. A.1 already specify the implemented operation. We will bring this distinction into the nearby Sec. 3.2 prose so the equation, update path, and attack generation cannot be conflated.

Q5. What benefits does REC's additional training provide for final robustness and historical retention?

A5. Thank you for asking us to relate the additional cost to measured benefits. The main-run training times are 25.72 h for REC and 13.11 h for USP-FSR-AT (1.96$\times$). In Table 6's three matched seeds, REC has the higher mean on five of six final metrics, although the margins are small and PGD-10 favors USP-FSR-AT:

Split CIFAR-10, three-seed mean (%)RECUSP-FSR-AT
Clean94.4694.38
FGSM77.9377.83
PGD-ℓ∞\ell_\infty-1071.4271.80
PGD-ℓ2\ell_2-1093.5793.56
BIM-1071.8271.40
RobustAvg78.6978.65

Our separate paired E1 audit addresses the retention question directly: it evaluates all 15 valid stage–task cells per method on Split CIFAR-10, seed 1993, Task-IL, with full-test random-start PGD-10:

E1 metricRECUSP-FSR-AT
Clean FAA (%)94.4994.14
PGD-10 FAA (%)71.9469.93
PGD-10 AIA (%)67.5463.94
PGD BWT (pp)+0.79-2.50
PGD forgetting F (pp)0.514.11

In this paired stream, REC improves PGD FAA by 2.01 pp, AIA by 3.59 pp (unrounded), and BWT by 3.29 pp, while reducing forgetting by 3.60 pp. Thus the additional computation accompanies stronger historical robust retention in the evaluated stream, even though the three-seed final RobustAvg is closely matched. We will add the paired E1 matrices and metric definitions to App. C.1 and connect retention with the reported training time in Sec. 4.1/App. E.

Q6. Do the retention and robustness trends persist across task orders and matched-seed strong attacks on Split CIFAR-100?

A6. We appreciate the suggestion to test whether the comparison persists beyond one task stream. We have extended the analysis in two directions: a complete ten-task Split CIFAR-100 benchmark and a paired alternative order on Split CIFAR-10. On CIFAR-100, REC obtains:

Split CIFAR-100, Task-IL (%)RECUSP-FSR-AT
Clean92.3985.33
PGD-ℓ∞\ell_\infty-1048.9747.18

Across all ten CIFAR-100 tasks, REC gains 7.06 pp Clean and 1.79 pp PGD-10 over USP-FSR-AT under the common CLIP ViT-B/16, 224×224224\times224 Task-IL and post-resize attack protocol. These are the main Table 1 results.

For Split CIFAR-10, we compare the original order, (4,2)∣(7,6)∣(0,3)∣(5,8)∣(9,1)(4,2)|(7,6)|(0,3)|(5,8)|(9,1), with an additional order, (3,5)∣(2,9)∣(0,7)∣(4,1)∣(6,8)(3,5)|(2,9)|(0,7)|(4,1)|(6,8):

Class orderMethodCleanPGD-10RobustAvg
OriginalREC95.0572.0779.32
OriginalUSP-FSR-AT93.4471.8278.35
Order AREC94.1071.2278.10
Order AUSP-FSR-AT93.8071.0078.00

On the original order, REC leads by 1.61 pp Clean, 0.25 pp PGD-10, and 0.97 pp RobustAvg; on Order A it remains higher on all three, by 0.30, 0.22, and 0.10 pp. The margins vary with order, but the direction is consistent in these two paired CIFAR-10 streams. We will add the Order A protocol and results to the Appendix and summarize both the CIFAR-100 scale comparison and order analysis in Sec. 4. The requested matched-seed strong attacks and alternative orders on CIFAR-100 are separate experiments; we will evaluate them and report their results in App. B.2 rather than infer them from the current tables.

Thank you very much for the author's response and the corresponding experimental additions, but I did not see the corresponding modifications in the PDF, so I am currently keeping the score unchanged.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their careful evaluations, constructive suggestions, and valuable discussions. Your feedback has helped us substantially improve the technical clarity and experimental evidence of our paper.

Our work addresses a central challenge in adversarial continual learning: acquiring robustness on incoming tasks while preserving robust knowledge from previous tasks without replaying historical samples. During the rebuttal, we provided additional clarifications, conducted new experiments, and incorporated the corresponding analyses into the revised manuscript. We summarize the main updates below.

1. Methodological Clarifications and Experimental Protocol. We have clarified the stage-wise backbone optimization, the complementary roles of DMEL and RASC, GradDiv's gradient formulation, and CRC's checkpoint selection and rollback rules. We have also expanded the training and evaluation protocols and clarified the implementation of adapted baselines, including FLAIR-AT (Tables 5 and 7).

2. Expanded Robustness Evaluation and Generalization. We have extended our evaluation beyond Split CIFAR-10/100 to a matched TinyImageNet setting, where REC achieves 83.10% Clean, 39.10% PGD-10, and 51.24% RobustAvg, outperforming the adapted FLAIR-AT baseline (Table 1). We additionally provide the complete ten-task Split CIFAR-100 comparison (Table 9), an alternative class-order experiment (Table 11), and stronger PGD-20/40 and AutoAttack evaluations on Split CIFAR-10 (Table 8).

3. Historical Robustness Retention and Computational Cost. We have conducted a paired five-stage evaluation, including complete Clean and PGD-10 accuracy matrices (Tables 13–15). REC achieves higher PGD-10 final average accuracy (71.94% vs. 69.93%) and substantially lower forgetting (0.51 vs. 4.11 percentage points) than USP-FSR-AT. We also report the measured training costs (Table 19), enabling a more comprehensive assessment of the robustness–retention–efficiency trade-off.

4. Component Contributions and Task-Agnostic Evaluation. We have strengthened the component analysis with DMEL and independent GradDiv ablations (Table 2), together with hyperparameter sensitivity experiments (Table 6). We further evaluate a Class-IL inference extension without task IDs, where REC achieves 46.17% Clean and 13.60% PGD-10, compared with 44.70% and 11.58% for USP-FSR-AT (Table 21).

We have incorporated these additional experiments, methodological clarifications, and corresponding discussions into the revised manuscript. We sincerely appreciate the reviewers' careful feedback and the Area Chair's time and consideration. Their suggestions have been invaluable in strengthening the clarity, rigor, and presentation of our work.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.