- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates dictionary learning (DL) for reducing overfitting in adversarial training of robust classifiers. The authors hypothesize that vanilla DL lacks robustness due to the L2 reconstruction loss learning excessive signal, and propose an L1 reconstruction penalty for correction. The L1-based loss is optimized using a novel iterative learning algorithm. The authors demonstrate that Elastic AT-DL improves robustness over vanilla AT-DL, and that Elastic AT-DL outperforms standard AT regularization techniques such as data augmentation and early stopping.
Critical strengths
The authors investigate a novel DL-based approach for reducing robust overfitting. The authors introduce a novel L1-based reconstruction loss to the standard DL loss and demonstrate that the new loss greatly improves the robustness of AT-based DL. The proposed approach demonstrates improved robsutness over established baselines such as data augmentation and early stopping.
Critical weaknesses
-
Section3.1 claims that “elastic dictionary learning improves robustness” over vanilla DL. The claim is about the robustness of DL under standard training. However, there is no experimental results comparing robustness of Elastic DL vs Vanilla DL under standard training.
-
The comparison misses strong augmentation-based baselines such as CutMix + Weight Averaging [a] and IDBH [b]. In addition, the comparison is mainly performed with resnet18, where overfitting is less prone to happen than larger models such as wideresnet. The authors conducted experiments on additional datasets with more resnet variants but comparison is limited to Elastic DL vs Vanilla DL (Table 7, 8, and 9), so the gain over baselines is unclear in these setups.
-
The paper misses computational cost comparision with ResNet + PGD-AT (table 6 only reports computational cost of Elastic DL vs Vanilla DL).
[a] Rebuffi, Sylvestre-Alvise, et al. "Fixing data augmentation to improve adversarial robustness." arXiv preprint arXiv:2103.01946 (2021).
[b] Li, Lin, and Michael Spratling. "Data augmentation alone can improve adversarial training." arXiv preprint arXiv:2301.09879 (2023).
Questions for authors
None
4 comments
W3. The paper misses computational cost comparision with ResNet + PGD-AT (table 6 only reports computational cost of Elastic DL vs Vanilla DL).
Response: Following the reviewer's suggestion, we extend the computational cost comparison to include the standard ResNet18 with PGD-AT (CIFAR-10, same hardware):
| Method | Train (h) | Params (M) | FLOPs (G) | Size (MB) |
|---|---|---|---|---|
| ResNet18 | 8.2 | 11.174 | 2.280 | 42.70 |
| Vanilla DL | 13.8 | 11.174 | 4.776 | 42.71 |
| Elastic DL | 15.2 | 11.174 | 4.776 | 42.71 |
Elastic DL has essentially the same number of parameters and model size as the standard ResNet18, since it reuses the same convolutional dictionaries and only adds one scalar per layer. The additional cost lies only in computation, from the unrolled RISTA iterations: compared with ResNet18 + PGD-AT, Elastic DL requires about training time and FLOPs. This overhead is moderate relative to the robustness gain, as Elastic DL improves AutoAttack accuracy from to () with higher clean accuracy (Table 3). In comparison, robustness gains of this magnitude are typically obtained with substantially larger backbones or millions of additional training images, which increase the training cost much more and, for larger backbones, also the model size. Moreover, the cost is flexible: it scales linearly with the number of unrolled iterations , and even already improves PGD accuracy from to .
Thank the authors for providing additional experiments and clarifications. I am inclined to maintain my original rating, mainly because of the novelty of this architectural direction and the solid CIFAR-10 results on ResNet18. However, the following remain major concerns:
- A central motivation of this work is to mitigate robust overfitting through Elastic DL, an approach orthogonal to existing augmentation-based strategies. However, Elastic DL with adversarial training is still much weaker than strong augmentation-based methods on WRN-34-10 while introducing considerable computational overhead.
- Although the authors argue that Elastic DL is complementary to established approaches, architectural orthogonality does not necessarily imply complementary gains. In particular, the benefits of combining Elastic DL with strong augmentation-based methods have not been demonstrated.
Overall, while the improvements over Vanilla DL are clear, the evidence remains insufficient to establish the competitiveness of Elastic DL as a general approach to mitigating robust overfitting.
We sincerely thank the reviewer for the thoughtful follow-up and for recognizing the novelty of this architectural direction and the solid results. We address the remaining concerns below.
1. Complementarity with strong data-centric methods. We agree that architectural orthogonality does not by itself imply complementary gains, which is why we empirically examined the combination of Elastic DL with strong defenses. In particular, PORT is a data-centric defense that leverages DDPM-generated data, in the same spirit as the generated-data setting that underlies the strongest results of [a]. Even on top of this strong data-centric baseline, Elastic DL further improves AutoAttack accuracy (55.14% → 56.30%, Table 4). Similarly, Elastic DL substantially improves HAT (53.16% → 59.07% AutoAttack accuracy, Table 3). These results provide direct evidence that the gains of Elastic DL are complementary to, rather than subsumed by, existing strong training strategies.
2. Comparison on WRN-34-10. Our WRN-34-10 experiment was designed to verify that the improvement of Elastic DL over Vanilla DL holds on a larger backbone, rather than to compete with augmentation methods whose training recipes are specifically tuned for WRN. In this setting, we directly reused the training configuration of our ResNet18 experiments without additional tuning, and Elastic DL consistently improved robustness under all attacks (+3.92% AutoAttack accuracy). Since Elastic DL modifies only the convolutional layers, it can be applied on top of these augmentation-based recipes, which we view as the most promising way to push absolute performance further.
3. Computational overhead. Elastic DL keeps the same number of parameters and model size as the standard backbone, and its computational cost is flexible along two dimensions. First, the number of unrolled iterations can be reduced; for example, already substantially improves robustness at a lower cost. Second, Elastic DL can be applied to only a subset of the convolutional layers rather than replacing all of them, which further reduces the overhead and allows practitioners to trade off robustness and efficiency according to their computational budget. We will clearly discuss this accuracy–robustness–cost trade-off in the final version.
We thank the reviewer again for the constructive feedback and the positive assessment, which have helped us improve the paper.
[a] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A. Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Fixing data augmentation to improve adversarial robustness. arXiv preprint arXiv:2103.01946, 2021.
[b] Lin Li and Michael Spratling. Data augmentation alone can improve adversarial training. In International Conference on Learning Representations (ICLR), 2023.
We sincerely thank the reviewer for the valuable comments. We address each point below.
Response: We thank the reviewer for this comment and clarify the scope of our claim. Throughout the paper, all robustness results of Vanilla DL and Elastic DL are obtained under adversarial training, and the claim in Section 3.1 should be understood in this setting. Under standard training, deep networks generally have essentially no robustness against adaptive attacks, which also holds for dictionary-learning-based networks: as shown in our preliminary study (Figure 1 and Appendix E.1), Vanilla DL under standard training is resilient to random noise, but its accuracy drops to nearly zero under adaptive PGD attacks. Therefore, comparing Vanilla DL and Elastic DL under standard training would not provide meaningful evidence about adversarial robustness, and adversarial training is the standard setting for evaluating it.
Response: We thank the reviewer for these suggestions and address each point below.
(1) Augmentation-based baselines. Elastic DL is an architectural modification, which is orthogonal to augmentation-based methods such as CutMix + Weight Averaging [a] and IDBH [b], rather than a competing alternative. These methods modify the training data or the optimization trajectory, while Elastic DL modifies the convolutional layers, so they can be directly combined. This is supported by our existing results: Elastic DL consistently improves a wide range of adversarial training methods (Table 3 and Figure 15), including PORT, which leverages additional DDPM-generated data in the same spirit as [a], improving its AutoAttack accuracy (Table 4). In Table 2, Elastic DL also outperforms the augmentation-based methods we compared (Cutout and Mixup) in mitigating robust overfitting.
(2) Larger backbones. Following the reviewer's suggestion, we evaluate Elastic DL on WideResNet-34-10 with PGD-AT on CIFAR-10:
Elastic DL consistently improves robustness on the larger backbone (e.g., +3.92% AutoAttack accuracy), showing that its benefit is not limited to ResNet18.
(3) Comparison with baselines in Tables 7–9. In these experiments, we focus on Vanilla DL as the baseline because it shares exactly the same architecture and training procedure with Elastic DL, differing only in the reconstruction objective, and thus directly isolates the effect of our contribution. Moreover, Vanilla DL is already a stronger baseline than the standard convolutional network: on ResNet18, it achieves higher clean and robust accuracy than the standard ResNet18 (e.g., 82.67%±0.50 vs. 80.60%±0.42 clean and 46.01%±0.31 vs. 43.77%±0.25 PGD-20 accuracy over three independent runs). Therefore, the consistent improvements of Elastic DL over Vanilla DL across datasets and backbones indicate improvements over the standard baselines as well.