acceptodds
Under review as a conference paper at ICLR 2027

Boosting Adversarial Robustness and Generalization with Dictionary Structure

Abstract

This work investigates a novel approach to boost adversarial robustness and generalization by incorporating structural prior into the design of deep learning models. Specifically, our study surprisingly reveals that existing dictionary learning-inspired convolutional neural networks (CNNs) provide a false sense of security against adversarial attacks. To address this, we propose Elastic Dictionary Learning Networks (EDLNets), a novel ResNet architecture that significantly enhances adversarial robustness and generalization. Extensive and reliable experiments demonstrate consistent and significant performance improvement on open robustness leaderboards such as RobustBench, surpassing state-of-the-art baselines. To the best of our knowledge, this is the first work to discover and validate that dictionary structure can reliably enhance deep learning robustness under strong adaptive attacks, unveiling a promising direction for future research. Our implementation is available at https://anonymous.4open.science/r/ElasticDL-6343.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Robustness, Dictionary Learning, Structural Prior

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YF7ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates dictionary learning (DL) for reducing overfitting in adversarial training of robust classifiers. The authors hypothesize that vanilla DL lacks robustness due to the L2 reconstruction loss learning excessive signal, and propose an L1 reconstruction penalty for correction. The L1-based loss is optimized using a novel iterative learning algorithm. The authors demonstrate that Elastic AT-DL improves robustness over vanilla AT-DL, and that Elastic AT-DL outperforms standard AT regularization techniques such as data augmentation and early stopping.

Critical strengths

The authors investigate a novel DL-based approach for reducing robust overfitting. The authors introduce a novel L1-based reconstruction loss to the standard DL loss and demonstrate that the new loss greatly improves the robustness of AT-based DL. The proposed approach demonstrates improved robsutness over established baselines such as data augmentation and early stopping.

Critical weaknesses
  • Section3.1 claims that “elastic dictionary learning improves robustness” over vanilla DL. The claim is about the robustness of DL under standard training. However, there is no experimental results comparing robustness of Elastic DL vs Vanilla DL under standard training.

  • The comparison misses strong augmentation-based baselines such as CutMix + Weight Averaging [a] and IDBH [b]. In addition, the comparison is mainly performed with resnet18, where overfitting is less prone to happen than larger models such as wideresnet. The authors conducted experiments on additional datasets with more resnet variants but comparison is limited to Elastic DL vs Vanilla DL (Table 7, 8, and 9), so the gain over baselines is unclear in these setups.

  • The paper misses computational cost comparision with ResNet + PGD-AT (table 6 only reports computational cost of Elastic DL vs Vanilla DL).

[a] Rebuffi, Sylvestre-Alvise, et al. "Fixing data augmentation to improve adversarial robustness." arXiv preprint arXiv:2103.01946 (2021).

[b] Li, Lin, and Michael Spratling. "Data augmentation alone can improve adversarial training." arXiv preprint arXiv:2301.09879 (2023).

Questions for authors

None

4 comments

We sincerely thank the reviewer for the valuable comments. We address each point below.

W1. Section3.1 claims that "elastic dictionary learning improves robustness" over vanilla DL. The claim is about the robustness of DL under standard training. However, there is no experimental results comparing robustness of Elastic DL vs Vanilla DL under standard training.

Response: We thank the reviewer for this comment and clarify the scope of our claim. Throughout the paper, all robustness results of Vanilla DL and Elastic DL are obtained under adversarial training, and the claim in Section 3.1 should be understood in this setting. Under standard training, deep networks generally have essentially no robustness against adaptive attacks, which also holds for dictionary-learning-based networks: as shown in our preliminary study (Figure 1 and Appendix E.1), Vanilla DL under standard training is resilient to random noise, but its accuracy drops to nearly zero under adaptive PGD attacks. Therefore, comparing Vanilla DL and Elastic DL under standard training would not provide meaningful evidence about adversarial robustness, and adversarial training is the standard setting for evaluating it.

W2. The comparison misses strong augmentation-based baselines such as CutMix + Weight Averaging [a] and IDBH [b]. In addition, the comparison is mainly performed with resnet18, where overfitting is less prone to happen than larger models such as wideresnet. The authors conducted experiments on additional datasets with more resnet variants but comparison is limited to Elastic DL vs Vanilla DL (Table 7, 8, and 9), so the gain over baselines is unclear in these setups.

Response: We thank the reviewer for these suggestions and address each point below.

(1) Augmentation-based baselines. Elastic DL is an architectural modification, which is orthogonal to augmentation-based methods such as CutMix + Weight Averaging [a] and IDBH [b], rather than a competing alternative. These methods modify the training data or the optimization trajectory, while Elastic DL modifies the convolutional layers, so they can be directly combined. This is supported by our existing results: Elastic DL consistently improves a wide range of adversarial training methods (Table 3 and Figure 15), including PORT, which leverages additional DDPM-generated data in the same spirit as [a], improving its AutoAttack accuracy (Table 4). In Table 2, Elastic DL also outperforms the augmentation-based methods we compared (Cutout and Mixup) in mitigating robust overfitting.

(2) Larger backbones. Following the reviewer's suggestion, we evaluate Elastic DL on WideResNet-34-10 with PGD-AT on CIFAR-10:

MethodNaturalPGDFGSMC&WAA
Vanilla DL + WRN-34-1085.1247.8656.4144.3746.20
Elastic DL + WRN-34-1085.3852.0460.7348.9550.12

Elastic DL consistently improves robustness on the larger backbone (e.g., +3.92%+3.92\% AutoAttack accuracy), showing that its benefit is not limited to ResNet18.

(3) Comparison with baselines in Tables 7–9. In these experiments, we focus on Vanilla DL as the baseline because it shares exactly the same architecture and training procedure with Elastic DL, differing only in the reconstruction objective, and thus directly isolates the effect of our contribution. Moreover, Vanilla DL is already a stronger baseline than the standard convolutional network: on ResNet18, it achieves higher clean and robust accuracy than the standard ResNet18 (e.g., 82.67%±0.5082.67\% \pm 0.50 vs. 80.60%±0.4280.60\% \pm 0.42 clean and 46.01%±0.3146.01\% \pm 0.31 vs. 43.77%±0.2543.77\% \pm 0.25 PGD-20 accuracy over three independent runs). Therefore, the consistent improvements of Elastic DL over Vanilla DL across datasets and backbones indicate improvements over the standard baselines as well.

W3. The paper misses computational cost comparision with ResNet + PGD-AT (table 6 only reports computational cost of Elastic DL vs Vanilla DL).

Response: Following the reviewer's suggestion, we extend the computational cost comparison to include the standard ResNet18 with PGD-AT (CIFAR-10, same hardware):

MethodTrain (h)Params (M)FLOPs (G)Size (MB)
ResNet188.211.1742.28042.70
Vanilla DL13.811.1744.77642.71
Elastic DL15.211.1744.77642.71

Elastic DL has essentially the same number of parameters and model size as the standard ResNet18, since it reuses the same convolutional dictionaries and only adds one scalar β\beta per layer. The additional cost lies only in computation, from the unrolled RISTA iterations: compared with ResNet18 + PGD-AT, Elastic DL requires about 1.9×1.9\times training time and 2.1×2.1\times FLOPs. This overhead is moderate relative to the robustness gain, as Elastic DL improves AutoAttack accuracy from 42.14%42.14\% to 52.90%52.90\% (+10.76%+10.76\%) with higher clean accuracy (Table 3). In comparison, robustness gains of this magnitude are typically obtained with substantially larger backbones or millions of additional training images, which increase the training cost much more and, for larger backbones, also the model size. Moreover, the cost is flexible: it scales linearly with the number of unrolled iterations TT, and even T=1T = 1 already improves PGD accuracy from 44.35%44.35\% to 50.18%50.18\%.

Thank the authors for providing additional experiments and clarifications. I am inclined to maintain my original rating, mainly because of the novelty of this architectural direction and the solid CIFAR-10 results on ResNet18. However, the following remain major concerns:

  • A central motivation of this work is to mitigate robust overfitting through Elastic DL, an approach orthogonal to existing augmentation-based strategies. However, Elastic DL with adversarial training is still much weaker than strong augmentation-based methods on WRN-34-10 while introducing considerable computational overhead.
  • Although the authors argue that Elastic DL is complementary to established approaches, architectural orthogonality does not necessarily imply complementary gains. In particular, the benefits of combining Elastic DL with strong augmentation-based methods have not been demonstrated.

Overall, while the improvements over Vanilla DL are clear, the evidence remains insufficient to establish the competitiveness of Elastic DL as a general approach to mitigating robust overfitting.

We sincerely thank the reviewer for the thoughtful follow-up and for recognizing the novelty of this architectural direction and the solid results. We address the remaining concerns below.

1. Complementarity with strong data-centric methods. We agree that architectural orthogonality does not by itself imply complementary gains, which is why we empirically examined the combination of Elastic DL with strong defenses. In particular, PORT is a data-centric defense that leverages DDPM-generated data, in the same spirit as the generated-data setting that underlies the strongest results of [a]. Even on top of this strong data-centric baseline, Elastic DL further improves AutoAttack accuracy (55.14% → 56.30%, Table 4). Similarly, Elastic DL substantially improves HAT (53.16% → 59.07% AutoAttack accuracy, Table 3). These results provide direct evidence that the gains of Elastic DL are complementary to, rather than subsumed by, existing strong training strategies.

2. Comparison on WRN-34-10. Our WRN-34-10 experiment was designed to verify that the improvement of Elastic DL over Vanilla DL holds on a larger backbone, rather than to compete with augmentation methods whose training recipes are specifically tuned for WRN. In this setting, we directly reused the training configuration of our ResNet18 experiments without additional tuning, and Elastic DL consistently improved robustness under all attacks (+3.92% AutoAttack accuracy). Since Elastic DL modifies only the convolutional layers, it can be applied on top of these augmentation-based recipes, which we view as the most promising way to push absolute performance further.

3. Computational overhead. Elastic DL keeps the same number of parameters and model size as the standard backbone, and its computational cost is flexible along two dimensions. First, the number of unrolled iterations TT can be reduced; for example, T=1T = 1 already substantially improves robustness at a lower cost. Second, Elastic DL can be applied to only a subset of the convolutional layers rather than replacing all of them, which further reduces the overhead and allows practitioners to trade off robustness and efficiency according to their computational budget. We will clearly discuss this accuracy–robustness–cost trade-off in the final version.

We thank the reviewer again for the constructive feedback and the positive assessment, which have helped us improve the paper.

[a] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A. Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Fixing data augmentation to improve adversarial robustness. arXiv preprint arXiv:2103.01946, 2021.

[b] Lin Li and Michael Spratling. Data augmentation alone can improve adversarial training. In International Conference on Learning Representations (ICLR), 2023.

Reviewer_tWAROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Elastic Dictionary Learning (Elastic DL) for adversarially robust image classification. It adds an ℓ1\ell_1 reconstruction term to the squared ℓ2\ell_2 objective of convolutional dictionary learning, derives a reweighted iterative update, and unrolls the update into network layers. The models are pretrained with Vanilla DL and then fine-tuned with Elastic DL. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet report improved robust accuracy across several backbones and adversarial-training methods.

Critical strengths
  1. The preliminary study identifies a relevant limitation of dictionary-learning networks: resistance to random noise does not translate into resistance to adaptive PGD attacks. This motivates examining the reconstruction objective in an adversarial setting.
  2. The reported AutoAttack gains over Vanilla DL extend across the tested datasets and backbones. Results with several adversarial-training methods provide initial evidence that Elastic DL can be combined with existing defenses.
Critical weaknesses
  1. The introduction characterizes prior dictionary-learning methods as mainly validated against random noise or universal perturbations, without discussing supervised sparse coding under adversarial perturbations [1]. Prior convolutional sparse coding has also used ℓ1\ell_1 data fidelity for noise restoration [2]. Neither work addresses the full setting studied here, but the paper should distinguish its contribution more precisely from these results and from established iterative reweighting methods [3].
  2. Figure 10 shows that Elastic DL trained from scratch does not achieve the main robustness gain. The reported results rely on pretraining Vanilla DL and then switching to Elastic DL. Continuing Vanilla DL from the same checkpoint under the identical fine-tuning schedule is needed to isolate the effect of the proposed objective.
  3. The paper does not report the AutoAttack sample count, seeds, or uncertainty estimates, and several attack settings are unspecified. The main Elastic DL model also has different PGD accuracies in Tables 2, 3, and 11. The authors should identify the checkpoints and protocols behind these results.
  4. The BPDA experiment specifically examines the soft-threshold operation. Although the reported white-box attacks differentiate through the reweighting operation, the paper does not separately assess whether it affects attack reliability. A targeted diagnostic, together with attack-strength sensitivity, would make the evaluation more convincing.
  5. The HAT and PORT comparisons do not sufficiently specify the baseline checkpoints, training data, implementation, and evaluation protocols. These details are needed to assess the claimed improvement over existing defenses.
  6. Lemma 3.1 uses weights based on 1/(2∣r∣)1/(2|r|), whereas the implemented update uses stabilized weights. The descent argument also treats one proximal-gradient step as an exact surrogate minimization. The stated guarantee therefore does not directly follow for the implemented algorithm.
  7. The proposed mechanism assumes that adversarial perturbations create influential reconstruction residuals, but bounded perturbations need not behave like large outliers. The paper does not report residual distributions, changes in the learned representations, or the layer-wise values of β\beta. Without this evidence and the matched training control in point 2, it is unclear why the full procedure improves robustness.

Minor:

  1. State the number of unrolled updates used in the evaluated networks and relate it to Figure 7.
  2. Report the perturbation budgets and full settings for FGSM, C&W, PGD-ℓ₂, and Tiny-ImageNet.
  3. Clarify that randomized smoothing certifies the smoothed classifier, rather than the deterministic model evaluated with AutoAttack.
  4. Distinguish the generalization claims supported by robust-overfitting results from those supported by out-of-distribution noise experiments.

References:

[1] Sulam, Jeremias, Ramchandran Muthukumar, and Raman Arora. Adversarial robustness of supervised sparse coding. In NeurIPS, 2020.

[2] Wohlberg, Brendt. Convolutional sparse representations as an image model for impulse noise restoration. IEEE IVMSP Workshop, 2016.

[3] Daubechies, Ingrid, Ronald DeVore, Massimo Fornasier, and C. Sinan Güntürk. Iteratively reweighted least squares minimization for sparse recovery. Communications on Pure and Applied Mathematics, 63(1):1–38, 2010.

Questions for authors
  1. How many images and runs were used for each AutoAttack result?
  2. Which checkpoints and attack settings produced the PGD results in Tables 2, 3, and 11?
  3. How does Vanilla DL perform when training continues from the same pre-switch checkpoint under the identical schedule?
  4. What are the learned layer-wise values of β\beta?
  5. Which training data and checkpoints were used for the HAT and PORT baselines?
  6. How does robustness change under an evaluation directed at the residual-dependent reweighting operation?
  7. What conditions guarantee descent for the stabilized weights and update used in the network?
9 comments

We sincerely thank the reviewer for the valuable comments. We address each point below.

W1. The introduction characterizes prior dictionary-learning methods as mainly validated against random noise or universal perturbations, without discussing supervised sparse coding under adversarial perturbations [1]. Prior convolutional sparse coding has also used ℓ1\ell_1 data fidelity for noise restoration [2]. Neither work addresses the full setting studied here, but the paper should distinguish its contribution more precisely from these results and from established iterative reweighting methods [3].

[1] Sulam, Jeremias, Ramchandran Muthukumar, and Raman Arora. Adversarial robustness of supervised sparse coding. In NeurIPS, 2020.

[2] Wohlberg, Brendt. Convolutional sparse representations as an image model for impulse noise restoration. IEEE IVMSP Workshop, 2016.

[3] Daubechies, Ingrid, Ronald DeVore, Massimo Fornasier, and C. Sinan Güntürk. Iteratively reweighted least squares minimization for sparse recovery. Communications on Pure and Applied Mathematics, 63(1):1–38, 2010.

Response: We thank the reviewer for pointing out these relevant works. We clarify the differences between our contribution and these works below:

  • [1] theoretically analyzes a single-layer sparse encoder with a linear classifier, providing robust generalization bounds and certificates under ℓ2\ell_2-bounded perturbations. In contrast, we embed dictionary structure into every layer of deep convolutional networks, train them end-to-end with adversarial training, and evaluate them empirically under strong adaptive attacks on large-scale benchmarks. Our finding that conventional ℓ2\ell_2-based sparse coding layers remain vulnerable to adaptive attacks in deep networks is complementary to [1].

  • [2] uses ℓ1\ell_1 data fidelity in convolutional sparse coding for impulse-noise restoration, which supports our motivation that ℓ1\ell_1 fidelity is robust to outlying noise. Our work differs in combining ℓ1\ell_1 and ℓ2\ell_2 reconstruction with a learnable layer-wise balance, unrolling the solver into trainable layers of a deep classifier, and targeting adaptive adversarial attacks rather than random noise.

  • [3] establishes iteratively reweighted least squares (IRLS), on which our reweighting builds; we do not claim reweighting itself as a contribution. Our contribution is to combine IRLS-style reweighting of the ℓ1\ell_1 reconstruction term with proximal steps for the ℓ1\ell_1 sparsity term, yielding an efficient solver for the doubly non-smooth elastic objective, and to unroll it into convolutional layers that integrate with existing adversarial training methods.

W2. Figure 10 shows that Elastic DL trained from scratch does not achieve the main robustness gain. The reported results rely on pretraining Vanilla DL and then switching to Elastic DL. Continuing Vanilla DL from the same checkpoint under the identical fine-tuning schedule is needed to isolate the effect of the proposed objective.

Response: We thank the reviewer for raising this point. We clarify that the Vanilla DL baseline in our paper already serves as exactly this control. Vanilla DL is trained for 200 epochs, which is identical to pretraining Vanilla DL for 150 epochs and then continuing to train it from the same checkpoint for the remaining 50 epochs under the same schedule. The Elastic DL model shares the same first 150 epochs and the same schedule for the last 50 epochs; the only difference is that the reconstruction objective is switched to the elastic one at epoch 150. Therefore, the comparison between Vanilla DL and Elastic DL isolates the effect of the proposed objective.

Under this matched comparison, continuing Vanilla DL does not improve robustness. In fact, Vanilla DL suffers from robust overfitting after epoch 100, and its test robust accuracy keeps decreasing during the last 50 epochs (Figure 2 and Figure 12). In contrast, switching to the elastic objective at epoch 150 immediately and substantially improves robustness, leading to 53.22%53.22\% vs. 45.64%45.64\% PGD accuracy and 52.90%52.90\% vs. 43.70%43.70\% AutoAttack accuracy, with comparable clean accuracy (83.57%83.57\% vs. 83.28%83.28\%; Table 3). This confirms that the robustness gain stems from the proposed objective rather than from the pretraining–fine-tuning schedule.

Regarding training from scratch (Figure 10), the lower performance is due to optimization instability in the early stage, when the backbone parameters and the balancing parameters β\beta are jointly optimized under a large learning rate. This is an optimization issue rather than a limitation of the elastic objective, and the pretraining–fine-tuning strategy is a simple and effective way to avoid it, with modest additional cost.

W3. The paper does not report the AutoAttack sample count, seeds, or uncertainty estimates, and several attack settings are unspecified. The main Elastic DL model also has different PGD accuracies in Tables 2, 3, and 11. The authors should identify the checkpoints and protocols behind these results.

Response: We thank the reviewer for this comment. Since Reviewer 5P6u raised similar concerns, we provide a unified clarification below.

(1) AutoAttack sample count and attack settings. All AutoAttack results are obtained with the official implementation (https://github.com/fra31/auto-attack) using the standard version (APGD-CE, APGD-T, FAB-T, and Square) on the full CIFAR-10 test set (10,000 images). We have added a new appendix detailing all attack settings, including the number of iterations, step sizes, random starts, and evaluation sample counts for PGD, FGSM, C&W, AutoAttack, and BPDA.

(2) Seeds and uncertainty estimates. We have trained the key ResNet18 models (standard ResNet18, Vanilla DL, and Elastic DL) with three independent random seeds and report mean ±\pm standard deviation:

ModelClean (%)PGD-20 (%)
ResNet1880.60 ±\pm 0.4243.77 ±\pm 0.25
Vanilla DL82.67 ±\pm 0.5046.01 ±\pm 0.31
Elastic DL (ours)83.44 ±\pm 0.1953.31 ±\pm 0.42

The standard deviations are all below 0.5%0.5\%, whereas Elastic DL improves PGD-20 accuracy by 7.30%7.30\% over Vanilla DL and 9.54%9.54\% over ResNet18. We have also added the within-run standard deviation over the last five epochs to Table 2.

(3) Checkpoints and protocols behind Tables 2, 3, and 11. The differences arise from different evaluation protocols:

  • Table 2 (54.94%54.94\%) studies robust overfitting, which requires evaluating robustness at every epoch. Following common practice in robust overfitting studies, it uses the same PGD-7 attack as in training, and FINAL is the average over the last five epochs. Since PGD-7 is weaker than PGD-20, the robust accuracy is slightly higher than in Table 3.

  • Table 3 (53.22%53.22\%) evaluates the final checkpoint (epoch 200) under the stronger PGD-20 attack.

  • Table 11 (53.29%53.29\%) evaluates the same final checkpoint under PGD-20 in a separate evaluation run; the small difference from Table 3 (0.07%0.07\%) is due to the random initialization of PGD.

W4. The BPDA experiment specifically examines the soft-threshold operation. Although the reported white-box attacks differentiate through the reweighting operation, the paper does not separately assess whether it affects attack reliability. A targeted diagnostic, together with attack-strength sensitivity, would make the evaluation more convincing.

Response: Following the reviewer's suggestion, we conduct a targeted diagnostic on the residual-dependent reweighting operation. We keep the forward pass exact, but replace the backward pass of the reweighting W=1/(∣r∣+ϵ)W = 1/(|r|+\epsilon) by an Expectation-over-Transformation (EOT) surrogate, i.e., the gradient of 1K∑k=1KW(r+ξk)\frac{1}{K}\sum_{k=1}^{K} W(r+\xi_k) with ξk∼N(0,σ2)\xi_k \sim \mathcal{N}(0,\sigma^2), K=8K = 8, and σ=2.5×10−2\sigma = 2.5\times10^{-2}. This smooths the reweighting in the backward pass and therefore removes any potentially ill-conditioned or obfuscated gradients caused by near-zero residuals. We evaluate Elastic DL (ResNet18, CIFAR-10, PGD-AT) with PGD-20 (α=2/255\alpha = 2/255) on the full test set at increasing ℓ∞\ell_\infty budgets:

Attack gradient8/2558/25516/25516/25532/25532/255
Exact (through the reweighting)53.4534.1123.86
EOT (σ=2.5×10−2\sigma = 2.5\times10^{-2}, K=8K=8)55.2436.6325.17

The "Exact" row is a separate evaluation run of the same checkpoint as in Table 3; the small differences (within 0.3%0.3\%) are due to the random initialization of PGD. We make the following observations:

  • The reweighting does not weaken the attack. At all budgets, the attack using the exact gradient through the reweighting is stronger than the attack using the smoothed EOT surrogate (e.g., 53.45%53.45\% vs. 55.24%55.24\% at 8/2558/255). If the reweighting caused gradient masking, the smoothed surrogate would yield a stronger attack. Hence, differentiating through the reweighting provides informative gradients, and the reported white-box results are not inflated by it.

  • Attack-strength sensitivity. The robust accuracy decreases monotonically as the budget increases from 8/2558/255 to 32/25532/255 under both gradients, which is the expected behavior of a reliable evaluation.

  • Consistency with existing evidence. This is consistent with our zero-order gradient analysis (Appendix E.3.4), where autograd gradients through the full network, including the reweighting, match finite-difference estimates with a negligible relative error (on the order of 10−310^{-3}), and with AutoAttack, whose gradient-free Square attack is unaffected by gradient masking and yields robust accuracy consistent with PGD at the standard budget (52.90%52.90\% vs. 53.22%53.22\%, Table 3).

W5. The HAT and PORT comparisons do not sufficiently specify the baseline checkpoints, training data, implementation, and evaluation protocols. These details are needed to assess the claimed improvement over existing defenses.

Response: We thank the reviewer for this comment and provide the full details below, which have been added to Appendix.

Checkpoints and implementation. All baselines are re-trained by us with ResNet18 on CIFAR-10 using the official implementations of HAT (https://github.com/imrahulr/hat) and PORT (https://github.com/inspire-group/proxy-distributions), rather than taken from released checkpoints. For each defense, "+ Vanilla DL" and "+ Elastic DL" only replace the convolutional layers with the corresponding dictionary learning layers, while keeping the training data, loss, and all hyperparameters identical to the baseline.

Training data.

  • HAT: the 50K CIFAR-10 training images, with the hyperparameters β=3.5\beta = 3.5 and γ=0.5\gamma = 0.5 following the original paper.

  • PORT: the 50K CIFAR-10 training images together with synthetic images generated by DDPM, following the original paper and its official implementation.

Matched training and evaluation. For each defense, the baseline, Vanilla DL, and Elastic DL models are trained under matched settings, i.e., with the same training data, defense-specific loss and hyperparameters, adversarial training budget, and training schedule. All models are evaluated under the same protocol as in Table 3, on the full CIFAR-10 test set with PGD-20 and AutoAttack (standard version). Therefore, the comparison is fair, and the improvements of HAT/PORT + Elastic DL over HAT/PORT can be attributed to the architectural change alone.

Reviewer_5P6uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Elastic Dictionary Learning Networks (EDLNets), which replace convolutional layers with unrolled sparse-coding modules combining squared ℓ2\ell_2 and ℓ1\ell_1 reconstruction losses with a sparsity penalty. A learned layer-wise parameter balances the reconstruction terms, and a reweighted iterative shrinkage-thresholding algorithm (RISTA) provides the updates. The approach aims to improve adversarial robustness by reducing sensitivity to reconstruction outliers. Experiments on CIFAR10, CIFAR100, and Tiny-ImageNet, using several ResNet backbones and adversarial training methods, report improved adversarial accuracy and reduced robust overfitting. Additional evaluations examine robustness under different attack budgets, transfer attacks, BPDA, and randomized smoothing.

Critical strengths
  1. Concrete integration of elastic sparse coding into convolutional networks. Sections 3.1–3.2 introduce a reconstruction objective with learned balancing weights and an unrolled RISTA implementation. This provides a practical architectural approach that can be combined with existing adversarial training methods.

  2. Substantial reported improvements in adversarial accuracy. Table 3 reports AutoAttack accuracy increasing from 42.14% to 52.90% for PGD adversarial training and from 53.16% to 59.07% for HAT, with improved clean accuracy in both cases. These results suggest that the proposed architecture can complement established training approaches.

  3. Evaluation across multiple datasets and ResNet depths. Figure 3 and Appendix E.3.1 compare Elastic DL with Vanilla DL on CIFAR10, CIFAR100, and Tiny-ImageNet using four ResNet depths. Tables 4 and 5 also examine multiple perturbation budgets under ℓ∞\ell_\infty and ℓ2\ell_2 attacks. These experiments provide evidence across several configurations within the studied CNN family.

  4. Multiple complementary checks of robustness. Figures 4–5 examine randomized smoothing and transfer attacks, while Appendices E.3.4 and E.3.5 provide gradient-consistency and BPDA evaluations. Together with AutoAttack, these checks broaden the evidence beyond a single attack and address potential concerns about the evaluation of unrolled sparse-coding layers.

Critical weaknesses
  1. The convergence guarantee does not cover the implemented RISTA update (major). Section 3.2 justifies descent by treating one proximal-gradient step as an exact minimization of Eq. (4), without specifying a sufficient step-size condition. Moreover, the implemented ϵ\epsilon-stabilized weights invalidate the original majorization argument and can increase the objective in Eq. (2). The guarantee therefore needs to address the actual stabilized update and its associated objective.

  2. The central empirical claims lack sufficient reproducibility evidence (major). Tables 2–9 provide neither the number of independent training runs nor uncertainty estimates. Table 2’s last-five-epoch average does not measure variability across runs. Important attack settings, including evaluation iterations, restarts, and sample counts, also remain unclear in Section 4.1 and Appendix E.3.5, while Figure 4 omits certification sample counts and confidence levels. Together, these gaps prevent a sufficiently reliable assessment of the claimed robustness improvements and reduction in robust overfitting.

  3. Missing core ablations leave the proposed design insufficiently justified (major). Sections 4.2 and 4.3 compare Elastic DL with Vanilla DL, but do not isolate pure ℓ1\ell_1 reconstruction, a fixed mixture, and learned balancing weights under matched training and unrolling settings. These comparisons are necessary to establish the benefits of combining reconstruction penalties and learning their balance. Sensitivity to unrolling depth and numerical stabilization also remains unclear.

  4. The reconstruction-outlier explanation lacks direct validation (moderate). Section 3.1 attributes the improvement to reduced sensitivity to reconstruction outliers, whereas Figure 8 compares separately generated input perturbations rather than reconstruction-residual distributions. Furthermore, the combined penalty retains unbounded residual sensitivity whenever β>0\beta>0. Direct residual measurements under matched perturbations would strengthen the connection between the proposed diagnosis and the observed improvements.

  5. Computational overhead relative to ordinary convolutions is unclear (moderate). Table 6 reports approximately 10% additional training time relative to Vanilla DL, which already performs iterative sparse inference. It does not establish the overhead relative to standard ResNet convolutions or report inference latency and memory requirements. These comparisons are important for assessing the practical accuracy–robustness–cost trade-off.

  6. Architectural generality remains limited to the ResNet family (minor). Section 4.1 and Figure 3 evaluate ResNet10, ResNet18, ResNet34, and ResNet50, without WideResNet or other CNN-family results. This provides useful within-family evidence but does not substantiate the Introduction’s broader claim of enhancing all convolutional architectures.

  7. Large-scale classification robustness remains untested (minor). The classification experiments cover CIFAR10, CIFAR100, and Tiny-ImageNet. Appendix E.3.7 provides ImageNet reconstruction examples rather than classification or adversarial-robustness results. ImageNet evaluation would help establish whether the robustness benefits and computational practicality of iterative dictionary layers persist at larger scale.

  8. Standardized corruption-robustness evidence is limited (minor). Figure 9 evaluates selected corruptions, but the manuscript does not clearly identify a standard CIFAR-C or ImageNet-C evaluation or specify its protocol. Such evaluation would strengthen the generalization claim by assessing reliability under common environmental degradations. This matters for safety-oriented applications because adversarial robustness alone does not establish robustness to naturally occurring distribution shifts.

  9. State-of-the-art wording exceeds the scope of the comparisons (minor). The conclusion’s broad performance claims rely on selected ResNet18 implementations of existing defenses rather than an overall leaderboard comparison. Improvements are also not uniform: Table 5 reports PORT’s ℓ2\ell_2 AutoAttack accuracy at budget 0.5 decreasing from 73.80% to 72.20%. The claims should reflect the evaluated configurations and observed trade-offs.

Questions for authors
  1. [Important] Could you clarify whether the ϵ\epsilon-stabilized RISTA update decreases the original objective in Eq. (2) or a modified objective, and provide a corresponding descent argument with sufficient step-size conditions? How does the single proximal-gradient step in Eq. (5) support the argument based on exact minimization in Eq. (4)?

  2. [Important] How many independent training runs underlie Tables 2–9? Could you provide means and standard deviations across independent runs for the key ResNet18 comparisons between standard convolutions, Vanilla DL, and Elastic DL, covering clean accuracy, AutoAttack accuracy, and the within-run best-to-final robustness gap?

  3. [Important] Could you specify the evaluation attack settings, including iterations, step sizes, restarts, sample counts, and AutoAttack configuration? Could you report whether the main robustness results remain stable with increased attack iterations and restarts? For Figure 4, what certification sample counts and confidence levels were used?

  4. [Important] Could you compare pure ℓ2\ell_2 reconstruction, pure ℓ1\ell_1 reconstruction, a fixed mixture, and the learned mixture under matched pretraining, fine-tuning, sparsity regularization, and unrolling settings? How sensitive are the robustness gains to the number of unrolled iterations and the stabilization parameter ϵ\epsilon?

  5. [Important] Could you compare clean and adversarial reconstruction-residual distributions for Vanilla DL and Elastic DL under matched perturbations? Do reductions in residual outliers track improvements in classification robustness across different balancing weights, and how does the remaining quadratic term affect the outlier-suppression explanation when β>0\beta>0?

  6. [Important] Could you extend Table 6 to include a standard convolutional ResNet18 baseline, reporting training time, inference latency, and peak memory under matched conditions? How does the robustness–cost trade-off change with the number of unrolled iterations?

  7. [Important] Have you evaluated Elastic DL on WideResNet or a CNN outside the ResNet family? If feasible, could you provide an additional backbone comparison during rebuttal and clarify which architectural properties may limit the broader applicability claimed in the Introduction?

  8. [Important] Are any full ImageNet classification and adversarial-robustness results available beyond the reconstruction examples in Appendix E.3.7? Could you clarify what evidence supports extending the method to larger-scale classification, including the computational implications of applying iterative dictionary layers to higher-resolution inputs?

  9. [Additional] Does Figure 9 follow a standard CIFAR-C or ImageNet-C evaluation protocol? Could you specify the dataset, corruption types, severity levels, and aggregation procedure, and, where feasible, provide a standardized CIFAR-10-C comparison to clarify whether the adversarial-robustness gains extend to common corruptions?

  10. [Additional] Which benchmark configurations support the conclusion’s state-of-the-art claim? Could you clarify its intended scope and discuss the settings where Elastic DL does not improve the original defense, including PORT’s lower ℓ2\ell_2 AutoAttack accuracy at budget 0.5 in Table 5?

12 comments

We sincerely thank the reviewer for the valuable comments. We address each point below.

W1. The convergence guarantee does not cover the implemented RISTA update (major). Section 3.2 justifies descent by treating one proximal-gradient step as an exact minimization of Eq. (4), without specifying a sufficient step-size condition. Moreover, the implemented ϵ\epsilon-stabilized weights invalidate the original majorization argument and can increase the objective in Eq. (2). The guarantee therefore needs to address the actual stabilized update and its associated objective.

Response: We thank the reviewer for the careful reading and address the two points below.

(1) Single proximal step and step-size condition. We thank the reviewer for pointing out this imprecision. Eq. (5) describes the update performed at each RISTA iteration: at iteration tt, the surrogate in Eq. (4) is constructed at ztz_t, and one proximal-gradient step is taken on it to obtain zt+1z_{t+1} (in our implementation, T=2T = 2 such iterations are unrolled in each layer). The descent argument in Section 3.2 was imprecisely phrased in terms of an exact minimizer of Eq. (4); however, exact minimization is not required for the descent chain: it suffices that each step does not increase the surrogate, i.e., L^(zt+1,zt)≤L^(zt,zt)\hat L(z_{t+1}, z_t) \le \hat L(z_t, z_t). By the standard descent lemma for proximal gradient methods [1], this holds whenever the step size satisfies

ηt≤1Lt,Lt=λmax⁡(Adiag(Dt)A∗),Dt=β1+(1−β)wt,\eta_t \le \frac{1}{L_t}, \qquad L_t = \lambda_{\max}\big(\mathcal{A} \mathrm{diag}(D_t) \mathcal{A}^*\big), \qquad D_t = \beta\mathbf{1} + (1-\beta) w_t,

where LtL_t is the Lipschitz constant of the gradient of the smooth (weighted quadratic) part of the surrogate in Eq. (4). Since LtL_t depends on the residual-dependent weights wtw_t, we compute it at each RISTA iteration by power iteration on the weighted operator: starting from a unit vector vv, we repeat u←A(Dt⊙A∗(v))u \leftarrow \mathcal{A}\big(D_t \odot \mathcal{A}^*(v)\big), v←u/∥u∥2v \leftarrow u/\Vert u\Vert _2, implemented with one transposed convolution, an elementwise product with DtD_t, and one convolution, and take Lt≈⟨v,A(Dt⊙A∗(v))⟩L_t \approx \langle v, \mathcal{A}(D_t\odot\mathcal{A}^*(v))\rangle. We then set ηt=0.9/Lt\eta_t = 0.9/L_t, so the condition ηt≤1/Lt\eta_t \le 1/L_t is satisfied with a safety margin.

(2) ϵ\epsilon-stabilized weights. Adding a small ϵ\epsilon (10−310^{-3}) to the reweighting, wt=1/(2(∣rt∣+ϵ))w_t = 1/(2(|r_t|+\epsilon)), is a common practice in iteratively reweighted methods [2] to avoid numerical blow-up when a residual entry approaches zero. It only perturbs the weights of near-zero residuals, and the stabilized update can be viewed as minimizing a slightly smoothed version of the ℓ1\ell_1 reconstruction term that converges to Eq. (2) as ϵ→0\epsilon \to 0; it does not affect the convergence behavior of the algorithm in practice. We empirically verified this in the submission: Figure 7 plots the overall objective in Eq. (2) and each of its terms across RISTA iterations of the implemented (stabilized) algorithm, showing that the objective decreases strictly and converges within three iterations.

Revision. We have revised Section 3.2 accordingly (changes highlighted in blue): the descent chain now relies on the descent lemma for a single proximal-gradient step instead of exact minimization of the surrogate, and the step size is written as ηt\eta_t in Eq. (5). We have also added a new Appendix, which states the step-size condition ηt≤1/Lt\eta_t \le 1/L_t and describes the power-iteration procedure used to compute LtL_t.

[1] Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1):183–202, 2009.

[2] Ingrid Daubechies, Ronald DeVore, Massimo Fornasier, and C. Sinan Güntürk. Iteratively reweighted least squares minimization for sparse recovery. Communications on Pure and Applied Mathematics, 63(1):1–38, 2010.

W2. The central empirical claims lack sufficient reproducibility evidence (major). Tables 2–9 provide neither the number of independent training runs nor uncertainty estimates. Table 2's last-five-epoch average does not measure variability across runs. Important attack settings, including evaluation iterations, restarts, and sample counts, also remain unclear in Section 4.1 and Appendix E.3.5, while Figure 4 omits certification sample counts and confidence levels. Together, these gaps prevent a sufficiently reliable assessment of the claimed robustness improvements and reduction in robust overfitting.

Response: We thank the reviewer for this important comment. We address each point below and will incorporate all clarifications into the revised manuscript.

(1) Uncertainty estimates for the key comparison. We note that reporting results from a single training run is common practice in the adversarial training literature, largely because adversarial training is computationally expensive (each training step requires generating multi-step adversarial examples). Moreover, once converged, adversarial training on CIFAR-10 is generally stable, so the run-to-run variability of the final accuracy is small relative to the differences between methods compared here. To nevertheless quantify variability for the central ResNet18 comparison (CIFAR-10, PGD-AT), we train each model three times with independent random seeds, evaluate each run on the full CIFAR-10 test set (10,000 images), and report mean ±\pm standard deviation in Table R1.

Clean and PGD-20 accuracy (%) of ResNet18 on CIFAR-10 with PGD-AT, mean ±\pm std over three independent runs.

ModelClean (%)PGD-20 (%)
ResNet1880.60 ±\pm 0.4243.77 ±\pm 0.25
Vanilla DL82.67 ±\pm 0.5046.01 ±\pm 0.31
Elastic DL (ours)83.44 ±\pm 0.1953.31 ±\pm 0.42

The standard deviations are all below 0.5%, whereas Elastic DL improves PGD-20 accuracy by 7.30% over Vanilla DL and 9.54% over ResNet18, i.e., more than an order of magnitude larger than the observed run-to-run variability. This confirms that the improvements are robust to training randomness.

(2) Table 2. Following the reviewer's suggestion, we additionally report in the revised Table 2 the standard deviation of the FINAL accuracy over the last five epochs (e.g., Elastic DL: robust FINAL 54.94±0.5154.94 \pm 0.51; Vanilla DL: 44.03±0.4244.03 \pm 0.42; best baseline, Cutout: 47.12±0.2347.12 \pm 0.23). For Early Stopping, FINAL corresponds to a single checkpoint, so no standard deviation is reported. Under this measure, the improvement of Elastic DL in robust accuracy over all baselines still far exceeds the fluctuation.

(3) Attack settings (Section 4.1 and Appendix E.3.5). We have added a new Appendix detailing all attack settings, referenced from Section 4.1:

  • Training: PGD-AT with ϵ=8/255\epsilon=8/255, step size α=2/255\alpha=2/255, 7 iterations, one random start. The robust accuracy in Table 2 is measured with the same PGD-7 attack on the full test set at every epoch.

  • PGD-ℓ∞\ell_\infty evaluation: 20 iterations, α=2/255\alpha=2/255, one random start (no restarts), ϵ=8/255\epsilon=8/255 unless otherwise specified, full test set (10,000 images).

  • FGSM: a single step of size ϵ\epsilon from a uniform random start in [−ϵ,ϵ][-\epsilon,\epsilon], full test set.

  • C&W: ℓ2\ell_2 formulation with c=1c=1, κ=0\kappa=0, 15 optimization steps, learning rate 0.010.01, full test set.

  • AutoAttack: we use the official implementation (https://github.com/fra31/auto-attack) with the standard version, i.e., the ensemble of APGD-CE, APGD-T, FAB-T, and Square, under the ℓ∞\ell_\infty threat model with ϵ=8/255\epsilon=8/255, evaluated on the full test set.

  • BPDA (Appendix E.3.5): PGD-ℓ∞\ell_\infty with 20 iterations, α=2/255\alpha=2/255, one random start, full test set, for ϵ∈{8,16,32,64}/255\epsilon\in\lbrace 8,16,32,64\rbrace /255.

(4) Certification settings (Figure 4). Certified accuracy is computed with the CERTIFY procedure of randomized smoothing [1], using N0=100N_0=100 Monte Carlo samples for class selection, N=10,000N=10,000 samples for estimation, and failure probability α=0.001\alpha=0.001 (i.e., 99.9% confidence). We certify 500 CIFAR-10 test images (every 20th image of the test set) at noise levels σ∈{0.1,0.15,0.2}\sigma\in\lbrace 0.1, 0.15, 0.2\rbrace.

[1] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning (ICML), pp. 1310–1320. PMLR, 2019.

W3. Missing core ablations leave the proposed design insufficiently justified (major). Sections 4.2 and 4.3 compare Elastic DL with Vanilla DL, but do not isolate pure ℓ1\ell_1 reconstruction, a fixed mixture, and learned balancing weights under matched training and unrolling settings. These comparisons are necessary to establish the benefits of combining reconstruction penalties and learning their balance. Sensitivity to unrolling depth and numerical stabilization also remains unclear.

Response: Following the reviewer's suggestion, we conduct ablations on ResNet18/CIFAR-10 with PGD-AT to isolate (1) the reconstruction objective and (2) the unrolling depth. All variants are trained under matched settings, differing only in the studied factor.

(1) Reconstruction objective.

Reconstruction objectiveCleanPGD-20 @8/2558/255PGD-20 @16/25516/255PGD-20 @32/25532/255
Pure ℓ2\ell_283.2845.6415.482.87
Pure ℓ1\ell_180.7151.3631.8521.94
Fixed mixture (β=0.5\beta = 0.5)82.6450.1228.0714.69
Learned layer-wise β\beta (Ours)83.5753.2234.1423.52

We make the following observations:

  • ℓ1\ell_1 reconstruction is the key source of robustness. Replacing the ℓ2\ell_2 reconstruction with ℓ1\ell_1 substantially improves robustness (e.g., 45.64%→51.36%45.64\% \to 51.36\% at 8/2558/255 and 2.87%→21.94%2.87\% \to 21.94\% at 32/25532/255), supporting our motivation that the ℓ2\ell_2 reconstruction is sensitive to adversarially induced outliers. Since all variants are trained under matched settings, the gain cannot be attributed to the training schedule itself.

  • Pure ℓ1\ell_1 sacrifices clean accuracy. Pure ℓ1\ell_1 reconstruction reduces clean accuracy by 2.57%2.57\% relative to pure ℓ2\ell_2, indicating that the ℓ2\ell_2 term remains important for reconstruction fidelity on clean inputs.

  • A fixed mixture is suboptimal. Fixing β=0.5\beta = 0.5 recovers most of the clean accuracy but yields lower robustness than pure ℓ1\ell_1, especially at larger budgets (14.69%14.69\% vs. 21.94%21.94\% at 32/25532/255).

  • Learning the balance gives the best of both. With learned layer-wise β\beta, Elastic DL achieves the highest clean accuracy and the highest robustness at all budgets, outperforming both pure objectives and the fixed mixture. This shows that combining the two reconstruction penalties and adapting their balance per layer are both necessary for the favorable accuracy–robustness trade-off.

(2) Unrolling depth TT. We vary the number of unrolled RISTA iterations TT per layer. Note that T=0T = 0 reduces each layer to a standard convolution z=A(x)\mathbf{z} = \mathcal{A}(\mathbf{x}), i.e., the standard ResNet18.

Unrolling depthCleanPGD-20 @8/2558/255PGD-20 @16/25516/255PGD-20 @32/25532/255
T=0T = 0 (ResNet18)80.9044.3514.781.99
T=1T = 182.7450.1827.4614.83
T=2T = 2 (Ours, default)83.5753.2234.1423.52
T=3T = 383.6253.4134.4723.89
T=4T = 483.4853.3534.5824.06

A single iteration (T=1T = 1) is not sufficient, as robustness remains clearly below that of T=2T = 2, especially at larger budgets. The performance converges at T=2T = 2: further increasing TT yields only marginal changes (within 0.6%0.6\%), consistent with the fast convergence of RISTA in Figure 7. We therefore use T=2T = 2 as the default for the consideration of efficiency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.