- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper examines the adversarial robustness of ensemble methods. Specifically testing if the models in the ensemble fail independently. The paper tests 6 CNN models on CIFAR-10, GTSRB and MNIST. The paper tests this by generating adversarial images on the teacher Resnet50 model (which is sued to distill the 6 student CNNs). To craft adversarial images MI-FGSM, DI²-FGSM, TI-FGSM and PGD attacks are used. The paper concludes that independence assumption overestimates ensemble robustness
Critical strengths
- Students trained from scratch, writing is easy enough to follow and an independent surrogate is used and three transfer-optimized attacks.
Critical weaknesses
-
My main concern is that the motivation is dated, the paper's setup up using ensemble defenses using works from 2017–2021 (Tramer 2017 [1], Pang 2019 [2], Kariyappa & Qureshi 2019 [3], DVERGE 2020 [4], TRS 2021 [5]) using attacks and models from 2017–2019. The community largely does not think that ensembles are valid defense against adversarial attacks. Paper such as TREND (2022) develop attacks specifically targeting ensembles and perform a thorough transferability analysis over a larger set of datasets and model architectures and optimizers. This debate was largely settled by 2020. Tramèr et al. 2020 broke 13 defenses, including EMPIR (ICLR 2020, weight quantized ensembles) and Pang et al.'s ADP. The field's progress has since come from adversarial training with more data, evaluated with AutoAttack [13] and RobustBench [14], not from ensembles. The paper's own Table 3 shows adversarial training removes almost all transfer, which weakens the case for studying ensembles.
-
Shared vulnerability across independently trained models is well documented [16, 17, 18, 19], including within ensembles [20, 21, 22]. Formal links between ensemble robustness and coinciding vulnerabilities exist [22, 23]. Correlated-error statistics (error consistency, disagreement, ensemble gain versus disagreement-to-error ratio, tandem loss) are established in ML [24, 25, 26].
[1] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, P. McDaniel. Ensemble Adversarial Training: Attacks and Defenses. ICLR 2018
[2] T. Pang, K. Xu, C. Du, N. Chen, J. Zhu. Improving Adversarial Robustness via Promoting Ensemble Diversity. ICML 2019
[3] S. Kariyappa, M. K. Qureshi. Improving Adversarial Robustness of Ensembles with Diversity Training.
[4] H. Yang, J. Zhang, H. Dong, N. Inkawhich, A. Gardner, A. Touchet, W. Wilkes, H. Berry, H. Li. DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles. NeurIPS 2020
[5] Z. Yang, L. Li, X. Xu, S. Zuo, Q. Chen, P. Zhou, B. Rubinstein, C. Zhang, B. Li. TRS: Transferability Reduced Ensemble via Promoting Gradient Diversity and Model Smoothness. NeurIPS 2021
[6] D. Ravikumar, S. Kodge, I. Garg, K. Roy. TREND: Transferability based Robust ENsemble Design. TAI 2022
[7] F. Tramer, N. Carlini, W. Brendel, A. Madry. On Adaptive Attacks to Adversarial Example Defenses. NeurIPS 2020.
[8] S. Sen, B. Ravindran, A. Raghunathan. EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness against Adversarial Attacks. ICLR 2020.
[9] S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, T. Mann. Fixing Data Augmentation to Improve Adversarial Robustness. arXiv:2103.01946, 2021.
[10] S. Gowal, S.-A. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, T. Mann. Improving Robustness using Generated Data. NeurIPS 2021
[11] Z. Wang, T. Pang, C. Du, M. Lin, W. Liu, S. Yan. Better Diffusion Models Further Improve Adversarial Training. ICML 2023. arXiv:2302.04638.
[12] B. R. Bartoldson, J. Diffenderfer, K. Parasyris, B. Kailkhura. Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies. ICML 2024.
[13] F. Croce, M. Hein. Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-free Attacks. ICML 2020.
[14] F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, M. Hein. RobustBench: a Standardized Adversarial Robustness Benchmark. NeurIPS 2021.
[16] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, A. Madry. Adversarial Examples Are Not Bugs, They Are Features. NeurIPS 2019.
[17] F. Tramèr, N. Papernot, I. Goodfellow, D. Boneh, P. McDaniel. The Space of Transferable Adversarial Examples. 2017. arXiv:1704.03453.
[18] A. Demontis, M. Melis, M. Pintor, M. Jagielski, B. Biggio, A. Oprea, C. Nita-Rotaru, F. Roli. Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attacks. USENIX Security 2019.
[19] J. M. Springer, M. Mitchell, G. T. Kenyon. A Little Robustness Goes a Long Way: Leveraging Robust Features for Targeted Transfer Attacks. NeurIPS 2021.
[20] C. Wiedeman, G. Wang. Disrupting Adversarial Transferability in Deep Neural Networks. Patterns, 2022. arXiv:2108.12492.
[21] W. Yao, Z. Zhang, H. Tang, Y. Gong. Understanding Model Ensemble in Transferable Adversarial Attack. ICML 2025. arXiv:2410.06851.
[22] H. Chen, Y. Zhang, Y. Dong, J. Zhu. Rethinking Model Ensemble in Transfer-based Adversarial Attacks. ICLR 2024.
[23] D. Zhang, H. Zhang, A. Courville, Y. Bengio, P. Ravikumar, A. S. Suggala. Building Robust Ensembles via Margin Boosting. ICML 2022.
[24] R. Geirhos, K. Meding, F. A. Wichmann. Beyond Accuracy: Quantifying Trial-by-Trial Behaviour of CNNs and Humans by Measuring Error Consistency. NeurIPS 2020.
[25] Y. Jiang, V. Nagarajan, C. Baek, J. Z. Kolter. Assessing Generalization of SGD via Disagreement. ICLR 2022.
[26] R. Theisen, H. Kim, Y. Yang, L. Hodgkinson, M. W. Mahoney. When Are Ensembles Really Effective? NeurIPS 2023.\
Questions for authors
- Pang et al. [2], Kariyappa & Qureshi [3], DVERGE [4] and TRS [5] all motivate diversity training by the observing that members are correlated. How does this paper's claim differ?
- Table 3 shows that adversarial training removes nearly all transfer, then why is the low-budget ensemble matter?