acceptodds
Under review as a conference paper at ICLR 2027

Beyond Independence: Modeling Correlated Adversarial Transfer In Model Ensembles

Abstract

Ensembles are widely believed to resist black-box transfer attacks: if adversarial examples transfer imperfectly, compromising several members at once should be rare. This argument silently assumes that transfer events across members are independent. We test that assumption directly, treating transfer against an ensemble as a vector of correlated binary events and measuring per-model transfer rates, pairwise correlations, and the overdispersion of the fooled count for six-model ensembles on CIFAR-10, GTSRB and MNIST under PGD transfer from a ResNet surrogate. In the regime where the ensemble argument is actually invoked, at small budgets where per-model transfer is a few percent and majority compromise is meant to be rare, independence is rejected wherever the data are well powered, including by an exact permutation test (): the observed rate of compromising a majority exceeds the independence prediction by more than an order of magnitude ( on CIFAR-10, on GTSRB, on MNIST at the smallest budget with events). As the budget rises and transfer saturates the gap closes and eventually reverses, exactly as it must; the danger zone for a defender is the low-budget corner, and that is where independence fails hardest. The failure is correctable with a single parameter: an exchangeable beta-binomial calibrated on the measured overdispersion matches the observed risk in every well-powered configuration, where independence fails in all of them, and, fit at one ensemble size, predicts majority-vote risk at other sizes to within a percentage point. The dependence is not an artifact of our training pipeline: it survives replacing distilled students with independently trained ones and the teacher surrogate with an unrelated model, and it persists under three transfer-optimized attacks. It also survives the standard fix: adversarial training lowers how often members are fooled but not how correlated their failures are, so the residual risk must still be read through the dependence structure. Ensemble robustness is governed by the dependence structure of failures, not by per-model transfer rates alone.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
adversarial robustness, transfer attacks, ensemble defenses, attack transferability, failure correlation, overdispersion, beta-binomial model, majority voting, black-box attacks, risk estimation, adversarial training, evaluation pitfalls

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vJFPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines the adversarial robustness of ensemble methods. Specifically testing if the models in the ensemble fail independently. The paper tests 6 CNN models on CIFAR-10, GTSRB and MNIST. The paper tests this by generating adversarial images on the teacher Resnet50 model (which is sued to distill the 6 student CNNs). To craft adversarial images MI-FGSM, DI²-FGSM, TI-FGSM and PGD attacks are used. The paper concludes that independence assumption overestimates ensemble robustness

Critical strengths
  • Students trained from scratch, writing is easy enough to follow and an independent surrogate is used and three transfer-optimized attacks.
Critical weaknesses
  • My main concern is that the motivation is dated, the paper's setup up using ensemble defenses using works from 2017–2021 (Tramer 2017 [1], Pang 2019 [2], Kariyappa & Qureshi 2019 [3], DVERGE 2020 [4], TRS 2021 [5]) using attacks and models from 2017–2019. The community largely does not think that ensembles are valid defense against adversarial attacks. Paper such as TREND (2022) develop attacks specifically targeting ensembles and perform a thorough transferability analysis over a larger set of datasets and model architectures and optimizers. This debate was largely settled by 2020. Tramèr et al. 2020 broke 13 defenses, including EMPIR (ICLR 2020, weight quantized ensembles) and Pang et al.'s ADP. The field's progress has since come from adversarial training with more data, evaluated with AutoAttack [13] and RobustBench [14], not from ensembles. The paper's own Table 3 shows adversarial training removes almost all transfer, which weakens the case for studying ensembles.

  • Shared vulnerability across independently trained models is well documented [16, 17, 18, 19], including within ensembles [20, 21, 22]. Formal links between ensemble robustness and coinciding vulnerabilities exist [22, 23]. Correlated-error statistics (error consistency, disagreement, ensemble gain versus disagreement-to-error ratio, tandem loss) are established in ML [24, 25, 26].

[1] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, P. McDaniel. Ensemble Adversarial Training: Attacks and Defenses. ICLR 2018
[2] T. Pang, K. Xu, C. Du, N. Chen, J. Zhu. Improving Adversarial Robustness via Promoting Ensemble Diversity. ICML 2019
[3] S. Kariyappa, M. K. Qureshi. Improving Adversarial Robustness of Ensembles with Diversity Training.
[4] H. Yang, J. Zhang, H. Dong, N. Inkawhich, A. Gardner, A. Touchet, W. Wilkes, H. Berry, H. Li. DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles. NeurIPS 2020
[5] Z. Yang, L. Li, X. Xu, S. Zuo, Q. Chen, P. Zhou, B. Rubinstein, C. Zhang, B. Li. TRS: Transferability Reduced Ensemble via Promoting Gradient Diversity and Model Smoothness. NeurIPS 2021
[6] D. Ravikumar, S. Kodge, I. Garg, K. Roy. TREND: Transferability based Robust ENsemble Design. TAI 2022
[7] F. Tramer, N. Carlini, W. Brendel, A. Madry. On Adaptive Attacks to Adversarial Example Defenses. NeurIPS 2020.
[8] S. Sen, B. Ravindran, A. Raghunathan. EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness against Adversarial Attacks. ICLR 2020.
[9] S.-A. Rebuffi, S. Gowal, D. A. Calian, F. Stimberg, O. Wiles, T. Mann. Fixing Data Augmentation to Improve Adversarial Robustness. arXiv:2103.01946, 2021.
[10] S. Gowal, S.-A. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, T. Mann. Improving Robustness using Generated Data. NeurIPS 2021
[11] Z. Wang, T. Pang, C. Du, M. Lin, W. Liu, S. Yan. Better Diffusion Models Further Improve Adversarial Training. ICML 2023. arXiv:2302.04638.
[12] B. R. Bartoldson, J. Diffenderfer, K. Parasyris, B. Kailkhura. Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies. ICML 2024.
[13] F. Croce, M. Hein. Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-free Attacks. ICML 2020.
[14] F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, M. Hein. RobustBench: a Standardized Adversarial Robustness Benchmark. NeurIPS 2021.
[16] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, A. Madry. Adversarial Examples Are Not Bugs, They Are Features. NeurIPS 2019.
[17] F. Tramèr, N. Papernot, I. Goodfellow, D. Boneh, P. McDaniel. The Space of Transferable Adversarial Examples. 2017. arXiv:1704.03453.
[18] A. Demontis, M. Melis, M. Pintor, M. Jagielski, B. Biggio, A. Oprea, C. Nita-Rotaru, F. Roli. Why Do Adversarial Attacks Transfer? Explaining Transferability of Evasion and Poisoning Attacks. USENIX Security 2019.
[19] J. M. Springer, M. Mitchell, G. T. Kenyon. A Little Robustness Goes a Long Way: Leveraging Robust Features for Targeted Transfer Attacks. NeurIPS 2021.
[20] C. Wiedeman, G. Wang. Disrupting Adversarial Transferability in Deep Neural Networks. Patterns, 2022. arXiv:2108.12492.
[21] W. Yao, Z. Zhang, H. Tang, Y. Gong. Understanding Model Ensemble in Transferable Adversarial Attack. ICML 2025. arXiv:2410.06851.
[22] H. Chen, Y. Zhang, Y. Dong, J. Zhu. Rethinking Model Ensemble in Transfer-based Adversarial Attacks. ICLR 2024.
[23] D. Zhang, H. Zhang, A. Courville, Y. Bengio, P. Ravikumar, A. S. Suggala. Building Robust Ensembles via Margin Boosting. ICML 2022.
[24] R. Geirhos, K. Meding, F. A. Wichmann. Beyond Accuracy: Quantifying Trial-by-Trial Behaviour of CNNs and Humans by Measuring Error Consistency. NeurIPS 2020.
[25] Y. Jiang, V. Nagarajan, C. Baek, J. Z. Kolter. Assessing Generalization of SGD via Disagreement. ICLR 2022.
[26] R. Theisen, H. Kim, Y. Yang, L. Hodgkinson, M. W. Mahoney. When Are Ensembles Really Effective? NeurIPS 2023.\

Questions for authors
  1. Pang et al. [2], Kariyappa & Qureshi [3], DVERGE [4] and TRS [5] all motivate diversity training by the observing that members are correlated. How does this paper's claim differ?
  2. Table 3 shows that adversarial training removes nearly all transfer, then why is the low-budget ensemble matter?
Reviewer_GQUxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether ensemble members fail independently under black-box transfer attacks. It shows that member failures are strongly correlated, especially at small attack budgets, which is exactly where ensembles are expected to offer the most protection. The authors propose a simple statistical model that accounts for this correlation and gives a more realistic estimate of ensemble risk.

Critical strengths

None

Critical weaknesses
  • The main finding is not surprising. The paper itself notes that models trained on the same data share adversarial subspaces, so it is expected that they will fail on the same inputs. Showing this confirms what the community already believed rather than revealing something new. The contribution lies in measuring how strong the effect is and offering a simple correction, which is useful but limited.

  • It argues against a claim nobody really makes. The paper frames ensemble defenses as resting on the assumption that members fail independently. In practice, people who build these defenses attack the full ensemble and report its robust accuracy, which already includes any correlated failures. Almost nobody multiplies per-member transfer rates together to estimate ensemble risk. The "hidden assumption" being exposed is closer to an informal intuition than a method anyone actually relies on.

  • The experiments are small and old. Everything is run on tiny images from CIFAR-10, MNIST and GTSRB, with compact CNN students attacked mainly from a ResNet-50 surrogate. There is nothing on larger or more realistic data such as ImageNet, nothing on modern architectures such as Vision Transformers, and no ensembles that mix different model families. Ensemble defenses are usually deployed in those settings, and the correlation could look quite different there.

  • The attacks are also old. The paper relies on PGD and three older momentum and augmentation-based variants. Stronger current attacks such as AutoAttack, query-based black-box attacks, and adaptive attacks designed with knowledge of the ensemble are not considered. Though the authors acknowledge this in the limitations, evaluating against such attacks is one of the major experiments the paper needs.

  • The most relevant experiment is missing. If the message is that ensemble defenses should be judged by how correlated their failures are, the natural step is to take existing diversity-promoting methods such as DVERGE, TRS or ADP and measure that directly. The paper does not, and defers it to future work. Without it, the paper offers a measurement tool but no evidence that any existing method is actually being misjudged.

Questions for authors
  • Does the measured correlation change when the ensemble members come from different architecture families, or when the models and data are larger and more modern (for example, ImageNet-scale with Vision Transformers)? Even one such configuration would show whether the finding generalizes beyond small CNNs on small images.
  • When the proposed dependence metric is applied to an existing diversity-promoting defense such as DVERGE or TRS, does the correlation actually drop compared to a standard ensemble?
  • Does the dependence persist under a stronger attack suite such as AutoAttack, or under an adaptive attack that targets the ensemble directly rather than a single surrogate?
Reviewer_cwqQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper argues that the usual case for ensemble defenses against black-box transfer attacks rests on an untested assumption: that members fail independently, so a single transferred perturbation should rarely fool a majority. They test this directly by treating transfer against an ensemble as a vector of correlated binary failures, and measuring the per-model transfer rate pˉ\bar{p}, the pairwise failure correlation ρ\rho, and the overdispersion DD of the fooled count. On CIFAR-10, GTSRB and MNIST under ℓ∞\ell_\infty PGD transfer from a surrogate, independence underestimates majority compromise by orders of magnitude at small budgets, which is exactly where the ensemble argument is used. The gap closes as ε\varepsilon grows and transfer saturates. They propose a one-parameter beta-binomial correction calibrated on DD, which matches the observed majority-vote risk and predicts it for smaller ensembles.

Critical strengths

S1. Mathematical rigor and statistical significance. The paper states its independence baseline exactly (the Poisson–binomial, Eq. 1) and its correction in closed form (the beta-binomial, Eq. 3), and it shows that the correction recovers the baseline as ρ→0\rho \to 0. It tests the central claim in two complementary ways: exact one-sided binomial tests against the independence null, and a distribution-free permutation test that preserves each member's marginal failure rate while destroying all dependence (Section 5.6). Independence is rejected at p<5×10−5p < 5 \times 10^{-5} at every well-powered rare-event budget. Uncertainty is reported throughout with Wilson intervals and bootstrap estimates. Configurations with fewer than ten majority-compromise events are flagged and excluded from conclusions, so the reported effects do not rest on rare-event noise.

S2. Studies an important problem in adversarial robustness. Ensembles are a widely used defense against black-box transfer attacks, and their appeal rests on how rarely members are expected to fail together. While this premise has been analyzed for randomized ensembles under worst-case attacks [1–5], the paper is the first, to my knowledge, to measure it directly for majority-vote ensembles under transfer. It shows that the premise fails worst at small perturbation budgets, exactly where defenders rely on it. This matters because robustness claims built on untested assumptions can give a false sense of security. Measuring failure dependence directly, rather than inferring it from per-model transfer rates, is a step toward more reliable robustness evaluation.


[1] Gnecco Heredia, L., Pydi, M. S., Meunier, L., Negrevergne, B., & Chevaleyre, Y. (2023). On the role of randomization in adversarially robust classification. Advances in Neural Information Processing Systems, 36, 79293-79319.

[2] Gnecco, L., Negrevergne, B., & Chevaleyre, Y. (2025, September). Lattice Climber Attack: Adversarial attacks for randomized mixtures of classifiers. In Proceedings of ECML PKDD 2025.

[3] Dbouk, H., & Shanbhag, N. (2023, July). On the robustness of randomized ensembles to adversarial perturbations. In International Conference on Machine Learning (pp. 7303-7328). PMLR.

[4] Dbouk, H., & Shanbhag, N. (2022, June). Adversarial vulnerability of randomized ensembles. In International Conference on Machine Learning (pp. 4890-4917). PMLR.

[5] Pinot, R., Ettedgui, R., Rizk, G., Chevaleyre, Y., & Atif, J. (2020, November). Randomization matters how to defend against strong adversarial attacks. In International Conference on Machine Learning (pp. 7717-7727). PMLR.

Critical weaknesses

W1. The paper overlooks the randomized-ensemble literature, which already frames robustness as overlap of vulnerable regions. The claim that the independence premise "is never stated, never measured" (line 52) does not hold. Work on randomized ensemble classifiers (RECs) [1–5] states this premise explicitly in worst-case form: an ensemble is robust at xx only if no single perturbation in the ε\varepsilon-ball fools many members at once. The paper's ρ\rho and DD measure this same overlap, evaluated at the surrogate's perturbation rather than in the worst case. The paper should position itself relative to [1–5] and state what the transfer setting adds beyond those worst-case analyses.

Questions for authors

Q1. What about diversity training? This papers tackled independently trained ensembles, and the authors acknowledge that diversity-promoting defenses are not evaluated (Section 6). Independently trained ensembles are already known to add little robustness [4, App. C.5], so a high ρ\rho is expected there. Could the authors report pˉ\bar{p}, ρ\rho and DD for at least one ensemble trained explicitly for diversity (e.g., DVERGE, TRS, or BARRE [3])? If DD stays well above 1, this supports the claim that dependence is "a property of the data and the attack geometry" (line 376). If DD approaches 1, the dependence is an artifact of independent training rather than of ensembles in general.

Misc.

  • Lines 213–215 refer to "the 15 member pairs of a six-model ensemble" before m=6m = 6 is introduced in Section 4. Would it be clearer to write (m2)\binom{m}{2} pairs here?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.