acceptodds
Under review as a conference paper at ICLR 2027

Uncovering the Limits of IBP-based Certified Training through Scaling

Abstract

While deep neural networks have enabled remarkable advances across a wide range of supervised learning tasks, their susceptibility to adversarial attacks remains a major concern. Certified training methods based on interval bound propagation (IBP) produce models with mathematically rigorous robustness guarantees, but typically suffer from degraded performance on clean data. While recent methods improved this trade-off by training on both adversarial examples and IBP bounds, progress has stalled, suggesting that IBP-based certified training may be approaching its limits. Here, we show that the progress that has been achieved came at the cost of severe overfitting, which traditional IBP training suffered less from. Thus, to uncover new performance limits of these methods, for the first time, we employ large-scale synthetic datasets and conduct the first systematic scaling study of IBP-based certified training across data and compute. We demonstrate that scaling yields performance improvements comparable to several years of algorithmic progress; for instance, on CIFAR-10 with , we obtain a model with 83.65% clean and 65.90% certified accuracy, substantially surpassing the previous state of the art of 79.87% and 64.54%, respectively. Analysing the resulting models, we find that these improvements are accompanied by changes in the distribution of unstable ReLUs and bound propagation tightness. At the same time, we reveal more fundamental limits of IBP-based certified training: performance quickly saturates with scaling, the gained robustness conflicts with propagation tightness, and stronger models incur increasingly prohibitive verification costs, making verification systems a bottleneck for certifying robust models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Robustness, Neural Network Verification, Certified Training, Interval Bound Propagation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rS7bOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the scaling effect in synthetic data and compute on certified training, particularly IBP-based methods.

Critical strengths

I would like to inform AC and other reviewers that the main topic of this paper, i.e., scaling effect in certified training, has been extensively studied in Altstidl et al. cited by this work. The main difference is that Altstidl et al. included more certified training algorithms and demonstrated the scaling effect in both synthetic data and compute, while this work restricts to the MTL-IBP only. Therefore, the only strength I can observe is an improvement performance (although presented in a significantly over-claimed way since it didn't compare to the fair evaluation result of CTBench) on CIFAR-10 ϵ=2/255\epsilon=2/255 for MTL-IBP, which Altstidl et al. didn't choose to study as a concrete case.

Critical weaknesses
  1. This work largely overlaps with Altstidl et al. and incorrectly stated the conclusion about MTL-IBP experiments done by them. While Altstidl et al. shows that MTL-IBP on CIFAR-10 ϵ=8/255\epsilon=8/255 does not benefit from scaling, this work does not establish the benefit for this setting as well. In fact, Fig 3.d did not see a consistent scaling benefit, and even the best number reported (35.58%) is very close to CTBench evaluation of the same algorithm and the same setting without any data or compute scaling (35.41%).

  2. This work does not establish overfitting in certified training but it claims to. The main result, namely Fig. 2, is completely unmeaningful. It presents compute scaling both as legend and as x-axis. It does not show any overfitting. The authors seem to even misunderstand the meaning of overfitting and believe that more compute decreases performance means overfitting, while there is no result indicating reducing train accuracy can improve test accuracy.

Questions for authors

See above.

4 comments
AuthorsOpenReview ↗

Author Response [1]

We, the authors, are very concerned about the factual inaccuracies in this review and are severely worried that several statements made by the reviewer may reflect misunderstandings of our submission and its relationship to prior work. Below, we address our concerns in detail so that the Area Chair and other reviewers can independently assess the made claims against the evidence presented in our submission.

Q: The work largely overlaps with Altstidl et al., 2024 [1].
A: We explicitly acknowledge Altstidl et al. and discuss their findings in Section 2.3 and differentiate our work from theirs in Appendix A.2. As explained there, the reviewer’s statement that the “main difference” is that they “included more certified training algorithms” mischaracterises the methodological distinction. Their main study investigates Lipschitz-based methods, which obtain guarantees through bounds on network Lipschitz constants (see, e.g., [2,3,4,5]). For this, these methods rely on specialised layers and may be restricted to the $\ell_2$ ball threat model [4,5]. Our study investigates an orthogonal approach to certified robustness, IBP-based training, where certification against input perturbations within $\ell_\infty$ balls relies on propagating reachability bounds through the network, potentially followed by branch-and-bound verification (see, e.g., [6,7,8]). Importantly, bound propagation is agnostic to the employed network architecture. Thus, these approaches differ strongly in the models they are applicable to, their training objectives and the downstream certification procedures.

We therefore view the scaling study of Altstidl et al. as a motivation to investigate the scaling behaviour of IBP-based certified training. While complementary approaches have been scaled successfully, a systematic study of IBP-based methods has remained missing, despite their state-of-the-art certified robustness on high-dimensional datasets such as TinyImageNet and at smaller perturbation radii on CIFAR-10 (see, e.g., [9,10,11]).

Moreover, our main contribution extends beyond the scaling study itself. Our work reveals that the apparent performance limits of IBP-based certified training were not simply attributable to weak training objectives, but were substantially driven by poor robust generalisation. Mitigating this limitation through synthetic data enabled substantial performance improvements, while further scaling revealed saturation and increasingly prohibitive verification costs as performance bottlenecks. These previously unexplored findings distinguish our contribution from merely demonstrating that additional data and compute can improve performance.

Q: Conclusion on MTL-IBP experiments of Altstidl et al., 2024 is incorrectly stated.
A: We respectfully disagree. Altstidl et al. explicitly conclude in Appendix H: “MTL-IBP, a convex bound propagation approach, cannot benefit from additional generated data.” Our manuscript accurately reflects their findings in Section 2.3, ll. 195-197.:

“However, their preliminary experiments with MTL-IBP reported catastrophic performance when scaling training data and compute, suggesting that these benefits do not trivially transfer to IBP-based certified training.”

This description is supported by their Table 10, where certified accuracy drops from 35.08% without generated data to 0.00% with 5 million generated images at 260 epochs. We therefore ask the reviewer to identify precisely which part of our account they consider incorrect.

Q: The work is restricted to MTL-IBP

A: Appendix C.2 evaluates the well-performing hybrid method SABR and traditional IBP training across the full compute and data scaling grid at both CIFAR-10 perturbation radii, where we observed similar results that support our main findings. We will state more clearly in a revised version of the manuscript that MTL-IBP is the focus of the main study, but not the only training method investigated.

In addition, in a general comment, we present new results on a ResNet and deeper and shallower convolutional networks, showing that our findings are not dependent on the employed architecture. We welcome concrete suggestions for further experiments that would strengthen the evaluation and are happy to provide additional results during the rebuttal period.

AuthorsOpenReview ↗

Author Response [2]

Q: Performance is overstated since it is not compared to CTBench
A: CTBench evaluates configurations at different operating points on the clean–certified accuracy trade-off, including configurations that favour certified accuracy at the expense of clean accuracy through different hyperparameters, as shown by Kaulen et al. [10]. Our scaling study uses the hyperparameters of the original authors to isolate the effects of additional data and compute. Nevertheless, the presented results also outperform those presented in CTBench on the Pareto frontier and we are happy to include a comparison to CTBench in a revised version of the manuscript.

Q: No consistent scaling benefit is observed at ε = 8/255.
A: Section 3.4 explicitly reports that certified performance at this radius saturates near previously achieved levels. Nevertheless, scaling substantially improves clean accuracy while maintaining competitive certified accuracy. Our strongest configuration achieves 57.27% clean and 35.58% certified accuracy, which Pareto-dominates all previously known results from the literature on the trade-off between clean and certified accuracy [10].

Q: Figure 2 is “completely unmeaningful” and does not establish overfitting.
A: We will clarify the caption and axis labelling of the figure. The legend identifies separate runs with total compute budgets of 1×, 2×, 5× and 10×, whereas the horizontal axis represents progress through each run as a percentage of its respective budget. As described in Section 3.1, we proportionally extend the warm-up and ramp-up phases with the overall training schedule and, therefore, each compute scale does induce a separate training trajectory.

Regarding overfitting, we follow the standard definition taught in introductory textbooks and lectures. Goodfellow, Bengio and Courville state in the book “Deep Learning”: “Overfitting occurs when the gap between the training error and test error is too large” [12]. Stanford’s CS231n course notes likewise explain: “The gap between the training and validation accuracy indicates the amount of overfitting” [13].

Figure 2 examines precisely this divergence: training certified accuracy increases while test certified accuracy deteriorates. The reviewer’s proposed requirement that “reducing train accuracy can improve test accuracy” is not a prerequisite for establishing overfitting. If this refers to model selection via early stopping, the plotted trajectories also show intervals where an earlier checkpoint has lower training certified accuracy but higher test certified accuracy (e.g., at 75 percent of the compute budget for the 10× trajectory). We would welcome clarification of the alternative definition on which the reviewer bases their position that “the authors seem to even misunderstand the meaning of overfitting” and authoritative references to this definition.

Finally, we reiterate that we appreciate substantive disagreement, but speculations about our understanding of basic machine learning concepts and the claims regarding potential inaccuracies in our account of prior work leave us very irritated. We therefore sincerely hope that the Area Chair and other reviewers can assess the significance of the present review in light of the corrections and evidence provided above.

References

[1] Altstidl et al., On the Scalability of Certified Adversarial Robustness with Generated Data, NeurIPS 2024.

[2] Zhang et al., Boosting the Certified Robustness of L-infinity Distance Nets, ICLR 2022

[3] Zhang et al., Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective, NeurIPS 2022.

[4] Xu et al., LOT: Layer-wise Orthogonal Training on Improving ℓ₂ Certified Robustness, NeurIPS 2022.

[5] Hu et al., Unlocking Deterministic Robustness Certification on ImageNet, NeurIPS 2023

[6] Gowal et al., Scalable Verified Training for Provably Robust Image Classification, ICCV 2019.

[7] Shi et al., Fast Certified Robust Training with Short Warmup, NeurIPS 2021.

[8] Zhang et al., Towards Stable and Efficient Training of Verifiably Robust Neural Networks, ICLR 2020.

[9] Mao et al., CTBench: A Library and Benchmark for Certified Training, ICML 2025

[10] Kaulen et al., Rethinking Evaluation Paradigms in IBP-based Certified Training, ICML 2026

[11] De Palma et al., Expressive Losses for Verified Robustness via Convex Combinations, ICLR 2024

[12] Goodfellow, Bengio and Courville, Deep Learning, 2016 (https://www.deeplearningbook.org/contents/ml.html)
Section 5.2, p. 110

[13] Stanford CS231n, Neural Networks Part 3: Learning and Evaluation, subsection “Train/Val accuracy”,https://cs231n.github.io/neural-networks-3/#accuracy

Dear authors,

After carefully reading the rebuttals, I respectfully point out that I did not misunderstand any major part of this work in my original review. I will only follow on the main points to make this discussion more constructive, although not all minor points are adequately addressed.

  1. Comparison with Altstidl et al.

Yes, they did more study around Lipschitz-based methods. However, it is clear that they also included studies about bound-based methods. In particular, they covered the ϵ=8/255\epsilon=8/255 setup, while this work only additionally covers ϵ=2/255\epsilon=2/255. I would like to highlight that this means roughly half of this work is merely an appendix of a prior work.

  1. Performance on 8/2558/255 and the argument of improved clean accuracy.

First, in certified training, the main metric is surely the certified accuracy: one would simply optimize clean accuracy if they want an improved clean accuracy. While it is of course also meaningful to improve the Pareto frontier, it is not meaningful when the Pareto frontier is limited to a specific interval, especially not to the right of the axis. I would like to highlight that CTBench uses 1/50 data and 1/5 compute to achieve 35.41% certified accuracy, while the best result reported in this work is 35.57%. How can one claim that scaling helps when much increased budget only improves 0.16% certified accuracy in a work discussing certified training?

  1. Definition of overfitting.

It is clear that the authors confused a behaviorial description with a definition. How large is "too large" in the context of train/test accuracy gap? For an example of precise textbook reference, check "An Introduction to Statistical Learning" where they describe overfitting as a reduced model family (in practice we often slightly abuse definition and also include stronger model regularization) would yield a better prediction performance. Unless this work proves that a model with lower train accuracy can improve the best test accuracy, it cannot conclude the model is overfitting. As a demonstration case, in a label-error setup, we often expect much higher train accuracy, while the test accuracy still consistently improves. One can even make this gap arbitrarily large by adjusting the label error rate.

Finally, I would like to explicitly state a potentially review-changing question for the authors. What do they believe is important in this work that the community should read this work? My counterargument would be, scaling up to 100x data and 10x compute improves the certified accuracy minimally for 1/2 experimental setup, while the only meaningful result is that for a specific ϵ=2/255\epsilon=2/255 (potentially also extend to other smaller ϵ\epsilon case but no proof), scaling by that much can further improve the accuracies. Even then, I do not believe such an empirical finding is novel enough for accept, given the certified training community already knows scaling helps accuracies, although the prior work's study is incomplete for a specific algorithm.

We thank the reviewer for their response. Several points raised were addressed in our original rebuttal; we clarify them below and provide concrete checkpoint results supporting our overfitting analysis.

Q: Altstidl et al. also conducted experiments on bound-based methods.

A: We explicitly acknowledged these experiments in our previous response. However, they reported catastrophic deterioration: Table 10 shows certified accuracy falling from 35.08% without generated data to 0.00% with 5 million generated images at 260 epochs. They concluded: “MTL-IBP, a convex bound propagation approach, cannot benefit from additional generated data.” Our study demonstrates that this negative conclusion does not hold generally and systematically investigates when scaling IBP-based methods succeeds.

However, we want to reiterate that the scaling study is not our primary contribution, but a means to mitigate the largely overlooked limit of robust overfitting in recent hybrid methods. Based on this, we conduct an extensive analysis of the more fundamental limitations, uncovering performance saturation at larger radii and increased verification costs. Thus, our work is not "merely an appendix of a prior work" but a comprehensive investigation that identifies and characterizes these limitations, provides insights into their underlying causes, and establishes practical guidelines for developing more effective and reliable methods.

Q: Performance on CIFAR-10 at ε = 8/255.

A: In both our manuscript, as well as in the original response, we acknowledge the limited improvement in certified accuracy at this radius and explicitly identify this saturation as a limitation of current approaches.

However, we argue that certified accuracy alone does not fully capture the trade-off between clean and certified accuracy. Therefore, we consider the substantial improvement in clean performance at this benchmark to represent a meaningful performance gain. As discussed by Kaulen et al. [1], changing hyperparameters can favour certified accuracy at the expense of clean accuracy without improving the generally achievable trade-off and refer to this work for a thorough discussion of the underlying evaluation problem.

Q: Definition of overfitting.

A: We thank the reviewer for clarifying their position. As mentioned in our previous response, Figure 2 also demonstrates the specific behaviour requested by the reviewer: earlier checkpoints have lower training certified accuracy but higher test certified accuracy than the final model. We give concrete evidence in the following table.

ComputeEarly → final epochTrain certified accuracyTest certified accuracyEarly test advantage
1×130 → 16061.0% → 61.7%53.0% → 52.4%+0.6 pp
2×200 → 32058.6% → 64.6%52.0% → 50.5%+1.5 pp
5×550 → 80064.8% → 67.9%50.2% → 48.0%+2.2 pp
10×900 → 160063.3% → 68.0%51.0% → 45.3%+5.7 pp

Thus, continued training improves training performance while worsening test performance, with increasingly severe deterioration under larger compute budgets. We would appreciate clarification of why these trajectories would not constitute robust overfitting.

Q: Why should the community read this work?

A: While IBP-based certified training achieves state-of-the-art performance across several benchmarks, progress has slowed, raising the question of whether current training objectives have reached their limits. We show that poor robust generalisation is a major, largely overlooked bottleneck: recent hybrid methods suffer severely from robust overfitting, which synthetic data mitigates, thereby enabling longer training schedules and substantial performance improvements.

We then investigate what limits progress once overfitting is reduced, revealing saturation of performance, especially at larger radii and increasingly difficult verification problems as important, more fundamental, limitations. Thus, the additional resources serve not merely to improve benchmark results, but to distinguish generalisation limitations from training and verification bottlenecks, providing guidance on where future advances are needed.

We thank the reviewer for the continued discussion and ask that the assessment be reconsidered in light of these concrete results and the analyses already provided in the submission and rebuttal.

Reference

[1] Kaulen et al., Rethinking Evaluation Paradigms in IBP-based Certified Training, ICML 2026.

Reviewer_gXFLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Summary: This paper reveals the bottleneck using IBP based certified training methods and concludes that the main reason comes from the overfitting. Though they propose to mitigate the overfitting issues through diffusion models generated extra training data, they meet another difficulty due to the bound tightness and verification complexity. Plenty of experimental results demonstrate these conclusions, but no clear theorem formulated on overfitting or scaling.

Critical strengths

Strong points:

  1. The motivations are clear: explain an important and under-explored question: whether certified-training plateaus come from optimization, generalization, or verification problems.

  2. Experimental results are informative: numerical improvements are meaningful and establish strong new results; Seed studies, method-transfer experiments, synthetic-data-quality tests, and longer verification budgets strengthen the conclusions; Experimental settings, software versions, hardware, schedules, and verification budgets are documented carefully.

Critical weaknesses

Weak points:

  1. Lacking theoretical propositions. There is no new certified-training algorithm or substantive theoretical result.
  2. Experimental results have limitations: (a) Experimental scope is not as broad and fundamental as claimed but limited to some special models and scales. (b) Authors fail to distinguish the unverified cases and verification timeout, causing confusion in the comparison of architectures. (c) The computational expense is very high, potentially limiting reproducibility. (d) The reliance on synthetic data raises practical questions about generator cost and dependence on generator quality. (e) Architecture and threat-model diversity remain limited.
Questions for authors

No other questions.

3 comments
AuthorsOpenReview ↗

Author Response [1]

We thank the reviewer for their assessment of our work. We are particularly grateful that our paper was found to address an “important and under-explored question” and that the numerical improvements were considered “meaningful” and recognised as establishing “strong new results”. We also appreciate the recognition that our additional experiments “strengthen the conclusions” and that the experimental setup is “documented carefully”.

However, we find it discouraging that one major criticism, namely that “there is no new certified-training algorithm or substantive theoretical result”, appears to concern the empirical nature of our contribution rather than the quality or significance of the investigation itself. We acknowledge that our work is primarily empirical, but respectfully maintain that a systematic experimental study can advance the field without introducing a new algorithm or theorem. Its contribution should instead be evaluated on the rigour of the experiments, the strength of the evidence and the significance of the resulting insights, which are qualities that the reviewer explicitly acknowledges.

Specifically, our study identifies robust overfitting as an important, yet largely overlooked, bottleneck in recent IBP-based certified training, demonstrates how mitigating it enables substantial performance improvements, and investigates the more fundamental limitations that emerge with further scaling. These findings clarify why progress has stalled and where future advances in training and verification are needed. In addition, the performance improvements enabled through our findings are comparable to years of newly proposed certified training algorithms (see Figure 1b). We therefore hope that our contribution can be assessed on the strength of this evidence and the insights it provides.

In the following, we address the remaining concerns regarding our study.

Q: Our experiments are limited to some special models and scales.
A: Our experiments were designed to systematically investigate how scaling affects state-of-the-art certified training. We therefore selected MTL-IBP and the CNN-7 architecture for our main study, since this combination achieved strongest results in related work (see, e.g., [1,2,3]). In Appendix C.2, we additionally investigate the well-performing hybrid method SABR and traditional IBP training. We also examine increased model capacity by doubling network widths. Regarding datasets, we chose TinyImageNet and CIFAR-10, which constitute the settings in which certified training techniques are usually evaluated (see, e.g., [1,2,3]).

Comparing our setup to other scaling studies, Altstidl et al. [4] investigate four data scaling settings while extending training schedules by two and three times for Lipschitz-based architectures; we investigate a grid of five data and for compute scaling factors up to 10 times the baseline compute budget. Regarding adversarial training, Wang et al. [5] primarily examine data quantity and training duration through separate ablations, whereas our grid directly explores their interaction. Bartoldson et al. [6] conduct an extensive scaling study, generally using each generated example once; our setup instead varies the available data pool independently of the optimisation budget.

We therefore maintain that our experiments are extensive, compared to the standards of the community, as clearly reflected in related work published at top-tier venues, enable the analysis of data and compute scaling both individually and jointly, and provide strong empirical support for our claims within the investigated settings.

Q: Unverified instances and unsolved instances are not adequately distinguished.
A: We regret that our explanation of certified accuracy was not clear enough and will update it in the revised manuscript. We defined certified accuracy as the proportion of certifiably robust inputs over all inputs, including certifiably robust, non-robust and unsolved instances, as usually done in related literature (see, e.g., [1,2,3]). Thus, unsolved instances are not excluded from reported results.

AuthorsOpenReview ↗

Author Response [2]

Q: The computational expense is high, thereby limiting reproducibility.
A: We acknowledge that a systematic scaling study requires substantial computational resources, particularly when evaluating certified accuracy using complete verification as done in our study. However, this expense should be distinguished from a lack of reproducibility: as the reviewer recognises, our experimental settings, software versions, hardware, schedules and verification budgets are documented carefully. We additionally provide the code, synthetic datasets and configurations needed to reproduce our experiments. Individual results can therefore be reproduced without repeating the entire scaling grid.

Moreover, quantifying the benefits and limitations of additional computational investment is a central purpose of our study. Our results show where increased resources improve performance, where gains saturate and where verification costs become prohibitive. These findings can help researchers allocate resources more effectively and avoid expensive scaling configurations that offer little additional benefit.

Q: What are the costs for generating data?
A: Generating the auxiliary datasets from pretrained checkpoints required approximately 173 GPU hours on an Nvidia H100 for 10 million TinyImageNet images and 167 GPU hours on an Nvidia H100 for 5 million discriminator-guided CIFAR-10 images. These computations are only required once, since the datasets could be used across training methods, and exclude the cost of training the diffusion models used for data generation themselves. We will include these costs in the final version of our manuscript to make the overall resource requirements more transparent.

Q: How dependent are results on the quality of generated data?
A: We respectfully refer the reviewer to Appendix C.4, which investigates this question by comparing EDM-generated data with samples from a discriminator-guided diffusion model achieving lower FID scores, indicating higher data quality. Across the investigated data and compute scales, higher-quality images do not consistently improve performance, yielding largely comparable results. This suggests that the benefits of data scaling can already be realised with the simpler EDM generator. We will reference these additional results more clearly in a revised version of the manuscript.

Q: Architecture and threat-model diversity is limited.
A: We aimed to investigate scaling in settings where IBP-based methods are competitive and commonly evaluated. Our main study therefore focuses on CNN-7, the de facto standard architecture for IBP-based certified training (see, e.g., [1,2,3]), and examines increased capacity by doubling its width. We evaluate CIFAR-10 and TinyImageNet under established threat models routinely used to demonstrate advances in this field (see, e.g., [1,2,3]). We therefore consider this experimental scope appropriate for supporting our findings within the certified training community.

In addition, in a general comment, we present new results on a ResNet and deeper and shallower convolutional networks, showing that our findings are not dependent on the employed architecture. We welcome concrete suggestions for further experiments that would strengthen the evaluation and are happy to provide additional results during the rebuttal period.

Finally, we thank the reviewer again for their thoughtful feedback. We hope that our explanations address the concerns raised and facilitate a discussion of the analyses and findings on their merits, recognising the empirical nature of our contribution.

References

[1] Mao et al., CTBench: A Library and Benchmark for Certified Training, ICML 2025

[2] Kaulen et al., Rethinking Evaluation Paradigms in IBP-based Certified Training, ICML 2026
[3] De Palma et al., Expressive Losses for Verified Robustness via Convex Combinations, ICLR 2024
[4] Altstidl et al., On the Scalability of Certified Adversarial Robustness with Generated Data, NeurIPS 2024.
[5] Wang et al., Better Diffusion Models Further Improve Adversarial Training, ICML 2023.
[6] Bartoldson et al., Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies, ICML 2024.

Reviewer_gXFLOpenReview ↗

Final comments

Thank you for the detailed response and additional experiments. While these strengthen the paper, my main concerns remain only partially addressed. I therefore maintain my original assessment.

Reviewer_5SWjOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Exploring the use of synthetic data and additional compute to improve produce improvements in the SOTA of IBP style certification performance.

Critical strengths

The authors demonstrate improvements to SOTA performance (although I hold concerns regarding both how SOTA is calculated, the confidence associated with that improvement, and the definition / generalizability of that confidence). Writing is generally of an acceptable standard, however there are some definite issues regarding specificity, presentation, and assumed knowledge (see weaknesses).

I'll also note that I appreciated the distinction between the temporal cost of algorithmic improvements (as in, moving between techniques) and the relative saturation there.

That "300 s per-instance cutoff" was documented as a part of S8 was interesting and more than I've seen in other IBP papers, but I'll also note that it then belies the question of what the overall computational resources (in terms of time) were for the study, why 300s per-instance was chosen, and what happened to the samples outside this range. For example: it may well be that the samples over 300s per instance were the hardest samples to certify for your approach. Were average performance results constructed including these excluded samples, or excluding?

Critical weaknesses

I have to say, that while I appreciate that the authors have managed to marginally improve performance, I must ask the purpose for which this matters. IBP style certifications are typically the largest of the family of certification techniques - but size isn't the limiting factor here. Rather, IBP style certifications are inherently hamstrung by the time to certify, the computational resources to certify, and architectural limitations. So I'm forced to see this work as a marginal conclusion that has limited value in moving forward the field, because it doesn't address a problem that's actually faced by IBP.

Or, to put the above in another way: what would the authors expect would be the impact of their work? Is this really moving IBP style certifications towards any kind of reliable, deployable reality? What does the work serve?

When it comes to the actual results themselves - I have to ask how transferable they are. They apply to a single model, a single training architecture, and two datasets - all of which are below what I would expect (although I note that my primarily community of practice is more straight AML and other styles of certification, rather than IBP). I also note that experimental data does not include any confidence intervals (which is notable for figures like Fig 5 lower-left and lower-right, which are quite noisey), and there's no significance tests to ensure that the relatively small observed performance deltas are statistically significant.

The points raised in the weaknesses section to this point underpin my broader concern with the paper - it moves SOTA slightly forward, but I'm not sure as a reader I would be able to take much from the work - even if I was working more actively in the IBP space. Spending more compute - and using more data - than prior approaches and improving on SOTA doesn't necessarily tell me anything, besides something that we already know in the ML space, in that hitting something with a bigger hammer likely flattens it out further than what would be achievable with a smaller hammer. However, exploring a bigger hammer doesn't help me to understand the why. What is happening when EDMs are used? Why are they able to produce this performance? Are there any downstream consequences for how the model performs? Would other approaches to data generation (eg. just adding randomness) approach the same performance with the same amounts of compute? There are so many interesting questions that could be answered, and right now, I'm not really seeing any of them.

I'll also note that given that EDMs are a key part of the generation process, they're barely discussed within the main body of the work. This would, to me, appear to be a rather odd choice.

General notes

  • Abstract is quite generic.
  • Choice of citations is odd at times. Croce & Hein and Madry being the only two citations for adversarial attacks?
  • Why is mathcal N mathcal P used for NP-completeness? This might just be a product of the communities in which I primarily read, but I'm more familiar with standard font NP-complete, but, well, this is some high pedantry.
  • "Best-performing methods are based on interval bound propagation (IBP) (Mirman et al., 2018; Gowal et al., 2019)" - I'd treat the claim that they're the best performing as also being one that's worth citing, as I think that otherwise other elements of the community would disagree here. Fundamentally, what is "best-performing", and how is it even measured?
  • This also applies to the earlier statement of "ibp...achieve state-of-the-art performance". IBP may produce SOTA in terms of certification size, however it also typically scales the worst with regards to the required computational requirements. Presenting certification strictly in terms of its size ignored the realities of operationalization, which surely is a part of the inspiration for certifications. As you note, the CNN-7 is the de facto standard for evaluating IBP certified training, yet by modern standards this is a tiny architecture - suggesting that SOTA is being set on a set of rules that inherently favor one approach. But this is me being a pedant, and potentially speaks to a long standing gripe I've had with SOTA evaluation in IBP style works.
  • I get that most readers will understand robust overfitting, but not all will - and at the moment, it's treated as assumed knowledge until S2.3.
  • Font choices for figure 3 are awful, at least on my screen - the black border makes it difficult to read to my eyes. And are the colours presenting the same data as the percentages, or something different? The colours are not explained, but you have a mid-green at 55.62 and a light at 54.10 - suggesting an incredibly tight colour scheme. It's odd either way.
  • Figure 19 has font that is way too small. Colours are not defined. Scales are not defined.
  • The appendices read like a being hit by a scatter gun. There's lots of things, but they're not well contextualized or explained. Look at Figure 31 for example - it gets half a paragraph, and there's no discussion covering the meaning or interpretation of the actual figures, or why there's such tightness between baseline, data only, and, in some cases data + compute.
Questions for authors
  • This is a broader question that exists beyond your work, but I ask this more for your intuition: if synthetic data can provide a point of mitigation against robust overfitting, how overfit is the data to the distribution from which the synthetic data is drawn?
  • How would this work generalize beyond De Palma's MTL-IBP? How would it generalize beyond CNN-7 architectures? What can be learned by this for certification approaches that can scale beyond CNN-7?
  • What value is there in a result that requires 100x data and 10x compute relative to prior approaches? Under what scenarios would this be an acceptable trade for a 3.83% and 2.15% increase over SOTA?
  • Did you evaluate any error bars? Confidence measures? P-scores to ensure significance for some of the claims?
  • For the 300s per instance cutoff, were average performance results constructed including these excluded samples, or excluding?
5 comments
AuthorsOpenReview ↗

Author Response [1]

We thank the reviewer for their detailed comments and are encouraged that the “improvements to SOTA performance” achieved through our scaling study were recognised. However, we maintain that the primary contribution of our work lies in an investigation of the limitations of IBP-based training. Specifically, our study identifies robust overfitting as an important, yet largely overlooked, bottleneck in recent IBP-based certified training, demonstrates how mitigating it enables substantial performance improvements, and investigates the more fundamental limitations that emerge with further scaling. These findings clarify why progress has stalled and where future advances in training and verification are needed.

One central concern raised by the reviewer appears to question the relevance of IBP-based certified training more broadly. In particular, the reviewer describes these techniques as “inherently hamstrung by the time to certify, the computational resources to certify, and architectural limitations” and consequently questions the value of further performance improvements. However, the limitations of IBP-based certified training are precisely what our study investigates. We therefore find it difficult to reconcile this criticism with our contribution. Rather than overlooking present limitations, we identify novel concrete limits in robust overfitting and verification cost, and show when they arise and how they constrain further progress.

We also respectfully disagree with the broader assessment of the value of improving IBP-based methods. These approaches achieve state-of-the-art certified robustness against $\ell_\infty$-bounded perturbations on high-dimensional datasets such as TinyImageNet and at smaller perturbation radii on CIFAR-10 (see, e.g., [1,2]). They do not require specialised layer types, and robustness certificates can be obtained efficiently for many inputs. Their continued relevance is also reflected in recent contributions published at leading machine learning venues (see, e.g., [1,2,3,4]). We therefore consider the performance improvements established by our study, which are comparable in magnitude to several years of algorithmic progress (see Figure 1b), valuable in their own right.

Similarly, we are surprised that the reviewer’s “broader concern with the paper” lies in the claim that “there are so many interesting questions that could be answered” regarding the reasons why data and compute scaling benefit performance, but they are “not really seeing any of them”. However, a substantial part of the paper is devoted to precisely such an analysis. Section 4.1 shows that additional synthetic data substantially reduces the certified generalisation gap, supporting improved robust generalisation as an explanation for the performance gains. Section 4.2 examines accompanying changes in the learned networks, revealing a shift in ReLU instability from the final classification layers towards the feature extractor. Section 4.3 further investigates why stronger models become harder to verify: although bounds improve for many inputs, scaling introduces additional challenging instances that require substantially more verification effort. Appendix C.4 also examines the effect of synthetic-data quality and finds that higher-quality generated data does not uniformly improve performance. We agree that simply showing that scaling data and compute is beneficial is not a very interesting finding on its own, and therefore intensively investigated the limits that remain once generalisation is improved. Thus, we are happy to explore additional questions that help understand the effects of scaling IBP-based training.

In the following, we address the remaining comments, questions and concerns raised by the reviewer in the order they were mentioned.

Q: What were the computational costs for verification?
A: We respectfully refer the reviewer to the PAR-10 numbers (Penalised Average Running Time, where unsolved instances are penalised by factor 10) presented in Section 4.3 (Verification Hardness), Appendix D.4 and the complete verification costs presented in Figure 26.

Q: How were instances that remained unsolved within the verification compute budget of 300s treated in calculating performance?
A: We regret that our explanation of certified accuracy was not clear enough and will update it in the revised manuscript. We defined certified accuracy as the proportion of certifiably robust inputs over all inputs, including certifiably robust, non-robust and unsolved instances, as usually done in related literature (see, e.g., [1,2,3]). Thus, unsolved instances are not excluded from reported results.

AuthorsOpenReview ↗

Author Response [2]

Q: Transferability of results is limited since “They apply to a single model, a single training architecture, and two datasets”.
A: We respectfully refer the reviewer to Appendix C.2, which evaluates the training methods SABR and traditional IBP across the whole data-compute scaling grid. These experiments demonstrate that the observed benefits extend beyond MTL-IBP. We additionally investigate two CNN-7 widths.

In addition, we present results on other architectures in a general comment, including a ResNet and deeper and shallower convolutional networks. We welcome concrete recommendations from the reviewer on further experiments that would strengthen the evaluation and are happy to provide additional results during the rebuttal period.

Q: Statistical significance is questionable.
A: We respectfully refer the reviewer to Appendix C.3 and Figure 9 for a study on the variance of the results. We report means and standard deviations across three seeds for the CIFAR-10 scaling grid using incomplete verification, and additionally repeat complete verification across three seeds for the strongest configurations. At $\epsilon$ = 2/255, these yield certified accuracies of 65.90%, 65.79% and 65.90%. Thus, the submission already provides evidence that the observed gains substantially exceed seed variance. We will reflect this more clearly in the main text of a revised version.

Q: SOTA is moved only slightly forward.
A: We respectfully refer the reviewer to Figure 1(b), showing that our observed improvements are comparable to several years of algorithmic progress in IBP-based certified training.

Q: What value is there in a performance improvement that requires 100× more data and 10× more compute?

A: First and foremost, we see our primary contribution in identifying that the apparent performance ceiling recently reached at smaller perturbation radii exemplified by limited algorithmic improvements in recent years cannot be attributed simply to weak training objectives induced by IBP. Instead, weak generalisation limited further progress, and there are more fundamental challenges that must be overcome that we identified in Section 4.4., most notably the verification bottleneck characterised by our work. We state this in our list of contributions at the end of the introduction, and we will further improve the presentation of our contributions in the revised manuscript.

Moreover, in safety-critical scenarios, whether the required guarantees can realistically be attained, even at substantial computational cost, may matter more than whether they can be obtained very efficiently. For example, an aircraft advisory system such as ACAS-Xu [5] or an autonomous vehicle must satisfy specified safety properties to support responsible deployment. In this context, our work shows that current approaches can be substantially improved through additional synthetic training data.

Finally, we want to thank the reviewer again for their detailed feedback and will incorporate the suggested changes regarding writing and the layout of the figures. We are confident that our provided explanations could address the concerns raised by the reviewer. Lastly, we sincerely hope that the provided clarifications can facilitate a discussion on the conducted analyses and their findings, and on the more fundamental limitations of IBP-based training uncovered by our work.

References
[1] Mao et al., CTBench: A Library and Benchmark for Certified Training, ICML 2025
[2] Kaulen et al., Rethinking Evaluation Paradigms in IBP-based Certified Training, ICML 2026
[3] De Palma et al., Expressive Losses for Verified Robustness via Convex Combinations, ICLR 2024
[4] Mao et al., Understanding Certified Training via Interval Bound Propagation, ICLR 2024
[5] Owen et al., “ACAS Xu: Integrated Collision Avoidance and Detect and Avoid Capability for UAS”, 38th Digital Avionics Systems Conference (DASC), 2019

I find it interesting that r7SB and I are both asking the same question (in my original review, and their rebuttal comment) - what is the purpose of this work?

We're two different reviewers who have taken different perspectives on the work, but the fact that both of us have been asking you this question could be argued to speak either to the narrative presentation of the work, or the fundamental value of the motivating research question behind the work. While I appreciate that you've attempted to answer this in your rebuttal responses, I am still not a believer enough in this work to change my score to a weak accept (although I may reflect on this further as additional rebuttal discussions come through).

I will also say that as someone who has been in the certification space for a long time, I find arguments like "Moreover, in safety-critical scenarios, whether the required guarantees can realistically be attained, even at substantial computational cost, may matter more than whether they can be obtained very efficiently......" mostly unconvincing (although I do acknowledge how common they are). Safety critical AI model deployments are happening now - and yet they do not employ certifications. Why? Well for one, real world deployments still need to take into account for latency and cost. Autonomous vehicles or AAS do not have the luxury of waiting significant amounts of time for a sample to be certified. And if the certificate is not tight, there is a follow on question of what value does the certification bring?

Fundamentally I still don't think you've answered the question as to what value does significantly increased compute and data requirements bring.

I appreciate the certification community has primarily been focused upon size, so perhaps my perspective is an outlier, but there are some that are starting to explore other avenues - e-values based certifications are looking at efficiency, and there's a neurips 26 paper on using certificates as a proxy for the WER in deployed ASR systems.

Also, since I'm here, could you please explain the 300s verification compute budget a little more? Thanks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Additional Results across Architectures

We thank the reviewers for their feedback, particularly the suggestion to investigate the generality of our findings across additional architectures. To address this, we have conducted experiments on deeper and shallower CNNs and a compact ResNet with two residual blocks (64/128 channels) and a 512-unit classification head, roughly matching CNN-7’s parameter count (~17M). We compare these results with those for CNN-7 and its wider variant already investigated in our study.

We report clean / certified accuracy (%). Clean accuracy is evaluated on all 10,000 test inputs, while certified accuracy is evaluated using α-CROWN on a fixed subset of 1,000 inputs to keep computational costs manageable. All results use seed 42.

CIFAR-10, ε = 2/255

ArchitectureBaseline1× data, 5× compute100× data, 1× compute100× data, 5× compute
CNN-579.08 / 55.5077.85 / 52.1080.75 / 62.1082.24 / 62.50
CNN-779.82 / 55.4079.05 / 52.1081.47 / 61.1082.72 / 61.20
CNN-7 Wide80.76 / 57.0080.55 / 52.1082.74 / 60.3084.02 / 59.10
CNN-979.60 / 56.9079.58 / 53.2081.10 / 59.5084.11 / 47.50
ResNet79.70 / 56.3078.55 / 50.8080.96 / 60.1082.79 / 62.50

These results show that our findings extend across architectures. Scaling data alone improves both clean and certified accuracy across all five architectures, whereas scaling compute alone reduces certified accuracy throughout. Joint scaling further improves clean accuracy relative to data-only scaling across all architectures, but its certification benefits vary. In particular, CNN-9 exhibits a substantial decline in certified accuracy. One possible explanation is that deeper networks accumulate larger approximation errors during bound propagation, making robustness harder to certify. However, the present results alone cannot distinguish this from a decline in underlying robustness.

CIFAR-10, ε = 8/255

ArchitectureBaseline1× data, 5× compute100× data, 1× compute100× data, 5× compute
CNN-551.76 / 37.0053.95 / 37.0053.06 / 37.8055.97 / 37.60
CNN-752.72 / 38.4055.85 / 36.1054.86 / 37.2056.11 / 36.50
CNN-7 Wide53.95 / 38.5055.59 / 36.9054.03 / 35.9056.96 / 36.20
CNN-951.90 / 37.6055.37 / 34.8052.23 / 35.2056.86 / 36.80
ResNet52.29 / 38.9054.11 / 36.2053.42 / 36.2057.22 / 29.60

At the larger radius, joint scaling improves clean accuracy across all architectures, while certified accuracy generally remains unchanged or declines. This is consistent with the limited certified gains reported in the main paper.

The additional experiments therefore extend our findings beyond CNN-7: scaling consistently improves the clean–certified accuracy trade-off at the smaller radius and clean accuracy at ε = 8/255. Importantly, different architectures do not yield decisive differences in performance. We will also run complete verification for the newly investigated settings and incorporate these findings into our study.

We thank the reviewers for suggesting this valuable addition and hope that these results address their concerns about transferability across architectures. We address the remaining concerns in our individual responses and welcome further discussion and concrete suggestions for additional experiments. Together with the evidence already presented in the submission, we believe these results strengthen the case for acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.