- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the scaling effect in synthetic data and compute on certified training, particularly IBP-based methods.
Critical strengths
I would like to inform AC and other reviewers that the main topic of this paper, i.e., scaling effect in certified training, has been extensively studied in Altstidl et al. cited by this work. The main difference is that Altstidl et al. included more certified training algorithms and demonstrated the scaling effect in both synthetic data and compute, while this work restricts to the MTL-IBP only. Therefore, the only strength I can observe is an improvement performance (although presented in a significantly over-claimed way since it didn't compare to the fair evaluation result of CTBench) on CIFAR-10 for MTL-IBP, which Altstidl et al. didn't choose to study as a concrete case.
Critical weaknesses
-
This work largely overlaps with Altstidl et al. and incorrectly stated the conclusion about MTL-IBP experiments done by them. While Altstidl et al. shows that MTL-IBP on CIFAR-10 does not benefit from scaling, this work does not establish the benefit for this setting as well. In fact, Fig 3.d did not see a consistent scaling benefit, and even the best number reported (35.58%) is very close to CTBench evaluation of the same algorithm and the same setting without any data or compute scaling (35.41%).
-
This work does not establish overfitting in certified training but it claims to. The main result, namely Fig. 2, is completely unmeaningful. It presents compute scaling both as legend and as x-axis. It does not show any overfitting. The authors seem to even misunderstand the meaning of overfitting and believe that more compute decreases performance means overfitting, while there is no result indicating reducing train accuracy can improve test accuracy.
Questions for authors
See above.
4 comments
Author Response [2]
Q: Performance is overstated since it is not compared to CTBench
A: CTBench evaluates configurations at different operating points on the clean–certified accuracy trade-off, including configurations that favour certified accuracy at the expense of clean accuracy through different hyperparameters, as shown by Kaulen et al. [10]. Our scaling study uses the hyperparameters of the original authors to isolate the effects of additional data and compute. Nevertheless, the presented results also outperform those presented in CTBench on the Pareto frontier and we are happy to include a comparison to CTBench in a revised version of the manuscript.
Q: No consistent scaling benefit is observed at ε = 8/255.
A: Section 3.4 explicitly reports that certified performance at this radius saturates near previously achieved levels. Nevertheless, scaling substantially improves clean accuracy while maintaining competitive certified accuracy. Our strongest configuration achieves 57.27% clean and 35.58% certified accuracy, which Pareto-dominates all previously known results from the literature on the trade-off between clean and certified accuracy [10].
Q: Figure 2 is “completely unmeaningful” and does not establish overfitting.
A: We will clarify the caption and axis labelling of the figure. The legend identifies separate runs with total compute budgets of 1×, 2×, 5× and 10×, whereas the horizontal axis represents progress through each run as a percentage of its respective budget. As described in Section 3.1, we proportionally extend the warm-up and ramp-up phases with the overall training schedule and, therefore, each compute scale does induce a separate training trajectory.
Regarding overfitting, we follow the standard definition taught in introductory textbooks and lectures. Goodfellow, Bengio and Courville state in the book “Deep Learning”: “Overfitting occurs when the gap between the training error and test error is too large” [12]. Stanford’s CS231n course notes likewise explain: “The gap between the training and validation accuracy indicates the amount of overfitting” [13].
Figure 2 examines precisely this divergence: training certified accuracy increases while test certified accuracy deteriorates. The reviewer’s proposed requirement that “reducing train accuracy can improve test accuracy” is not a prerequisite for establishing overfitting. If this refers to model selection via early stopping, the plotted trajectories also show intervals where an earlier checkpoint has lower training certified accuracy but higher test certified accuracy (e.g., at 75 percent of the compute budget for the 10× trajectory). We would welcome clarification of the alternative definition on which the reviewer bases their position that “the authors seem to even misunderstand the meaning of overfitting” and authoritative references to this definition.
Finally, we reiterate that we appreciate substantive disagreement, but speculations about our understanding of basic machine learning concepts and the claims regarding potential inaccuracies in our account of prior work leave us very irritated. We therefore sincerely hope that the Area Chair and other reviewers can assess the significance of the present review in light of the corrections and evidence provided above.
References
[1] Altstidl et al., On the Scalability of Certified Adversarial Robustness with Generated Data, NeurIPS 2024.
[2] Zhang et al., Boosting the Certified Robustness of L-infinity Distance Nets, ICLR 2022
[3] Zhang et al., Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective, NeurIPS 2022.
[4] Xu et al., LOT: Layer-wise Orthogonal Training on Improving ℓ₂ Certified Robustness, NeurIPS 2022.
[5] Hu et al., Unlocking Deterministic Robustness Certification on ImageNet, NeurIPS 2023
[6] Gowal et al., Scalable Verified Training for Provably Robust Image Classification, ICCV 2019.
[7] Shi et al., Fast Certified Robust Training with Short Warmup, NeurIPS 2021.
[8] Zhang et al., Towards Stable and Efficient Training of Verifiably Robust Neural Networks, ICLR 2020.
[9] Mao et al., CTBench: A Library and Benchmark for Certified Training, ICML 2025
[10] Kaulen et al., Rethinking Evaluation Paradigms in IBP-based Certified Training, ICML 2026
[11] De Palma et al., Expressive Losses for Verified Robustness via Convex Combinations, ICLR 2024
[12] Goodfellow, Bengio and Courville, Deep Learning, 2016 (https://www.deeplearningbook.org/contents/ml.html)
Section 5.2, p. 110
[13] Stanford CS231n, Neural Networks Part 3: Learning and Evaluation, subsection “Train/Val accuracy”,https://cs231n.github.io/neural-networks-3/#accuracy
Dear authors,
After carefully reading the rebuttals, I respectfully point out that I did not misunderstand any major part of this work in my original review. I will only follow on the main points to make this discussion more constructive, although not all minor points are adequately addressed.
- Comparison with Altstidl et al.
Yes, they did more study around Lipschitz-based methods. However, it is clear that they also included studies about bound-based methods. In particular, they covered the setup, while this work only additionally covers . I would like to highlight that this means roughly half of this work is merely an appendix of a prior work.
- Performance on and the argument of improved clean accuracy.
First, in certified training, the main metric is surely the certified accuracy: one would simply optimize clean accuracy if they want an improved clean accuracy. While it is of course also meaningful to improve the Pareto frontier, it is not meaningful when the Pareto frontier is limited to a specific interval, especially not to the right of the axis. I would like to highlight that CTBench uses 1/50 data and 1/5 compute to achieve 35.41% certified accuracy, while the best result reported in this work is 35.57%. How can one claim that scaling helps when much increased budget only improves 0.16% certified accuracy in a work discussing certified training?
- Definition of overfitting.
It is clear that the authors confused a behaviorial description with a definition. How large is "too large" in the context of train/test accuracy gap? For an example of precise textbook reference, check "An Introduction to Statistical Learning" where they describe overfitting as a reduced model family (in practice we often slightly abuse definition and also include stronger model regularization) would yield a better prediction performance. Unless this work proves that a model with lower train accuracy can improve the best test accuracy, it cannot conclude the model is overfitting. As a demonstration case, in a label-error setup, we often expect much higher train accuracy, while the test accuracy still consistently improves. One can even make this gap arbitrarily large by adjusting the label error rate.
Finally, I would like to explicitly state a potentially review-changing question for the authors. What do they believe is important in this work that the community should read this work? My counterargument would be, scaling up to 100x data and 10x compute improves the certified accuracy minimally for 1/2 experimental setup, while the only meaningful result is that for a specific (potentially also extend to other smaller case but no proof), scaling by that much can further improve the accuracies. Even then, I do not believe such an empirical finding is novel enough for accept, given the certified training community already knows scaling helps accuracies, although the prior work's study is incomplete for a specific algorithm.
We thank the reviewer for their response. Several points raised were addressed in our original rebuttal; we clarify them below and provide concrete checkpoint results supporting our overfitting analysis.
Q: Altstidl et al. also conducted experiments on bound-based methods.
A: We explicitly acknowledged these experiments in our previous response. However, they reported catastrophic deterioration: Table 10 shows certified accuracy falling from 35.08% without generated data to 0.00% with 5 million generated images at 260 epochs. They concluded: “MTL-IBP, a convex bound propagation approach, cannot benefit from additional generated data.” Our study demonstrates that this negative conclusion does not hold generally and systematically investigates when scaling IBP-based methods succeeds.
However, we want to reiterate that the scaling study is not our primary contribution, but a means to mitigate the largely overlooked limit of robust overfitting in recent hybrid methods. Based on this, we conduct an extensive analysis of the more fundamental limitations, uncovering performance saturation at larger radii and increased verification costs. Thus, our work is not "merely an appendix of a prior work" but a comprehensive investigation that identifies and characterizes these limitations, provides insights into their underlying causes, and establishes practical guidelines for developing more effective and reliable methods.
Q: Performance on CIFAR-10 at ε = 8/255.
A: In both our manuscript, as well as in the original response, we acknowledge the limited improvement in certified accuracy at this radius and explicitly identify this saturation as a limitation of current approaches.
However, we argue that certified accuracy alone does not fully capture the trade-off between clean and certified accuracy. Therefore, we consider the substantial improvement in clean performance at this benchmark to represent a meaningful performance gain. As discussed by Kaulen et al. [1], changing hyperparameters can favour certified accuracy at the expense of clean accuracy without improving the generally achievable trade-off and refer to this work for a thorough discussion of the underlying evaluation problem.
Q: Definition of overfitting.
A: We thank the reviewer for clarifying their position. As mentioned in our previous response, Figure 2 also demonstrates the specific behaviour requested by the reviewer: earlier checkpoints have lower training certified accuracy but higher test certified accuracy than the final model. We give concrete evidence in the following table.
| Compute | Early → final epoch | Train certified accuracy | Test certified accuracy | Early test advantage |
|---|---|---|---|---|
| 1× | 130 → 160 | 61.0% → 61.7% | 53.0% → 52.4% | +0.6 pp |
| 2× | 200 → 320 | 58.6% → 64.6% | 52.0% → 50.5% | +1.5 pp |
| 5× | 550 → 800 | 64.8% → 67.9% | 50.2% → 48.0% | +2.2 pp |
| 10× | 900 → 1600 | 63.3% → 68.0% | 51.0% → 45.3% | +5.7 pp |
Thus, continued training improves training performance while worsening test performance, with increasingly severe deterioration under larger compute budgets. We would appreciate clarification of why these trajectories would not constitute robust overfitting.
Q: Why should the community read this work?
A: While IBP-based certified training achieves state-of-the-art performance across several benchmarks, progress has slowed, raising the question of whether current training objectives have reached their limits. We show that poor robust generalisation is a major, largely overlooked bottleneck: recent hybrid methods suffer severely from robust overfitting, which synthetic data mitigates, thereby enabling longer training schedules and substantial performance improvements.
We then investigate what limits progress once overfitting is reduced, revealing saturation of performance, especially at larger radii and increasingly difficult verification problems as important, more fundamental, limitations. Thus, the additional resources serve not merely to improve benchmark results, but to distinguish generalisation limitations from training and verification bottlenecks, providing guidance on where future advances are needed.
We thank the reviewer for the continued discussion and ask that the assessment be reconsidered in light of these concrete results and the analyses already provided in the submission and rebuttal.
Reference
[1] Kaulen et al., Rethinking Evaluation Paradigms in IBP-based Certified Training, ICML 2026.
Author Response [1]
We, the authors, are very concerned about the factual inaccuracies in this review and are severely worried that several statements made by the reviewer may reflect misunderstandings of our submission and its relationship to prior work. Below, we address our concerns in detail so that the Area Chair and other reviewers can independently assess the made claims against the evidence presented in our submission.
Q: The work largely overlaps with Altstidl et al., 2024 [1].
A: We explicitly acknowledge Altstidl et al. and discuss their findings in Section 2.3 and differentiate our work from theirs in Appendix A.2. As explained there, the reviewer’s statement that the “main difference” is that they “included more certified training algorithms” mischaracterises the methodological distinction. Their main study investigates Lipschitz-based methods, which obtain guarantees through bounds on network Lipschitz constants (see, e.g., [2,3,4,5]). For this, these methods rely on specialised layers and may be restricted to the $\ell_2$ ball threat model [4,5]. Our study investigates an orthogonal approach to certified robustness, IBP-based training, where certification against input perturbations within $\ell_\infty$ balls relies on propagating reachability bounds through the network, potentially followed by branch-and-bound verification (see, e.g., [6,7,8]). Importantly, bound propagation is agnostic to the employed network architecture. Thus, these approaches differ strongly in the models they are applicable to, their training objectives and the downstream certification procedures.
We therefore view the scaling study of Altstidl et al. as a motivation to investigate the scaling behaviour of IBP-based certified training. While complementary approaches have been scaled successfully, a systematic study of IBP-based methods has remained missing, despite their state-of-the-art certified robustness on high-dimensional datasets such as TinyImageNet and at smaller perturbation radii on CIFAR-10 (see, e.g., [9,10,11]).
Moreover, our main contribution extends beyond the scaling study itself. Our work reveals that the apparent performance limits of IBP-based certified training were not simply attributable to weak training objectives, but were substantially driven by poor robust generalisation. Mitigating this limitation through synthetic data enabled substantial performance improvements, while further scaling revealed saturation and increasingly prohibitive verification costs as performance bottlenecks. These previously unexplored findings distinguish our contribution from merely demonstrating that additional data and compute can improve performance.
Q: Conclusion on MTL-IBP experiments of Altstidl et al., 2024 is incorrectly stated.
A: We respectfully disagree. Altstidl et al. explicitly conclude in Appendix H: “MTL-IBP, a convex bound propagation approach, cannot benefit from additional generated data.” Our manuscript accurately reflects their findings in Section 2.3, ll. 195-197.:
“However, their preliminary experiments with MTL-IBP reported catastrophic performance when scaling training data and compute, suggesting that these benefits do not trivially transfer to IBP-based certified training.”
This description is supported by their Table 10, where certified accuracy drops from 35.08% without generated data to 0.00% with 5 million generated images at 260 epochs. We therefore ask the reviewer to identify precisely which part of our account they consider incorrect.
Q: The work is restricted to MTL-IBP
A: Appendix C.2 evaluates the well-performing hybrid method SABR and traditional IBP training across the full compute and data scaling grid at both CIFAR-10 perturbation radii, where we observed similar results that support our main findings. We will state more clearly in a revised version of the manuscript that MTL-IBP is the focus of the main study, but not the only training method investigated.
In addition, in a general comment, we present new results on a ResNet and deeper and shallower convolutional networks, showing that our findings are not dependent on the employed architecture. We welcome concrete suggestions for further experiments that would strengthen the evaluation and are happy to provide additional results during the rebuttal period.