- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates how data uncertainty affects model prunability. On MNIST and CIFAR-10, the authors observe that for larger models, the number of parameters remaining after pruning first increases and then decreases as uncertainty increases.
Critical strengths
The authors not only add noise to the data but also attempt to construct samples with lower uncertainty using generated data, which is an interesting perspective. Experiments with models of different widths, along with comparisons at fixed accuracy levels, also help explain the observed trends.
Critical weaknesses
-
The paper aims to investigate whether the required model size changes when the optimal decision rule remains unchanged. However, the main results use different accuracy thresholds at different noise levels. At higher noise levels, a relaxed accuracy threshold may lead to smaller selected models. Therefore, the observed increase-then-decrease trend cannot be directly interpreted as a change in the required model capacity.
-
The number of parameters remaining after pruning depends heavily on the specific training and pruning methods. Although Figure 2 uses a fixed accuracy threshold, it still cannot rule out differences in fitting and optimization across noise levels. In the appendix, simply changing the pruning criterion leads to substantial changes in the results. Therefore, there is insufficient evidence to use this measure as a proxy for the intrinsic capacity required by the task.
-
The control of uncertainty is also insufficient. Datasets with the same average entropy may have very different noise distributions. In the generated-data experiments, images are selected directly based on the predictive entropy of an ensemble model. The low-entropy group may simply contain samples that are easier to classify, rather than samples with only reduced aleatoric uncertainty.
Questions for authors
See Weakness
2 comments
- The control of uncertainty is also insufficient. Datasets with the same average entropy may have very different noise distributions.
We do not disagree that the shape of the noise distribution could also be a factor that modulates pruning behavior in both the expansion and contraction regime. However, our results suggest that, given a family of noise distributions indexed by aleatoric uncertainty, varying aleatoric uncertainty produces two regimes of behavior. Both uniform label noise and distribution shaping already represent two different noise distributions in support of this claim.
In addition to these uncertainty modifiers, we have also implemented a form of label noise that we call boundary-blur label noise, which preferentially corrupts examples with higher-entropy conditional label distributions. Intuitively, examples that look more uncertain are more likely to have their label flipped, which is more aligned with the forms of label noise one might naturally observe.
We describe this experiment and its results in Appendix D. The results show that this form of noise also induces an expansion and a contraction regime, providing another data point in support of our claims.
In the generated-data experiments, images are selected directly based on the predictive entropy of an ensemble model. The low-entropy group may simply contain samples that are easier to classify, rather than samples with only reduced aleatoric uncertainty.
We agree that reducing aleatoric uncertainty through distribution shaping could also change the difficulty of individual examples. In the original submitted PDF, we explicitly anticipated this concern and designed the finetuning experiment in Section 8.2 and Appendix F.5 to test whether our results merely reflect an easier training distribution.
In this experiment, we compared networks trained and pruned on the reduced- and original-uncertainty distributions. We then finetuned both sets of networks on the same original-uncertainty distribution, keeping their pruning masks fixed, and evaluate them on the same held-out test distribution. This methodology allows us to determine whether the smaller networks obtained using reduced-uncertainty data can perform comparably on the original, more challenging distribution.
Our results show that finetuning substantially closes the accuracy gap between the two sets of networks. Averaged across architectures and pruning checkpoints, the gap decreases from 0.573 to 0.132 percentage points on MNIST and 0.780 to 0.347 percentage points on CIFAR-10. For larger MNIST architectures, networks originally trained on reduced-uncertainty data even outperform those trained on the original-uncertainty distribution after finetuning.
These findings demonstrate that the smaller networks obtained through distribution shaping are not merely specialized to an easier evaluation distribution. While we do not claim that example difficulty and aleatoric uncertainty can be perfectly disentangled, the proposed alternative explanation is explicitly investigated in our paper, and the experimental evidence argues against attributing our results solely to reduced task difficulty.
Thank you for raising this distinction. Target relaxation is explicitly one of the two mechanisms at the center of our hypothesis. This is stated in the abstract and Introduction, investigated experimentally in Section 6, and used to explain the two regimes in Section 7. Indeed, we explicitly state on page 2 that "target relaxation follows directly from our evaluation criterion."
We do not claim that target relaxation is surprising. Rather, the surprising finding is that pruned network size initially increases despite target relaxation, due to the competing effect of uncertainty overhead, which we isolate experimentally in Section 5.
Target relaxation is part of our explanation for the contraction regime. Our empirical claim concerns how uncertainty overhead and target relaxation jointly affect pruned network size under the specified training and pruning procedure. We agree that this observation alone does not establish a change in the intrinsic capacity required by the task.
We have since dropped the claim that one can use pruned network size as a proxy for the intrinsic capacity required by the task. See the revised introduction.
We agree that pruned network size depends on the training and pruning procedures; however, this dependence does not undermine our central empirical finding. Our experiments investigate how aleatoric uncertainty affects the compressibility of networks under a fixed training and pruning procedure. Figure 2 holds the architecture, accuracy target, and training and pruning protocols fixed across uncertainty levels. Differences in fitting and optimization induced by aleatoric uncertainty are therefore potential mechanisms underlying uncertainty overhead, rather than confounding variables that invalidate the observation.
The clean-label Taylor-pruning appendix in the original submission contained a preliminary comparison whose methodology had not been sufficiently validated. We have withdrawn that analysis from the revised paper and do not rely on it to support the two regimes hypothesis. Determining whether a first-order Taylor criterion outperforms iterative magnitude pruning in the presence of aleatoric uncertainty could be interesting future work, since prior work has shown that iterative magnitude pruning performs at least as well as first-order Taylor pruning [1, 2], but this work is out of scope of the current submission.
If you are referring to Appendix G.2, where we consider selecting the highest-accuracy sparse model as an alternative selection criterion, then this appendix actually provides further support of the two regimes hypothesis. Under this alternative criterion, we still observe an expansion and contraction regime.
[1] Yeom, S.-K., Seegerer, P., Lapuschkin, S., Binder, A., Wiedemann, S., Müller, K.-R., & Samek, W. (2021). Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning. Pattern Recognition.
[2] Janusz, M., Wojnar, T., Li, Y., Benini, L., & Adamczewski, K. (2025). One Shot vs. Iterative: Rethinking Pruning Strategies for Model Compression. European Conference on Artificial Intelligence (ECAI).