acceptodds
Under review as a conference paper at ICLR 2027

The Two Regimes of Pruning Under Aleatoric Uncertainty

Abstract

Nearly every machine learning problem grounded in the natural world exhibits irreducible uncertainty in its data-generating process, a property known as _aleatoric uncertainty_. Aleatoric uncertainty can leave the optimal decision rule unchanged, which raises a natural question: should the size of a neural network approximating this decision rule also remain unchanged? We study this question through _pruning_—the process of removing parameters from a model to obtain a smaller model—which we use as a proxy for the minimum model size needed to approximate the optimal decision rule. We find that pruned network size is not invariant to aleatoric uncertainty. Instead, sufficiently overparameterized neural networks exhibit two regimes of behavior: increasing aleatoric uncertainty initially increases pruned network size (the _expansion regime_), but beyond a critical point, further uncertainty decreases it (the _contraction regime_). Through ablations, we find that these two regimes arise from two competing effects: increasing aleatoric uncertainty increases the pruned network size required for a fixed performance target, while simultaneously lowering the performance target set by the dense model.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Pruning, Aleatoric Uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jvYcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how data uncertainty affects model prunability. On MNIST and CIFAR-10, the authors observe that for larger models, the number of parameters remaining after pruning first increases and then decreases as uncertainty increases.

Critical strengths

The authors not only add noise to the data but also attempt to construct samples with lower uncertainty using generated data, which is an interesting perspective. Experiments with models of different widths, along with comparisons at fixed accuracy levels, also help explain the observed trends.

Critical weaknesses
  1. The paper aims to investigate whether the required model size changes when the optimal decision rule remains unchanged. However, the main results use different accuracy thresholds at different noise levels. At higher noise levels, a relaxed accuracy threshold may lead to smaller selected models. Therefore, the observed increase-then-decrease trend cannot be directly interpreted as a change in the required model capacity.

  2. The number of parameters remaining after pruning depends heavily on the specific training and pruning methods. Although Figure 2 uses a fixed accuracy threshold, it still cannot rule out differences in fitting and optimization across noise levels. In the appendix, simply changing the pruning criterion leads to substantial changes in the results. Therefore, there is insufficient evidence to use this measure as a proxy for the intrinsic capacity required by the task.

  3. The control of uncertainty is also insufficient. Datasets with the same average entropy may have very different noise distributions. In the generated-data experiments, images are selected directly based on the predictive entropy of an ensemble model. The low-entropy group may simply contain samples that are easier to classify, rather than samples with only reduced aleatoric uncertainty.

Questions for authors

See Weakness

2 comments
  1. The paper aims to investigate whether the required model size changes when the optimal decision rule remains unchanged. However, the main results use different accuracy thresholds at different noise levels. At higher noise levels, a relaxed accuracy threshold may lead to smaller selected models. Therefore, the observed increase-then-decrease trend cannot be directly interpreted as a change in the required model capacity.

Thank you for raising this distinction. Target relaxation is explicitly one of the two mechanisms at the center of our hypothesis. This is stated in the abstract and Introduction, investigated experimentally in Section 6, and used to explain the two regimes in Section 7. Indeed, we explicitly state on page 2 that "target relaxation follows directly from our evaluation criterion."

We do not claim that target relaxation is surprising. Rather, the surprising finding is that pruned network size initially increases despite target relaxation, due to the competing effect of uncertainty overhead, which we isolate experimentally in Section 5.

Target relaxation is part of our explanation for the contraction regime. Our empirical claim concerns how uncertainty overhead and target relaxation jointly affect pruned network size under the specified training and pruning procedure. We agree that this observation alone does not establish a change in the intrinsic capacity required by the task.

  1. The number of parameters remaining after pruning depends heavily on the specific training and pruning methods. Although Figure 2 uses a fixed accuracy threshold, it still cannot rule out differences in fitting and optimization across noise levels. In the appendix, simply changing the pruning criterion leads to substantial changes in the results. Therefore, there is insufficient evidence to use this measure as a proxy for the intrinsic capacity required by the task.

We have since dropped the claim that one can use pruned network size as a proxy for the intrinsic capacity required by the task. See the revised introduction.

We agree that pruned network size depends on the training and pruning procedures; however, this dependence does not undermine our central empirical finding. Our experiments investigate how aleatoric uncertainty affects the compressibility of networks under a fixed training and pruning procedure. Figure 2 holds the architecture, accuracy target, and training and pruning protocols fixed across uncertainty levels. Differences in fitting and optimization induced by aleatoric uncertainty are therefore potential mechanisms underlying uncertainty overhead, rather than confounding variables that invalidate the observation.

The clean-label Taylor-pruning appendix in the original submission contained a preliminary comparison whose methodology had not been sufficiently validated. We have withdrawn that analysis from the revised paper and do not rely on it to support the two regimes hypothesis. Determining whether a first-order Taylor criterion outperforms iterative magnitude pruning in the presence of aleatoric uncertainty could be interesting future work, since prior work has shown that iterative magnitude pruning performs at least as well as first-order Taylor pruning [1, 2], but this work is out of scope of the current submission.

If you are referring to Appendix G.2, where we consider selecting the highest-accuracy sparse model as an alternative selection criterion, then this appendix actually provides further support of the two regimes hypothesis. Under this alternative criterion, we still observe an expansion and contraction regime.

[1] Yeom, S.-K., Seegerer, P., Lapuschkin, S., Binder, A., Wiedemann, S., Müller, K.-R., & Samek, W. (2021). Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning. Pattern Recognition.

[2] Janusz, M., Wojnar, T., Li, Y., Benini, L., & Adamczewski, K. (2025). One Shot vs. Iterative: Rethinking Pruning Strategies for Model Compression. European Conference on Artificial Intelligence (ECAI).

  1. The control of uncertainty is also insufficient. Datasets with the same average entropy may have very different noise distributions.

We do not disagree that the shape of the noise distribution could also be a factor that modulates pruning behavior in both the expansion and contraction regime. However, our results suggest that, given a family of noise distributions indexed by aleatoric uncertainty, varying aleatoric uncertainty produces two regimes of behavior. Both uniform label noise and distribution shaping already represent two different noise distributions in support of this claim.

In addition to these uncertainty modifiers, we have also implemented a form of label noise that we call boundary-blur label noise, which preferentially corrupts examples with higher-entropy conditional label distributions. Intuitively, examples that look more uncertain are more likely to have their label flipped, which is more aligned with the forms of label noise one might naturally observe.

We describe this experiment and its results in Appendix D. The results show that this form of noise also induces an expansion and a contraction regime, providing another data point in support of our claims.

In the generated-data experiments, images are selected directly based on the predictive entropy of an ensemble model. The low-entropy group may simply contain samples that are easier to classify, rather than samples with only reduced aleatoric uncertainty.

We agree that reducing aleatoric uncertainty through distribution shaping could also change the difficulty of individual examples. In the original submitted PDF, we explicitly anticipated this concern and designed the finetuning experiment in Section 8.2 and Appendix F.5 to test whether our results merely reflect an easier training distribution.

In this experiment, we compared networks trained and pruned on the reduced- and original-uncertainty distributions. We then finetuned both sets of networks on the same original-uncertainty distribution, keeping their pruning masks fixed, and evaluate them on the same held-out test distribution. This methodology allows us to determine whether the smaller networks obtained using reduced-uncertainty data can perform comparably on the original, more challenging distribution.

Our results show that finetuning substantially closes the accuracy gap between the two sets of networks. Averaged across architectures and pruning checkpoints, the gap decreases from 0.573 to 0.132 percentage points on MNIST and 0.780 to 0.347 percentage points on CIFAR-10. For larger MNIST architectures, networks originally trained on reduced-uncertainty data even outperform those trained on the original-uncertainty distribution after finetuning.

These findings demonstrate that the smaller networks obtained through distribution shaping are not merely specialized to an easier evaluation distribution. While we do not claim that example difficulty and aleatoric uncertainty can be perfectly disentangled, the proposed alternative explanation is explicitly investigated in our paper, and the experimental evidence argues against attributing our results solely to reduced task difficulty.

Reviewer_u9AsOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents an experimental investigation on the neural network size as a function of aleatoric uncertainty. The authors train several model variants on vision tasks (MNIST and CIFAR10) and calculate which size (expressed in # of parameters) is needed to match the original model's performance. The authors then conclude that the relationship between pruned model size and aleatoric uncertainty follows an inverted U shape (two different regimes):

  • a low aleatoric uncertainty causes the pruned network size to increase
  • following a certain limit amount of uncertainty, the regime changes and the pruned network size decreases
Critical strengths
  • The analysis of this paper is novel.
  • Despite not training the model on large datasets and limiting themselves to vision, the authors present a lot of experiments (as a combination of different hyperparameters) to their proof
Critical weaknesses
  1. Limited importance of the findings: the scientific importance of these findings seems very limited and does not lead to any practical advance:

    1. The inverted U-shape curve can partly be explained by "target relaxation"
    2. The authors do not state how their contributions benefit the scientific field
    3. The authors do not present an investigation on "uncertainty overhead"
  2. Supposedly-limited human contribution: the AI statement seems particularly dry and it does not allow to understand what is the amount of human contribution to this paper. The statement indicates that I tools were assisting with "proposing and redefining hypotheses, developing theoretical argument and proofs, designing experiments, processing data, implementing and running experiment code, interpreting results, conducting literature review, and writing portions of the paper". If these tasks were ran mostly by AIs, then it means the authors' contribution is merely the review of the AI work and the author should be the AI that produced this work. (Ref. Question 1) The authors should state more clearly what was their role in the work to better contextualize this statement and understand whether there was any substantial amount of human contribution on this work.

  3. Figure 1 inflates the findings: Figure 1 is partially conflicting with Figure 4. Figure 1 could apply only to the largest models (which references the "sufficiently overparameterized" as to the abstract. The caption of Figure 1 should report this, otherwise it seems this double regime behavior is generic. Even for the 2nd largest models (orange shade in Fig. 4) the inverted U-shape seems not to be there, rather being an approximately straight line (or very slightly increasing) with a sharp decrease after the 0.3-1.0 nats point has been crossed. Figure 1, as it is presented, seems like an idealization of a trend which, in reality, is observed only in the biggest MLP/CNN and is much milder for the bigger models, and completely absent for the smallest.

  4. Conflation of epistemic and aleatoric uncertainty: while the authors claim to control aleatoric uncertainty, the approach of label noising to induce aleatoric uncertainty has an effect of increasing epistemic uncertainty if the dataset size remains constant. Label noising reduces the effective sample size and the model requires more data to learn the same classification rule. As per Hüllermeier & Waegeman (2021), Ch. 2.4, the epistemic uncertainty is conditioned by the (effective) sample size. The authors should vary the dataset size as a function of label noising magnitude to really isolate the effect of aleatoric and epistemic uncertainty from their experiments.

  5. Lack of clarity: some parts of the paper need a deeper review to simplify the writing, add important information, and increase clarity. Some examples:

    1. Caption for figure 1 should indicate that the pruned networks are trained until matching accuracy of the unpruned model. This is the "cover figure" of the paper, so information must be clear and not be inferred from the main text.
    2. The definitions of uncertainty overhead and target relaxation can be explained by using a clearer language and, possibly, improving the typesetting, e.g., by using a bullet point, and possibly accompanying the definitions with a figure or example.
    3. In the Uncertainty Estimation paragraph of Sec. 2, aleatoric and epistemic uncertainty are explained using natural language, but without referring to them as aleatoric and epistemic uncertainty. In the same paragraph, (approximately) Bayesian approaches are introduced but without saying that they can be used for improving uncertainty estimation. Additionally, the authors do not state that these are not the only approaches for uncertainty estimation (while it seems that uncertainty estimation can only be done via BNNs or ensembles). Additionally, even simple neural networks for classification can be used for uncertainty quantification by using the predictive entropy or the max of the output simplex, without recourse to BNNs or ensembles. Of course, ensembles provides a simple way for disentangling the uncertainties, but the authors should motivate their choices.
    4. Appendix H is not cited in the main text - this is important as it better contextualizes the limitations of the work. It should be cited in the conclusions.
    5. It is unclear which type of pruning is operated: the authors state they do global magnitude pruning with learning rate rewind, which should point to the reference being Renda et al. (2020), where this type of methodology is formalized. The Lottery Ticket Hypothesis papers cited in the present work operate weight rewind (which appears not to be the case here). The paper from Han et al. introducing iterative magnitude pruning with finetuning should be Han et al. (2015) - see references below - and not the Han et al. from 2016 cited in the paper. Han et al. (2015) do finetuning while keeping a low learning rate. In the present papers, the authors state they operate "global magnitude pruning with learning rate rewind"; it would seem the authors don't do any fine-tuning, but full retraining after pruning, if we go by the litearture; however, sifting through appendix B.6, this would appear to be false, as it is stated that only 60 epochs of fine-tuning are operated. The authors should clarify which methodology has been applied, be clearer in the text, and attribute the appropriate reference for this type of pruning method → See question #2.
    6. In Sec. 4, paragraph "An ensemble estimate...", you reference (1) without saying it's an equation, the correct crossreference should be "Eq. (1)". I suggest using the package cleveref, command \Cref for avoiding these small mistakes.
    7. The authors use different terminology throughout the paper: "uncertainty modifiers" (line 150) vs "uncertainty transformers" (line 140), "target relaxation" (line 76) vs "reference relaxation" (line 118).
    8. The color palette choice for some figures (e.g., Figures 3, 4, 12) is suboptimal since the yellow lines can be hard to see, especially on printed paper. Consider using different palettes where the lighter colors are a bit darker. The issue is even more prevalent when error bands are drawn as some are neigh-invisible due to the color choice.

My recommendation and evaluation on the soundness stem mainly from the limited importance and overstatement of the findings, the small insight into the uncertainty overhead phenomenon, and the missed considerations on epistemic uncertainty.

References:

Han, Song, et al. "Learning both weights and connections for efficient neural network." Advances in neural information processing systems 28 (2015).

Hüllermeier, Eyke, and Willem Waegeman. "Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods." Machine learning 110.3 (2021): 457-506.

Renda, Alex, Jonathan Frankle, and Michael Carbin. "Comparing rewinding and fine-tuning in neural network pruning." arXiv preprint arXiv:2003.02389 (2020).

Questions for authors
  1. Expand the AI Statement to clearly identify which tasks were carried out by humans and which by GenAI tools and how the human component intervened in the project (see concern #2).
  2. Which is the correct pruning schedule that was operated? Full retraining, fine-tuning with/without learning rate rewind? (See concern "Lack of clarity" → Point 5).
  3. In Sec. 4, paragraph "An ensemble estimate...", why is it necessary to use cross-validation-style folds for estimating aleatoric uncertainty? Can't it be estimated directly from the empirical version of Eq. (1), i.e., by taking the average entropy of the classifier?
  4. What do the terms "student" and "teacher" refer to in the appendices? Usually, these terms indicate the usage of knowledge distillation, not pruning. Consider using other terms instead to avoid confusion.
2 comments

The inverted U-shape curve can partly be explained by "target relaxation"

Thank you for raising this distinction. Target relaxation is not an alternative explanation for our findings; it is explicitly one of the two mechanisms at the center of our hypothesis. This is stated in the abstract and Introduction, investigated experimentally in Section 6, and used to explain the two regimes in Section 7.

We do not claim that target relaxation is surprising. Rather, the surprising finding is that pruned network size initially increases despite target relaxation, due to the competing effect of uncertainty overhead, which we isolate experimentally in Section 5.

Target relaxation is part of our explanation for the contraction regime. Our empirical claim concerns how uncertainty overhead and target relaxation jointly affect pruned network size under the specified training and pruning procedure.

The authors do not present an investigation on "uncertainty overhead"

We have conducted an experiment in which label noise is resampled every minibatch or pruning round (see Appendix I in the revised PDF). Resampling substantially reduces uncertainty overhead and, at some accuracy targets, even improves prunability relative to clean labels. These results motivate our magnitude crowding conjecture: repeated exposure to the same noisy examples causes weights that memorize them to grow large enough to survive magnitude pruning, displacing weights important for correctly classifying clean examples. We have added a discussion of this conjecture in Section 9 to the revised paper.

Supposedly-limited human contribution: the AI statement seems particularly dry and it does not allow to understand what is the amount of human contribution to this paper.

We have expanded our AI use statement.

Figure 1 inflates the findings: Figure 1 is partially conflicting with Figure 4. ...

Note that the y axis in Figure 4 is log-scale, so small visual differences correspond to large absolute differences. For example, the difference in pruned network size between the MNIST 1% and 0% label noise condition for the 2nd-largest model is ~2.7x, despite looking insignificant on Figure 4.

With that being said, we have updated the caption of Figure 1 to make it clear that it is an idealized depiction of the two regimes hypothesis and that it is specific to sufficiently overparameterized models.

“ Conflation of epistemic and aleatoric uncertainty: while the authors claim to control aleatoric uncertainty, the approach of label noising to induce aleatoric uncertainty has an effect of increasing epistemic uncertainty if the dataset size remains constant.

Thank you for suggesting this experiment. The discussion by Hüllermeier and Waegeman does not establish a connection between aleatoric uncertainty and the number of observations required to approximate the Bayes decision rule; they refer to the general phenomenon that increasing the number of examples a learner sees generally lowers approximation uncertainty. However, we take the reviewer’s point, as there is precedent in the literature for the notion of requiring more examples in the presence of noise [1, 2].

The mapping from label noise to dataset size that the authors of [2] derive is based on information theory, and does not readily apply to neural networks. Absent of an established function from aleatoric uncertainty to dataset size, we sweep over larger dataset sizes and observe whether our results hold for larger dataset sizes as well. Since we already train on the full datasets for MNIST and CIFAR-10, we perform this experiment on synthetic benchmarks where one can easily generate more data.

The results are in Appendix F. While the trends appear to vary for moderately overparameterized networks across dataset sizes, we still observe the two regimes of pruning for the largest and/or second-largest architecture size at each dataset size. For some moderately overparameterized architecture sizes, increasing the dataset size appears to mitigate the expansion incurred from uncertainty overhead. Taken together, these results suggest that for smaller models, more training and more data can mitigate the effects of uncertainty overhead, but for large enough models, the effects of aleatoric uncertainty are persistent.

However, a confounder in these experiments is that we do not control for the amount of compute we supply for each dataset size, since we hold the number of epochs fixed. In the final revision of our paper, we will include experiments where we match the amount of compute across dataset sizes.

[1] Angluin, D., & Laird, P. D. (1988). Learning from noisy examples. Machine Learning.

[2] Aslam, J. A., & Decatur, S. E. (1996). On the sample complexity of noise-tolerant learning. Information Processing Letters.

In the Uncertainty Estimation paragraph of Sec. 2, aleatoric and epistemic uncertainty are explained using natural language, but without referring to them as aleatoric and epistemic uncertainty. In the same paragraph, (approximately) Bayesian approaches are introduced but without saying that they can be used for improving uncertainty estimation.

Thank you for these suggestions. We have updated the related work to reflect them.

Appendix H is not cited in the main text - this is important as it better contextualizes the limitations of the work. It should be cited in the conclusions.

We have added a reference to the limitations appendix in the new discussion section.

It is unclear which type of pruning is operated: the authors state they do global magnitude pruning with learning rate rewind, which should point to the reference being Renda et al. (2020), where this type of methodology is formalized.

We perform global iterative magnitude pruning with learning rate rewinding, following Renda et al. (2020). We have added a citation to this work in the “Training and Pruning” paragraph of Section 4. Additionally, to prevent confusion, we have changed the word “finetuning” to “retraining,” where relevant, to make it clear that we do not use a low learning rate while retraining after pruning.

We have cited the Han et al. (2016) paper in the related work, because we wanted to emphasize that pruning can be used to lower memory footprints, which is the goal of the 2016 version.

In Sec. 4, paragraph "An ensemble estimate...", you reference (1) without saying it's an equation.

We have corrected this in the revised PDF.

The authors use different terminology throughout the paper: "uncertainty modifiers" (line 150) vs "uncertainty transformers" (line 140), "target relaxation" (line 76) vs "reference relaxation" (line 118).

Thank you for bringing this to our attention. “Uncertainty transformers” and “reference relaxation” is deprecated terminology from an earlier draft of our paper. We have removed any traces of the terms in the revised PDF.

The color palette choice for some figures (e.g., Figures 3, 4, 12) is suboptimal since the yellow lines can be hard to see, especially on printed paper. Consider using different palettes where the lighter colors are a bit darker. The issue is even more prevalent when error bands are drawn as some are neigh-invisible due to the color choice.

Thank you for pointing this out. We have shifted the color palettes to be darker and darkened the error bands in the revised PDF.

In Sec. 4, paragraph "An ensemble estimate...", why is it necessary to use cross-validation-style folds for estimating aleatoric uncertainty? Can't it be estimated directly from the empirical version of Eq. (1), i.e., by taking the average entropy of the classifier?

We use cross-fitting to obtain out-of-sample uncertainty estimates for every training example, since classifiers may underestimate aleatoric uncertainty on their own training data due to memorization. We have added this clarification to the “Estimating Aleatoric Uncertainty” paragraph of Section 4.

What do the terms "student" and "teacher" refer to in the appendices? Usually, these terms indicate the usage of knowledge distillation, not pruning. Consider using other terms instead to avoid confusion.

This is another case of outdated terminology from an earlier draft. We have removed any mentions of “student” or “teacher” in the revised PDF.

Reviewer_k5jdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

To be honest, I’m not sure if I understood what the paper is really about. I think it is very poorly presented. The paper tries to connect aleatoric uncertainty to the size of the network. The paper doesn’t really provide any convincing motivation for why this connection is interesting, but it seems as if they find some strong signal in their experiments. I think the paper tries to formulate a hypothesis about the relationship of aleatoric uncertainty and pruned network size, but I don’t find any precise formulation of their hypothesis. When the paper talks about “Aleatoric Uncertainty” in their hypothesis, I’m not sure if they refer to (a) true aleatoric uncertainty (b) estimated aleatoric uncertainty (estimated with your specific pipeline?) (c) you only restrict yourself to settings where estimated and aleatoric uncertainty agree, and you don’t have any hypothesis about which of these 2 quantities is the relevant one once they disagree. If this is an open question, openly communicate this. Do you have any evidence that they roughly agree in your setting?

When you talk about the “network size” in your hypothesis, I'm not sure if you refer to: (a) the network size resulting from your very specific training and pruning pipeline (b) do you have any evidence that this generalizes to other training and pruning pipelines? (c) in the abstract, you mention “which we use as a proxy for the minimum model size needed to approximate the optimal decision rule”. Do you have any reason to believe that this is a good proxy?

What do you even mean by “optimal decision rule” here? (a) Optimal on the true distribution (infinitely many data points)? (b) Optimal with respect to the posterior of a given finite training dataset? (c) Do the experiments in sections 5,6, and 7 correspond to different definitions of “optimal decision rule”? For section 5, one could mainly argue about proxies for the true optimal decision boundary. For sections 6 and 7, not, in my opinion.

Based on the abstract, it sounds as if there are two regimes where increasing the aleatoric uncertainty first increases the network's size, but then further increasing the aleatoric uncertainty decreases the network's size. However, this is only the case in Section 7. In Section 5, I don’t see the phenomenon, but I think only the results in Section 5 have a chance to approximate the minimum model size needed to approximate the true optimal decision rule. Section 7 obviously measures something different.

Overall, the paper feels more like a list of experimental results without a clear general hypothesis. It feels more like: “if you follow this very specific complicated pipeline, these plots show the results”, but I don’t really understand what the scientific questions were that inspired this very specific experimental setup.

Critical strengths

There seems to be some quite strong signal in the data that the authors seem to care a lot about. However, it is not clear to me why the authors care so much about this, or what the bigger picture is behind it. The signal seems to be strong and qualitatively similar among MNIST and CIFAR, so there might be a general pattern/phenomena behind it. There might be some interesting thoughts hidden between the lines.

Critical weaknesses

W1: The presentation of this paper is really poor.

W1a: The motivation, hypothesis, goal, conclusion, and potential impact of this paper are very unclear to me. What is the goal of this paper? Why did you write it? Do you see any concrete future path for how this could be beneficial for real-world application? Did you just observe this phenomenon and thought it might be interesting to others but didn’t really know why? It looks like you designed a very specific pipeline very deliberately (to isolate certain phenomena?)? On the one hand, it feels like you designed these experiments very deliberately, but on the other hand, they feel very arbitrary to me personally since I don’t know what goal you pursue via these experiments. Which observations, experiments, intuition, theory, or literature inspired you to design these experiments? If your experiments were inspired by some other literature, don’t assume that your readers are familiar with this line of work; instead, explain to the reader what their motivation was and how your work contributes to this line of work. Which of your results agree with the literature, and which results are in tension with the literature? Why are they in tension? What was the motivation of this other literature? What did they do differently? What are your hypotheses for explanations why your results are different? What still feels surprising and paradoxical to you, and what seems very expexctable/logial/explainable to you? If there is a paradox, are you trying to introduce a new paradox which still feels paradoxical, or are you resolving/explaining existing paradoxes, or are you introducing a paradox and directly resolve/explain it yourself? If there is a paradox, explain both why it is suprising/paradox/unexpected, and how it can be resolved/eplained (which intuition, theory, or empirically observed rules help to explain it). What are the central hypotheses of your paper? If you have a hypothesis, try to formulate it as precisely as possible. If you still have uncertaitny/ambiguity about certain parts of the hypothesis, formulate them clearly (e.g., how general is your hypothesis, what are the assumptions). What is the empirical evidence for your hypothesis? For which parts of the hypothesis do you still lack empirical evidence? Do you have theoretical/inutiviae explanations for your hypothesis? For which parts of the hypothesis? How does this hypothesis relate to other hypotheses/theories/beliefs/experimtents/discussions in the literature?

You write “This result runs counter to prior work by Xue et al. (2024), which shows that in the overparameterized regime and at high levels of label noise, increasing parameters can worsen test accuracy.”. Is this the core of your results? Should your paper be seen as a response to Xue et al. (2024)? I have never heard of Xue et al. (2024). Can you explain the main motivation, idea, and results of Xue et al. (2024), if your paper should be seen as a response to their paper?

W1b: Sometimes it is not clear to me what exactly you are plotting. You introduce a quite precise notation at the start, but then you don’t use it. Please tell me, for each plot, what you show on the x-axis and what you show on the y-axis in your mathematical notation. How is α0\alpha_0 defined? Please use the notation from the start to define it. If it’s the estimated aleatoric uncertainty it depends on the size of the models? Does each model size have a different α0\alpha_0? How much do they disagree? Or do you average over all models from different sizes? Line 198 is formulated unnecessarily complicated and contradicts line 202. Lines 198 and 202 define the range of η\eta differently. Why don’t you simply compute pη(y∣x)=(1−η)p(y∣x)+η/Kp_\eta(y|x)=(1-\eta)p(y|x) + \eta/K for every η∈[0,1]\eta\in[0,1] instead of Line 198. Then you can write “Concretely, uniform label noise on K ≥ 2 classes replaces each label with a uniformly sampled label with probability η” in Lines 195-196 (i.e., remove the word “different“). W1a is much more of a problem than W1b. In Line 204, you write “expected corrupted test accuracy as a function of model performance on the uncorrupted test set”. When do you actually use the expected corrupted test accuracy? For most of your plots, I had the impression that you are actually just using the uncorrupted test accuracy?

W2: specific to natural data distributions? This weakness is very much connected to weakness W1a of a missing formulation of the hypothesis. You emphasize the use of “natural data” in your paper. Do you think that your phonomena/hypothesis is specific to natural data and would be hard to reproduce with simple synthetic data? It would be nice to discuss which phenomena that you observe are very obvious and would reproduce with basically any dataset including simple synthetic data; and which phenomena/hypothesis are more mysterious/paradox/surprising that cannot easily be reproduced in simplistic synthetic settings. For example, Fig 3&5 feel very obvious/expected to me. I think this could be easily reproduced with simple synthetic settings such as multidimensional Gaussian linear logistic regression. In such a simplistic setting, one could maybe even compute this analytically. If you reduce the target accuracy, one can obviously prune to smaller model sizes. That’s not surprising - do you agree? Do you think the other results from the paper would also reproduce in such simple settings? Repeating these experiments for some synthetic datasets would help a lot to refine the assumptions of your hypothesis. This would also help to better understand the mechanisms behind it. For synthetic datasets, you also have access to quantities like true aleatoric uncertainty or the true optimal decision boundary.

W3: Missing stability analysis Lines 450-457 feel very hacky. You use a different rule to select the target accuracy for these experiments compared to the other experiments. I understand that the std is too small in this experiment. But for the other experiments with the uniform label noise, you should also try if your results would significantly change if you always take the same fixed std instead of recomputing it for every level of aleatoric uncertainty and every model size.

Are the results stable in the number of training observations? What role does weight-decay play? Increasing weight decay would probably decrease the size of the pruned network. Would your results look very different without weight-decay? How about other datasets (e.g. simpler synthetic datasets as mentioned in W3 or more modern realistic datasets)? How about other pruning/reduction/compression algorithms? How about other training pipelines? The abstract sounds quite general, but your experiments are very specific. Which experiments you should add also depends on what your scientific question or your hypothesis is. So the first step is to clearly define your scientific question(s) and/or hypotheses (see W1a).

Questions for authors

Q1: I see multiple phenomena that seem to be quite expected to me:

  1. Adding noise to the training data hurts the test accuracy
  2. If you start from a worse test accuracy, it is harder to achieve a certain target accuracy when you want to prune a lot
  3. if you lower the target accuracy, you can prune more Do you agree that these statements are extremely plausible? Do you mention these basic intuitions somewhere in the paper to guide the reader? Do any of your experiments contradict these statements? Which of your experiments can be explained by these rules? Which aspects of your results go beyond these explanations (i.e cannot simply be explained by these simple rules). Is the goal of your paper to show a very surprising phenomenon that cannot be explained by basic existing intuition? Or is the goal of your paper to give a simple explanation of a question which was previously puzzling?

I don’t really understand what you want to express in Lines 79-80: “For sufficiently overparameterized models, we identify two regimes of pruning induced by aleatoric uncertainty: an expansion regime and a contraction regime (Section 7).” Expansion and contraction of what? The true optimal decision boundary doesn’t change when you change the aleatoric uncertainty (agreed?). Thus, the smallest number of neurons to express this fixed true optimal decision boundary also doesn’t change when changing the aleatoric uncertainty. What is your hypothesis? Something like this: the “prunability” of the decision boundaries found by AdamW changes in a non-monotonic way depending on the aleatoric uncertainty: small aleatoric uncertainty => AdamW finds a decision boundary that can be approximated well by pruning; medium aleatoric uncertainty => AdamW finds a decision boundary that cannot be approximated well by pruning; high aleatoric uncertainty => AdamW finds a decision boundary that can be approximated well by pruning.

I don’t really understand lines 307 to 310: “This result runs counter to prior work by Xue et al. (2024), which shows that in the overparameterized regime and at high levels of label noise, increasing parameters can worsen test accuracy. These results suggest that pruned network size decreases more sharply for overparameterized models than it does for underparameterized models.” Which of these results are yours and which are the results of Xue et al. (2024). Can you explain more about what Xue et al. (2024) showed and what their motivation was?

Do you also apply Distribution Shaping to the test dataset in Section 8?

5 comments

I think the paper tries to formulate a hypothesis about the relationship of aleatoric uncertainty and pruned network size, but I don’t find any precise formulation of their hypothesis.

We agree that it would clarify our writing if we codified our observations as a hypothesis. In the revised draft, we have included a precise formulation of the two regimes hypothesis at the end of Section 3, and we have updated Section 1 and the abstract to introduce the two regimes hypothesis.

When the paper talks about “Aleatoric Uncertainty” in their hypothesis, I’m not sure if they refer to (a) true aleatoric uncertainty (b) estimated aleatoric uncertainty (estimated with your specific pipeline?) (c) you only restrict yourself to settings where estimated and aleatoric uncertainty agree, and you don’t have any hypothesis about which of these 2 quantities is the relevant one once they disagree. If this is an open question, openly communicate this. Do you have any evidence that they roughly agree in your setting?

Our hypothesis concerns true aleatoric uncertainty, which we estimate using deep ensembles where necessary. In Sections 5–7, we prescribe the label-noise rate directly, so the observed trends against label-noise rate do not depend on the ensemble uncertainty estimator. The numerical conditional-entropy coordinates for natural data still depend on an estimate of the original conditional label distribution. For the synthetic benchmarks, clean labels are deterministic, so conditional entropy after label corruption can be calculated exactly.

Prior work provides empirical evidence that aleatoric uncertainty estimation techniques achieve moderate correlations with human-label uncertainty, with deep ensembles specifically achieving ~0.34 and ~0.43 Spearman rank correlation with human-label uncertainty on CIFAR-10 and ImageNet, respectively [1]. Rank correlations are most relevant to our experiments because we seek to characterize how pruned network size changes as aleatoric uncertainty increases, rather than predict its exact dependence on aleatoric uncertainty. Thus, we require only that estimated uncertainty vary monotonically with true uncertainty.

Because deep ensembles correlate only moderately with human-label uncertainty, we avoid relying on fine-grained differences in estimated aleatoric uncertainty. Instead, our primary comparison in Section 8 uses the original-data uncertainty level and the lowest level attainable by our generative methodology, maximizing their separation to reduce sensitivity to estimation error.

[1] Mucsányi, B., Kirchhof, M., & Oh, S. J. (2024). Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks. Advances in Neural Information Processing Systems.

When you talk about the “network size” in your hypothesis, I'm not sure if you refer to: (a) the network size resulting from your very specific training and pruning pipeline (b) do you have any evidence that this generalizes to other training and pruning pipelines?

In our statement of the two regimes hypothesis, the pruned network size is the sparsest network achieved through iterative magnitude pruning that still matches dense network performance. Iterative magnitude pruning has been used to make advances in the science of deep learning (e.g., the lottery ticket hypothesis, linear mode connectivity, sparse double descent, and pruning-identified exemplars) [1, 2, 3, 4].

Recent developments in the pruning literature have consisted of one-shot methods, such as SparseGPT, Wanda, and ALPS [5, 6, 7]. We evaluate one such approach, Wanda, in Appendix H. We find that Wanda is largely invariant to aleatoric uncertainty at every architecture size, although the average sparsity is very low, ranging from an average of 11% to 38% across architecture sizes. Iterative magnitude pruning achieves multiple orders of magnitude smaller network sizes at low aleatoric uncertainties and only begins to approach the large pruned network sizes achieved by Wanda when aleatoric uncertainty is high. We will include comparisons to all three techniques in the final version of our work.

[1] Frankle, J., & Carbin, M. (2019). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. International Conference on Learning Representations.

[2] Frankle, J., Dziugaite, G. K., Roy, D. M., & Carbin, M. (2020). Linear Mode Connectivity and the Lottery Ticket Hypothesis. International Conference on Machine Learning.

[3] He, Z., Xie, Z., Zhu, Q., & Qin, Z. (2022). Sparse Double Descent: Where Network Pruning Aggravates Overfitting. International Conference on Machine Learning.

[4] Hooker, S., Courville, A., Clark, G., Dauphin, Y., & Frome, A. (2020). What Do Compressed Deep Neural Networks Forget? arXiv:1911.05248.

[5] Frantar, E., & Alistarh, D. (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. Proceedings of the 40th International Conference on Machine Learning.

[6] Sun, M., Liu, Z., Bair, A., & Kolter, J. Z. (2024). A Simple and Effective Pruning Approach for Large Language Models. International Conference on Learning Representations.

[7] Meng, X., Behdin, K., Wang, H., & Mazumder, R. (2024). ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models. Advances in Neural Information Processing Systems.

(c) in the abstract, you mention “which we use as a proxy for the minimum model size needed to approximate the optimal decision rule”. Do you have any reason to believe that this is a good proxy?

We have since dropped this as a claim in our revised introduction.

What do you even mean by “optimal decision rule” here? (a) Optimal on the true distribution (infinitely many data points)? (b) Optimal with respect to the posterior of a given finite training dataset? (c) Do the experiments in sections 5,6, and 7 correspond to different definitions of “optimal decision rule”? For section 5, one could mainly argue about proxies for the true optimal decision boundary. For sections 6 and 7, not, in my opinion.

Our definition of an optimal decision rule is a “Bayes decision rule,” defined in Section 3 to operate over the entire support of the data distribution. All sections in the paper refer to this definition of optimal decision rule.

You write “This result runs counter to prior work by Xue et al. (2024), which shows that in the overparameterized regime and at high levels of label noise, increasing parameters can worsen test accuracy.”. Is this the core of your results? Should your paper be seen as a response to Xue et al. (2024)? I have never heard of Xue et al. (2024). Can you explain the main motivation, idea, and results of Xue et al. (2024), if your paper should be seen as a response to their paper?

Our paper is not meant to be a response to Xue et al. We have since determined that a proper comparison of our results to the results of Xue et al. would require more space, and we do not believe the observation is sufficiently valuable. We have removed it from the revised PDF.

W1b: Sometimes it is not clear to me what exactly you are plotting. You introduce a quite precise notation at the start, but then you don’t use it. Please tell me, for each plot, what you show on the x-axis and what you show on the y-axis in your mathematical notation.

In the revised PDF, we have included math notation on all relevant axis labels, to more explicitly relate the plotted quantities to the quantities we formally define.

How is  defined? Please use the notation from the start to define it. If it’s the estimated aleatoric uncertainty it depends on the size of the models? Does each model size have a different ? How much do they disagree? Or do you average over all models from different sizes?

\alpha_0 refers to the aleatoric uncertainty of the original data distribution. We have added a definition in Section 3 on line 161.

W2: specific to natural data distributions? This weakness is very much connected to weakness W1a of a missing formulation of the hypothesis. You emphasize theuse of “natural data” in your paper. Do you think that your phonomena/hypothesis is specific to natural data and would be hard to reproduce with simple synthetic data? It would be nice to discuss which phenomena that you observe are very obvious and would reproduce with basically any dataset including simple synthetic data; and which phenomena/hypothesis are more mysterious/paradox/surprising that cannot easily be reproduced in simplistic synthetic settings. For example, Fig 3&5 feel very obvious/expected to me. I think this could be easily reproduced with simple synthetic settings such as multidimensional Gaussian linear logistic regression. In such a simplistic setting, one could maybe even compute this analytically. If you reduce the target accuracy, one can obviously prune to smaller model sizes. That’s not surprising - do you agree? Do you think the other results from the paper would also reproduce in such simple settings? Repeating these experiments for some synthetic datasets would help a lot to refine the assumptions of your hypothesis. This would also help to better understand the mechanisms behind it. For synthetic datasets, you also have access to quantities like true aleatoric uncertainty or the true optimal decision boundary.

We agree that evaluating our hypothesis on synthetic data makes sense as a setting where one can exactly control the aleatoric uncertainty and where one can calculate the optimal test accuracy. We have introduced three synthetic classification tasks in Appendix E—linear binary, quadratic binary, and prototype multiclass—and we have evaluated our hypothesis on these tasks using uniform label noise as an uncertainty modifier.

Linear binary is a binary classification task where a random hyperplane through the origin determines each input’s label. Quadratic binary is a binary classification task where the sign of a random quadratic function of the input determines each label. Prototype multiclass is a ten-way classification task where each input takes the class of the nearest of ten random prototype points. All three tasks draw 512-dimensional standard Gaussian inputs and assign clean labels algorithmically and deterministically. As a result, the unmodified data has zero aleatoric uncertainty, and we can compute exactly the aleatoric uncertainty that each level of uniform label noise induces.

Figure 19 in Appendix E shows the analogous result to Figure 4. We find that for all three synthetic benchmarks, the largest models exhibit the two regimes of pruning, except for the largest model on linear binary, which only appears to exhibit expansion.

Comments on the paper

1 comment

We would like to thank all of the reviewers for the time they put into reading our submission. The questions you all asked forced us to rethink the motivation for our work and sharpen our observations, leading to a significantly better paper. Again, thank you.

All three reviewers commented on the lack of motivation and/or the lack of explanation of the scientific significance of our results. We have since rewritten the introduction with a new framing of our results that contextualizes them in the conventional wisdom established by prior work, and we have added a discussion section (Section 9) where we discuss the implications of our findings and a potential explanation for uncertainty overhead.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.