The Two Regimes of Pruning Under Aleatoric Uncertainty

Nearly every machine learning problem grounded in the natural world exhibits irreducible uncertainty in its data-generating process, a property known as _aleatoric uncertainty_. Aleatoric uncertainty can leave the optimal decision rule unchanged, which raises a natural question: should the size of a neural network approximating this decision rule also remain unchanged? We study this question through _pruning_—the process of removing parameters from a model to obtain a smaller model—which we use as a proxy for the minimum model size needed to approximate the optimal decision rule. We find that pruned network size is not invariant to aleatoric uncertainty. Instead, sufficiently overparameterized neural networks exhibit two regimes of behavior: increasing aleatoric uncertainty initially increases pruned network size (the _expansion regime_), but beyond a critical point, further uncertainty decreases it (the _contraction regime_). Through ablations, we find that these two regimes arise from two competing effects: increasing aleatoric uncertainty increases the pruned network size required for a fixed performance target, while simultaneously lowering the performance target set by the dense model.