acceptodds
Withdrawn from ICLR 2027

How Sparsity Allocation Shapes Label-Free Post-Pruning Recoverability

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Post-training pruning involves two coupled design choices: how sparsity is allocated across layers and how the resulting sparse model is repaired. We study whether these choices can be treated independently when recovery must be label-free, using only a small unlabeled calibration set. Across ResNet-18/34/50 on CIFAR-10, CIFAR-100, and Imagenette, we combine four unstructured sparsity-allocation strategies with the same suite of gradient-free repair methods. We find that allocation changes both the attainable repaired accuracy and the repair method that works best, with the strongest interaction appearing in a repair-sensitive transition regime between stable recovery and collapse. Across 27 main ResNet settings, the best repair method changes across allocations in 21 settings, and allocation changes the recoverability regime in 13. We further introduce LDMS-σ, a calibration-only diagnostic based on dense–pruned activation statistics, which selects the best repaired allocation in 25 of 27 settings and remains within one percentage point in 26. These results show that post-pruning recoverability is a property of the allocation–repair pair rather than either component alone.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
neural network pruning, sparsity allocation, post-pruning repair, sparse neural networks, model compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_g5UtOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares different allocation/correction pairings for single-shot post-training pruning with label-free data. Selection strategies for the allocation method are further studied.

Critical strengths

None

Critical weaknesses

Limited novelty : While a central claim of the paper is to shed new light on the allocation-repair pairing, this is hardly a new research direction. Lazerevich 2021 [1]--cited in the paper--already reports similar comparison across different sparsity selection criterion and correction methods. Moreover, their comparisons across allocation strategies already include LAMP and two magnitude variants. With respect to competitive allocation, the current paper only adds ERK (2020) to this existing benchmarking. Given that layerwise sparsity allocation is a very active topic, it is hard to justify not adding more recent proposed methods such as [2], [3] to list only two.

Flawed methodology : A fundamental practice in ML is to report results on an held-out test set after tuning on a train set. However, the results used to compare selection strategies in the main text are the ones LDMS-σ\sigma was tuned on, as evidenced by Table 4, Appendix C and Appendix I. Limited results on an held-out set is solely reported, without reference, in Appendix I and show severe deterioration of the criterion. On top of that, three different ϵ\epsilon variables (sharing the same notation) are introduced in the definition of LDMS-σ\sigma--without precising to which numerical values they were set nor explaining how they were selected.

Weak evaluations: Reported results are overall very limited and restricted to convolution architectures paired with small datasets; meaningful results on ImageNet-1k and transformer models are entirely missing. Moreover, base dense performance is often not reported and standard deviations are omitted from the main text--a verification in the Appendix shows those are often significant.

Other major issue: Lines 296-298 state that LDMS-σ\sigma selection is unchanged across a sweep. Reported numbers for that sweep in Table 4 of the Appendix appear to contradict directly this statement. Direct reference to this table is missing from the main text.


References:

[1]: Ivan Lazarevich, Alexander Kozlov, Nikita Malinin: Post-training deep neural network pruning via layer-wise calibration. ICCVW 2021

[2]: Kaixin Xu, Zhe Wang, Xue Geng, Min Wu, Xiaoli Li, Weisi Lin: Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networks. ICCV 2023

[3]: Ruihao Gong, Yang Yong, Zining Wang, Jinyang Guo, Xiuying Wei, Yuqing Ma, Xianglong Liu: Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes. AAAI 2024

Questions for authors

Additional suggestions to the authors:

  • The reported results on ImageNet-1K in Table 19 are vacuous, with no method achieving relevant performance and by being limited to a single seed. Lazarevich (2021) reports significant results in the same setting with sparsity set at 50%, which could serve as a good starting point.

  • The authors should further justify the need for a pre-repair selection strategy such as LDMS-σ\sigma which requires an already trained and pruned network to be evaluated. The remaining steps, repair and evaluation, are already very cheap compared to training.

  • Emphasis should be put on presenting message-carrying figures rather than good looking figures--Figures 1 and 2 both carry very little information. In particular for Figure 1, the Unlabeled Calibration Set pictogram of (a) appears misplaced and missing axis definitions/graduations make (d) hard to interpret.

Reviewer_6QBrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how sparsity allocation affects label-free post-pruning recoverability. The authors evaluate four allocation strategies with six repair methods across ResNets on CIFAR-10, CIFAR-100, and Imagenet. The results show that allocation affects both the best repair method and the attainable accuracy. The paper also proposes LDMS-δ\delta, a diagnostic based on activation statistics, which selects the best repaired allocation in 25 of 27 main settings.

Critical strengths
  1. The paper studies a practical problem in post-training pruning. The results show that models with the same global sparsity can have very different accuracy after repair, depending on how sparsity is allocated across layers.
  2. The evaluation covers several architectures, datasets, and sparsity levels. Using the same set of repair methods for different allocations provides a useful comparison of how allocation affects recoverability.
  3. LDMS-δ\delta is a simple method that uses only unlabeled calibration data. It achieves good results on the main ResNet experiments without requiring additional training or parameter tuning.
Critical weaknesses
  1. Although the best repair changes across allocations in 21 of 27 settings, this number drops to 12 when methods within one percentage point are considered equivalent. This makes the practical importance of some changes less convincing.
  2. LDMS-δ\delta does not perform as consistently on other architectures. It selects the best allocation in only 11 of 24 VGG16-BN settings and 7 of 11 DenseNet-121 settings. Although the accuracy loss is often small, the results raise questions about how well the method generalizes.
  3. The main result assumes that the best repair method is known for each allocation. However, when LDMS-δ\delta is combined with a fixed ASR-q50 repair method, the complete pipeline matches the best allocation–repair combination in only 12 of 27 settings. This is a significant difference for practical deployment.
  4. The three recoverability regimes are defined using manually chosen thresholds. While the authors provide some sensitivity analysis, it is unclear whether these regimes represent distinct properties of pruned networks or mainly serve to organize the experimental results.
Questions for authors

Please see weakness.

Reviewer_jMusOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper study label-free post-training pruning of CNNs using a small unlabeled calibration set. It crosses four sparsity allocations (ERK, LAMP, Uniform, Global) with six repair methods: no repair, BN recalibration (Shomron & Weiser, 2020), layerwise scaling, channelwise variance matching, and ASR-q50 and Clipped ASR (Zhan et al., arXiv 2026). It evaluates ResNet-18/34/50 on CIFAR-10, CIFAR-100, and Imagenette at 90-95% sparsity (27 settings). The authors report that the best repair method changes with allocation in 21/27 settings. They also propose LDMS-σ, a calibration-only diagnostic that selects the best allocation in 25/27 settings. They conclude that recoverability is a property of the allocation-repair pair.

Critical strengths
  1. The experimental grid is systematic, uses 3 seeds, and includes sensitivity analyses (thresholds, near-ties, calibration size). Failure cases are disclosed.
  2. The observation that allocation strongly affects repaired accuracy at fixed sparsity (eg. 66.7% vs 22.9% in Table 1) is useful and a cheap calibration-only way to rank allocations is a practical idea.
Critical weaknesses
  1. The claim is broader than the evidence.

    • The paper tested "Repair" is one family. Channelwise, ASR-q50, and Clip are the same idea with and without shrinkage and clipping, and they come from a single arXiv preprint. The 21/27 count falls to 12/27 at a 1-pp tolerance (Table 6).
    • There is no comparison with established calibration-based pruning and reconstruction methods such as OBC (Frantar & Alistarh, NeurIPS 2022) or AdaPrune (Hubara et al., NeurIPS 2021), and related work omits this line.
    • The appendix tests do not support the pair claim. On ImageNet-100, Clip is best for both allocations. On ImageNet-1K, the best accuracy is only 7.38% vs dense 73.69% - complete collaps, and ASR is below BN for all allocations. In DenseNet, BN or layerwise is often best.
    • Scale is mostly limited to small datasets. The author can include LLM-scale evidence to support their pair cliam, even though the cited post-training pruning work (SparseGPT, Wanda) targets LLMs.
  2. The diagnostic evidence is weak.

    • Channel CV and Always-LAMP have identical rows in Table 3, without any explination.
    • LDMS-σ was chosen among variants on the same 27 settings, with no clear gain over LDMS-var. Transfer drops on held-out architectures (VGG 11/24, DenseNet 7/11, max regret 6.99 pp).
Questions for authors
  1. Does the interaction hold for OBC or AdaPrune-style reconstruction on a subset of settings?
  2. How often does the best repair family change when ASR-q50, Clip, and Channelwise are counted as one?
  3. Table 19 shows ImageNet-1K collapsing (best 7.38% vs 73.69% dense) under all six repairs, and ASR-q50 falls below BN. Since the collapse regime is defined relative to the evaluated repair suite, this may reflect the limits of statistic-based repair and not of the sparse model. The authos may check/report whether a calibration-based reconstruction method (eg. OBC) retains substantially higher accuracy under the same allocations, and report LDMS-σ's selection on this setting.
  4. Were LDMS-σ's form and weighting fixed before seeing the 27 settings? Why are the Channel CV and Always-LAMP rows identical?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.