acceptodds
Under review as a conference paper at ICLR 2027

Caffeinate: Recovering Dormant Subnetworks in Modern Convolutional Neural Networks

Abstract

One-shot pruning is understood to degrade accuracy exclusively due to the removal of critical weights from a model. Evidence from batch-normalized CNNs supports a different account. The pruned model retains a high-performing sub-network that signal collapse, which is a progressive, depth-wise decay of activation variance, renders dormant, and recalibrating the BatchNorm statistics recovers it. Modern convolutional architectures, such as ConvNeXt, FocalNet, HorNet, and RDNet, adopt LayerNorm instead of BatchNorm, which computes statistics per sample at inference time. LayerNorm contains no stored statistics to recalibrate. We demonstrate that signal collapse exists in these networks and that damage due to pruning lies in the per-channel output moments, the subspace the LayerNorm affine parameters control. We develop Caffeinate, a closed-form correction of LayerNorm's affine parameters, which is applied once to a pruned network to restore the original per-channel output moments from only forward passes, without gradients or labels. Since Caffeinate rewrites only the affine parameters, it can be applied to any one-shot pruning mask. We validate this on magnitude pruning (MP) and CHITA, a state-of-the-art impact-based pruning method which uses second order information to select which parameters to prune and to update the unpruned parameters. Caffeinate recovers up to accuracy over MP across twelve ImageNet networks. MP corrected by Caffeinate outperforms CHITA on ten of twelve models at and all twelve models at , by up to accuracy, with a memory footprint of – GB against CHITA's – GB. When applied to CHITA, Caffeinate improves the accuracy of the pruned model by up to . This demonstrates that dormant high-performing sub-networks exist after one-shot pruning, regardless of the pruning method, and that correcting per-channel output moments can recover accuracy.

Primary area:
optimization
Keywords:
LayerNorm, one shot pruning, Modern CNNs, dormant subnetworks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Y7BvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops CAFFEINATE, a closed-form correction of LayerNorm’s affine parameters, which is applied once to a pruned net work to restore the original per-channel output moments from only forward passes, without gradients or labels. It seems that this is a derivative work of REFLOW[1] in LN CNN.

[1] https://arxiv.org/pdf/2502.15790

Critical strengths
  • This paper is well-written.
  • The idea makes sense.
  • The performance is very good, as shown in Table 1.
  • I do not see any other similar work in the field of CNN pruning, nor does my agent. I would like to vote for acceptance because of the simplicity and effectiveness.
Critical weaknesses
  • There are similar ideas in the field of quantization[1,2]. Scaling the activations is part of these works. The authors could add some discussion of these works.

[1] SmoothQuant: Xiao et al., "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models", ICML 2023.

[2] AWQ: Lin et al., "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration", MLSys 2024.

Questions for authors

Please check the Critical Weaknesses.

Reviewer_CLasOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a signal collapse beyond batch normalization based CNNs and demonstrates that one-shot pruning of layer normalization based CNNs causes the signal variance ratio at most layers to fall below one, with the ratio decreasing with sparsity and layer depth. This implies that dormant high-performing sub-networks exist after one-shot pruning, regardless of the pruning method, and that correcting per-channel output moments can recover accuracy. The authors introduce a closed-form, forward-pass only, label-free correction of LN affine parameters which is applied once to a pruned network to restore the original per-channel output moments from only forward passes, without gradients or labels.

Critical strengths

Since the correction focuses and rewrites only the LN affines of a pruned model, it is easy to adapt to whatever mask a LN network pruner produces. This can be seen as there are improvements on both MP and CHITA models in Table 1.

The correction is lightweight with a footprint of 2.5-2.8 GB against CHITA’s 25-64GB. This is because it is a closed-form correction and applies only once instead of an iterative loop.

Critical weaknesses

My primary concern is the insufficient motivation for applying the proposed pruning framework to CNNs. Given the existence of one-shot, post-training pruning methods, such as SNOWS, that reduce vision model inference costs without requiring retraining, the authors need to establish a compelling justification for why a separate approach for CNNs is necessary. In particular, the paper does not sufficiently clarify what fundamental limitations of existing pruning methods make them inadequate for CNNs, what unique challenges the proposed method addresses, or what substantial advantages it offers over established alternatives. Without a clear problem statement and convincing evidence of its necessity, the contribution risks appearing incremental rather than addressing a significant gap in the literature. The authors should explicitly identify the limitations of existing approaches, explain why these limitations matter specifically for CNNs, and provide rigorous experimental comparisons demonstrating that the proposed method offers meaningful improvements over existing post-training pruning techniques in terms of inference efficiency, computational cost, and model accuracy.

The metric does not evaluate the accuracy claim it makes. It states that there would be an accuracy drop at 30-60% sparsity but never investigates that claim.

Questions for authors

Please see "Critical Weaknesses" Section.

Reviewer_hbHHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates accuracy recovery after one-shot pruning in modern LayerNorm-based CNNs. The authors show that signal collapse, previously studied by REFLOW in BatchNorm-based networks, also occurs in this setting. They propose CAFFEINATE, which uses a small unlabeled calibration set to sequentially correct LayerNorm’s affine parameters through a per-channel closed-form solution, matching the original network’s output means and standard deviations on the calibration set without gradients or retraining. Experiments across twelve ImageNet models show that CAFFEINATE substantially recovers accuracy after magnitude pruning(90–99% of the dense models top-1 accuracy at 30% sparsity), outperforms CHITA at higher sparsities with a much smaller memory footprint (2.5–2.8 GB versus 25–64 GB, and further improves accuracy when applied on top of CHITA.

Critical strengths

A simple and effective method with practical value. CAFFEINATE uses a small unlabeled calibration set to correct LayerNorm’s affine parameters in closed form, without gradients, retraining, or modifications to convolutional and linear weights. This lightweight correction substantially recovers accuracy after pruning and further improves CHITA, demonstrating its effectiveness and compatibility with existing pruning methods.

Broad empirical evaluation. The experiments cover twelve ImageNet models spanning four architecture families, multiple sparsity levels, and both magnitude pruning and CHITA, demonstrating the method’s effectiveness across the evaluated architectures and pruning settings.

Critical weaknesses

Incremental novelty relative to REFLOW. The core ideas of signal collapse, dormant subnetworks, and activation statistics recovery through unlabeled forward-pass calibration are inherited from REFLOW. The main contribution is extending this approach to LayerNorm-based CNNs, which lack running statistics, by correcting the affine parameters. Both the crosswise-affine identity and the resulting closed-form solution are relatively straightforward, limiting the conceptual novelty.

Limited correction capacity and incomplete recovery at higher sparsities. Per-channel affine correction can restore means and standard deviations, but does not guarantee recovery of feature content or sample-level representations. At higher sparsities, corrected models remain substantially below dense-model accuracy. The advantage of CHITA+CAFFEINATE over MP+CAFFEINATE also indicates that moment correction alone cannot fully recover the accuracy lost due to pruning. Thus, recovery remains incomplete at higher sparsities (CAFFEINATE improves magnitude-pruned ConvNeXt-T from 0.27% to 34.59% top-1 accuracy, still far below the dense model’s 82.13%)

Questions for authors

CHITA+CAFFEINATE outperforms MP+CAFFEINATE at high sparsities, suggesting that moment correction alone does not address all pruning-induced degradation. Is there a sparsity regime beyond which moment correction offers diminishing gains or fails to maintain practically useful accuracy across different models?

How much accuracy does layer-by-layer sequential correction gain over simultaneous correction using statistics from the uncorrected pruned network? Could layers be corrected in groups, by stage or module, to reduce the number of passes over the calibration set? For networks with branches and residual connections, how is the correction order determined, and which dependencies affect the results?

How robust is CAFFEINATE to limited calibration data and distribution shift? Can the authors evaluate sensitivity to genuine domain differences between calibration and evaluation images? Does accuracy recovery remain reliable when calibration data are scarce or drawn from a different distribution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.