- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies which parts of a network structured pruning removes first, and why pruning can improve generalization. The authors find that the BN scale and weight norms of the deepest block shrink much more than those of other blocks, so standard pruning scores remove this block first. They call it a noise sink that holds the network's overfitting capacity, and show that the same pattern moves to the next block once the deepest one is removed.
Critical strengths
The paper is clearly written and easy to follow. The observation is consistent across seeds, architectures and datasets, and the sequential block removal experiment is a nice way to probe it.
Critical weaknesses
- The novelty is limited. That deep layers learn task-specific features, that memorization tends to happen in later layers, and that pruning removes deep layers first are all known results. The paper mainly connects these ideas, and the added insight is modest.
- The noise sink explanation is not well supported. Shrinking weights suggest that a block is barely used, not that it stores memorized noise. The paper's own controls also point the other way: the effect grows with more data and does not change with label noise, which is the opposite of what a memorization story would predict. A simpler explanation fits all the observations: the deepest block is redundant, so weight decay shrinks it. The paper also never directly tests memorization.
- The main findings are weaker than presented. The 53.33% threshold is simply the share of channels in the deepest block, so it follows from arithmetic. The “sharp generalization transition” at this threshold is mostly a rise in training loss, while test loss stays flat. Test accuracy is rarely reported, and the low accuracy on full CIFAR-10 suggests the models are not well tuned.
Questions for authors
- Does removing the deepest block mainly hurt the model’s fit on mislabeled examples?
- Does the data-size trend remain if the total number of training steps is held fixed?
- Do the findings hold with test accuracy as the metric and with a standard training recipe?
2 comments
response
I thank the authors for the detailed response and the additional experiments, especially E4 and the SGD comparison. However, these results mainly support the simpler explanation I raised, and the noise sink framing has been withdrawn. What remains is that Adam with coupled L2 shrinks layers with small gradients, which is a known effect and largely disappears under SGD.
Since the core message has changed a lot and is still not well supported, I will maintain my original rating.
Response to Reviewer EBqK (R1)
We thank Reviewer EBqK for the careful reading and direct experimental questions. The revision addresses all three weaknesses and answers all three questions.
W1 — Limited novelty
We agree that deep layers are task-specific, memorization concentrates in later layers, and magnitude pruning removes deep channels first are individually known. The contribution is not any one of these observations in isolation — it is the mechanistic account of why this happens and the predictive value that emerges.
Three things were not known. First, why Block 4 specifically collapses while shallower blocks do not, even though all layers use the same optimizer: E6 (§4.1) shows that Block 4 develops the smallest gradient norms during training, causing Adam to amplify its updates far more aggressively — by over 2800× relative to Block 1 — allowing weight decay to dominate. The SGD result — where the collapse essentially disappears — is a direct confirming prediction of this account. Second, prior work does not separate optimizer-driven (L2) collapse from structure-dependent (γ) collapse; E3 (§4) isolates them experimentally for the first time using a random-label control. Third, the transition threshold can be computed from channel counts alone and predicted before any experiment, then confirmed on a new architecture (E7, §4.2).
W2 — "Noise sink" framing
The reviewer is correct on both counts. The "noise sink / memorization" framing was directly contradicted by two of our own results: the collapse does not change with label noise, and it grows with more data rather than less. We have retired this framing entirely — the phrase does not appear in the revised manuscript.
The revised frame is depth-dependent dual collapse as a topological gradient-dynamics property. Under this account, both results become coherent rather than contradictory. Block 4's statistics are determined by optimizer dynamics, not by the learning signal — so changing the label noise changes what the network memorizes but not how Adam treats Block 4, and the statistics are unaffected as observed. The data-volume trend is resolved by E4 (§4): when total gradient steps are held fixed across data fractions, Sγ is flat across the full range. The original trend was entirely a training-duration effect — more data meant more gradient steps, which means more Adam amplification. The reviewer's observation is now supporting evidence for the revised account.
W3 — Threshold, training loss, test accuracy
On the threshold: the reviewer is correct that 53.33% is a simple fraction, and that is the source of its value. Because it comes from channel counts alone, it can be predicted before any experiment. E7 (§4.2) tests this prospectively on ResNet-56, which has a different architecture and a different predicted threshold (57.14%), and confirms that the γ criterion targets the predicted stage first. The arithmetic observation is exactly the mechanism that makes the prediction generalisable.
On the transition being mainly a training-loss event: the reviewer is correct. The revised manuscript now states this explicitly — the transition is primarily a training-loss spike — and we have removed any implication that test loss shows a sharp cliff.
On test accuracy: test accuracy has been added to all main tables. The non-monotonic trajectory is confirmed by test accuracy in the overparameterized regime.
Q1 — Does removing the deepest block mainly hurt fit on mislabeled examples?
No. The label-noise experiment (Appendix) varied noise from 0% to 100% of training labels. Block 4's weight statistics are invariant to noise level across the full range — if Block 4 were the locus of mislabeled-example fit, they would degrade as noise increases. The optimizer-dynamics account explains why: Block 4's collapse is determined by gradient depth and Adam's amplification, not by the content of the labels.
Q2 — Does the data-size trend remain if the total number of training steps is held fixed?
No. E4 (§4) addresses this directly. When total gradient steps are equalized across data fractions from 5% to 100%, Sγ is flat across the entire range. The trend entirely disappears once the training-duration confound is removed.
Q3 — Do the findings hold with test accuracy and a standard training recipe?
Yes on both counts. Test accuracy is now reported in all main tables and confirms the non-monotonic trajectory in the overparameterized regime. The full-CIFAR-10 experiment (§5.2) uses the standard 45k/5k train/validation split and achieves 74.3% test accuracy. Test accuracy remains stable across pruning ratios while AN improves monotonically, confirming the pattern holds under the standard recipe.
We hope to have addressed all your concerns and would greatly appreciate your re-evaluation of the score in light of the new experiments and revised presentation.