acceptodds
Under review as a conference paper at ICLR 2027

WHERE DOES OVERFITTING LIVE? DEPTHD-EPENDENT STRUCTURE IN PRUNED NEURAL NETWORKS

Abstract

Although structured pruning is known to improve generalization in overparameterized networks, which parameters carry the overfitting burden—and why pruning consistently finds them—remains unclear. We show that, during training, the deepest active block undergoes a simultaneous collapse in both BN scale factors and channel L2-norms that is absent in shallower blocks: overfittingrelated capacity concentrates toward the network’s deepest stage, which we term the noise sink. On ResNet-18, all three criteria exclusively target the deepest stage until the architecturally determined 53.33% threshold; on ResNet-34, exclusivity is criterion-dependent; on VGG-16, the two deepest stages are cotargeted from the first cuts. Crossing the stage-exhaustion boundary triggers a sharp, criterion-dependent generalization transition. Sequential block removal further reveals that the noise sink role is not fixed to any specific block: after the deepest block is removed, the next deepest undergoes the same dual collapse and becomes the new pruning target, confirmed on ResNet-18, ResNet-34, and VGG-16. Controlled experiments show that Block 4 suppression strengthens monotonically with training data volume, the L2 component persists under SGD+momentum, and the pattern is invariant to label noise—indicating a general property of over-parameterized network convergence rather than a small-data or optimizer artifact. Code is available at https://anonymous.4open. science/r/depth-structure-pruning-CB93/.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Structured pruning, Channel pruning, Overparameterization, Generalization, Batch normalization, Lottery ticket hypothesis, Network slimming, Depth / residual networks.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EBqKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies which parts of a network structured pruning removes first, and why pruning can improve generalization. The authors find that the BN scale and weight norms of the deepest block shrink much more than those of other blocks, so standard pruning scores remove this block first. They call it a noise sink that holds the network's overfitting capacity, and show that the same pattern moves to the next block once the deepest one is removed.

Critical strengths

The paper is clearly written and easy to follow. The observation is consistent across seeds, architectures and datasets, and the sequential block removal experiment is a nice way to probe it.

Critical weaknesses
  • The novelty is limited. That deep layers learn task-specific features, that memorization tends to happen in later layers, and that pruning removes deep layers first are all known results. The paper mainly connects these ideas, and the added insight is modest.
  • The noise sink explanation is not well supported. Shrinking weights suggest that a block is barely used, not that it stores memorized noise. The paper's own controls also point the other way: the effect grows with more data and does not change with label noise, which is the opposite of what a memorization story would predict. A simpler explanation fits all the observations: the deepest block is redundant, so weight decay shrinks it. The paper also never directly tests memorization.
  • The main findings are weaker than presented. The 53.33% threshold is simply the share of channels in the deepest block, so it follows from arithmetic. The “sharp generalization transition” at this threshold is mostly a rise in training loss, while test loss stays flat. Test accuracy is rarely reported, and the low accuracy on full CIFAR-10 suggests the models are not well tuned.
Questions for authors
  • Does removing the deepest block mainly hurt the model’s fit on mislabeled examples?
  • Does the data-size trend remain if the total number of training steps is held fixed?
  • Do the findings hold with test accuracy as the metric and with a standard training recipe?
2 comments
AuthorsOpenReview ↗

Response to Reviewer EBqK (R1)

We thank Reviewer EBqK for the careful reading and direct experimental questions. The revision addresses all three weaknesses and answers all three questions.


W1 — Limited novelty

We agree that deep layers are task-specific, memorization concentrates in later layers, and magnitude pruning removes deep channels first are individually known. The contribution is not any one of these observations in isolation — it is the mechanistic account of why this happens and the predictive value that emerges.

Three things were not known. First, why Block 4 specifically collapses while shallower blocks do not, even though all layers use the same optimizer: E6 (§4.1) shows that Block 4 develops the smallest gradient norms during training, causing Adam to amplify its updates far more aggressively — by over 2800× relative to Block 1 — allowing weight decay to dominate. The SGD result — where the collapse essentially disappears — is a direct confirming prediction of this account. Second, prior work does not separate optimizer-driven (L2L_2) collapse from structure-dependent (γ\gamma) collapse; E3 (§4) isolates them experimentally for the first time using a random-label control. Third, the transition threshold can be computed from channel counts alone and predicted before any experiment, then confirmed on a new architecture (E7, §4.2).


W2 — "Noise sink" framing

The reviewer is correct on both counts. The "noise sink / memorization" framing was directly contradicted by two of our own results: the collapse does not change with label noise, and it grows with more data rather than less. We have retired this framing entirely — the phrase does not appear in the revised manuscript.

The revised frame is depth-dependent dual collapse as a topological gradient-dynamics property. Under this account, both results become coherent rather than contradictory. Block 4's statistics are determined by optimizer dynamics, not by the learning signal — so changing the label noise changes what the network memorizes but not how Adam treats Block 4, and the statistics are unaffected as observed. The data-volume trend is resolved by E4 (§4): when total gradient steps are held fixed across data fractions, SγS_\gamma is flat across the full range. The original trend was entirely a training-duration effect — more data meant more gradient steps, which means more Adam amplification. The reviewer's observation is now supporting evidence for the revised account.


W3 — Threshold, training loss, test accuracy

On the threshold: the reviewer is correct that 53.33% is a simple fraction, and that is the source of its value. Because it comes from channel counts alone, it can be predicted before any experiment. E7 (§4.2) tests this prospectively on ResNet-56, which has a different architecture and a different predicted threshold (57.14%), and confirms that the γ\gamma criterion targets the predicted stage first. The arithmetic observation is exactly the mechanism that makes the prediction generalisable.

On the transition being mainly a training-loss event: the reviewer is correct. The revised manuscript now states this explicitly — the transition is primarily a training-loss spike — and we have removed any implication that test loss shows a sharp cliff.

On test accuracy: test accuracy has been added to all main tables. The non-monotonic trajectory is confirmed by test accuracy in the overparameterized regime.


Q1 — Does removing the deepest block mainly hurt fit on mislabeled examples?

No. The label-noise experiment (Appendix) varied noise from 0% to 100% of training labels. Block 4's weight statistics are invariant to noise level across the full range — if Block 4 were the locus of mislabeled-example fit, they would degrade as noise increases. The optimizer-dynamics account explains why: Block 4's collapse is determined by gradient depth and Adam's amplification, not by the content of the labels.


Q2 — Does the data-size trend remain if the total number of training steps is held fixed?

No. E4 (§4) addresses this directly. When total gradient steps are equalized across data fractions from 5% to 100%, SγS_\gamma is flat across the entire range. The trend entirely disappears once the training-duration confound is removed.


Q3 — Do the findings hold with test accuracy and a standard training recipe?

Yes on both counts. Test accuracy is now reported in all main tables and confirms the non-monotonic trajectory in the overparameterized regime. The full-CIFAR-10 experiment (§5.2) uses the standard 45k/5k train/validation split and achieves 74.3% test accuracy. Test accuracy remains stable across pruning ratios while AN improves monotonically, confirming the pattern holds under the standard recipe.


We hope to have addressed all your concerns and would greatly appreciate your re-evaluation of the score in light of the new experiments and revised presentation.

Reviewer_EBqKOpenReview ↗

response

I thank the authors for the detailed response and the additional experiments, especially E4 and the SGD comparison. However, these results mainly support the simpler explanation I raised, and the noise sink framing has been withdrawn. What remains is that Adam with coupled L2 shrinks layers with small gradients, which is a known effect and largely disappears under SGD.

Since the core message has changed a lot and is still not well supported, I will maintain my original rating.

Reviewer_d3yxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the interesting topic of structural sensitivity in the over-fitted regime, specifically the idea that overfitting is concentrated towards deeper structures as demonstrated by the effect of prunning later blocks in resnet (and layers in VGG) on Cifar10.

The paper analyzes the training, validation and test loss and the ("AN") difference between validation and training loss as function of pruning amount and order.

*** update upon rebuttal *** The original paper read as AI slop. The rebuttal similarly seems to be heavily AI driven without remedy of the core issues and contradictory prose and poorly grounded experimental design and readability. The authors fail to show the deltas in the paper in a track changes and audit of what survives and what does not. Upon checking some of the updated experiments --- contradictions arise.

The paper is thus flagged for ethics review.

It is noted that even if not problematic from a slop/ethics perspective, it fails novelty, rigor and generalization and thus remains clear reject.

Critical strengths

The topic of overfitting and its relationship to pruning is of considerable interest, as is the distribution of excess versus signal in the overparametrized regime. The idea that one can have fine grained understanding of the sensitivity of the generalization gap (or AN) on pruning location is interesting.

Critical weaknesses

The paper suffers numerous weaknesses.

(a) It reads as if it may have been written heavily with AI and poorly audited by the authors --- canonic examples include:

  • The introduction of results, terminology and jargon (as early as the abstract) that the reader can not understand without having prior context of the body of the work: "the architecturally determined 53.33% threshold", "BN", " stages are cotargeted from the first cuts", "the stage-exhaustion boundary " etc have no way of being interpreted when dropped as if already known. This reads as a manuscript building on the author's previous discussions with the AI, but not conveyed as a self contained linearly introduced artifact.
  • Strong statements and superfluous terminology not supported by correspondingly strong evidence --- most notably the main claim of "deepest block concentration" does not exhibit connection to overfitting. At best the deepest block is readily pruned with correspondingly small norms for magnitude pruning, not a new result. Similarly the graphs --- beyond being exceedingly small with similar colors for different losses such that they are quite hard to read --- do not support sharp transition claims.

(b) The experiments do not rigorously establish the privilege of later layers and distinguish the observed phenomena from other to-be-controlled phenomena such as signal decay or vanilla overparamatarization. For example --- previous work examining per depth dependence of optimal pruning is not cited or thoroughly considered. Moreover, the sequential pruning experiment do not show baselines and counterfactuals needed for the substantiation of the claims --- such as (i) from scratch trained corresponding networks of same size (ii) different order of block removal (ii) depth privilege relative to similar width downscaling (iv) importantly, the counterfactual showing no depth privilege in pure memorization case (random data).

(d) More generally, the methodology poorly supports the claims: for example, while removing excess capacity does increase generalization under these settings, it is not clear that this is not a sub-optimality of the chosen training configurations' comparison. One can note in particular that in figure 3 while early stopping would result in different minimal test loss, the last run has not converged yet. It is also not expected for constant hyperparams to be stable across different effective sized networks.

(e) Even if the experiments were well constructed as to support the claims --- they are of limited data scale and diversity for the strength of the claim: Transformers and Imagenet in the image domain are required at minimum, and transformers and language modeling are required to convince of claim generalization.

Questions for authors

See above weaknesses that require addressing both structural and material as well as with respect to clear and minimally taxing readability.

1 comment
AuthorsOpenReview ↗

Response to Reviewer d3yx (R1)

We thank Reviewer d3yx for the frank and detailed critique. The revision directly addresses all five concerns.


W1(a) — Writing quality and self-contained presentation

The reviewer's critique of the presentation was valid and we have acted on it fully. The abstract and introduction have been rewritten so that every term is defined before it is used — including "BN γ\gamma," "AN index," and "stage-exhaustion boundary." The "noise sink / memorization" language, which generated the objection about strong claims without correspondingly strong evidence, has been retired entirely. All main figures have been regenerated with larger panels when possible, and test accuracy is now reported alongside test loss in the main tables.


W1(b) — Novelty: deepest-block concentration and magnitude pruning

We agree that deep layers being task-specific, and magnitude pruning targeting small-norm channels, are known results. What is new is the mechanistic account of why the norms become small in the deepest block specifically — and the predictive value that follows from it.

Three things were not known before this work. First, why Block 4 specifically and not Block 3: E6 (§4.1) is the first direct measurement showing that Block 4 develops the smallest gradient norms during training, which causes Adam to amplify its updates far more aggressively than other blocks (over 2800× relative to Block 1). This is a property of gradient depth and adaptive optimizers, not a restatement of magnitude pruning. Second, the collapse role migrates: after removing Block 4, Block 3 becomes the most suppressed block — the collapse follows topological depth, not block identity. Prior work documents depth-ordered pruning; it does not document migration. Third, the transition threshold can be predicted from channel counts alone before running any experiment, and confirmed prospectively on a new architecture (E7, §4.2).


W2(b) — Missing counterfactuals

All four requested counterfactuals have been added.

Smaller networks trained from scratch (E2) reproduce the suppression ordering without any pruning, confirming the effect is not an artifact of the pruning protocol. Reversing the removal order (E1) shows that removing the shallowest block leaves the deepest block's statistics intact, while removing the deepest block eliminates the collapse — establishing a causal direction. A width-matched comparison (E5) shows that reducing capacity by the same amount as depth-reduction, but through width rather than block removal, leaves suppression near its full-model level. And training on random labels (E3) reproduces the L2L_2 collapse but not the γ\gamma collapse — directly answering the pure memorization counterfactual: γ\gamma suppression requires a structured training signal.


W3(d) — Sub-optimality; convergence; hyperparameter stability

The from-scratch experiment (E2) rules out sub-optimality: the suppression ordering is reproduced in models trained from scratch for 200 epochs, with no pruning procedure involved.

On convergence, the reviewer correctly identifies that the final phase (one block remaining) may not fully converge within 100 epochs. We now say this explicitly in the manuscript. The claim rests on the direction and relative ordering of the dynamics immediately after each removal, not on terminal values.

On hyperparameters, the concern is valid. We acknowledge in the E5 discussion that constant hyperparameters are a potential confound. The key mitigating factor is that all three configurations in E5 share the same training protocol, so whatever bias exists affects them equally — the consistent ordering (full model suppressed more than width-reduced, which is suppressed more than depth-reduced) is attributable to topology rather than tuning.


W4(e) — Scale and diversity (Transformers, ImageNet)

We acknowledge this as a genuine limitation. The BN γ\gamma metric does not directly transfer to Transformers, which use LayerNorm — an analogous collapse statistic would need to be defined and validated before the mechanistic account can be tested in that setting, and we see this as a meaningful open problem rather than a simple extension. Beyond the main experiments, TinyImageNet (64×64, 200 classes) confirms the dual collapse at higher resolution, extending coverage beyond 32×32 imagery. The E6 account also makes a specific testable prediction for other architectures — gradient norm decreasing with depth, Adam amplification increasing with depth — that could serve as a roadmap for future work. We have framed the present contribution explicitly as a mechanistic analysis of the convolutional, overparameterized regime and added this to the Limitations section.


We hope to have addressed all your concerns and would greatly appreciate your re-evaluation of the score in light of the new experiments and revised presentation.

Reviewer_oM8pOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates where overfitting-related capacity is concentrated in neural networks and how structured pruning affects generalization. The authors observe that the deepest blocks of ResNets show a substantial decrease in BN scaling factors and weight L2 norms during training. They refer to this phenomenon as a "noise sink" and further show that similar suppression appears in the next deepest block after the original one is removed. The paper also identifies architecture-dependent pruning thresholds associated with changes in generalization performance.

Critical strengths

pros

The paper studies an interesting question about the relationship between network depth, pruning, and overfitting.

The analysis goes beyond pruning performance and examines how channel statistics evolve during training. I find the sequential block removal experiments particularly interesting.

Critical weaknesses

Cons

  1. The decrease in BN scaling factors and weight norms does not necessarily indicate that deeper channels are responsible for overfitting. The improvement in generalization after removing these channels could also result from ordinary capacity reduction. Moreover, the much weaker suppression under SGD suggests that the observed pattern may be partly influenced by optimization dynamics. More direct evidence is needed to distinguish these explanations.

  2. The 53.33% threshold for ResNet-18 follows directly from the fraction of prunable channels in Block 4, and its connection to generalization depends on the pruning criterion. The authors also attribute a sharp transition to the removal of a single channel in Block 2, but this explanation lacks a controlled experiment isolating that channel's effect.

Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer oM8p (R1)

We thank Reviewer oM8p for the careful reading. The revision directly addresses both concerns.


W1 — Capacity reduction vs. optimizer dynamics

The reviewer is right to raise this. We now address it with three complementary experiments.

The capacity concern is answered by a matched comparison (E5): a width-reduced four-block model with the same parameter count as the depth-reduced model retains near-full suppression, while removing the topologically deepest block reduces it. Same parameter reduction, different topology — different outcome. Capacity alone is not the driver.

The optimizer-dynamics point is also correct, and we now confirm and explain it directly (E6). Block 4 consistently develops the smallest gradient norms during pre-training, which causes Adam to amplify its updates far more aggressively than any other block — by over 2800× relative to Block 1 by epoch 100. This allows weight decay to dominate and suppress γ\gamma. The SGD result fits precisely: SGD has no such amplification mechanism, and the collapse essentially disappears under it. What the reviewer read as a weakness is now the mechanistic account.

A third experiment (E3) dissociates the two collapse components. Training on entirely random labels reproduces the L2L_2 norm collapse but not the γ\gamma collapse. A pure weight-decay-on-redundant-channels story cannot explain why one measure collapses without structure and the other does not.


W2 — Threshold follows from arithmetic; Block-2 isolation missing

On the threshold: the reviewer is correct that 53.33% is a simple fraction, and that is exactly the source of its value. Because it comes from channel counts alone, it can be predicted before running any experiment. We test this prospectively on ResNet-56, which has a different channel distribution (predicted threshold: 57.14%), and confirm that the γ\gamma criterion targets the deepest stage first at the predicted zone, with almost nothing selected from the shallowest stage (E7, Section 4.2). The arithmetic the reviewer questions is what makes the prediction work across architectures.

On Block-2 isolation: we ran the requested controlled experiment. Block-2 channels are excluded from the selection pool entirely, and the same total number of channels is pruned from the remaining blocks. The generalization trajectory is statistically indistinguishable from the standard case — AN at 54% differs by less than 1% across four seeds. Block-2 is not the cause. The revised manuscript replaces the original single-channel causal claim with the correct mechanism: depletion completeness. Under L2L_2, Block 4 channels are uniformly suppressed and therefore exhausted as a unit before any other block appears in the ranking, producing a hard boundary. Under γ\gamma, the distribution bridges Blocks 3 and 4, spreading the transition over a range of ratios.


We hope to have addressed all your concerns and would greatly appreciate your re-evaluation of the score in light of the new experiments and revised presentation.

I thank the authors for their detailed response and additional experiments. These additions make the empirical analysis more strong, including E5, E6 and the controlled experiments on block, however, I still have some concerns:

first, E5 and E6 is helpful on understanding the mechanism of the suppression, but ther is still some gap between suppression and overfitting, and I think that the large effective learning rate does not directly prove that weight decay is dominat in training.

second, regarding on the pruning threshold, there is a new question regarding Table 3, the training loss change a lot at the sparsity level of 52% and 45%, but the testing loss just changes a little, may be this result is hepful on understanding the transition of generalization gap, not the generalization, as the later one is often approximately equal to testing loss.

Overall, I appreciate the detailed response and additional experiment, but I think the paper still lack support for some core conclusion, so I am willing to maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.