acceptodds
Under review as a conference paper at ICLR 2027

Downsampling Blocks and Residual Connections Propagate Memorization Entangled with Generalization in ResNets

Abstract

Residual networks (ResNets) are among the most widely used deep learning models due to their ability to train very deep neural networks through a stable gradient flow enabled by residual/skip connections. However, no prior work has studied how different model pathways propagate memorization and generalization within the ResNet architecture. Hence, in this work, we investigate which pathways in ResNets are responsible for propagating memorization and generalization. We find that downsampling blocks and residual connections propagate memorization along with generalization, supported by activation-level analysis. We conduct a block-wise Fisher information analysis to elucidate the dominant role played by downsampling blocks in propagating memorization. We also provide a weight-level analysis to show that weights contributing to memorization and generalization are entangled and primarily concentrated in downsampling blocks. We validate our findings across three interpretability strategies, six architectures, and six datasets. Together, our results provide a clear map of how memorization and generalization propagate in the ResNet architecture, which has never been articulated in the literature.

Primary area:
interpretability and explainable AI
Keywords:
Downsampling block, Model memorization, Residual connections, ResNet

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ibQmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The submission examines how residual connections and main branches in downsampling and non-downsampling blocks of ResNets relate to label memorization (via injected label noise) and generalization (test accuracy). The main evidence is single-pathway ablation (zero/mean/optimal) at each block, followed by a full forward pass and measurement of memorization score and test accuracy. This is supplemented by activation-norm ratios between pathways, block-wise Fisher trace, and Taylor-importance-based weight ranking/pruning. From these, the authors claim that downsampling blocks and residual connections “propagate” memorization together with generalization, and that memorization- and generalization-relevant weights are entangled and concentrated in downsampling blocks.

Critical strengths

None

Critical weaknesses

1. The central measurement cannot separate memorization from general network breakdown.

Ablating a pathway that carries most of the representation (the identity stream in non-downsampling blocks, or either branch at the stage-transition bottleneck) degrades the network as a whole. Any output metric will drop under such an intervention. Co-movement of memorization score and test accuracy is therefore the null outcome, not evidence of entanglement, and the paper never tests against this null. In most non-downsampling residual ablations, both metrics collapse to near chance (e.g., Fig. 2), where no differential effect could be observed even in principle.

Optimal ablation does not address this. It removes the off-distribution penalty of a fixed replacement, but an input-independent constant cannot restore the input-dependent information carried by the residual stream. Agreement across the three ablation strategies is thus expected and does not constitute robustness.

The secondary analyses share the same issue:

  • The activation ratio profiles for mislabeled samples (Fig. 4a) and test samples (Fig. 4b) are nearly identical. This indicates that the measure reflects architectural norm structure (residual-stream accumulation and rescaling at projection shortcuts), not anything specific to memorization.
  • Block-wise Tr(F) and first-order Taylor scores are not defined differentially with respect to clean training samples. Their concentration in downsampling blocks therefore does not establish a memorization-specific role.

A claim of memorization-specific involvement would require a differential design: graded (dose-response) ablation, a breakdown of post-ablation predictions on mislabeled samples, and a margin-matched clean control. Without these, the main conclusion does not follow from the evidence.


2. Organization and presentation obscure rather than support the claims.

Sections 5–8 are each a single short analysis presented as a top-level section. Their weight does not justify this structure, and they would read more clearly as subsections of one experimental section.

The abstract states that no prior work has studied this question, while the related work section hedges with to the best of our knowledge. The abstract should be equally careful, particularly given the closely related lesion analysis of Veit et al. (2016) [1].

The appendix (118 figures) appears not to have been checked figure by figure:

  • Figs. 52, 54, …, 70 consist of captions only, each duplicating the preceding plot.

In addition, the evaluation is described as covering six architectures and six datasets, but Appendix A.3 lists roughly a dozen fixed model–dataset pairs, and some appendix figures use pairs not on that list. Since the “consistent across all settings” claim rests on the appendix, these issues materially reduce confidence in it.

In its current form, the submission does not appear to have been prepared with the care expected for review.

[1] Veit et al., Residual networks behave like ensembles of relatively shallow networks. NeurIPS, 29, 2016.

Questions for authors
  1. Separating memorization from general breakdown. The ablated pathways (especially the identity stream in non-downsampling blocks) carry information needed for any prediction. Their removal is therefore expected to reduce both memorization score and test accuracy, regardless of where memorization is implemented. What evidence in the paper distinguishes this pathway propagates memorization from this pathway is necessary for the network to function at all? If none, the central claim should be restated accordingly.

  2. Post-ablation predictions on mislabeled samples. After each ablation, what fraction of mislabeled training samples (a) retain the noisy label, (b) revert to the true label, or (c) go to another class? A drop in memorization score caused by (c) is indistinguishable from network failure. Only (b), together with preserved accuracy on clean training samples, would indicate a memorization-specific effect.

  3. Differential control. Can the authors show that memorized samples degrade faster than margin-matched clean training samples under graded ablation (e.g., scaling a pathway by α∈[0,1]\alpha \in [0,1])? The same question applies to the Fisher and Taylor analyses: are the reported concentrations in downsampling blocks still present when importance is computed differentially, i.e., mislabeled minus clean training samples?

  4. Appendix. Several captions contain errors indicating the figures were not individually checked (see Weakness 2). Since the claim of consistency across settings rests on the appendix, please verify and correct these figures.

Reviewer_A2nzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates how memorization and generalization propagate through ResNet architectures by independently ablating the residual/skip connections and main branches of individual blocks. The effects are measured using a memorization score (the fraction of label-corrupted training samples still predicted according to their noisy labels) and test accuracy.

The authors report four main findings: (1) both pathways contribute to memorization and generalization in downsampling blocks; (2) residual connections are the primary carriers of both in non-downsampling blocks, while main branches generally have limited impact; (3) activation-norm ratios, block-wise Fisher information traces, and Taylor-based weight-importance analyses provide additional evidence for the importance of downsampling blocks; and (4) highly ranked memorization and generalization weights exhibit substantial overlap, with pruning either set degrading both metrics, suggesting that the two are entangled.

The experiments cover selected combinations of six ResNet-family architectures and six datasets, with three label-noise levels (2%, 5%, and 10%), three ablation strategies (zero, mean, and optimal), and three random seeds.

Critical strengths
  • Systematic pathway-level analysis: The paper independently examines residual connections and main branches across downsampling and non-downsampling blocks, providing a more fine-grained characterization of their contributions to memorization and generalization.

  • Meaningful extension of prior work: The pathway-level investigation naturally extends Veit et al. (2016) and Maini et al. (2023). While Veit et al. studied the effects of main-branch ablations on test accuracy and Maini et al. investigated the localization of memorization across layers, this work explicitly examines the roles of both pathways in memorization and generalization.

  • Broad experimental validation: The findings are evaluated across selected combinations of six ResNet-family architectures, six datasets, three label-noise levels (2%, 5%, and 10%), and three random seeds, providing evidence that the observed trends are not limited to a single experimental configuration.

  • Multiple ablation strategies: The use of zero, mean, and optimal ablation reduces the likelihood that the reported patterns are artifacts of a particular intervention strategy.

  • Additional controls and weight-level analysis: The authors include a random-pruning baseline (Fig. 89) and explicitly quantify the overlap between memorization- and generalization-important weights (Fig. 88), strengthening the empirical investigation beyond pathway-level ablations.

Critical weaknesses
  • Limited memorization-specific insight: The main ablation results may largely reflect the ResNet architecture rather than a memorization-specific mechanism. In non-downsampling blocks, skip connections preserve the entire incoming representation, so their importance is expected. The limited effect of removing main branches was also established by Veit et al. (2016). The paper does not adequately distinguish general disruption of network predictions from disruption specifically attributable to memorization.

  • Insufficient evidence for memorization–generalization entanglement: Pruning highly important weights damages both metrics, but this does not necessarily establish that memorization and generalization are inseparable.

  • Architectural confounding in downsampling blocks: The importance attributed to downsampling may instead arise from stage transitions or projection shortcuts. Notably, the first block of the conv2 stage does not perform spatial downsampling but is grouped with downsampling blocks and often exhibits high importance.

Questions for authors
  • Initialization and controls. Appendix A does not state whether models are trained from scratch or from pretrained weights; the small datasets and lr = 2e-5 suggest fine-tuning, in which case the localization could reflect pretraining/fine-tuning dynamics. Please clarify, and add a from-scratch run and a 0%-noise baseline.

  • Block 1. The first conv2 block does not downsample yet shows among the largest effects. Is the operative factor the projection shortcut rather than downsampling?

  • Differential effects on memorization and generalization: The ablation results generally show that memorization score and test accuracy decrease together, which may simply reflect disruption of pathways important for overall network performance. A more informative analysis would examine whether individual pathways disproportionately affect one metric relative to the other. Could the authors report the per-block changes in memorization score and test accuracy, normalized by their respective baseline values, and analyze whether any pathways exhibit selective sensitivity to memorization or generalization? This would help distinguish memorization-specific effects from general representational dependence.

Reviewer_Qea4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Asks which ResNet pathways (identity skip, projection skip, main branch) carry memorization vs generalization. Noisy labels (2/5/10%), per-block zero/mean/optimal ablation, Fisher trace, Taylor weight importance. Claims downsampling blocks + residual paths carry both; main branch in non-downsampling blocks negligible. Six ResNet variants, six datasets.

Critical strengths
  1. Important, under-explored question: how the residual stream routes memorization, not just which layer.
  2. Downsampling-block signal consistent across models, datasets, diagnostics. Worth explaining.
  3. Three ablation types, three noise rates, random-pruning control.
  4. Confidence-vs-accuracy effect in last blocks: interesting side note.
Critical weaknesses
  1. Ablations show necessity, not localization. Memorization and generalization always drop together.
  2. No dissociation between the two. The central claim is untested.
  3. “Downsampling” effect confounded with architectural bottleneck effects.
  4. Fisher and importance analyses correlational and possibly scale-driven.
  5. ResNet family only, so the claim can’t be generalized or attributed
  6. Key setup details missing
Questions for authors
  1. What evidence separates “stores memorization” from “needed for the network to work”?
  2. Does the effect survive when downsampling is decoupled from residual blocks?
  3. Does it appear in non-ResNet or synthetic settings? Especially have a look at Allen-Zhou work and the Cagnetta/Wyatt RHM work. This might be a better avenue to show what you are looking for.
  4. Pretrained or from scratch models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.