- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The submission examines how residual connections and main branches in downsampling and non-downsampling blocks of ResNets relate to label memorization (via injected label noise) and generalization (test accuracy). The main evidence is single-pathway ablation (zero/mean/optimal) at each block, followed by a full forward pass and measurement of memorization score and test accuracy. This is supplemented by activation-norm ratios between pathways, block-wise Fisher trace, and Taylor-importance-based weight ranking/pruning. From these, the authors claim that downsampling blocks and residual connections “propagate” memorization together with generalization, and that memorization- and generalization-relevant weights are entangled and concentrated in downsampling blocks.
Critical strengths
None
Critical weaknesses
1. The central measurement cannot separate memorization from general network breakdown.
Ablating a pathway that carries most of the representation (the identity stream in non-downsampling blocks, or either branch at the stage-transition bottleneck) degrades the network as a whole. Any output metric will drop under such an intervention. Co-movement of memorization score and test accuracy is therefore the null outcome, not evidence of entanglement, and the paper never tests against this null. In most non-downsampling residual ablations, both metrics collapse to near chance (e.g., Fig. 2), where no differential effect could be observed even in principle.
Optimal ablation does not address this. It removes the off-distribution penalty of a fixed replacement, but an input-independent constant cannot restore the input-dependent information carried by the residual stream. Agreement across the three ablation strategies is thus expected and does not constitute robustness.
The secondary analyses share the same issue:
- The activation ratio profiles for mislabeled samples (Fig. 4a) and test samples (Fig. 4b) are nearly identical. This indicates that the measure reflects architectural norm structure (residual-stream accumulation and rescaling at projection shortcuts), not anything specific to memorization.
- Block-wise Tr(F) and first-order Taylor scores are not defined differentially with respect to clean training samples. Their concentration in downsampling blocks therefore does not establish a memorization-specific role.
A claim of memorization-specific involvement would require a differential design: graded (dose-response) ablation, a breakdown of post-ablation predictions on mislabeled samples, and a margin-matched clean control. Without these, the main conclusion does not follow from the evidence.
2. Organization and presentation obscure rather than support the claims.
Sections 5–8 are each a single short analysis presented as a top-level section. Their weight does not justify this structure, and they would read more clearly as subsections of one experimental section.
The abstract states that no prior work has studied this question, while the related work section hedges with to the best of our knowledge. The abstract should be equally careful, particularly given the closely related lesion analysis of Veit et al. (2016) [1].
The appendix (118 figures) appears not to have been checked figure by figure:
- Figs. 52, 54, …, 70 consist of captions only, each duplicating the preceding plot.
In addition, the evaluation is described as covering six architectures and six datasets, but Appendix A.3 lists roughly a dozen fixed model–dataset pairs, and some appendix figures use pairs not on that list. Since the “consistent across all settings” claim rests on the appendix, these issues materially reduce confidence in it.
In its current form, the submission does not appear to have been prepared with the care expected for review.
[1] Veit et al., Residual networks behave like ensembles of relatively shallow networks. NeurIPS, 29, 2016.
Questions for authors
-
Separating memorization from general breakdown. The ablated pathways (especially the identity stream in non-downsampling blocks) carry information needed for any prediction. Their removal is therefore expected to reduce both memorization score and test accuracy, regardless of where memorization is implemented. What evidence in the paper distinguishes this pathway propagates memorization from this pathway is necessary for the network to function at all? If none, the central claim should be restated accordingly.
-
Post-ablation predictions on mislabeled samples. After each ablation, what fraction of mislabeled training samples (a) retain the noisy label, (b) revert to the true label, or (c) go to another class? A drop in memorization score caused by (c) is indistinguishable from network failure. Only (b), together with preserved accuracy on clean training samples, would indicate a memorization-specific effect.
-
Differential control. Can the authors show that memorized samples degrade faster than margin-matched clean training samples under graded ablation (e.g., scaling a pathway by )? The same question applies to the Fisher and Taylor analyses: are the reported concentrations in downsampling blocks still present when importance is computed differentially, i.e., mislabeled minus clean training samples?
-
Appendix. Several captions contain errors indicating the figures were not individually checked (see Weakness 2). Since the claim of consistency across settings rests on the appendix, please verify and correct these figures.