- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how stochastic noise in SGD affects the evolution of the Neural Tangent Kernel (NTK) and its kernel-target alignment (KTA). Empirically, the authors show that stronger stochasticity enhances KTA across MLP, ResNet-18, and tiny ViT architectures. Theoretically, using a two-layer diagonal linear network with label noise, the authors adapt the stochastic collapse framework to show that noise suppresses weakly aligned modes. Finally, Section 5 explores the relationship between low-rank bias and KTA, demonstrating via an orthogonal regularization experiment that rank reduction alone does not guarantee alignment.
Critical strengths
-
Extending the study of KTA evolution from deterministic GD at the Edge of Stability (EoS) to SGD is well-motivated. Controlling for matched training loss across batch sizes and learning rates makes the empirical observations in Section 3 solid.
-
The distinction between low-rank bias (dimensionality reduction) and KTA (directional alignment of surviving modes) is the most interesting part of the paper. The orthogonal regularization experiment effectively demonstrates that KTA can increase even when rank reduction is explicitly prevented.
Critical weaknesses
-
The theoretical analysis relies on a diagonal linear network with whitened data (), balanced initialization (), and crucially, Assumption 4.4, which requires the initial neuron product and the target alignment to share the exact same sign. Under standard random initialization, half of the modes will have opposite signs (). If Assumption 4.4 is removed, the drift term in the SDE reverses sign for negatively aligned modes (), meaning modes with strong target signals () are driven toward the origin fastest and become trapped by the absorbing invariant set () before they can flip signs. Thus, Assumption 4.4 is not merely a technical convenience; without it, the core theoretical conclusion can completely reverse.
-
The SDE formulation and mode-collapse dynamics in Section 4 largely follow the established derivations of Stochastic Collapse (Chen et al., 2023). The paper's distinct theoretical addition is mainly substituting this collapse result into a Cauchy-Schwarz-style algebraic condition for KTA (Proposition 4.5), which feels somewhat incremental for a top-tier venue.
-
While Section 5 is conceptually engaging, its theoretical formulation () is a purely static, deterministic matrix optimization problem that contains no SGD dynamics, batch size, or noise scale. It relies on Hessian trace as an external surrogate rather than deriving the interplay between low-rank bias and KTA directly from stochastic dynamics.
-
Due to the computational cost of full NTK evaluation, all empirical validations are restricted to binary classification subsets (labels 0 vs. 1) of CIFAR-10 and Imagenette, leaving open whether the findings hold in standard multi-class regimes.
Questions for authors
[Q1] How can the theory in Section 4 account for the 50% of modes where under random initialization? Why wouldn't stochastic collapse prematurely trap strongly target-aligned modes at the saddle point as they attempt to cross zero to flip signs?
[Q2] Can the authors provide a more direct theoretical bridge in Section 5 that connects SGD noise scale ( or batch size ) to directional preservation, rather than relying on the static surrogate of minimizing ?
7 comments
Weaknesses
1. The theoretical analysis relies on a diagonal linear network with whitened data, balanced initialization, and crucially, Assumption 4.4, which requires the initial neuron product and the target alignment to share the exact same sign.
2. The SDE formulation and mode-collapse dynamics in Section 4 largely follow the established derivations of Stochastic Collapse (Chen et al., 2023).
3. While Section 5 is conceptually engaging, its theoretical formulation is a purely static, deterministic matrix optimization problem that contains no SGD dynamics, batch size, or noise scale.
Response (Weaknesses 1–3): We acknowledge the limitations of the previous theoretical framework.
To address these concerns, we have substantially revised Section 4 to consider general two-layer linear neural networks without assuming whitened inputs or imposing the restrictive sign-alignment condition. Furthermore, our revised analysis no longer relies on the stochastic collapse framework of Chen et al. (2023).
Instead, building upon the implicit regularization framework of Smith et al. [2], we investigate the approximate finite-time dynamics of mini-batch SGD and explicitly incorporate the influence of stochastic noise on kernel–target alignment. This approach allows us to move beyond the static, deterministic optimization problem considered in the previous version and to investigate how the noise scale affects the evolution of kernel–target alignment.
We believe that these revisions substantially improve both the generality and the practical relevance of our theoretical framework while establishing a clearer connection between stochastic optimization and kernel–target alignment.
4. Due to the computational cost of full NTK evaluation, all empirical validations are restricted to binary classification subsets.
Response: We acknowledge that the substantial computational cost of full NTK evaluation has restricted our empirical analysis to binary classification subsets.
To address this concern, we conducted additional experiments using multi-class neural networks trained under the mean squared error (MSE) loss. However, these experiments generally resulted in limited predictive performance, with test accuracies below 55% in the settings we examined. Consequently, we considered these results insufficient for a meaningful investigation of kernel–target alignment and decided to retain our focus on binary classification, where the models achieve more reliable predictive performance.
We acknowledge that this restriction limits the generality of our empirical findings. Extending our analysis to multi-class classification with sufficiently accurate models remains an important direction for future work.
References
[1] Lyu, Kaifeng, et al. "Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking." International Conference on Learning Representations (2024).
[2] Smith, Samuel L., et al. "On the Origin of Implicit Regularization in Stochastic Gradient Descent." arXiv preprint arXiv:2101.12176 (2021).
Thank you for the detailed response and substantial revisions. I appreciate the authors' efforts to address my concerns, particularly the reformulated theoretical analysis. While the revised manuscript represents a meaningful improvement, I still have some reservations about the connection between the theoretical framework and actual SGD dynamics, as well as the limited empirical generality. Therefore, I will maintain my original rating of 2.
Dear Reviewer,
Thank you very much for your thoughtful and constructive review. We sincerely appreciate the time and effort you have devoted to carefully evaluating our manuscript.
In response to your valuable feedback, we have substantially revised Section 4 and developed a new theoretical framework to investigate the mechanisms governing kernel–target alignment in two-layer linear neural networks. We have also conducted additional experiments and strengthened the connection between our theoretical analysis and empirical observations. Below, we provide detailed responses to the questions and weaknesses raised in your review.
Questions
1. How can the theory in Section 4 account for the 50% of modes where under random initialization? Why wouldn't stochastic collapse prematurely trap strongly target-aligned modes at the saddle point as they attempt to cross zero to flip signs?
Response: We thank the reviewer for identifying this important limitation of our previous theoretical analysis. We acknowledge that the sign-alignment assumption imposed in the previous version was restrictive and did not adequately account for the behavior of randomly initialized modes with opposite signs.
To address this concern, we have substantially reformulated our theoretical framework. In particular, our revised analysis considers general two-layer linear neural networks with symmetric random initialization, yielding zero initial network outputs. Such zero-output initialization schemes have also been employed in previous theoretical studies, including Lyu et al. [1].
Importantly, our revised analysis no longer relies on the stochastic collapse mechanism or the restrictive sign-alignment assumption. Instead, we investigate kernel–target alignment through the implicit regularization induced by SGD. We believe that this reformulation addresses the underlying concern regarding the applicability of our previous assumptions to randomly initialized networks.
2. Can the authors provide a more direct theoretical bridge in Section 5?
Response: We thank the reviewer for emphasizing the need for a more direct connection between SGD dynamics and kernel–target alignment.
In the revised manuscript, we have substantially extended the analysis previously presented in Section 5 by incorporating the implicit regularization framework of Smith et al. [2]. Rather than relying solely on the static characterization of Hessian-trace minimization, we investigate the approximate finite-time dynamics induced by mini-batch SGD.
In particular, our revised analysis characterizes the evolution of kernel–target alignment under SGD-induced implicit regularization and relates the rate of this evolution to the scale of stochastic noise. We believe that these results establish a more explicit theoretical connection between mini-batch stochasticity and the observed changes in kernel–target alignment.