acceptodds
Withdrawn from ICLR 2027

On the Effect of Stochastic Noise on Kernel-Target Alignment: An NTK Analysis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neural Tangent Kernel (NTK) has emerged as an important framework to understand the behavior of neural networks. An important notion of the kernel is kernel–target alignment, which measures how much the top eigenfunctions of NTK are aligned to the target function. In a rich feature learning regime, it is observed that NTK evolves to increase the kernel-target alignment. Recent work has shown that this particular evolution is prominent during the edge of stability (EoS) from a large learning rate of gradient descent (GD). In this work, we examine the evolution of NTK under stochastic gradient descent (SGD). Across networks of various architectures, we observe that stochastic noise (e.g. from a small batch size of SGD) can induce a larger kernel-target alignment. Theoretically, in a toy example of a diagonal neural network, we show that stochastic noise can selectively suppress weakly target-aligned modes and thereby increase the kernel-target alignment. We also connect such mode suppression to low-rank bias of SGD. Empirically, we observe that low-rank bias is one of the elements that are correlated with the kernel-target alignment. However, we also find that rank reduction alone does not determine kernel-target alignment; rather, the effect depends critically on whether the remaining spectral directions align to the target.

Primary area:
learning theory
Keywords:
Neural Tangent Kernel, Feature Learning, SGD

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vZHsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how stochastic noise in SGD affects the evolution of the Neural Tangent Kernel (NTK) and its kernel-target alignment (KTA). Empirically, the authors show that stronger stochasticity enhances KTA across MLP, ResNet-18, and tiny ViT architectures. Theoretically, using a two-layer diagonal linear network with label noise, the authors adapt the stochastic collapse framework to show that noise suppresses weakly aligned modes. Finally, Section 5 explores the relationship between low-rank bias and KTA, demonstrating via an orthogonal regularization experiment that rank reduction alone does not guarantee alignment.

Critical strengths
  • Extending the study of KTA evolution from deterministic GD at the Edge of Stability (EoS) to SGD is well-motivated. Controlling for matched training loss across batch sizes and learning rates makes the empirical observations in Section 3 solid.

  • The distinction between low-rank bias (dimensionality reduction) and KTA (directional alignment of surviving modes) is the most interesting part of the paper. The orthogonal regularization experiment effectively demonstrates that KTA can increase even when rank reduction is explicitly prevented.

Critical weaknesses
  • The theoretical analysis relies on a diagonal linear network with whitened data (Σxx=I\Sigma_{xx} = I), balanced initialization (ai2=bi2a_i^2 = b_i^2), and crucially, Assumption 4.4, which requires the initial neuron product si(0)s_i(0) and the target alignment μi\mu_i to share the exact same sign. Under standard random initialization, half of the modes will have opposite signs (sign(si(0))≠sign(μi)\text{sign}(s_i(0)) \neq \text{sign}(\mu_i)). If Assumption 4.4 is removed, the drift term in the SDE reverses sign for negatively aligned modes (Drift(dlog⁡∣si∣)≈−2μi−τ\text{Drift}(d \log \vert{}s_i\vert{}) \approx -2\mu_i - \tau), meaning modes with strong target signals (μi≫0\mu_i \gg 0) are driven toward the origin fastest and become trapped by the absorbing invariant set (ai=bi=0a_i=b_i=0) before they can flip signs. Thus, Assumption 4.4 is not merely a technical convenience; without it, the core theoretical conclusion can completely reverse.

  • The SDE formulation and mode-collapse dynamics in Section 4 largely follow the established derivations of Stochastic Collapse (Chen et al., 2023). The paper's distinct theoretical addition is mainly substituting this collapse result into a Cauchy-Schwarz-style algebraic condition for KTA (Proposition 4.5), which feels somewhat incremental for a top-tier venue.

  • While Section 5 is conceptually engaging, its theoretical formulation (min⁡a,Wtr(H) s.t. W⊤a=β\min_{a,W} \text{tr}(H) \text{ s.t. } W^\top a = \beta) is a purely static, deterministic matrix optimization problem that contains no SGD dynamics, batch size, or noise scale. It relies on Hessian trace as an external surrogate rather than deriving the interplay between low-rank bias and KTA directly from stochastic dynamics.

  • Due to the computational cost of full NTK evaluation, all empirical validations are restricted to binary classification subsets (labels 0 vs. 1) of CIFAR-10 and Imagenette, leaving open whether the findings hold in standard multi-class regimes.

Questions for authors

[Q1] How can the theory in Section 4 account for the 50% of modes where sign(si(0))=−sign(μi)\text{sign}(s_i(0)) = -\text{sign}(\mu_i) under random initialization? Why wouldn't stochastic collapse prematurely trap strongly target-aligned modes at the saddle point si=0s_i=0 as they attempt to cross zero to flip signs?

[Q2] Can the authors provide a more direct theoretical bridge in Section 5 that connects SGD noise scale (τ\tau or batch size BB) to directional preservation, rather than relying on the static surrogate of minimizing tr(H)\text{tr}(H)?

7 comments

Dear Reviewer,

Thank you very much for your thoughtful and constructive review. We sincerely appreciate the time and effort you have devoted to carefully evaluating our manuscript.

In response to your valuable feedback, we have substantially revised Section 4 and developed a new theoretical framework to investigate the mechanisms governing kernel–target alignment in two-layer linear neural networks. We have also conducted additional experiments and strengthened the connection between our theoretical analysis and empirical observations. Below, we provide detailed responses to the questions and weaknesses raised in your review.

Questions

1. How can the theory in Section 4 account for the 50% of modes where under random initialization? Why wouldn't stochastic collapse prematurely trap strongly target-aligned modes at the saddle point as they attempt to cross zero to flip signs?

Response: We thank the reviewer for identifying this important limitation of our previous theoretical analysis. We acknowledge that the sign-alignment assumption imposed in the previous version was restrictive and did not adequately account for the behavior of randomly initialized modes with opposite signs.

To address this concern, we have substantially reformulated our theoretical framework. In particular, our revised analysis considers general two-layer linear neural networks with symmetric random initialization, yielding zero initial network outputs. Such zero-output initialization schemes have also been employed in previous theoretical studies, including Lyu et al. [1].

Importantly, our revised analysis no longer relies on the stochastic collapse mechanism or the restrictive sign-alignment assumption. Instead, we investigate kernel–target alignment through the implicit regularization induced by SGD. We believe that this reformulation addresses the underlying concern regarding the applicability of our previous assumptions to randomly initialized networks.

2. Can the authors provide a more direct theoretical bridge in Section 5?

Response: We thank the reviewer for emphasizing the need for a more direct connection between SGD dynamics and kernel–target alignment.

In the revised manuscript, we have substantially extended the analysis previously presented in Section 5 by incorporating the implicit regularization framework of Smith et al. [2]. Rather than relying solely on the static characterization of Hessian-trace minimization, we investigate the approximate finite-time dynamics induced by mini-batch SGD.

In particular, our revised analysis characterizes the evolution of kernel–target alignment under SGD-induced implicit regularization and relates the rate of this evolution to the scale of stochastic noise. We believe that these results establish a more explicit theoretical connection between mini-batch stochasticity and the observed changes in kernel–target alignment.

Weaknesses

1. The theoretical analysis relies on a diagonal linear network with whitened data, balanced initialization, and crucially, Assumption 4.4, which requires the initial neuron product and the target alignment to share the exact same sign.

2. The SDE formulation and mode-collapse dynamics in Section 4 largely follow the established derivations of Stochastic Collapse (Chen et al., 2023).

3. While Section 5 is conceptually engaging, its theoretical formulation is a purely static, deterministic matrix optimization problem that contains no SGD dynamics, batch size, or noise scale.

Response (Weaknesses 1–3): We acknowledge the limitations of the previous theoretical framework.

To address these concerns, we have substantially revised Section 4 to consider general two-layer linear neural networks without assuming whitened inputs or imposing the restrictive sign-alignment condition. Furthermore, our revised analysis no longer relies on the stochastic collapse framework of Chen et al. (2023).

Instead, building upon the implicit regularization framework of Smith et al. [2], we investigate the approximate finite-time dynamics of mini-batch SGD and explicitly incorporate the influence of stochastic noise on kernel–target alignment. This approach allows us to move beyond the static, deterministic optimization problem considered in the previous version and to investigate how the noise scale affects the evolution of kernel–target alignment.

We believe that these revisions substantially improve both the generality and the practical relevance of our theoretical framework while establishing a clearer connection between stochastic optimization and kernel–target alignment.

4. Due to the computational cost of full NTK evaluation, all empirical validations are restricted to binary classification subsets.

Response: We acknowledge that the substantial computational cost of full NTK evaluation has restricted our empirical analysis to binary classification subsets.

To address this concern, we conducted additional experiments using multi-class neural networks trained under the mean squared error (MSE) loss. However, these experiments generally resulted in limited predictive performance, with test accuracies below 55% in the settings we examined. Consequently, we considered these results insufficient for a meaningful investigation of kernel–target alignment and decided to retain our focus on binary classification, where the models achieve more reliable predictive performance.

We acknowledge that this restriction limits the generality of our empirical findings. Extending our analysis to multi-class classification with sufficiently accurate models remains an important direction for future work.


References

[1] Lyu, Kaifeng, et al. "Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking." International Conference on Learning Representations (2024).

[2] Smith, Samuel L., et al. "On the Origin of Implicit Regularization in Stochastic Gradient Descent." arXiv preprint arXiv:2101.12176 (2021).

Thank you for the detailed response and substantial revisions. I appreciate the authors' efforts to address my concerns, particularly the reformulated theoretical analysis. While the revised manuscript represents a meaningful improvement, I still have some reservations about the connection between the theoretical framework and actual SGD dynamics, as well as the limited empirical generality. Therefore, I will maintain my original rating of 2.

Reviewer_pCnwOpenReview ↗
Rating
1(−2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the connection between stochasticity during training (from mini-batch gradient estimates and a finite learning rate) and kernel-data alignment of the neural tangent kernel (NTK). The authors measure empirically that increased stochasticity during training from decreased batch sizes and increased learning rates increases the NTK alignment. For a simple diagonal linear two-layer network, a suppression of non-aligned eigenfunctions of the NTK is derived for the case of training under label noise. This result is extended to mini-batch noise under some assumptions. Furthermore, the correlation between a low-rank bias of stochastic gradient descent and kernel-target alignment is studied. This is observed empirically for two-layer ReLU networks and theoretically for a two-layer linear network. However, the authors note that the low rank of the first-layer weight matrices can only explain part of the observed kernel alignment.

Critical strengths
  1. The empirical observations seem to support the claimed correlation between the improved kernel alignment and the stochasticity of the learning algorithm. Although stochasticity does not explain the full extend of the kernel alignment, there is an interesting effect here.
  2. The toy examples that the authors work through theoretically show the expected behavior and also capture the empirically observed harmfulness of too much stochasticity.
Critical weaknesses
  1. The toy problems chosen for the theoretical analysis are very simple. The analysis in Section 4 pertains only to diagonal linear networks. As equation (6) shows, these models boil down to a linear combination of the input features and I am not sure what the effect of the second layer is (see question below). The analysis in Section 5 is restricted to linear two-layer ReLU networks which are also very simple close to an optimal solution.
  2. The connection between the analyses in Sections 4 and 5 is not clear to me. Section 4 seems to analyze the impact of stochasticity in the toy problem. However, the observed suppression seems to be just the low-rank observation from Section 5. In Section 5 on the other hand, the stochasticity is not treated directly but is indirectly introduced by the low-rank assumption. Given that this was essentially the result of Section 4, it seems like a very strong assumption. Overall, it remains unclear how exactly these two analyses are related to each other.
Questions for authors
  1. Could you comment on the relation between Sections 4 and 5 that I outlined in weakness 2?
  2. Have you measured what fraction of the variance in the kernel alignment is explained by the low rank in Section 5.1?
  3. What is the significance of the second layer in equation (6)? It looks to me like the network has to learn a factor aibia_ib_i but does not gain more expressivity compared to a simpler model with just bib_i.
2 comments

Dear Reviewer,

Thank you very much for your thoughtful and constructive review. We sincerely appreciate the time and effort you have devoted to carefully evaluating our manuscript.

In response to your valuable feedback, we have substantially revised Section 4 and developed a new theoretical framework to investigate the mechanisms governing kernel–target alignment in two-layer linear neural networks. We have also conducted additional experiments and strengthened the connection between our theoretical analysis and empirical observations. Below, we provide detailed responses to the questions and weaknesses raised in your review.

Questions

1. Could you comment on the relation between Sections 4 and 5 that I outlined in weakness 2?

Response: We thank the reviewer for highlighting the need to establish a clearer connection between the theoretical analysis and the empirical observations concerning low-rank bias.

In the revised manuscript, we have substantially extended the analysis previously presented in Section 5, placing greater emphasis on the relationship between low-rank bias and kernel–target alignment. In particular, our revised theoretical framework investigates how SGD-induced implicit regularization influences kernel–target alignment and clarifies the role of low-rank structure in this process.

We believe that these revisions provide a more coherent connection between the theoretical mechanism and the empirical findings, thereby addressing the reviewer's concern regarding the consistency of the manuscript.

2. Have you measured what fraction of the variance in the kernel alignment is explained by the low rank in Section 5.1?

Response: We thank the reviewer for this insightful question. While our revised analysis does not directly quantify the fraction of variance in kernel–target alignment attributable to low-rank bias, it provides a more explicit theoretical characterization of the relationship between the two.

Specifically, we investigate the conditions under which low-rank structure can contribute to an increase in kernel–target alignment, as well as cases in which such an increase is not guaranteed. This analysis clarifies both the explanatory role and the limitations of low-rank bias in determining kernel–target alignment.

We believe that these results provide a more rigorous understanding of the relationship between low-rank bias and kernel–target alignment, although a direct quantitative decomposition of the variance remains an interesting direction for further investigation.

3. What is the significance of the second layer in equation (6)?

Response: We thank the reviewer for raising this question. In the revised manuscript, we have substantially reformulated our theoretical analysis. Rather than relying on stochastic collapse, we now investigate kernel–target alignment through the implicit regularization induced by SGD, following the framework of Smith et al. [1].

Importantly, the second-layer weights continue to play a significant role in determining kernel–target alignment in two-layer linear neural networks. Our revised analysis explicitly considers the evolution of these weights and their contribution to the dynamics of the NTK and its alignment with the target.

We believe that this reformulation provides a clearer theoretical explanation of the role of the second layer and its interaction with SGD-induced implicit regularization.

Weaknesses

1. The toy problems chosen for the theoretical analysis are very simple. The analysis in Section 4 pertains only to diagonal linear networks.

Response: We acknowledge that the theoretical setting in the previous version of our manuscript was highly simplified, particularly due to its restriction to diagonal linear networks and whitened inputs.

To address this limitation, we have substantially revised Section 4 to consider general two-layer linear neural networks without assuming whitened inputs. This extension allows our analysis to accommodate nontrivial input covariance structures and provides a broader characterization of the mechanisms governing kernel–target alignment.

Furthermore, we have conducted additional experiments using two-layer linear neural networks to examine the predictions of our revised theoretical framework. The empirical results are consistent with our theoretical findings, providing additional support for the proposed mechanism.

Although two-layer linear networks remain simplified models compared with practical deep nonlinear architectures, we believe that the revised analysis substantially improves the generality of our theoretical results and strengthens their connection to the empirical observations.


References

[1] Smith, Samuel L., et al. "On the Origin of Implicit Regularization in Stochastic Gradient Descent." arXiv preprint arXiv:2101.12176 (2021).

I want to thank the authors for their substantial review and rebuttals. However, the new version of the manuscript constitutes such a substantial rewrite of the original submission that it exceeds my capacity at the rebuttal stage to evaluate it carefully. Therefore, I recommend rejection at this point and resubmission at a later stage. I will lower my score accordingly.

I want to emphasize that this is not based on an evaluation of the quality of the new version of the manuscript.

Reviewer_3KaJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper attempts to investigate the relationship between SGD and the evolution of the NTK and kernel-task alignment. Experiments on binary classification tasks using ResNet-18 report higher alignment for smaller batch sizes. The authors analyze a diagonal linear network under label noise and give conditions under which suppressing weakly target-aligned modes increases alignment. They also provide a local correspondence with minibatch noise near small initialization. Finally, experiments on two-layer ReLU networks examine the relationship between weight-matrix stable rank and NTK alignment, illustrating that rank reduction alone does not determine alignment. The reported relationship between alignment and test performance varies across architectures.

Critical strengths

The research question is important and interesting.

The paper is relatively easy to follow.

Critical weaknesses

The paper suffers from serious flaws that prevent me from recommending acceptance.

(W1) The analysis in part 3, and in particular Fig 1, uses an equal number of epochs rather than an equal number of optimization steps; the improved kernel-task alignment due to a smaller batch size is confounded with optimization steps.

(W2) The abstract and introduction discuss edge of stability (EoS) and say that the NTK evolves at the EoS, but EoS is never mentioned in the body of the work. Moreover, the theoretical analysis is done in continuous time, where EoS cannot appear.

(W3) Below Eq. 9, the manuscript reads:

Under these dynamics, we find that the balance between ai and bi is preserved throughout training. It can be confirmed by checking the evolution of Ri = a2 i − b2 i . By Itoˆ’s lemma, dRi = −τ Ridt and Ri = a2 i − b2 i = 0 remains during training.

This is not an original result, and it should be appropriately attributed to [1,2]

(W4) The theory is extremely cartoonish, so much so that it is hard to recognize real data or model in it, and no bridge between the theory and experiment is offered. I think the field has largely moved away from such simple models. No relation between theory and experiments is ever demonstrated.

(W5) It's been argued that SGD noise is insignificant in an online setting [3]. Given the overwhelming popularity of online learning in modern ML I expect a discussion of this issue.

(W6) Positioning compared to related works. [4] tracks the evolution of the NTK under SGD (including discrete time and EoS) in a richer model but is never cited. [5] solves a two-layer linear network (non-diagonal) on anisotropic data, without strong assumptions on the initialization, and extracts neural scaling laws. The contribution compared to Atanasov et al., 2022, [4], [5] and especially Baratin et al., 2021 is unclear.

[1] Saxe A M, McClelland J L and Ganguli S 2014 Exact solutions to the nonlinear dynamics of learning in deep linear neural networks Online: http://arxiv.org/abs/1312.6120

[2] Chen F, Kunin D, Yamamura A and Ganguli S 2024 Stochastic collapse: how gradient noise attracts SGD dynamics towards simpler subnetworks* J. Stat. Mech. 2024 104023

[3] Vyas N, Morwani D, Zhao R, Kaplun G, Kakade S and Barak B 2023 Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning Online: http://arxiv.org/abs/2306.08590

[4] Agarwala A and Pennington J 2025 High dimensional analysis reveals conservative sharpening and a stochastic edge of stability Online: http://arxiv.org/abs/2404.19261

[5] Bordelon B, Atanasov A and Pehlevan C 2024 How Feature Learning Can Improve Neural Scaling Laws Online: http://arxiv.org/abs/2409.17858

Questions for authors
  1. Could the authors frame their contribution compared to Baratin et al., 2021 (cited in the submission) more clearly?

  2. Can you plot Figures 1 and 2 as a function of training steps?

  3. Can you repeat the experiments in an online setting (e.g., using CIFAR5M)?

5 comments

Dear Reviewer,

Thank you very much for your thoughtful and constructive review. We sincerely appreciate the time and effort you have devoted to carefully evaluating our manuscript.

In response to your valuable feedback, we have substantially revised Section 4 and developed a new theoretical framework for investigating kernel–target alignment in two-layer linear neural networks. We have also conducted additional experiments and clarified the contributions and limitations of our work. Below, we address the questions and weaknesses raised in your review.

Questions

1. Could the authors frame their contribution compared to Baratin et al., 2021 (cited in the submission) more clearly?

Response: We thank the reviewer for highlighting the need to clarify our contribution relative to Baratin et al. (2021). Their work investigates the evolution of neural tangent kernels (NTKs) through centered kernel–target alignment, providing important insights into feature learning and generalization. However, to the best of our knowledge, their analysis does not explicitly examine how mini-batch size influences the evolution of kernel–target alignment.

In contrast, our primary contribution is to empirically identify the relationship between mini-batch size and kernel–target alignment and to theoretically investigate how SGD-induced implicit regularization affects its finite-time evolution. We believe that this distinction clarifies our contribution relative to their work.

2. Can you plot Figures 1 and 2 as a function of training steps?

Response: We thank the reviewer for this valuable suggestion. In response, we have revised Figure 1 in Section 3 to present the results as functions of the number of SGD steps, rather than training epochs. The results remain consistent with our previous observations: smaller batch sizes generally yield higher kernel–target alignment, even when compared at the same number of training steps. This additional analysis provides further evidence that the observed batch-size dependence is not solely attributable to differences in the number of parameter updates per epoch.

3. Can you repeat the experiments in an online setting (e.g., using CIFAR5M)?

Response: We thank the reviewer for suggesting this important extension. We have conducted additional experiments using CIFAR-5M to investigate whether the relationship between mini-batch size and kernel–target alignment extends to online learning. The results are presented in Appendix B.3.

Unlike our previous observations, these experiments do not demonstrate a clear dependence of kernel–target alignment on mini-batch size. We acknowledge that this finding suggests a potential limitation in the generalizability of our results.

In particular, our theoretical framework is motivated by the implicit regularization analysis of Smith et al. [1], which considers a regime in which data fitting occurs on a relatively fast timescale compared with the slower dynamics induced by implicit regularization. In online learning, however, this timescale separation is not necessarily satisfied, as the model continuously encounters newly sampled data. Consequently, the assumptions underlying our two-timescale analysis do not directly extend to online learning.

We therefore acknowledge that extending our theoretical framework to online learning remains an open question and an important direction for future investigation.

Weaknesses

1. The abstract and introduction discuss edge of stability (EoS) and say that the NTK evolves at the EoS, but EoS is never mentioned in the body of the work. Moreover, the theoretical analysis is done in continuous time, where EoS cannot appear.

Response: We acknowledge that our theoretical analysis assumes a sufficiently small learning rate and considers a continuous-time approximation of the optimization dynamics. Consequently, our framework does not directly characterize the discrete-time phenomena associated with the edge of stability (EoS), which remains a limitation of our analysis.

We also attempted to empirically investigate the influence of larger learning rates on kernel–target alignment. However, in our experimental setting, learning rates exceeding 0.005 frequently resulted in unstable optimization dynamics, including divergence of the training loss and numerical instability (NaN values). These observations made it difficult to systematically characterize kernel–target alignment in the large-learning-rate regime.

For these reasons, we have chosen to focus on the sufficiently small-learning-rate regime, where the continuous-time approximation is more appropriate. We acknowledge that extending our theoretical analysis to the EoS regime would require a different theoretical treatment and remains an important direction for future work.

2. Below Eq. 9, the manuscript reads: … This is not an original result, and it should be appropriately attributed to [1,2].

Response: We acknowledge that the previous version of our manuscript relied on results concerning stochastic collapse without adequately distinguishing established findings from our original contributions. We sincerely apologize for this oversight.

In the revised manuscript, we have substantially reformulated our theoretical analysis, which no longer relies on stochastic collapse. Instead, we investigate kernel–target alignment through a two-timescale framework motivated by the SGD-induced implicit regularization characterized by Smith et al. [1].

We acknowledge that timescale separation introduces nontrivial assumptions. Nevertheless, related approaches have been employed in previous theoretical studies of SGD, including the analysis of isotropic noise and label-noise SGD by Li et al. [2].

Building upon these established frameworks, our contribution is to investigate how SGD-induced implicit regularization affects the finite-time evolution of kernel–target alignment in two-layer linear neural networks. We have revised the theoretical presentation and attribution to more clearly distinguish our contributions from existing results.

3. The theory is extremely cartoonish, so much so that it is hard to recognize real data or model in it, and no bridge between the theory and experiment is offered.

Response: We acknowledge that the theoretical setting in the previous version was highly simplified and that the connection between our theoretical predictions and empirical observations was insufficiently established.

To address these concerns, we have substantially revised Section 4 to consider general two-layer linear neural networks without assuming whitened inputs. This extension allows our theoretical framework to accommodate nontrivial input covariance structures and provides a broader characterization of the dynamics governing kernel–target alignment.

Furthermore, we have conducted additional experiments using two-layer linear neural networks to directly examine the predictions of our revised theoretical analysis. The empirical results are consistent with our theoretical findings, providing additional support for the proposed mechanism connecting SGD-induced implicit regularization and kernel–target alignment.

Although two-layer linear networks remain simplified compared with the deep nonlinear architectures used in our main experiments, we believe that the revised framework provides a more meaningful connection between the theoretical mechanism and empirical observations.

4. Positioning compared to related works.

Response: Bordelon et al. (2025) investigate how feature learning influences neural scaling laws, particularly by improving the convergence rates of difficult tasks outside the initial kernel's reproducing kernel Hilbert space (RKHS). In contrast, our work focuses on how SGD-induced implicit regularization influences the evolution of NTK–target alignment through slow dynamics near a minimizer manifold.

Agarwala and Pennington (2025) investigate the influence of SGD noise on loss-landscape curvature and optimization stability, including conservative sharpening and the stochastic edge of stability. While their analysis primarily concerns optimization stability, our work examines the relationship between stochastic noise, low-rank bias, and kernel–target alignment.

Furthermore, Atanasov et al. (2022) investigate feature-learning dynamics under deterministic gradient flow, whereas our analysis considers how mini-batch stochasticity perturbs the underlying gradient-flow dynamics through SGD-induced implicit regularization.

Thus, although these studies share the broader objective of understanding feature evolution and optimization dynamics, our work specifically investigates the mechanism connecting SGD-induced implicit regularization and kernel–target alignment. We have revised the Related Work section in Appendix A to clarify these distinctions.


References

[1] Smith, Samuel L., et al. "On the Origin of Implicit Regularization in Stochastic Gradient Descent." arXiv preprint arXiv:2101.12176 (2021).

[2] Li, Zhiyuan, Tianhao Wang, and Sanjeev Arora. "What Happens after SGD Reaches Zero Loss?—A Mathematical Framework." arXiv preprint arXiv:2110.06914 (2021).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.