acceptodds
Withdrawn from ICLR 2027

SGD-Induced Parameter Drift in the Saddle-to-Saddle Dynamics of Deep Linear Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Stochastic gradient descent (SGD) and gradient descent (GD) can remain close in function space while selecting distinct parametrization. We study this separation in deep linear teacher–student regression across initialization scales, in both online and offline training. Empirically, SGD and GD learn similar end-to-end maps throughout training, and SGD does not achieve lower population loss. However, for small initialization, where the network undergoes saddle-to-saddle dynamics, we find that SGD systematically redistributes scale across layers. In the absence of label noise, once a teacher mode is learned, its output-layer singular value grows while the corresponding singular values in earlier layers shrink, leaving the mode's end-to-end contribution approximately constant. This corresponds to motion along the rescaling symmetries of the network. Layer imbalances are not conserved by SGD; near rank-deficient saddles, their mean drift is a gradient flow that reduces the gradient noise. For Gaussian inputs with teacher-aligned covariances and decoupled learned modes, we show that this flow is driven by the residual error of the unlearned modes, and prove that SGD follows it while near a saddle. The flow predicts that the output-layer singular value grows as , independent of depth, while the earlier-layer singular values decay as , where is the depth and is a learning-rate-scaled training time. With label noise, the layer singular values flow towards a unique residual-dependent fixed point until the next mode is learned. This fixed point shifts as additional modes are learned and, at full fit, recovers the noise equilibrium of Ziyin et al. (2024). These results characterize an implicit bias of SGD toward particular parametrization that can be substantial even when the learned function remains close to that of GD.

Primary area:
optimization
Keywords:
stochastic gradient descent, implicit bias, deep linear networks, optimization dynamics, gradient noise, feature learning, saddle-to-saddle dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UqQMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the difference between SGD and GD training trajectories in parameter space. More specifically, they show that if we train a deep linear network to fit a linear function, while the SGD and GD training trajectories are close in function space, SGD introduces drift in parameter space: GD approximately preserves the layer balance; SGD with no label noise redistributes the norm toward the last layer; and SGD with label noise shifts the norms of different layers toward an equilibrium.

Critical strengths

The paper is quite clean. Within the studied setting, they do provide convincing theoretical analysis and experimental results.

Critical weaknesses

This paper considers only linear networks, which have very strong structural properties.

In particular, for linear networks in the analyzed setting, different singular values and subspaces are essentially decoupled. The theoretical analysis heavily relies on these properties, and it is not clear whether the intuition obtained this way carries over to nonlinear networks.

Questions for authors

Can you (at least partially) reproduce the observed phenomena in nonlinear networks, empirically or theoretically?

For example, if we train a fully connected bias-free ReLU network to fit this linear target function, can we observe similar drift? In this setting, the layers are still homogeneous, so we can still shift norms between layers and study how they change during training.

Reviewer_CGrEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the differences in parameterizations learned by SGD and GD, even though they are still close in function space. The authors consider a deep linear teacher–student setting under regression across initialization scales, in both online and offline training. For small initialization, where the network undergoes saddle-to-saddle dynamics, the paper shows SGD systematically redistributes scale across layers. For Gaussian inputs with teacher-aligned covariances and decoupled learned modes, the authors show that the flow is driven by the residual error of the unlearned modes and prove that SGD follows it near a saddle. These results characterize an implicit bias of SGD toward particular parametrizations that can be substantial even when the learned function remains close to that of GD.

Critical strengths
  1. The authors characterize how SGD redistributes scale across layers near rank-deficient saddles. Proposition 4.1 guarantees that SGD follows the predicted flow while remaining near a saddle.

  2. The asymptotic scaling laws show that output-layer singular values grow as $ \tau^{1/4} $, independently of network depth, while earlier-layer singular values decay as $ \tau^{-1/[4(L-1)]} $, where $ L $ is the depth and τ=η2t\tau = \eta^2 t is a learning-rate-scaled training time.

  3. The experiments compare SGD and GD across lazy, mean-field, and saddle-to-saddle regimes, including online and offline training, different batch sizes, and singular input covariances.

Critical weaknesses
  1. The connection between the functions learned by SGD and GD in Section 3 and the saddle-specific theoretical analysis in Section 4 could be clearer. The current presentation makes it difficult to see how the initial empirical findings motivate the paper’s main theoretical contribution.

  2. The paper does not establish the teacher–student setting before discussing the experimental results. The description is provided across Sections 3-4 and Appendix A, making the results and their connection to the theory difficult to follow.

  3. The authors show that SGD and GD learn similar functions but different parametrizations. However, the theoretical analysis focuses mainly on parameter drift near saddle points. It remains unclear how the theory connects to the empirical observations.

  4. Proposition 4.1 characterizes parameter drift only while SGD remains near a decoupled saddle. However, the paper does not establish whether SGD stays in this regime long enough for the predicted drift to become significant.

  5. The authors connect parameter drift to sharpening, fine-tuning, and interpretability in Section 5. But the paper lacks an explanation of experiments to evaluate the consequences.

Questions for authors

Questions:

  1. Please refer to the weaknesses section above.

  2. The connection between the functions learned by SGD and GD in Section 3 and the saddle-specific theoretical analysis in Section 4 could be clearer. Could you please discuss this?

  3. As the theory focuses on parameter drift near saddle points, please expand on how the empirical observations connect to Proposition 4.1.

Reviewer_XLeUOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper identifies a key difference between GD and SGD in the setting of saddle-to-saddle dynamics: SGD redistributes the magnitudes of weights at saddles, while GD leave the weights unchanged.

Critical strengths
  1. The paper is insightful: it establishes the implicit bias of SGD in redistributing the magnitudes in DLN.
Critical weaknesses
  1. The presentation needs to be improved.
  • The three regimes of the experiments in Section 3 should be introduced in greater detail; as a less-known concept, the saddle to saddle dynamics should be bear an informal introduction; the motivation to include three regimes should be discussed. There should also be a summary of experiment observations.
  • Section 4 is more like a list of plain statements rather than an organized explanation. I cannot understand the presentation at all without the help of AI. Specifically, the most important theoretical observation that only the last imbalance drifts (Line 372) is not directly observable from Eq. (4), and requires better explanations than the current Line 327.
  • Section 4 and 5 should be combined. At its current state, Section 4 is hard to understand, and the reader can find it hard to locate the exact evidence to support the claims in Section 5.
  1. The difference between the settings with and without label noise does not seem to be supported by the experiments in Section 3.
Questions for authors
  1. How is Eq. (8), especially the fourth-root, derived from Eq. (7)?
  2. As is claimed in the introduction, label noise gives the drift a moving fixed point. How to interpret that fixed point?
  3. What are the potential upsides / downsides of the SGD-induced parameter drift, in both settings with and without label noise?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.