acceptodds
Under review as a conference paper at ICLR 2027

Mind the Drift: Diagonal Linear Networks Under Large Learning Rates

Abstract

Large learning rates can qualitatively change the trajectory of neural network training, often pushing optimization into regimes far from classical gradient-flow behavior. The Edge of Stability (EoS) offers a valuable lens on the dynamics such learning rates induce. We study corresponding dynamics in diagonal linear networks, where we uncover a competition between two distinct implicit biases that jointly determine the sparsity of the recovered solution in regression settings. Complementary to the Gain, which captures the average discretization error accumulated by Gradient Descent relative to Gradient Flow, we derive a closely associated but overlooked quantity: the Drift. Under large learning rates, it describes an imbalance between different discretization errors and represents a systematic shift in the optimization trajectory. While the Gain grows monotonically in certain regimes, and can bias towards denser, flatter interpolators, the impact of the Drift depends on its alignment with potential solutions, which can either counteract or reinforce the effect of the Gain. Consequently, its behavior drives model selection, particularly during early training epochs. To validate our theoretical insights, we introduce an intervention that actively steers the Gain to recover sharper, sparser solutions. Thus, our analysis reveals that large learning rates do not universally hinder the recovery of sparse solutions. On the contrary, they can be harnessed to control the implicit bias of training.

Primary area:
optimization
Keywords:
Implicit Bias, training dynamics, optimization, theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rR6ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends the analysis of the implicit bias of gradient descent in diagonal linear networks developed by Even et al. (2023). In particular, it revisits the shifted reference point appearing in the Bregman-divergence characterization of the limiting solution. While Even et al. argued that this shift could be approximately ignored, the authors show that its contribution to the implicit regularizer can remain significant. They introduce the notion of Drift to characterize this signed correction and study its interaction with the previously identified Gain. They also derive a sparse-recovery bound, investigate the evolution of these quantities during training, and introduce a modified optimization method, Gain Control (GC), intended to favor sparse solutions.

Critical strengths
  • The main contribution is showing that the Drift term, which Even et al. treated as a perturbation, can significantly affect the implicit bias at large learning rates. In particular, the paper highlights a limitation of the previous argument: showing that the shifted reference point is small is not sufficient to conclude that its contribution to the implicit regularizer is negligible.
  • The qualitative study of the interaction between Drift and Gain is interesting. In particular, the analysis of Drift accumulation under consistent gradient signs and its cancellation under approximately alternating gradients provides useful intuition about the training dynamics.
  • The proposed Gain Control update is an interesting attempt to exploit these insights to influence the implicit bias.
Critical weaknesses
  • The contributions appear somewhat incremental, and the relationship with previous work is not always described precisely. For example, at line 182, the authors state that Even et al. "do not further analyze the shift in β~0\tilde{\beta}_0." However, Even et al. explicitly introduce the corresponding signed correction in their time-varying potential (Eq. 7 and Proposition 2) and discuss why the shifted reference point might be neglected. This discussion relies on Proposition 16, which establishes ∣β~0∣≤α2|\tilde{\beta}_0|\leq\alpha^2 under its assumptions, and Proposition 19, which shows that β~0→0\tilde{\beta}_0\to0 as γ→0\gamma\to0. The present paper provides a useful refinement of that interpretation, but the underlying correction was already part of the previous characterization.

  • The practical relevance of the Drift mechanism in the large-learning-rate regime is not entirely clear to me. The authors observe that Drift approximately saturates once the training dynamics enter the EoS regime, while Gain continues accumulating (Figures 3, 5, and 9). This suggests that the relative contribution of Drift to the implicit regularizer may become negligible compared with Gain, potentially recovering the picture suggested by Even et al. While Figure 8 illustrates that Drift can still affect solution selection in a particular example, it is unclear how significant this effect remains for larger learning rates and longer training. Furthermore, in the three-coordinate experiment, the larger learning rate leads to a less sparse solution. Similarly, in the transformer experiment, increasing the learning rate at fixed batch size leads to lower validation accuracy. These observations raise questions about the practical relevance of the proposed mechanism in the aggressive-learning-rate regime that motivates the paper.

  • While Theorem 2.4 provides a sparse-recovery bound, it depends on the final effective scales and Drift, which are themselves determined by the optimization trajectory. It is therefore unclear to what extent this result predicts the effect of the learning rate on recovery or identifies settings in which Drift successfully compensates for Gain. More generally, the analysis of the training phases relies on assumptions about gradient signs, rather than establishing when these patterns arise.

  • The regression experiments are relatively narrow. A more extensive comparison with GD and appropriately tuned SGD would help establish the benefits of Gain Control. The practical relevance of the transformer experiments is also unclear, particularly because the measurements are obtained with Muon rather than the GD dynamics analyzed theoretically.

  • Minor comments:

-The discussion following Theorem 2.4 could be expanded. In its current form, the interpretation is difficult without consulting the appendix. In particular, it would be helpful to clarify which assumptions would make the bound informative about learning-rate-dependent sparse recovery.

  • The related work should discuss Nacson et al., Implicit Bias of the Step Size in Linear Diagonal Neural Networks (ICML 2022). This work already shows that large learning rates can induce sparsity-promoting behavior for GD on non-centered data.
Questions for authors
  1. Could the authors extend the Gain–Drift analysis to SGD? This would be particularly interesting given the differences between GD and SGD established by Even et al.

  2. Could the authors comment further on the assumption at line 261 that "the predictor gradients maintain a consistent sign in each coordinate"? Under which conditions does this hold in practice? What can be said about Drift accumulation when gradient signs change?

  3. Since Drift appears to saturate while Gain continues to accumulate in the EoS regime, could the authors clarify under which conditions Drift remains significant relative to Gain in determining the final implicit bias? In particular, could the authors provide evidence that this effect remains important at large learning rates, beyond the specific toy example? This seems important for understanding the practical relevance of the proposed analysis.

2 comments

On extending the Gain–Drift analysis to SGD in light of the GD–SGD differences established by Even et al.

We thank the reviewer for this suggestion. We agree that understanding how stochasticity affects the competition between Gain and Drift is an interesting direction, particularly in light of the differences identified by Even et al. (2023).

A natural starting point is to examine conditional moments of the stochastic predictor gradient. Write Gk,i=gk,i+ξk,iG_{k,i}=g_{k,i}+\xi_{k,i}, where gk,i=∇iL(βk)g_{k,i}=\nabla_i L(\beta_k) and Ek[ξk,i]=0\mathbb E_k[\xi_{k,i}]=0, with expectations conditional on the training history. In the regime ∣γkGk,i∣≤ρ<1|\gamma_kG_{k,i}|\leq\rho<1, the Drift expansion gives

Ek[ΔDk,i]=23γk3Ek[Gk,i3]+Oρ!(γk5Ek[∣Gk,i∣5]),\mathbb E_k[\Delta D_{k,i}]=\frac{2}{3}\gamma_k^3\mathbb E_k[G_{k,i}^3]+O_\rho\\!\left(\gamma_k^5\mathbb E_k[|G_{k,i}|^5]\right),

where

Ek[Gk,i3]=gk,i3+3gk,iEk[ξk,i2]+Ek[ξk,i3].\mathbb E_k[G_{k,i}^3]=g_{k,i}^3+3g_{k,i}\mathbb E_k[\xi_{k,i}^2]+\mathbb E_k[\xi_{k,i}^3].

Thus, expected Drift depends on both the noise variance and its third centered moment, whereas Gain depends at leading order on the second moment of the stochastic gradient. In particular, unbiasedness alone does not imply that stochasticity leaves expected Drift unchanged.

This calculation provides a concrete starting point for an SGD extension. Establishing how these contributions accumulate or cancel over training would additionally require controlling the evolving conditional moments and fluctuations around their conditional expectations. We therefore leave this stochastic extension of the phase-dependent Gain–Drift analysis to future work, while theoretically focusing the present study on GD.

On when predictor gradients maintain coordinatewise sign coherence and how sign changes affect Drift accumulation

The Drift increment is cubic to leading order in the product of the learning rate and the loss gradient: d(z)=23z3+O(z5),z=γ∇L(β).d(z)=\frac{2}{3}z^3+O(z^5), \qquad z=\gamma\nabla L(\beta). More generally, d(z)d(z) is an odd function. Within the regime ∥zk∥∞≤ρ<1\|z_k\|_\infty\le\rho<1, consecutive iterations with the same sign in the gradient of the loss therefore produce Drift increments that accumulate coordinatewise, whereas opposite-sign gradients produce increments that partially cancel, with exact cancellation when their magnitudes are equal. Lemma 2.5 quantifies accumulation during a sign-coherent phase. Lemma 2.6 quantifies cancellation when consecutive gradients approximately reverse, bounding the remaining Drift increment in terms of the mismatch between the two steps. Thus, near period-two oscillations, the additional Drift can be small even when the individual gradients remain substantial. In contrast, the Gain increment is even and nonnegative in this regime, so the contributions from the two steps add up rather than cancel. This explains how oscillatory training can suppress further Drift accumulation while allowing Gain to continue accumulating.

On when accumulated Drift remains significant relative to Gain in the final implicit bias, and the evidence at large learning rates beyond the toy example

Saturation of Drift means that its subsequent increments become small. It does not remove the Drift already accumulated. Under the convergence assumptions of Proposition 2.3, the final interpolator minimizes $ \psi_{a_\infty}(\beta)+\frac12\langle D_\infty,\beta\rangle $ Consequently, Drift accumulated during the initial phase (i.e. before or during the onset of EoS) remains a linear contribution to the final implicit regularizer, even when later oscillations largely cancel its additional increments. If the initially accumulated Drift is significant and suitably aligned with the final solution, its contribution remains significant even though later drift increments are small. Furthermore, our experiments with DeiT-Small trained on ImageNet for 300 epochs using Muon provide preliminary evidence of this persistence beyond diagonal linear networks. Across three learning-rate and batch-size configurations, the analogous Drift quantity tracked for the query–key pairs accumulates substantially early in training grows more in magnitude relative to the Gain (Fig. 7). The configuration with the largest final Drift also exhibits the lowest aggregate query–key effective rank (Table 2), indicating greater concentration of spectral mass. These observations suggest that early Drift accumulation may remain relevant to the structure of learned representations in practical architectures.

I thank the authors for their reply. After reading the other reviews and the authors’ rebuttal, I have decided to maintain my overall rating.

Reviewer_QL9YOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies overparametrizd regression problem X\beta = y with two-layer diagonal linear network. The authors introduce "Drift" and obtain a sparse recovery theorem, given convergence of GD, which gives an error bound characterized by the Gain and the Drift (Theorem 2.4), where "Gain" is an amount introduced by Even et al. In addition, the authors study how the drift behaves during EoS phases (lemma 2.5-2.6). The authors introduce gain control algorithm and proves its convergence to the same interpolator \beta_\infty as that of GD (Proposition 3.3 and Theorem 3.4).

Critical strengths

The paper introduces "drift" and uses that to quantify the error of sparse recovery problem together with the concept of "gain" introduced by Even et al (Theorem 2.4). The paper also explains a likely scenario behind the Edge of Stability phenomenon through the lens of drifty by Lemma 2.5 and 2.6.

Critical weaknesses

The paper seems to be largely motivated by Even et al. In fact, their setting is same as that of Even et al.'s. Moreover, their EoS analyses appears to give theoretical analyses to the heuristic observation given in Section 5 of Even et al. Therefore, explicit comparison between Even et al.'s results is essential to assess their contribution. With this regard, I have a couple concerns:

  1. The paper introduces the drift and it is the central object of their study. The quantity may not be necessarily new. In fact it has already essentially appeared as \phi_k in Section 3.2, Even et al. (https://arxiv.org/pdf/2302.08982). Therefore Proposition 2.3 is essentially algebraic reformulation of Theorem 1 and Proposition 2 of Even et al.
  2. The error bound of Theorem 2.4 appears new but it has two limitations: i) it assumes the convergence of GD and ii) only an upper bound is obtained. For i), it does seem that convergence is neither discussed in their paper nor follow from an existing result especially at the EoS regime. For ii), the authors claim that the Gain and the drift competes does not seem to follow only from the upper bound.
  3. The authors seem to claim that Lemma 2.5 and 2.6 together explains the mechanism behind the EoS, namely, the progressive sharpening phase and oscillation phase, respectively. If my understanding is correct, what is actually stated in Lemma 2.5 and 2.6 seem to be very misleading. Lemma 2.5 assumes coherence of signs by \sigma_jz_{k,j}\geq 0. Without the sign-coherence assumption, the claimed drift accumulation would not hold. Therefore, Lemma 2.5 actually assumes important mechanism rather than proving it. Similarly, (eq-11) of Lemma 2.6 itself does not explain the claimed drift cancellation at EoS. As stated in the last sentence of Lemma 2.6, (eq-11) can be used to show drift cancellation if you assume z_{k+1} \approx -z_k. Therefore, Lemma 2.6 does not explain why oscillation occurs. Rather, it is a rather elementary consequence assuming oscillation. Moreover, neither theorem explains how exactly the phase shift from the progressive sharpening phase to the oscillatory phase. Therefore, it seems to me that that the argument in section 2.2 remains largely heuristic rather than a complete theory.
  4. This is related to 2). The authors introduced the Gain Control algorithm in section 3 and illustrated the effectiveness of the Gain and Drift. I wonder why they did not study GD instead, which appears to be where their motivation came from.
Questions for authors
  1. Could the authors provide a more explicit comparison with Even et al. (2023), particularly regarding the novelty of the Drift and the mathematical implications of Proposition 2.3 relative to their Theorem 1 and Proposition 2?

  2. Can the authors establish convergence of ordinary GD for learning rates that lead to the EoS dynamics considered in the paper? If this is currently beyond the scope of the analysis, what is the main mathematical obstruction?

  3. Can the authors derive the gradient behavior assumed or invoked in Lemmas 2.5 and 2.6 from EoS dynamics, or otherwise establish a quantitative relationship between EoS and Drift accumulation or saturation?

  4. Can the authors identify a theoretical result showing that the phase-dependent evolution of Gain and Drift leads to a change in the selected interpolator, rather than only providing an endpoint recovery bound conditional on their final values?

5 comments

On whether the upper bound establishes competition between Gain and Drift

We thank the reviewer for raising this distinction. We agree that an increase in an upper bound does not, by itself, imply an increase in the actual recovery error. However, the competition between Gain and Drift is explicitly encoded in the signed cancellation within the NSP bound, rather than inferred solely from changes in its magnitude.

Specifically, Theorem 2.4 gives

∥β^−β⋆∥1≤(1+c)AScsinh⁡(cRS+∥DSc∥∞).\lVert \widehat{\beta}-\beta^\star \rVert_1 \le (1+c)A_{S^c}\sinh\left(cR_S+\lVert D_{S^c}\rVert_\infty\right).

where

\left\\| \operatorname{arsinh}\left( \frac{\beta_S^\star e^{Q_S}}{\alpha^2} \right)+D_S \right\\|_\infty.$$ The Gain-dependent contribution and Drift are added **before taking the norm**, so their alignment directly determines whether they cancel or reinforce one another. To make this quantitative, denote $$g_S(Q_S)= \operatorname{arsinh}\left( \frac{\beta_S^\star e^{Q_S}}{\alpha^2} \right).$$ At fixed Gain and off-support Drift, consider $D_S=-\lambda g_S(Q_S)$, with $0\le\lambda\le1$. Then $$R_S=(1-\lambda)\\|g_S(Q_S)\\|_\infty,$$ and hence $$\lVert \widehat{\beta}-\beta^\star \rVert_1 \le (1+c)A_{S^c}\sinh\left(c(1-\lambda)\lVert g_S(Q_S)\rVert_\infty+\lVert D_{S^c}\rVert_\infty\right).$$ Thus, increasing the opposing Drift tightens the recovery guarantee, whereas reinforcing Drift enlarges the support mismatch. Moreover, increasing support Gain increases the magnitude of each coordinate of $g_S$. Preserving the same signed mismatch requires the compensating change $$D'_S-D_S = -\left[g_S(Q'_S)-g_S(Q_S)\right].$$ This identifies an explicit quantitative trade-off between the two contributions. Therefore, Theorem 2.4 establishes this competition at the level of the recovery guarantee. Its effect on the selected interpolator is supported separately by the explicit analysis in Section 2.2 and the fixed-Gain Drift ablation in Figure 2. ### On whether Lemmas 2.5 and 2.6 explain the emergence and transition of EoS phases or characterize Drift under assumed gradient patterns The Edge of Stability is an independent phenomena arising out of large learning rates. This is separate from Gain and Drift which are not at all responsible for triggering these phases. Rather, Gain and Drift are post-trajectory quantities that we use to draw insights regarding the "sparsity" of the learned solution. To do this, we first show the direct relation between them. Then, in order to explain their competition and highlight under which circumstances we can recover a sparse solution, we use the framework by Damian et al. (2023) that argues about two training phases provided by the Edge of Stability (EoS). EoS fits to our problem, since it concerns the gradients during training and Gain and Drift also depend on the gradient sequences. ### On the novelty of Drift and Proposition 2.3 relative to Theorem 1 and Proposition 2 of Even et al. (2023) Even et al. (2023) state that under strict conditions on the learning rate the shifted reference point $\tilde\beta_0$ can be ignored and the implicit bias of GD can be \emph{completely} represented by encoding the effect of the Gain into the potential function of the gradient flow. In this work, we investigate whether this claim is true under larger learning rates, and -- contrary to the claim by Even et al. -- find that this reference point cannot be ignored. It is needed to sufficiently describe the implicit bias and the selected solution. We highlight this distinction in Figure 8 in Appendix A.1, which we have now moved to the main section for better exposition as Fig 2 in the revision. The pink line represents the solution selected by minimizing the implicit bias potential ignoring the Drift term, as mentioned in Even et. al (2023). This only encapsulates the effects of the gain. The dotted trajectory represents the solution selected by adding the drift term to the implicit bias and accurately represents the one selected by Gradient Descent (blue line). We show that in both settings where the training reaches the edge of stability threshold or converges below it, the Drift term is necessary to accurately describe the implicit bias.

On convergence of ordinary GD in the EoS regime and the main mathematical obstruction to proving it

Establishing convergence for GD in EoS settings is beyond the scope of our work. We assume convergence, as we are primarily interested in accurately describing the implicit bias under large learning settings using post-hoc quantities and gathering insights. The main mathematical obstruction is controlling the interaction between oscillations and evolving curvature once standard descent guarantees cease to apply. For (F(u,v)=L(u⊙v))(F(u,v)=L(u\odot v)), the GD update has Jacobian (I−γ∇2F)(I-\gamma\nabla^2F), with linearized multiplier (1−γλ)(1-\gamma\lambda) along a Hessian eigenvector of eigenvalue (λ)(\lambda). When (γλ≥2)(\gamma\lambda\ge2), this multiplier has magnitude at least one, so the update is no longer locally contractive in that direction. Nevertheless, nonlinear effects can counteract this instability. The oscillations themselves can reduce curvature and restore local stability, as described by the self-stabilization mechanism of Damian et al. (2023).

On deriving the gradient behavior in Lemmas 2.5 and 2.6 from EoS dynamics and quantitatively linking it to Drift accumulation or saturation

The assumptions of Lemma 2.5 and 2.6 cannot be derived from EoS dynamics. Rather, they are simple assumptions on the signs of the gradient sequence that help to provide an upper bound of the accumulated Drift. Consecutive training steps can either have gradients that are sign-coherent or have opposite signs, thereby invoking either lemma.

On whether the phase-dependent evolution of Gain and Drift provably changes the selected interpolator

We thank the reviewer for this valuable direction. However, it is beyond the scope of the current work. Gain and Drift are fundamentally post-hoc quantities that provide novel insights into the training dynamics on their own. In our work, we extend the work of Even. et al. (2023) towards large learning rate regimes and investigate the implicit bias. We also show that due to the quadratic and cubic nature of these quantities, they either continue to accumulate or saturate during the different phases of EoS, thereby affecting the sparsity of the learned solution. This provides insights that are novel on their own. The Edge of stability framework does not provide estimates of the gradients or their signs. Intuitively, the oscillatory phase suggests gradient signs to be flipped for consecutive training iterations. We use this heuristic in Lemma 2.6 to reason about the saturation of Drift. However, to draw a formal conclusion, the evolution of gradients encountered need to be studied separately in Edge of Stability regimes first, which we reserve as an interesting direction for future works. A promising starting point is the Central Flows framework (Cohen et al., 2025), which models the time-averaged dynamics of oscillatory optimization trajectories; adapting this framework to characterize the evolution of Gain and Drift and its consequences for interpolator selection is an interesting direction for future work.

Thanks for your response. I appreciate clarifying the limitation of your analysis.

  1. I agree that Theorem 2.4 captures the possibility of algebraic cancellation between Gain-dependent and Drift-dependent terms within the recovery upper bound. However, my concern is that Theorem 2.4 itself does not establish the claimed competition between Gain and Drift in determining the selected interpolator.

  2. Thanks for clarifying that neither Lemma 2.5 nor 2.6 is meant to explain the emergence of EoS itself. But then what is the significance of these lemmas? They may be useful auxiliary estimates but given that the assumptions cannot be derived from EoS as the authors stated, it is hard to understand what the actual significance of these lemmas are.

  3. For the novelty of the Drift and the mathematical implications of Proposition 2.3 relative to Theorem 1 and Proposition 2 of Even et al., I don't think the authors directly answered to my concern. I think the contribution should be more explicitly attributed to Even et al.

Reviewer_4eYPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies implicit bias in diagonal linear networks at large learning rates. Building on Even et al. (2023), it analyzes a signed correction called Drift that can reinforce or oppose Gain’s effect on sparse recovery. It provides a recovery error bound and conditional results on Drift accumulation and cancellation. The paper also proposes Gain Control, a modified update that redistributes Gain across features. It characterizes the update’s implicit bias and proves convergence for sufficiently small learning rates.


Reference

Even et al. (S)GD over Diagonal Linear Networks: Implicit Bias, Large Stepsizes and Edge of Stability. NeurIPS 2023.

Critical strengths

None

Critical weaknesses

Overall, the paper is difficult to follow, and its contribution beyond prior work is limited.

The implicit bias potential established by Even et al. (2023) already contains a linear correction equivalent to Drift, up to a scaling factor. Proposition 2.3 therefore reformulates an existing characterization. Theorem 2.4 adds a recovery bound, but it depends on the final Gain and Drift values accumulated during training.

Lemmas 2.5 and 2.6 assume that gradients retain their signs or approximately reverse signs between consecutive steps. The paper does not establish conditions on the data, initialization, or learning rate that ensure these behaviors. Moreover, Lemma 2.6 bounds Drift by the sum of reversal errors. Small errors at individual steps do not ensure that Drift saturates. These results therefore provide limited guidance on when large-step GD improves sparse recovery.

Gain Control (GC) provides limited support for the proposed explanation of standard GD/SGD. Its convergence guarantee requires sufficiently small learning rates and does not cover the large-step regime of interest. GC also changes both Gain and Drift, but the experiments do not separate their contributions to improved recovery. It therefore remains unclear what GC reveals about standard training beyond the existing implicit bias characterization.

Questions for authors

Do the reported trajectories satisfy the assumptions of Lemmas 2.5 and 2.6? Specifically, do the gradients retain their signs during the claimed accumulation phase, and does ∥zk∥∞≤ρ<1\lVert z_k\rVert_\infty\le\rho<1 hold for a fixed ρ\rho over each analyzed interval?

Does Lemma 2.6 quantitatively explain the observed saturation of Drift during EoS? Its bound depends on the sum of gradient-reversal errors, which can grow even when each error is small. For the oscillatory interval in Figure 9, could you compare this bound with the measured change in Drift as the interval grows?

1 comment

On Proposition 2.3 being a reformulation of an existing characterization

We agree with the reviewer on this. The central goal of this paper is to study this linear characterization. In order to ease readability and referring to this quantity and to make its contribution to the implicit bias more explicit, we redefined it as a proposition. We have revised this to make its reference to the previous work clearer.

On Theorem 2.4’s recovery bound depending on the final accumulated Gain and Drift

Gain and Drift have been defined throughout as post-trajectory statistics. Similar to Even et al, (2023), we argue that even though they are post-trajectory, understanding their effects on the training dynamics reveals insights on the sparsity of the recovered solution. While Even et al. (2023) was heuristic in making this connection, we explicitly relate these quantities with the sparsity of the solution.

On what GC reveals about standard training beyond the existing implicit-bias characterization

Our main goal was to show that implicit bias and post -trajectory quantities can be used to design and maneuver the dynamics suitably. GC's update step reverses the qualitative effect of both Gain and Drift. Therefore, in contrast to the behavior of GD in similar settings, Gain now promotes sparsity while Drift opposes it. However, since the dynamics reaches the Edge of stability and oscillates around the stability threshold, the Gain keeps accumulating while the Drift approximately cancels. This drives the dynamics towards the sparse interpolator.

On whether the reported trajectories satisfy the assumptions of Lemmas 2.5 and 2.6

While the trajectory for the toy example satisfies the assumptions, it does not do so for the experiments in Fig 6. Lemmas 2.5 and 2.6 quantitatively describe the accumulation of Drift in simplified settings. We make no assumptions on trajectories during general training runs. Here, the lemmas provide a conservative upper bound. During a training run, Drift may or may not monotonically increase during the progressive sharpening phase or perfectly cancel out during the oscillatory phase. The lemmas are a first step in understanding the accumulation of Drift, which we associate with the two phases of training under large learning rates.

On whether Lemma 2.6 quantitatively explains Drift saturation and how its bound compares with the measured Drift change in Figure 9

Lemma 2.6 bounds the drift increment in the oscillatory phase. This, combined with the overall shrinkage of gradients, qualitatively explains the observed saturation of Drift during EoS. A more formal statement would need to bound the overall cumulative error ∑l=0m−1∣∣el∣∣∞\sum_{l=0}^{m-1} ||e_l||_\infty, and establish convergence of the drift increment sequence. This would require some additional assumptions on the gradient estimates and the overall trajectory. We compare the upper bound in Lemma 2.6 with the measured change in Drift in Fig 3, where we show the bound's quantitative tightness in the oscillatory interval.

Comments on the paper

1 comment

We thank all reviewers for their valuable feedback. In the following, we would like to highlight our contributions and summarize the rebuttal

  1. Role of early training in solution selection: We establish the importance of Drift—an algebraic term previously identified by Even et al. (2023)—in describing GD’s implicit bias in two-layer diagonal linear networks at large learning rates. Drift accumulates substantially during early training, developing a directional bias that can favour sparse solutions. This early bias persists throughout training and helps determine the final solution’s properties. Figure 2 demonstrates why Gain alone is insufficient to explain solution selection, while Thm. 2.4 and Sec. 2.2 characterize how Gain and Drift jointly influence sparsity. Rebuttal: We agree with explicitly crediting Even et al. for the algebraic term; reviewers acknowledge the novelty of our analysis of its previously unexplored role.
  2. Connection to the Edge of Stability: Drift can plateau during the oscillatory phase of EoS while Gain continues to accumulate, shifting the bias towards denser solutions. Approximate gradient reversals cause the leading cubic Drift increments to cancel, while quadratic Gain increments reinforce each other. Under their respective assumptions, Lemma 2.5 bounds Drift accumulation during sign-coherent dynamics, while Lemma 2.6 quantifies cancellation under approximate gradient reversals, supporting a plateau when accumulated reversal errors remain small. Figures 1 and 5 provide evidence consistent with this mechanism and its effect on sparsity. Rebuttal: We clarify the lemmas’ assumptions and their role in formalizing the observed connection to EoS.
  3. Gain Control as an implicit bias-first optimiser: We further propose an update step inspired from the behaviour of these post hoc quantities. Gain Control achieves better performance on simple linear regression problems in recovering sparse solutions than (S)GD. We additionally prove convergence (Thm 3.4). Overall, we introduce a first step in drawing actionable insights from implicit bias, and motivating optimiser design geared towards recovering desirable solution properties. Rebuttal: The reviewers acknowledge that this is an interesting concept.
  4. Practical Relevance: Our experiments with DeiT-Small trained on ImageNet for 300 epochs using Muon (an increasingly popular modern optimiser) provide preliminary evidence of the persistence of implicit bias developed by the Drift beyond diagonal linear networks. Across three learning-rate and batch-size configurations, the analogous Drift quantity tracked for the query–key pairs accumulates substantially early in training grows more in magnitude relative to the Gain (Fig. 7). The configuration with the largest final Drift also exhibits the lowest aggregate query–key effective rank (Table 2), indicating greater concentration of spectral mass. These observations suggest that early Drift accumulation may remain relevant to the structure of learned representations in practical architectures

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.