- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper extends the analysis of the implicit bias of gradient descent in diagonal linear networks developed by Even et al. (2023). In particular, it revisits the shifted reference point appearing in the Bregman-divergence characterization of the limiting solution. While Even et al. argued that this shift could be approximately ignored, the authors show that its contribution to the implicit regularizer can remain significant. They introduce the notion of Drift to characterize this signed correction and study its interaction with the previously identified Gain. They also derive a sparse-recovery bound, investigate the evolution of these quantities during training, and introduce a modified optimization method, Gain Control (GC), intended to favor sparse solutions.
Critical strengths
- The main contribution is showing that the Drift term, which Even et al. treated as a perturbation, can significantly affect the implicit bias at large learning rates. In particular, the paper highlights a limitation of the previous argument: showing that the shifted reference point is small is not sufficient to conclude that its contribution to the implicit regularizer is negligible.
- The qualitative study of the interaction between Drift and Gain is interesting. In particular, the analysis of Drift accumulation under consistent gradient signs and its cancellation under approximately alternating gradients provides useful intuition about the training dynamics.
- The proposed Gain Control update is an interesting attempt to exploit these insights to influence the implicit bias.
Critical weaknesses
-
The contributions appear somewhat incremental, and the relationship with previous work is not always described precisely. For example, at line 182, the authors state that Even et al. "do not further analyze the shift in ." However, Even et al. explicitly introduce the corresponding signed correction in their time-varying potential (Eq. 7 and Proposition 2) and discuss why the shifted reference point might be neglected. This discussion relies on Proposition 16, which establishes under its assumptions, and Proposition 19, which shows that as . The present paper provides a useful refinement of that interpretation, but the underlying correction was already part of the previous characterization.
-
The practical relevance of the Drift mechanism in the large-learning-rate regime is not entirely clear to me. The authors observe that Drift approximately saturates once the training dynamics enter the EoS regime, while Gain continues accumulating (Figures 3, 5, and 9). This suggests that the relative contribution of Drift to the implicit regularizer may become negligible compared with Gain, potentially recovering the picture suggested by Even et al. While Figure 8 illustrates that Drift can still affect solution selection in a particular example, it is unclear how significant this effect remains for larger learning rates and longer training. Furthermore, in the three-coordinate experiment, the larger learning rate leads to a less sparse solution. Similarly, in the transformer experiment, increasing the learning rate at fixed batch size leads to lower validation accuracy. These observations raise questions about the practical relevance of the proposed mechanism in the aggressive-learning-rate regime that motivates the paper.
-
While Theorem 2.4 provides a sparse-recovery bound, it depends on the final effective scales and Drift, which are themselves determined by the optimization trajectory. It is therefore unclear to what extent this result predicts the effect of the learning rate on recovery or identifies settings in which Drift successfully compensates for Gain. More generally, the analysis of the training phases relies on assumptions about gradient signs, rather than establishing when these patterns arise.
-
The regression experiments are relatively narrow. A more extensive comparison with GD and appropriately tuned SGD would help establish the benefits of Gain Control. The practical relevance of the transformer experiments is also unclear, particularly because the measurements are obtained with Muon rather than the GD dynamics analyzed theoretically.
-
Minor comments:
-The discussion following Theorem 2.4 could be expanded. In its current form, the interpretation is difficult without consulting the appendix. In particular, it would be helpful to clarify which assumptions would make the bound informative about learning-rate-dependent sparse recovery.
- The related work should discuss Nacson et al., Implicit Bias of the Step Size in Linear Diagonal Neural Networks (ICML 2022). This work already shows that large learning rates can induce sparsity-promoting behavior for GD on non-centered data.
Questions for authors
-
Could the authors extend the Gain–Drift analysis to SGD? This would be particularly interesting given the differences between GD and SGD established by Even et al.
-
Could the authors comment further on the assumption at line 261 that "the predictor gradients maintain a consistent sign in each coordinate"? Under which conditions does this hold in practice? What can be said about Drift accumulation when gradient signs change?
-
Since Drift appears to saturate while Gain continues to accumulate in the EoS regime, could the authors clarify under which conditions Drift remains significant relative to Gain in determining the final implicit bias? In particular, could the authors provide evidence that this effect remains important at large learning rates, beyond the specific toy example? This seems important for understanding the practical relevance of the proposed analysis.
2 comments
I thank the authors for their reply. After reading the other reviews and the authors’ rebuttal, I have decided to maintain my overall rating.
On extending the Gain–Drift analysis to SGD in light of the GD–SGD differences established by Even et al.
We thank the reviewer for this suggestion. We agree that understanding how stochasticity affects the competition between Gain and Drift is an interesting direction, particularly in light of the differences identified by Even et al. (2023).
A natural starting point is to examine conditional moments of the stochastic predictor gradient. Write Gk,i=gk,i+ξk,i, where gk,i=∇iL(βk) and Ek[ξk,i]=0, with expectations conditional on the training history. In the regime ∣γkGk,i∣≤ρ<1, the Drift expansion gives
Ek[ΔDk,i]=32γk3Ek[Gk,i3]+Oρ!(γk5Ek[∣Gk,i∣5]),
where
Ek[Gk,i3]=gk,i3+3gk,iEk[ξk,i2]+Ek[ξk,i3].
Thus, expected Drift depends on both the noise variance and its third centered moment, whereas Gain depends at leading order on the second moment of the stochastic gradient. In particular, unbiasedness alone does not imply that stochasticity leaves expected Drift unchanged.
This calculation provides a concrete starting point for an SGD extension. Establishing how these contributions accumulate or cancel over training would additionally require controlling the evolving conditional moments and fluctuations around their conditional expectations. We therefore leave this stochastic extension of the phase-dependent Gain–Drift analysis to future work, while theoretically focusing the present study on GD.
On when predictor gradients maintain coordinatewise sign coherence and how sign changes affect Drift accumulation
The Drift increment is cubic to leading order in the product of the learning rate and the loss gradient: d(z)=32z3+O(z5),z=γ∇L(β). More generally, d(z) is an odd function. Within the regime ∥zk∥∞≤ρ<1, consecutive iterations with the same sign in the gradient of the loss therefore produce Drift increments that accumulate coordinatewise, whereas opposite-sign gradients produce increments that partially cancel, with exact cancellation when their magnitudes are equal. Lemma 2.5 quantifies accumulation during a sign-coherent phase. Lemma 2.6 quantifies cancellation when consecutive gradients approximately reverse, bounding the remaining Drift increment in terms of the mismatch between the two steps. Thus, near period-two oscillations, the additional Drift can be small even when the individual gradients remain substantial. In contrast, the Gain increment is even and nonnegative in this regime, so the contributions from the two steps add up rather than cancel. This explains how oscillatory training can suppress further Drift accumulation while allowing Gain to continue accumulating.
On when accumulated Drift remains significant relative to Gain in the final implicit bias, and the evidence at large learning rates beyond the toy example
Saturation of Drift means that its subsequent increments become small. It does not remove the Drift already accumulated. Under the convergence assumptions of Proposition 2.3, the final interpolator minimizes $ \psi_{a_\infty}(\beta)+\frac12\langle D_\infty,\beta\rangle $ Consequently, Drift accumulated during the initial phase (i.e. before or during the onset of EoS) remains a linear contribution to the final implicit regularizer, even when later oscillations largely cancel its additional increments. If the initially accumulated Drift is significant and suitably aligned with the final solution, its contribution remains significant even though later drift increments are small. Furthermore, our experiments with DeiT-Small trained on ImageNet for 300 epochs using Muon provide preliminary evidence of this persistence beyond diagonal linear networks. Across three learning-rate and batch-size configurations, the analogous Drift quantity tracked for the query–key pairs accumulates substantially early in training grows more in magnitude relative to the Gain (Fig. 7). The configuration with the largest final Drift also exhibits the lowest aggregate query–key effective rank (Table 2), indicating greater concentration of spectral mass. These observations suggest that early Drift accumulation may remain relevant to the structure of learned representations in practical architectures.