acceptodds
Under review as a conference paper at ICLR 2027

Mirror Descent at the Edge of Stability

24 views

Abstract

In many deep learning settings, gradient descent has been shown to operate at the edge of stability – a regime where the largest eigenvalue of the loss Hessian hovers around . We study this phenomenon in mirror descent, a generalization of gradient descent. As mirror descent is non-Euclidean, existing explanations of the edge of stability phase do not extend to this setting. We generalize the existing notion of sharpness to introduce mirror sharpness, which measures the curvature of the loss with respect to the mirror potential. Empirically, we observe that the mirror sharpness hovers slightly above the threshold. Furthermore, we derive a dual central flow ODE for mirror descent that models iterates in continuous time at the onset of edge of stability. Our ODE provides insights on the difficulties of extending existing arguments for why sharpness stabilizes at from gradient descent to mirror descent in full generality.

Primary area:
optimization
Keywords:
edge of stability, non-convex optimization, mirror descent, central flows

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eRWiOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
3
Read review
Paper summary

The manuscripts extends the quantities involved in the edge of stability regime for gradient descent to mirror descent such as sharpness and a central flow description. The work empirically verifies that the proposed mirror sharpness is reached by standard training and hovers around it. Moreover, it is highlighted that the self stabilization analog may not extend to mirror descent.

Critical strengths

The empirical verification of reaching the corresponding stability threshold is useful as it corroborates the introduced EoS extensions. This could be valuable for practitioners to take this into account when using mirror descent for controlling implicit regularization.

Critical weaknesses
  1. The main weakness of the current manuscript is that most statements are direct extensions of the edge of stability for gradient descent. This significantly limits both the conceptual and technical novelty of the work. The descent lemma, definition of sharpness and reparameterization connection follow from the connection between mirror descent and Riemannian gradient descent (see questions). Moreover, Lemma 10 for stabilization requires additional guidance as the experiments do not illustrate the finding of beta having the possibility of being negative.

  2. The relevance of the work is limited as mirror descent is not the default optimizer in deep learning. For instance, the work does not provide guidance on how the EoS regime affects the resulting implicit regularization. Furthermore, no use-case is given or studied for mirror descent in the considered experimental deep learning setups.

The work is promising but in its current form does not meet the acceptance bar of ICLR. I believe it requires both improved theoretical analysis and repositioning with respect to literature on why mirror descent is relevant to study.

Questions for authors

Can the authors construct an example for Lemma 10 such that beta is negative?

Previous work has established connections between reparameterizations, mirror flow, and Riemannian gradient flow as discussed in Section 3 of the manuscript. Can the authors discuss the relationship to their work:

  1. Ehsan Amid, Manfred K. Warmuth "Reparameterizing Mirror Descent as Gradient Descent" (2020)
  2. Li, Zhiyuan et al. “Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent.” (NeurIPS 2022)

Discussing related work on implicit bias of mirror descent that established max-margin results for the classification setting covering the hyperbolic entropy and going beyond the power mirror maps would strengthen the manuscript and is more related to experimental setup than regression:

  1. Jacobs, Tom and Guido Montúfar. “Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning.” (2026)
  2. Pesme, Scott et al. “Implicit Bias of Mirror Flow on Separable Data.” (NeurIPS 2024)
5 comments

We thank the reviewer for their comments, and we would like to address their concerns below.


Weaknesses

W1a: “The main weakness of the current manuscript is that most statements are direct extensions of the edge of stability for gradient descent. This significantly limits both the conceptual and technical novelty of the work. The descent lemma, definition of sharpness and reparameterization connection follow from the connection between mirror descent and Riemannian gradient descent.”

We respectfully disagree with these points from the reviewer. First, as gradient descent is a special case of mirror descent, any result for mirror descent must recover prior results; thus, we do not believe this fact in itself to be a weakness that limits the novelty of this work. We still believe our work to be novel and significant, as extending to the general case of mirror descent reveals fundamental differences in how we must treat the EoS phase (e.g., Lemma 10) Additionally, this work broadens our understanding of where EoS may take place and what it looks like (as Reviewer zPTY states) while Reviewer Tzt8 finds our notion of sharpness to be new and useful.

We also do not believe the results described to follow easily from the connection between mirror descent and Riemannian gradient descent. See our response to Q2 for a more in-depth elaboration on this point. Furthermore, while we use the literature on Riemannian gradient descent to motivate why our definition of sharpness is sensible, we do not believe this connection itself to be completely trivial. A priori, to the authors, there were other notions of sharpness that appear sensible, such as the ℓ2\ell_2 sharpness in the dual space since updates are linear in the dual space.

W1b: “Lemma 10 for stabilization requires additional guidance as the experiments do not illustrate the finding of beta having the possibility of being negative”

We agree that determining if realistic optimization scenarios can have β<0\beta < 0 is a very important direction for future work, as we have further elaborated on in the response to Question 1. However, we do not believe that this limits its significance, as it is a theoretical negative result that we believe to be of interest to the optimization community, as the result implies that deriving a self-stabilization result using the current central flows framework would be difficult without extra assumptions.

W2a: “The relevance of the work is limited as mirror descent is not the default optimizer in deep learning”

We agree that mirror descent is not the default optimizer in deep learning; however, we do not believe that to limit the relevance of this work. Our motivation is that mirror descent is a relatively simple and natural setting in which we can test which aspects of the self-stabilization theory depend on Euclidean (and non-adaptive) geometry. The vast majority of works along this line study full-batch gradient descent extensively (see [CKLKT21] and [DNL23]) as we ought to better understand the simpler cases before tackling more complex adaptive optimizers.

W2b: “For instance, the work does not provide guidance on how the EoS regime affects the resulting implicit regularization.”

We took this question to mean how EoS may affect regularization in the context of generalization (as EoS itself is a form of implicit bias as we show in this paper); please let us know if this was not the intent. It is not well-understood how different notions of sharpness are connected to generalization for realistic data distribution — see [ACMHF23]. We do not tackle this issue in the work because we believe a better understanding needs to be reached before considering how mirror sharpness can factor into this.

W2c: “Furthermore, no use-case is given or studied for mirror descent in the considered experimental deep learning setups.”

We agree that more “experimental” deep learning setups may be interesting to consider as follow-up work; however, we see this paper’s main contribution as being a theoretical one.

W3: I believe it requires both improved theoretical analysis and repositioning with respect to literature on why mirror descent is relevant to study.

We have updated the manuscript in Lines 49-55 with some more justification on why mirror descent is important to study.

Questions

Q1: Can the authors construct an example for Lemma 10 such that beta is negative?

We are somewhat unsure how to interpret this question: if the question is asking for a theoretical example, yes. Our proof for Lemma 10 is a constructive one that shows for any mirror map Φ\Phi, we can construct a loss such that βΦ<0\beta^\Phi < 0 at a certain point. If the reviewer is referring to if we can construct an example of a successful optimization trajectory using mirror descent where βΦ<0\beta^\Phi < 0, we have outlined this as future work in line 485, and discuss this matter below in greater detail.

We have not been able to find such a trajectory empirically; as we show in Appendix G, all of our examples of edge of stability consistently display βΦ>0\beta^\Phi > 0. One may, instead, ask if it is possible to construct an explicit counterexample. In the simplest case where LL is a quadratic, the answer is no, since all examples of mirror descent reduce to gradient descent in this case as we show in Theorem 4, so we know that βΦ≥0\beta^\Phi \geq 0. Thus, we would need to turn to losses with non-zero third derivatives. In general, it is not feasible to explicitly write out such an optimization trajectory, especially in the discrete setting, and thus we believe that reasoning about them would require substantially new understandings.

Q2: Can the authors discuss the relationship to their work: [AW20] Ehsan Amid, Manfred K. Warmuth "Reparameterizing Mirror Descent as Gradient Descent" (2020) [LWLA22] Li, Zhiyuan et al. “Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent.” (NeurIPS 2022)

We do not see our work as having any particularly significant connections to these works beyond studying some notion of reparameterizations of mirror descent/flow.

The most fundamental difference is that [AW20] and [LWLA22] both study when continuous mirror flow can be reparameterized into gradient descent, whereas our result in Section 3 shows that for a quadratic loss function, the discrete update can be reparameterized. In fact, [AW20, Section 5] explicitly claims, in their conclusion, that their theorem “holds only in the continuous-time and the equivalence relation breaks down after discretization.”

Furthermore, our primary goal in section 3 is not to make novel connections between reparameterizations of mirror descent, but to aid the reader in understanding why our definition of mirror sharpness is sensible and highlight the effect of third-order terms. It is precisely because we cannot find this result in the literature, as we mention in the text [lines 192-193], that we felt it important to state and prove Theorem 4 from scratch, which identifies a simple reparameterization for discrete mirror descent.

Thus, we believe that including a comparison to [AW20] and [LWLA22] would detract from the clarity of section 3. If the reviewer sees a way Theorem 4 directly follows from these works, as suggested in Weakness 1, we would be happy to cite and adapt.

Q3: Discussing related work on implicit bias of mirror descent that established max-margin results for the classification setting covering the hyperbolic entropy and going beyond the power mirror maps would strengthen the manuscript and is more related to experimental setup than regression: Jacobs, Tom and Guido Montúfar. “Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning.” (2026) Pesme, Scott et al. “Implicit Bias of Mirror Flow on Separable Data.” (NeurIPS 2024)

We thank the reviewer for these references and have incorporated them into a revision of the related works section.

References

[ACMHF23]: Maksym Andriushchenko, Francesco Croce, Maximilian Muller, Matthias Hein, and Nicolas Flammarion. A Modern Look at the Relationship between Sharpness and Generalization. ICML 2023.

[CKLKT21]: Jeremy Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. ICLR 2021.

[DNL23]: Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability. ICLR 2022.

Thank you for the detailed response. I appreciate the additional motivation and the clarification that Theorem 4 concerns a discrete-time reparameterization. I agree that the empirical observations are interesting. Nevertheless, I think the theoretical results relatively direct, and I am not yet convinced that the theoretical depth or demonstrated implications are sufficient.

To clarify the implicit regularization question, I would like to know how operating at EoS interacts with the parameter-space bias induced by the mirror map, not generalization. For instance, hyperbolic entropy is used to promote sparsity [1], does entering the EoS regime preserve or change this behavior? A brief discussion of this interaction would help establish the implications of the proposed framework. Moreover, it would also help address the practical relevance of the framework, connecting to Reviewer Tzt8’s follow-up comment.

For Q1, I understand the construction in Lemma 10. My concern still remains if this obstruction can arise along an actual mirror descent trajectory near EoS as the proof is by construction. An example, potentially using a regression objective that is nonquadratic in the parameters, would strengthen this connection. Alternatively, a clearer discussion in the revised manuscript of what the pointwise construction does and does not imply for actual trajectories would help delimit the result.

I keep my score for now.

[1] Jacobs et al., Hyperbolic aware minimization: Implicit bias for sparsity. ICLR 2026.

Reviewer_Tzt8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The edge-of-stability phenomenon is well-known for gradient descent but much less understood for mirror descent. This paper fills that gap with theoretical and empirical studies. The authors first define mirror sharpness as the largest generalized eigenvalue of the loss Hessian w.r.t. the mirror potential Hessian, and show experimentally that it drifts up to 2/η2/\eta and oscillates around it. They then adapt the central flow framework to the dual coordinates of MD and show that the resulting flow can track the observed trajectories better than a conventional Riemannian gradient flow. Finally, they identify an additional geometry-dependent term in the stabilization coefficient and construct examples where this coefficient is negative, showing a limitation of directly extending the usual Euclidean self-stabilization argument to general MD.

Critical strengths
  1. The observation that mirror sharpness tracks 2/η2/\eta whereas ordinary Euclidean sharpness need not is an interesting empirical finding. The experiments make a persuasive case that this is not merely the standard Euclidean EoS phenomenon plotted in different coordinates.
  2. The new sharpness notion (Definition 1) is useful and a natural extension. (Although this should be no surprising for those familiar with mirror descent, because it is closely related to [1])
  3. The authors give a obstruction to the standard self-stabilization explanation by Lemma 10. Such negative evidence is a useful contribution and in my view, the most interesting theoretical contribution. It clarifies why the existing self-stabilization argument for GD cannot automatically be transferred to arbitrary mirror geometries.

[1] Relatively-Smooth Convex Optimization by First-Order Methods, and Applications. Haihao Lu, Robert M. Freund, and Yurii Nesterov. SIAM Journal on Optimization, Vol. 28, Iss. 1 (2018)

Critical weaknesses
  1. As mentioned earlier, Definition 1 is closely related to the Hessian characterization of relative smoothness [1], as well as existing preconditioned/generalized sharpness notions. The paper distinguishes MD from fixed geometry steepest descent, but this connection to relative smoothness should be acknowledged and the novelty stated more precisely. Similarly, the dual central flow largely adapts the modeling strategy of Cohen et al. (2024). Because the condition SΦ=2/ηS^\Phi=2/\eta is imposed in the construction, the fact that the modeled flow remains near this threshold does not independently explain why the actual MD iterates reach and maintain EoS. The new geometry-dependent stabilization term is valuable, but the work stops short of a predictive theory of the EoS mechanism.
  2. The scope of the validation and the consistency of the numerical description leave some uncertainty. The experiments primarily concern full-batch training of relatively small networks on a small CIFAR-10 subset, using coordinatewise nonquadratic potentials. This is a reasonable first testbed and similar setups are adopted by well-known EoS papers. However, it does not cover some important applications of MD, particularly entropy-based MD on the probability simplex, nor stochastic training (present in Cohen et al. (2020)). I would originally expect some practical applications of MD in deep learning, such as policy optimization in RL [2], or language modeling experiments [3]. The current experiments are limited to vision tasks. (again, I am aware that this is common for EoS papers but I expect more)
  3. This is not necessarily a weakness, but the central flow validation would also be more convincing if it directly tested the predicted oscillation direction and variance, rather than only the sharpness and trajectory distance. Also, the equation in lines 763-765 appears to omit a factor of η\eta.

[2] Mirror Descent Policy Optimization. Manan Tomar, Lior Shani, Yonathan Efroni, Mohammad Ghavamzadeh. ICLR 2022

[3] DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, Adams Wei Yu. NeurIPS 2023

Questions for authors
  1. Does lines 758-760 require βΦ>0\beta^\Phi>0? In Lemma 10 β\beta could be negative.
  2. Could the authors compare the predicted unstable direction and oscillation variance with measurements from the actual MD iterates? This would help distinguish a predictive model of MD dynamics from a flow that tracks the EoS threshold partly because the threshold is enforced by construction.
5 comments

We thank the reviewer for their comments, and we would like to address their concerns below.


Weaknesses

W1a: As mentioned earlier, Definition 1 is closely related to the Hessian characterization of relative smoothness [1], as well as existing preconditioned/generalized sharpness notions. The paper distinguishes MD from fixed geometry steepest descent, but this connection to relative smoothness should be acknowledged and the novelty stated more precisely.

We appreciate the reviewer’s observations that our definition of mirror sharpness naturally extends prior notions of preconditioned/generalized sharpness. We have already discussed such connections in Lines 156-165 and Lines 218-226—please let us know if this feels insufficient.

We agree that our work builds on the relatively new viewpoint of relative smoothness in terms of how we think about the Hessian and its geometry; we have included a revision citing connections to the closely related [1]. One main difference, however, is that we are interested in exploring sharpness, a local quantity, rather than a global quantity in this paper.

W1b: “the modeled flow remains near this threshold does not independently explain why the actual MD iterates reach and maintain EoS”

Yes, we agree that such an explanation of why iterates even reaches EoS is highly desirable. This problem is highly difficult and not much is known even in the simplest setting of full-batch gradient descent, and, to our knowledge, all works studying the EoS phase in detail (e.g., [DNL23], [CDTKL25]) make this assumption—in particular, [CDTKL25] also enforces SΦ=2/ηS^\Phi = 2/\eta in their construction. We use the central flow framework to show that it is currently insufficient in fully explaining this self-stabilization behavior (i.e., that EoS is maintained), rather than explain why MD iterates reach EoS.

W2a: “The scope of the validation and the consistency of the numerical description leave some uncertainty” “The current experiments are limited to vision tasks.”

Thank you for raising this point—we agree with it to some extent. As the reviewer notes, in this paper, our experimental setup is consistent in scale and setup with well-known EoS papers. We acknowledge that such experiments are relatively small because computing Hessians are extremely expensive, but we note that our experiments are of similar scale to those in the seminal works [CKLKT21] and [CDTKL25].

Since these previous works have already shown that EoS manifests across a diverse set of problem tasks, we felt it important to present a single problem task to more easily compare differences between the mirror maps we selected. We believe our experiments on this front are already quite robust (as reviewer zPTY points out), spanning over two losses (MSE or cross-entropy), three architectures (MLP, CNN, vision transformer), four mirror maps, and three learning rates for each possible configuration of the above.

In our revision, we have included Appendix H detailing an additional experiment using a character prediction task based on the Tiny Shakespeare dataset, similar to [ICCG26]. We also observe similar EoS behavior in our experiments on this character prediction task.

Weaknesses (cont.)

W2b: “However, it does not cover some important applications of MD, particularly entropy-based MD on the probability simplex, nor stochastic training (present in Cohen et al. (2020)). I would originally expect some practical applications of MD in deep learning, such as policy optimization in RL [2], or language modeling experiments [3].”

We thank the reviewer for these interesting directions. We believe these topics to be out of scope of our paper, which studies full-batch mirror descent on Rn\mathbb{R}^n; however, such topics are significant and could be the subject of possible extensions. Below, we explain why we believe that highly nontrivial work is required.

  1. Entropy-based MD. We do not currently know of any problems that exhibit progressive sharpening where the parameters must be constrained to the simplex; thus, we are unable to study the EoS phase empirically in entropy-based mirror descent. Understanding what kinds of non-convex problems manifest EoS is a very important and active area of research, but our focus on this paper is on how the optimizer affects the EoS stage.

A finer point is that we are not sure what the right definition for mirror sharpness is when our parameter space is the simplex. Consider our definition of mirror sharpness over Rn\mathbb{R}^n:

SΦ=max⁡∥u∥G−1=1⟨u,Hu⟩.S^{\Phi} = \max_{\|u\|_{G^{-1}} = 1} \langle u, Hu \rangle.

In the Euclidean case, we identify uu with the direction of the unstable oscillations, and keeping u∈Rnu \in \mathbb{R}^n remains sensible in the non-Euclidean case, as our experiments show. Thus, it seems very tempting to enforce that uu must be a tangent vector with respect to the simplex in the case of entropy. However, we believe that such an extension would require more additional work to be justified because it is still unclear what this uu represents in non-Euclidean settings, as is suggested in [ICCG26, Appendix C].

  1. Stochastic settings. In our paper, we find that the extension of prior arguments to mirror descent already have unresolved challenges. Although we find the topic of stochastic mirror descent important, we believe that a better understanding of the simpler cases must be obtained before thinking about how stochasticity interacts with non-Euclidean geometries.

  2. Practical examples. Regarding the specific practical examples the reviewer has pointed out:

The paper [2] discusses solving an optimization problem to approximate a mirror descent step, rather than applying an exact mirror descent step. We believe it to be an interesting problem whether or not these approximate steps also have an associated EoS phenomena; however, as we have shown that the characterization of EoS is heavily dependent on the geometry of the optimizer, we believe it would require additional work to answer.

To our best understanding, it seems that the objective in Equation 1 of [3] is a min-max problem over parameters, where the ones on a probability simplex are updated using a mirror descent step. See our response to item 1 about entropy. We note that such a problem (i.e., maximizing over α\alpha) is linear over α\alpha and thus, as an intuitive guess, we do not expect to see any EoS behavior in this setup. Furthermore, it is also an interesting question whether the min-max problem setting in general also exhibits EoS (e.g., a concurrent submission to ICLR 2027 discusses this).

We hope that our above points clarify why we believe these directions to be substantial undertakings that are out of scope of this paper.

W3a: “the central flow validation would also be more convincing if it directly tested the predicted oscillation direction and variance”

See our response to Q2.

W3b: “Also, the equation in lines 763-765 appears to omit a factor of η\eta.”

Thank you for pointing out this typo. We have revised this and included the factor of η\eta that was dropped.

Questions

Q1: Does lines 758-760 require β>0\beta > 0? In Lemma 10 could β\beta be negative.

Thank you for the question—we appreciate your close attention to details on the derivations of the central flow. Yes, this is correct and we have stated this explicitly.

Q2: Could the authors compare the predicted unstable direction and oscillation variance with measurements from the actual MD iterates? This would help distinguish a predictive model of MD dynamics from a flow that tracks the EoS threshold partly because the threshold is enforced by construction.

We are happy to provide additional experimental results. We have plotted the difference between the actual mirror descent iterates and our central flow model projected onto the predicted unstable direction, as well as compared our predicted variance (from the dual central flow model) σ2\sigma^2 with that of the empirical values (from actual mirror descent) for the experiment shown in the main text on an ℓ1.5\ell_{1.5} potential with a CNN (Figures 3, 22). See https://anonymous.4open.science/r/supplemental-mirror-descent-at-the-edge-of-stability-C6F9/additional-experimental-infl.png.

References

[1]: Relatively-Smooth Convex Optimization by First-Order Methods, and Applications. Haihao Lu, Robert M. Freund, and Yurii Nesterov. SIAM Journal on Optimization, Vol. 28, Iss. 1 (2018).

[2]: Mirror Descent Policy Optimization. Manan Tomar, Lior Shani, Yonathan Efroni, Mohammad Ghavamzadeh. ICLR 2022.

[3]: DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma, Adams Wei Yu. NeurIPS 2023.

[CDTKL25]: Jeremy Cohen, Alex Damian, Ameet Talwalkar, J. Zico Kolter, and Jason D. Lee. Understanding Optimization in Deep Learning with Central Flows. ICLR 2025.

[CKLKT21]: Jeremy Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. ICLR 2021.

[DNL23]: Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability. ICLR 2022.

[ICCG26]: Rustem Islamov, Michael Crawshaw, Jeremy Cohen, and Robert M. Gower. Non-Euclidean Gradient Descent Operates at the Edge of Stability. ICLR 2026.

[MKWDG24]: Aaron Mishkin, Ahmed Khaled, Yuanhao Wang, Aaron Defazio, Robert M. Gower. Directional Smoothness and Gradient Methods: Convergence and Adaptivity. NeurIPS 2024.

Reviewer_zPTYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors extend the notion of sharpness from Euclidean gradient descent to mirror descent, in which setting the geometry is determined by the Hessian of the mirror potential and the sharpness is given by the largest generalized eigenvalue of the loss Hessian with respect to this mirror potential Hessian. They then demonstrate empirically that the mirror sharpness self-stablises around 2/η2/\eta when training neural networks with mirror descent. Finally, the authors derive a generalisation of the central flow due to Cohen et al in the mirror descent setting; the authors observe that the central flow is a superior approximation to mirror descent than the mirror Riemannian gradient flow, and caution that the central flow is not sufficient ground for the self-stablisation observed empirically, unlike in the Euclidean case.

Critical strengths
  1. The derivations and experiments are, to my knowledge, novel, and indicate robust edge of stability phenomena in mirror descent. They will be useful in obtaining a more general framing of where edge of stability phenomena may be expected and what form they should take.
  2. The observation of the insufficiency of central flow to derive self-stabilization in the mirror descent setting is a surprising negative result. I believe this will be an important negative result for future theory to build on and as such is of interest to the community.
Critical weaknesses

None

Questions for authors

While the mirror descent considered by the authors provides valuable further observations and theorising on edge of stability, the consideration of mirror descent is never motivated. I would appreciate a brief couple of sentences addressing why mirror descent is a setting of interest. The paper itself would also benefit from a brief motivation of this kind.

2 comments

We thank the reviewer for their comments, and we would like to address their concern below.


Questions

Q1: “The consideration of mirror descent is never motivated. I would appreciate a brief couple of sentences addressing why mirror descent is a setting of interest.”

Thank you for this suggestion—we have revised the introduction including some high-level motivations on mirror descent and added the following paragraph in Lines 49-55 of our revision:

We believe the study of mirror descent to be important for two main reasons. First, we view understanding this regime to be a first step in building towards a theory that explains the \emph{mechanisms} of EoS in adaptively preconditioned optimizers (such as those in [CGKAM+23]). While these optimizers differ from mirror descent, historically, the developments of several adaptive preconditioning methods like AdaGrad [DHS11] and Shampoo [GKS18] are based on the lens of (online) mirror descent. Second, mirror descent is a relatively tractable testbed for understanding fundamental differences in the EoS phenomenon between Euclidean and non-Euclidean regimes, as the Bregman geometry is well-structured and the update is entirely explicit.

Please let us know if you had any other points in mind---we would be happy to include them.

References

[CGKAM+23]: Jeremy Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, Zachary Nado, George E. Dahl, and Justin Gilmer. Adaptive Gradient Methods at the Edge of Stability. In NeurIPS 2023 Workshop Heavy Tails in Machine Learning.

[DHS11]: John Duchi, Elad Hazan, and Yoram Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. JMLR 2011.

[GKS18]: Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned Stochastic Tensor Optimization. ICLR 2018.

Thank you for your changes. I think they add nice motivation.

I have read the other reviews and retain the view that the paper should be accepted. My score remains a 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.