- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The manuscripts extends the quantities involved in the edge of stability regime for gradient descent to mirror descent such as sharpness and a central flow description. The work empirically verifies that the proposed mirror sharpness is reached by standard training and hovers around it. Moreover, it is highlighted that the self stabilization analog may not extend to mirror descent.
Critical strengths
The empirical verification of reaching the corresponding stability threshold is useful as it corroborates the introduced EoS extensions. This could be valuable for practitioners to take this into account when using mirror descent for controlling implicit regularization.
Critical weaknesses
-
The main weakness of the current manuscript is that most statements are direct extensions of the edge of stability for gradient descent. This significantly limits both the conceptual and technical novelty of the work. The descent lemma, definition of sharpness and reparameterization connection follow from the connection between mirror descent and Riemannian gradient descent (see questions). Moreover, Lemma 10 for stabilization requires additional guidance as the experiments do not illustrate the finding of beta having the possibility of being negative.
-
The relevance of the work is limited as mirror descent is not the default optimizer in deep learning. For instance, the work does not provide guidance on how the EoS regime affects the resulting implicit regularization. Furthermore, no use-case is given or studied for mirror descent in the considered experimental deep learning setups.
The work is promising but in its current form does not meet the acceptance bar of ICLR. I believe it requires both improved theoretical analysis and repositioning with respect to literature on why mirror descent is relevant to study.
Questions for authors
Can the authors construct an example for Lemma 10 such that beta is negative?
Previous work has established connections between reparameterizations, mirror flow, and Riemannian gradient flow as discussed in Section 3 of the manuscript. Can the authors discuss the relationship to their work:
- Ehsan Amid, Manfred K. Warmuth "Reparameterizing Mirror Descent as Gradient Descent" (2020)
- Li, Zhiyuan et al. “Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent.” (NeurIPS 2022)
Discussing related work on implicit bias of mirror descent that established max-margin results for the classification setting covering the hyperbolic entropy and going beyond the power mirror maps would strengthen the manuscript and is more related to experimental setup than regression:
- Jacobs, Tom and Guido Montúfar. “Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning.” (2026)
- Pesme, Scott et al. “Implicit Bias of Mirror Flow on Separable Data.” (NeurIPS 2024)
5 comments
Questions
Q1: Can the authors construct an example for Lemma 10 such that beta is negative?
We are somewhat unsure how to interpret this question: if the question is asking for a theoretical example, yes. Our proof for Lemma 10 is a constructive one that shows for any mirror map , we can construct a loss such that at a certain point. If the reviewer is referring to if we can construct an example of a successful optimization trajectory using mirror descent where , we have outlined this as future work in line 485, and discuss this matter below in greater detail.
We have not been able to find such a trajectory empirically; as we show in Appendix G, all of our examples of edge of stability consistently display . One may, instead, ask if it is possible to construct an explicit counterexample. In the simplest case where is a quadratic, the answer is no, since all examples of mirror descent reduce to gradient descent in this case as we show in Theorem 4, so we know that . Thus, we would need to turn to losses with non-zero third derivatives. In general, it is not feasible to explicitly write out such an optimization trajectory, especially in the discrete setting, and thus we believe that reasoning about them would require substantially new understandings.
Q2: Can the authors discuss the relationship to their work: [AW20] Ehsan Amid, Manfred K. Warmuth "Reparameterizing Mirror Descent as Gradient Descent" (2020) [LWLA22] Li, Zhiyuan et al. “Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent.” (NeurIPS 2022)
We do not see our work as having any particularly significant connections to these works beyond studying some notion of reparameterizations of mirror descent/flow.
The most fundamental difference is that [AW20] and [LWLA22] both study when continuous mirror flow can be reparameterized into gradient descent, whereas our result in Section 3 shows that for a quadratic loss function, the discrete update can be reparameterized. In fact, [AW20, Section 5] explicitly claims, in their conclusion, that their theorem “holds only in the continuous-time and the equivalence relation breaks down after discretization.”
Furthermore, our primary goal in section 3 is not to make novel connections between reparameterizations of mirror descent, but to aid the reader in understanding why our definition of mirror sharpness is sensible and highlight the effect of third-order terms. It is precisely because we cannot find this result in the literature, as we mention in the text [lines 192-193], that we felt it important to state and prove Theorem 4 from scratch, which identifies a simple reparameterization for discrete mirror descent.
Thus, we believe that including a comparison to [AW20] and [LWLA22] would detract from the clarity of section 3. If the reviewer sees a way Theorem 4 directly follows from these works, as suggested in Weakness 1, we would be happy to cite and adapt.
Q3: Discussing related work on implicit bias of mirror descent that established max-margin results for the classification setting covering the hyperbolic entropy and going beyond the power mirror maps would strengthen the manuscript and is more related to experimental setup than regression: Jacobs, Tom and Guido Montúfar. “Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning.” (2026) Pesme, Scott et al. “Implicit Bias of Mirror Flow on Separable Data.” (NeurIPS 2024)
We thank the reviewer for these references and have incorporated them into a revision of the related works section.
References
[ACMHF23]: Maksym Andriushchenko, Francesco Croce, Maximilian Muller, Matthias Hein, and Nicolas Flammarion. A Modern Look at the Relationship between Sharpness and Generalization. ICML 2023.
[CKLKT21]: Jeremy Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. ICLR 2021.
[DNL23]: Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability. ICLR 2022.
Thank you for the detailed response. I appreciate the additional motivation and the clarification that Theorem 4 concerns a discrete-time reparameterization. I agree that the empirical observations are interesting. Nevertheless, I think the theoretical results relatively direct, and I am not yet convinced that the theoretical depth or demonstrated implications are sufficient.
To clarify the implicit regularization question, I would like to know how operating at EoS interacts with the parameter-space bias induced by the mirror map, not generalization. For instance, hyperbolic entropy is used to promote sparsity [1], does entering the EoS regime preserve or change this behavior? A brief discussion of this interaction would help establish the implications of the proposed framework. Moreover, it would also help address the practical relevance of the framework, connecting to Reviewer Tzt8’s follow-up comment.
For Q1, I understand the construction in Lemma 10. My concern still remains if this obstruction can arise along an actual mirror descent trajectory near EoS as the proof is by construction. An example, potentially using a regression objective that is nonquadratic in the parameters, would strengthen this connection. Alternatively, a clearer discussion in the revised manuscript of what the pointwise construction does and does not imply for actual trajectories would help delimit the result.
I keep my score for now.
[1] Jacobs et al., Hyperbolic aware minimization: Implicit bias for sparsity. ICLR 2026.
We thank the reviewer for their comments, and we would like to address their concerns below.
Weaknesses
We respectfully disagree with these points from the reviewer. First, as gradient descent is a special case of mirror descent, any result for mirror descent must recover prior results; thus, we do not believe this fact in itself to be a weakness that limits the novelty of this work. We still believe our work to be novel and significant, as extending to the general case of mirror descent reveals fundamental differences in how we must treat the EoS phase (e.g., Lemma 10) Additionally, this work broadens our understanding of where EoS may take place and what it looks like (as Reviewer zPTY states) while Reviewer Tzt8 finds our notion of sharpness to be new and useful.
We also do not believe the results described to follow easily from the connection between mirror descent and Riemannian gradient descent. See our response to Q2 for a more in-depth elaboration on this point. Furthermore, while we use the literature on Riemannian gradient descent to motivate why our definition of sharpness is sensible, we do not believe this connection itself to be completely trivial. A priori, to the authors, there were other notions of sharpness that appear sensible, such as the ℓ2 sharpness in the dual space since updates are linear in the dual space.
We agree that determining if realistic optimization scenarios can have β<0 is a very important direction for future work, as we have further elaborated on in the response to Question 1. However, we do not believe that this limits its significance, as it is a theoretical negative result that we believe to be of interest to the optimization community, as the result implies that deriving a self-stabilization result using the current central flows framework would be difficult without extra assumptions.
We agree that mirror descent is not the default optimizer in deep learning; however, we do not believe that to limit the relevance of this work. Our motivation is that mirror descent is a relatively simple and natural setting in which we can test which aspects of the self-stabilization theory depend on Euclidean (and non-adaptive) geometry. The vast majority of works along this line study full-batch gradient descent extensively (see [CKLKT21] and [DNL23]) as we ought to better understand the simpler cases before tackling more complex adaptive optimizers.
We took this question to mean how EoS may affect regularization in the context of generalization (as EoS itself is a form of implicit bias as we show in this paper); please let us know if this was not the intent. It is not well-understood how different notions of sharpness are connected to generalization for realistic data distribution — see [ACMHF23]. We do not tackle this issue in the work because we believe a better understanding needs to be reached before considering how mirror sharpness can factor into this.
We agree that more “experimental” deep learning setups may be interesting to consider as follow-up work; however, we see this paper’s main contribution as being a theoretical one.
We have updated the manuscript in Lines 49-55 with some more justification on why mirror descent is important to study.