- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether gradient interventions based on curvature can effectively control sharpness at the Edge of Stability. The authors study damping and amplification under full-batch gradient descent 2 period system proposed in prior works and extend the investigation to stochastic optimization, and compare the dynamics with Sharpness-Aware Minimization (SAM).
Critical strengths
Stated limitations: The paper states limitations of its results to avoid potentail misunderstanding of the interpretation.
Critical weaknesses
- Some part is really hard to understand:
For example: "2-orbit system" "stochastic mismatch," and "effective control response" and especially for the paragraph "Stochastic mismatch between the measured curvature direction and the effective control response".The work can be benefit from better editing with more illustration or narrative or simply properly introduce the definition/terminology before using them
-
Can the author clarify those packages or at least explain how to calculate those stated quantities before using them? For example: ARPACK top eigenvalue and TOST used in calculating the experiment results.
-
The proof in the appendix is really hard to read. I am not sure about the correctness of the proof. The original dynamic system from self-stabilization [1] is derived through the loss landscape assumption with discrete optimization simplification with specification of the manifold being modeled. However, within the paper did not build the orbit on this system but instead directly modify it. Based on the proof, I suppose there are serious correctness issues regarding the proof and the claim.
[1] Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. In The Eleventh International Conference on Learning Representations, 2023.
-
The goal of lemma 1 is unclear. If it is not attractive, then why is it the dynamics of the real optimization process?
-
The paper has a lot of scattered experiments with no coherent purpose given. It will be a lot easier to read if the author can integrate them properly to deliver the idea behind. The work can also benefit from better illustration of all the quantities being calculated or demonstrated. Currently, I uncertain what exactly is the definition of the quantities used in the main content.
Questions for authors
All questions are included in the weakness part.