- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The authors build on prior work identifying that small-step SGD implicitly minimizes a loss objective augmented by a gradient fluctuation term. The authors investigate under which conditions the minimization of this gradient fluctuation term simultaneously minimizes the objective sharpness (which, for the authors, means the Hessian trace). The authors derive general upper and lower bounds on this trace-sharpness in terms of the residual covariance and the gradient fluctuation. The authors then prove, in the setting of a deep linear network with a noisy teacher, that minimization of the gradient fluctuation simultaneously minimizes the sharpness if and only if the teacher label noise is isotropic. The authors also empirically validate the claim that anisotropic label noise leads to sharpness in both the linear network setting of their theory, and for MLPs, RNNs and transformers.
Critical strengths
- The result that SGD can provably seek sharp solutions is somewhat surprising and will be of interest to those working on the link between optimization and sharpness.
- The theoretical results, which rely on somewhat strong hypotheses, are supplemented by experiments indicating that the insights obtained via theory might also hold in more practical settings.
Critical weaknesses
The authors propose an explanation of progressive sharpening based on their theory: anisotropic label noise causes SGD to converge to sharper solutions provided the initialization is flatter. More generally, it is tautologically true that if the sharpness is smaller at initialization than at convergence, then some sharpening must of course have occurred. The authors' proposed explanation appears to be a special case of this tautological statement. While I don't argue with the content of the claim (indeed, it is obvious), I think that labeling it as an explanation/mechanism for progressive sharpening is mistaken. Progressive sharpening rather refers to near-monotonic increase of the sharpness from the start of training until the onset of the edge of stability; so far as I can tell, nothing in the authors' theory prevents significant departures from the near-monotonic trend that is usually referred to as progressive sharpening.
Questions for authors
I am willing to raise my score if the authors commit to moderating their claims about progressive sharpening (or convince me that I am mistaken in my critique of these claims).
I would also encourage a more thorough literature review in the Deep Linear Networks section of Appendix A. It appears to be missing a number of relevant works such as https://arxiv.org/pdf/2110.03677, https://arxiv.org/pdf/2502.20531, https://arxiv.org/pdf/2510.17506
2 comments
Thank you for your changes. I have raised my score to a 3.
(W) The authors propose an explanation of progressive sharpening based on their theory: anisotropic label noise causes SGD to converge to sharper solutions provided the initialization is flatter. More generally, it is tautologically true that if the sharpness is smaller at initialization than at convergence, then some sharpening must of course have occurred. The authors' proposed explanation appears to be a special case of this tautological statement. While I don't argue with the content of the claim (indeed, it is obvious), I think that labeling it as an explanation/mechanism for progressive sharpening is mistaken. Progressive sharpening rather refers to near-monotonic increase of the sharpness from the start of training until the onset of the edge of stability; so far as I can tell, nothing in the authors' theory prevents significant departures from the near-monotonic trend that is usually referred to as progressive sharpening.
Reply to weakness: We thank the reviewer for this helpful clarification. We agree that our theoretical results characterize the sharpness selected at convergence. Our theoretical approach is to compare the initial sharpness with the predicted final sharpness, thereby determining whether training exhibits an overall sharpening or flattening effect. As the reviewer correctly points out, this does not guarantee monotonicity along the training trajectory. However, our experiments (Figures 2 and 4) do exhibit an approximately monotonic increase in sharpness, consistent with the conventional notion of progressive sharpening. To avoid confusion, we have added a clarification in Section 5 explicitly distinguishing these two aspects. We now emphasize that our theory explains the noise-dependent increase in the selected sharpness.
(Q) I would also encourage a more thorough literature review in the Deep Linear Networks section of Appendix A. It appears to be missing a number of relevant works such as https://arxiv.org/pdf/2110.03677, https://arxiv.org/pdf/2502.20531, https://arxiv.org/pdf/2510.17506.
Reply to Question: We thank the reviewer for suggesting these relevant works. We have expanded the Deep Linear Networks paragraph in Appendix A to include these literatures. The revision has been incorporated into the updated PDF.