acceptodds
Under review as a conference paper at ICLR 2027

Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model

Abstract

A large body of theory and empirical work hypothesizes a connection between the flatness of a neural network's loss landscape during training and its performance. However, there have been conceptually opposite pieces of evidence regarding when SGD prefers flatter or sharper solutions. In this work, we partially but causally clarify the flatness-seeking behavior of SGD by identifying and exactly solving an analytically solvable model that exhibits both flattening and sharpening behavior. In this model, the SGD training has no a priori preference for flatness, but only a preference for minimal gradient fluctuations. This leads to the insight that, at least within this model, it is data distribution that uniquely determines the sharpness at convergence, and that a flat minimum is preferred if and only if the noise in the labels is isotropic across all output dimensions. When the noise in the labels is anisotropic, the model instead prefers sharpness and can converge to an arbitrarily sharp solution, depending on the imbalance in the noise in the labels spectrum. We reproduce this key insight in controlled settings with different model architectures such as MLP, RNN, and transformers.

Primary area:
learning theory
Keywords:
deep linear network, implicit bias, sharpness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UZd2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The authors build on prior work identifying that small-step SGD implicitly minimizes a loss objective augmented by a gradient fluctuation term. The authors investigate under which conditions the minimization of this gradient fluctuation term simultaneously minimizes the objective sharpness (which, for the authors, means the Hessian trace). The authors derive general upper and lower bounds on this trace-sharpness in terms of the residual covariance and the gradient fluctuation. The authors then prove, in the setting of a deep linear network with a noisy teacher, that minimization of the gradient fluctuation simultaneously minimizes the sharpness if and only if the teacher label noise is isotropic. The authors also empirically validate the claim that anisotropic label noise leads to sharpness in both the linear network setting of their theory, and for MLPs, RNNs and transformers.

Critical strengths
  1. The result that SGD can provably seek sharp solutions is somewhat surprising and will be of interest to those working on the link between optimization and sharpness.
  2. The theoretical results, which rely on somewhat strong hypotheses, are supplemented by experiments indicating that the insights obtained via theory might also hold in more practical settings.
Critical weaknesses

The authors propose an explanation of progressive sharpening based on their theory: anisotropic label noise causes SGD to converge to sharper solutions provided the initialization is flatter. More generally, it is tautologically true that if the sharpness is smaller at initialization than at convergence, then some sharpening must of course have occurred. The authors' proposed explanation appears to be a special case of this tautological statement. While I don't argue with the content of the claim (indeed, it is obvious), I think that labeling it as an explanation/mechanism for progressive sharpening is mistaken. Progressive sharpening rather refers to near-monotonic increase of the sharpness from the start of training until the onset of the edge of stability; so far as I can tell, nothing in the authors' theory prevents significant departures from the near-monotonic trend that is usually referred to as progressive sharpening.

Questions for authors

I am willing to raise my score if the authors commit to moderating their claims about progressive sharpening (or convince me that I am mistaken in my critique of these claims).

I would also encourage a more thorough literature review in the Deep Linear Networks section of Appendix A. It appears to be missing a number of relevant works such as https://arxiv.org/pdf/2110.03677, https://arxiv.org/pdf/2502.20531, https://arxiv.org/pdf/2510.17506

2 comments

(W) The authors propose an explanation of progressive sharpening based on their theory: anisotropic label noise causes SGD to converge to sharper solutions provided the initialization is flatter. More generally, it is tautologically true that if the sharpness is smaller at initialization than at convergence, then some sharpening must of course have occurred. The authors' proposed explanation appears to be a special case of this tautological statement. While I don't argue with the content of the claim (indeed, it is obvious), I think that labeling it as an explanation/mechanism for progressive sharpening is mistaken. Progressive sharpening rather refers to near-monotonic increase of the sharpness from the start of training until the onset of the edge of stability; so far as I can tell, nothing in the authors' theory prevents significant departures from the near-monotonic trend that is usually referred to as progressive sharpening.

Reply to weakness: We thank the reviewer for this helpful clarification. We agree that our theoretical results characterize the sharpness selected at convergence. Our theoretical approach is to compare the initial sharpness with the predicted final sharpness, thereby determining whether training exhibits an overall sharpening or flattening effect. As the reviewer correctly points out, this does not guarantee monotonicity along the training trajectory. However, our experiments (Figures 2 and 4) do exhibit an approximately monotonic increase in sharpness, consistent with the conventional notion of progressive sharpening. To avoid confusion, we have added a clarification in Section 5 explicitly distinguishing these two aspects. We now emphasize that our theory explains the noise-dependent increase in the selected sharpness.

(Q) I would also encourage a more thorough literature review in the Deep Linear Networks section of Appendix A. It appears to be missing a number of relevant works such as https://arxiv.org/pdf/2110.03677, https://arxiv.org/pdf/2502.20531, https://arxiv.org/pdf/2510.17506.

Reply to Question: We thank the reviewer for suggesting these relevant works. We have expanded the Deep Linear Networks paragraph in Appendix A to include these literatures. The revision has been incorporated into the updated PDF.

Thank you for your changes. I have raised my score to a 3.

Reviewer_3AkBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates when SGD selects flat or sharp minima through an analytically solvable deep linear network model. Building on the minimum-gradient-fluctuation principle, the authors derive closed-form expressions for the Hessian sharpness of selected solutions at arbitrary depth, characterize its dependence on label-noise anisotropy, and establish additional spectral properties. Experiments on linear networks, MLPs, RNNs, and Transformers are presented to support the predicted relationship between noise anisotropy and sharpness.

Critical strengths

S1. Analytical characterization. The paper derives explicit arbitrary-depth characterizations of selected Hessian sharpness, including quantitative bounds and spectral properties.

S2. Conceptual clarity and empirical coverage. The paper clearly distinguishes progressive sharpening along training trajectories from implicit selection among minima. Experiments across several nonlinear architectures also broaden the empirical evidence for the relationship between noise anisotropy and sharpness.

Critical weaknesses

W1. Substantial overlap with prior work. The central mechanism and several qualitative findings—including minimum-fluctuation selection, noise-dependent sharpening and flattening, and initialization-independent noise equilibria—were already investigated by Ziyin et al. (2024, §5.2; 2025, §5.2, Theorem 10). Moreover, Ziyin et al. (2025, Theorem 12) already characterize the arbitrary-depth minimum-fluctuation solutions used in this paper. The claimed new mechanism of progressive sharpening therefore substantially overlaps with these earlier results.

W2. Limited significance of the theoretical extension. The main selected-sharpness formula appears to follow relatively directly from evaluating established Hessian expressions at previously characterized solutions, drawing on existing deep matrix-factorization analyses (Gatmiry et al., 2023; Kamber & Parhi, 2025). Although the additional bounds and spectral properties are formally new, it remains unclear what substantial technical challenge or previously unexplained phenomenon they resolve. The nonlinear experiments broaden the architectures studied but largely reproduce an anisotropy–sharpness relationship already examined by Ziyin et al. (2025, Appendix A.6). Section 6 consequently overstates the distinction from that work by suggesting that nonlinear relevance had not been investigated.

Questions for authors

Q1. Given that Ziyin et al. (2025) already characterize arbitrary-depth minimum-fluctuation solutions and explain noise-dependent sharpening and flattening, what substantive new mathematical challenge or previously unexplained phenomenon is resolved by the present extension?

Q2. How does the proposed minimum-fluctuation selection principle extend to finite-data interpolation regimes? For squared loss, any interpolating solution θ⋆\theta^\star satisfies ∇θℓi(θ⋆)=0\nabla_\theta \ell_i(\theta^\star)=0 for all training samples ii, and hence V(θ⋆)=1n∑i=1n∣∇θℓi(θ⋆)∣2=0.V(\theta^⋆)=\frac{1}{n}\sum_{i=1}^{n}|\nabla_\theta\ell_i(\theta^⋆)|^2=0. Thus, the fluctuation term in Eq. (5) cannot distinguish among interpolating minima, even when they have different Hessian sharpness. How does the proposed framework account for SGD's implicit selection in this common overparameterized setting, where trajectory-dependent effects may remain important?

Q3. In the proposed deep linear model, fixed residual connections can be absorbed into the unconstrained weights through Zl=Wl+RlZ_l=W_l+R_l, yielding an ordinary deep linear network with an identical loss landscape up to parameter translation. Given that the claimed invariance to residual connections follows directly from this elementary reparameterization, what substantive insight does this result provide about residual architectures, particularly nonlinear networks where the equivalence no longer holds? Why is this emphasized as a meaningful finding rather than treated as an immediate corollary?

3 comments

Reply to W1, W2 and Q1: We thank the reviewer for the careful comparison with prior work. We agree that our work builds on the results established in Ziyin et al. (2024, 2025). However, our contributions go beyond extending their results. We establish a general theoretical explanation of the relationship between gradient fluctuation and Hessian sharpness, and provide a quantitative characterization of their discrepancy in an exactly solvable model.

  1. A general theoretical framework beyond linear networks (Section 3).

An important contribution of our work is the general analysis in Section 3.3, which applies to arbitrary regression models. Specifically, we derive the relationship T(θ)=Tr⁡Γ(θ),V(θ)=Tr⁡[ΣϵΓ(θ)]T(\theta)=\operatorname{Tr}\Gamma(\theta),V(\theta)=\operatorname{Tr}[\Sigma_\epsilon\Gamma(\theta)], where Γ(θ)\Gamma(\theta) characterizes the sensitivity of model outputs to parameter perturbations. This reveals a fundamental distinction: Hessian sharpness measures total parameter sensitivity, whereas gradient fluctuation weights this sensitivity according to the label-noise covariance.

Importantly, Eqs. (14)–(16) establish that these two quantities can induce opposite orderings: a solution with smaller gradient fluctuation can have larger Hessian sharpness. This provides a general, architecture-independent explanation for why the minimum-fluctuation principle does not necessarily imply flatness selection. Unlike the specific linear-network calculations in prior work, this relationship does not rely on linearity, network depth, or a particular parameterization. Sections 4–5 then construct an exactly solvable model in which this general possibility is realized and quantitatively characterized.

  1. Exact quantitative characterization of the sharpness discrepancy (Sections 4–5).

Compared with existing work, we establish several new results:

2.1. Exact selected and minimal sharpness (Theorem 4.1). We derive closed-form expressions for both the sharpness selected by minimal fluctuation and the minimal achievable sharpness, enabling a direct quantitative comparison between the two objectives at arbitrary depth.

2.2. A necessary and sufficient condition for flatness selection (Corollary 5.2). We prove that isotropic noise is not merely an example where SGD selects flat minima; it is the necessary and sufficient condition for achieving the flattest solution in our model.

2.3. Quantitative scaling of excess sharpness (Corollary 5.1). We prove TTmin⁡=Θ(κ(Σϵ))\frac{T}{T_{\min}}=\Theta\left(\sqrt{\kappa(\Sigma_\epsilon)}\right) with constants independent of network depth. It shows that the fluctuation bias can select solutions arbitrarily sharper than the flattest minimum and precisely quantifies how this discrepancy depends on noise anisotropy.

2.4. Hessian spectral properties (Theorem 4.1, Theorem B.5, Corollary 5.3). We establish that all minimum-fluctuation solutions share the same Hessian spectrum, derive explicit nonzero eigenvalues under simultaneous diagonalization, and prove the corresponding κ(Σϵ)\sqrt{\kappa(\Sigma_\epsilon)} scaling for the largest Hessian eigenvalue.

These results answer a question not resolved by the previous results: how far from the flattest possible minimum does fluctuation minimization lead, and under what conditions does this discrepancy disappear?

  1. Relation to existing deep matrix-factorization results.

We agree that known Hessian expressions facilitate the calculation of sharpness. However, Neither Gatmiry et al. (2023) nor Kamber & Parhi (2025) establishes the noise-dependent discrepancy between minimum-fluctuation selection and global sharpness minimization. Our results require comparing these two distinct optimization objectives, characterizing their coincidence conditions, and establishing depth-uniform bounds. The significance therefore lies not merely in evaluating an existing Hessian formula, but in the resulting quantitative characterization of SGD's implicit selection.

  1. Additional empirical evidence beyond prior work.

Compared with Ziyin et al. (2025), our experiments go substantially further by systematically varying noise anisotropy across MLPs, RNNs, Transformers, and Vision Transformers, including both regression and classification settings. In particular, Figure 4 quantitatively compares SGD's converged sharpness with the theoretical prediction, the globally minimal Hessian trace, and the minimum possible largest Hessian eigenvalue. The SGD-to-GD intervention further isolates the role of stochasticity in the observed sharpening.

In summary, our main contribution is not a new minimum-fluctuation principle, but a general explanation of its relationship to sharpness and a quantitative theory of when and by how much fluctuation minimization deviates from flatness minimization.

(Q2) How does the proposed minimum-fluctuation selection principle extend to finite-data interpolation regimes? For squared loss, any interpolating solution θ∗\theta^* satisfies ∇θℓi(θ∗)=0\nabla_\theta\ell_i(\theta^*)=0 for all training samples ii, and hence V(θ∗)=1n∑i=1n∣∇θℓi(θ∗)∣2=0V(\theta^*)=\frac{1}{n}\sum_{i=1}^n|\nabla_\theta\ell_i(\theta^*)|^2=0. Thus, the fluctuation term in Eq. (5) cannot distinguish among interpolating minima, even when they have different Hessian sharpness. How does the proposed framework account for SGD's implicit selection in this common overparameterized setting, where trajectory-dependent effects may remain important?

Reply to Q2: We agree with the reviewer that this is our limitation. Our analytical results instead consider population-risk minimization with irreducible label noise, where the per-sample residuals remain nonzero, so the gradient fluctuation retains a nontrivial dependence on the network parameters. This is precisely the mechanism underlying our selection results. In interpolation regimes, the stochastic gradient noise vanishes interpolation, and selection may depend on the optimization trajectory before interpolation is reached. This is an important future direction.

(Q3) In the proposed deep linear model, fixed residual connections can be absorbed into the unconstrained weights through Zℓ=Wℓ+RℓZ_\ell=W_\ell+R_\ell, yielding an ordinary deep linear network with an identical loss landscape up to parameter translation. Given that the claimed invariance to residual connections follows directly from this elementary reparameterization, what substantive insight does this result provide about residual architectures, particularly nonlinear networks where the equivalence no longer holds? Why is this emphasized as a meaningful finding rather than treated as an immediate corollary?

Reply to Q3: We agree that this does not constitute a new mechanism specific to residual architectures. Its role is to clarify the scope of the solvable model: fixed additive residual terms do not alter the selected sharpness in this setting. We also agree that the argument does not extend directly to nonlinear residual networks, and we will clarify that our nonlinear experiments provide empirical evidence rather than an analytical extension.

Thank you for the detailed response and for clarifying the relationship to prior work. Your explanation has helped me better appreciate the significance of the quantitative results. In particular, I see value in developing a systematic understanding of minimum-fluctuation selection within an exactly solvable model, including when it coincides with flatness minimization, how far the selected solution can depart from the flattest minimum, and what spectral properties it exhibits. Although my concerns about the overlap with prior work remain somewhat relevant, I now place greater weight on the depth and precision of the understanding established within this setting. I also appreciate your explicit acknowledgment of the limitations concerning finite-data interpolation and the interpretation of residual connections. Taking these clarifications into account, I find the resulting quantitative characterization sufficiently informative to support weak acceptance and have raised my score to 3.

Reviewer_sYDyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tries to understand whether SGD prefers sharp or flat solutions. The sharpness definition used here is mostly the Hessian trace. For that, it analyzes a class of deep linear residual networks in the gradient flow limit (i.e., away from the edge of stability). The main tool is the minimal fluctuation principle. SGD in the gradient flow limit does not minimize or maximize sharpness but minimizes the entropic loss. Depending on model and data, this loss might prefer sharp or flat solutions.

Critical strengths

The paper is very cleanly written and strikes a nice balance between outlining the theory and presenting the mathematical formalism and giving an intuitive insight into the theory. I especially enjoyed Section 5, where every main consequence of the theorem is explained. I think the main strength of the paper is for other theory-oriented papers, as it shows that a lot of terms in the literature that are often used synonymously are actually different. Furthermore, I think a large class of people connect, e.g., progressive sharpening to the edge of stability, but the paper here cleanly shows that there are other possible mechanisms.

Critical weaknesses

In my opinion, the main weakness of the work is in its scope and missing empirical verifications. While the authors have extensive experiments on the different label noise settings also in nonlinear architectures, to me, e.g., the claim that the converged sharpness is independent of initialization is never shown to be true or wrong on nonlinear architectures empirically. Label noise is also not commonly used in real-world architectures which strongly limits the scope of the paper.

Questions for authors
  • Can you connect your notion of sharpness to generalization?
  • For nonlinear architectures, is the sharpness of the converged model also independent of the initialization?
  • How does muP vs. NTK parameterization change the proofs and notion of sharpness?
  • What happens with the sharpness notion and the insights when we have finite dataset size? We should, e.g., see some form of double descent across width. Can this model be extended to capture the under- to overparametrized transition?
1 comment

(W1) In my opinion, the main weakness of the work is in its scope and missing empirical verifications. While the authors have extensive experiments on the different label noise settings also in nonlinear architectures, to me, e.g., the claim that the converged sharpness is independent of initialization is never shown to be true or wrong on nonlinear architectures empirically. Label noise is also not commonly used in real-world architectures which strongly limits the scope of the paper.

(Q2) For nonlinear architectures, is the sharpness of the converged model also independent of the initialization?

Reply to W1 and Q2: We thank the reviewer for raising this question. While our theoretical result on initialization independence applies to deep linear networks, our nonlinear experiments provide empirical evidence for robustness to random initialization. In particular, Figures 2 and 5 already report results averaged over five independent trials, with relatively small variability toward convergence. We have additionally included error bands across independent random initializations in Figures 3 and 9, covering MLPs, RNNs, and Transformers. The results show that the final sharpness exhibits certain variability across random seeds under the standard initialization schemes, but the overall trend is very clear.

(W2) Label noise is also not commonly used in real-world architectures which strongly limits the scope of the paper.

Reply to W2: We would like to clarify that label noise in our theoretical framework is a natural statistical modeling assumption, not an additional procedure that must be explicitly applied when training neural networks. The model y=Vx+ϵy=Vx+\epsilon describes irreducible uncertainty in the observed targets, which can arise from measurement errors, incomplete observations, or variability in annotations. Such uncertainty can exist even when no artificial noise is added during training.

Furthermore, our analysis in Appendix B.9 shows that even when the original labels are noiseless, noise in the input features can induce an effective residual covariance, which can itself be anisotropic. Thus, the proposed mechanism is not restricted to explicitly corrupted labels.

In our experiments, we deliberately introduce controlled label noise to isolate the effect of its anisotropy on sharpness. We agree that our exact theoretical predictions rely on idealized assumptions and do not directly establish the same behavior for all real-world datasets. Nevertheless, we believe that the noisy-teacher setting provides a natural and informative model for understanding how data uncertainty influences SGD's implicit selection of sharpness.

(Q1) Can you connect your notion of sharpness to generalization?

Reply to Q1: Our analysis focuses on SGD's implicit selection among minima rather than generalization. In fact, in our deep linear model, all global minima represent the same predictor fθ(x)=Vxf_\theta(x)=Vx, and therefore have identical generalization loss, despite potentially having arbitrarily different Hessian sharpness. This highlights that sharpness and generalization are distinct properties, and that a preference for flatter solutions does not necessarily imply better generalization.

(Q3) How does muP vs. NTK parameterization change the proofs and notion of sharpness?

Reply to Q3: We thank the reviewer for this interesting question. We have added Appendix B.10, where we analytically investigate the effect of layer-wise learning-rate scaling on sharpness in our deep linear model. Interestingly, we find that minimum-fluctuation selection achieves the globally minimal Euclidean sharpness if and only if the label noise is isotropic and all layer-wise learning rates are equal, under the full-row-rank assumption. Thus, parameterization and learning-rate scaling can affect flatness selection even when the data distribution is unchanged.

We discuss NTK- and μP\mu P scalings in Appendix B.10. These calculations characterize the corresponding finite-width selected solutions.

(Q4) What happens with the sharpness notion and the insights when we have finite dataset size? We should, e.g., see some form of double descent across width. Can this model be extended to capture the under- to overparametrized transition?

Reply to Q4: Our deep linear model considers population-risk minimization with irreducible label noise. For finite datasets in the interpolation regime, the per-sample gradients vanish at the interpolating solutions, and consequently the fluctuation term in Eq. (5) cannot distinguish among them. Characterizing SGD's implicit selection in this regime requires accounting for the optimization trajectory before interpolation. We agree that extending our framework across the interpolation threshold, and investigating its relationship with double descent, would be valuable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.