acceptodds
Under review as a conference paper at ICLR 2027

Observation Does Not Imply Control: The Dynamics of Directional Curvature Interventions at the Edge of Stability

Abstract

Curvature estimation has made sharp directions increasingly accessible in neural-network optimization, but a measured direction need not respond predictably to intervention. We study directional gradient edits at the Edge of Stability (EoS). In a full-batch MLP, fixed-divisor edits provide an interventional test of known preconditioned-stability algebra: conditional on trajectories that exhibit the corresponding sign-flip orbit, late raw sharpness tracks its predicted location , while orbit survival is separate and is not explained by the reduced model. In stochastic ResNet-18/CIFAR-100 training, an engaged rank-1 attenuation arm and wider raw-divisor top-eigenspace rescalings through show no statistically resolved reduction in the tested sharpness diagnostics at the canonical operating point; the wider rescaling rule is not unconditional pure damping because below-target directions receive bounded sign-reversed amplification. We use SAM as a sharpness-reduction control: the tested first-order corrections do not reproduce its ResNet response, and the sampled linSAM-to-SAM interpolation associates lower sharpness with increasing weight on the finite re-query residual bundle. This pathwise decomposition does not isolate a unique nonlinear mechanism and does not transfer in the same form to the LayerNorm ViT control. These results distinguish observing curvature from predicting the response to a specific curvature intervention, within the tested systems, measurements, and intervention rules.

Primary area:
optimization
Keywords:
edge of stability, sharpness, Hessian eigenvalue, curvature, gradient interventions, optimization, sharpness-aware minimization, deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_499aOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether gradient interventions based on curvature can effectively control sharpness at the Edge of Stability. The authors study damping and amplification under full-batch gradient descent 2 period system proposed in prior works and extend the investigation to stochastic optimization, and compare the dynamics with Sharpness-Aware Minimization (SAM).

Critical strengths

Stated limitations: The paper states limitations of its results to avoid potentail misunderstanding of the interpretation.

Critical weaknesses
  1. Some part is really hard to understand:

For example: "2-orbit system" "stochastic mismatch," and "effective control response" and especially for the paragraph "Stochastic mismatch between the measured curvature direction and the effective control response".The work can be benefit from better editing with more illustration or narrative or simply properly introduce the definition/terminology before using them

  1. Can the author clarify those packages or at least explain how to calculate those stated quantities before using them? For example: ARPACK top eigenvalue and TOST used in calculating the experiment results.

  2. The proof in the appendix is really hard to read. I am not sure about the correctness of the proof. The original dynamic system from self-stabilization [1] is derived through the loss landscape assumption with discrete optimization simplification with specification of the manifold being modeled. However, within the paper did not build the orbit on this system but instead directly modify it. Based on the proof, I suppose there are serious correctness issues regarding the proof and the claim.

[1] Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. In The Eleventh International Conference on Learning Representations, 2023.

  1. The goal of lemma 1 is unclear. If it is not attractive, then why is it the dynamics of the real optimization process?

  2. The paper has a lot of scattered experiments with no coherent purpose given. It will be a lot easier to read if the author can integrate them properly to deliver the idea behind. The work can also benefit from better illustration of all the quantities being calculated or demonstrated. Currently, I uncertain what exactly is the definition of the quantities used in the main content.

Questions for authors

All questions are included in the weakness part.

Reviewer_PA3NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3(+1)
Read review
Paper summary

The paper studies whether modifying gradients along sharp Hessian directions can reliably reduce loss sharpness. In full-batch training, damping can instead increase sharpness, consistent with a shifted stability threshold. In stochastic ResNet training, the tested directional edits produce no statistically significant sharpness reduction, while SAM does. Comparing SAM with its linear approximation reveals different behavior on ResNet and ViT, limiting a general explanation of SAM's effect.

The main contribution is a controlled experimental study showing that intervention outcomes depend on the optimizer and training regime. Validation includes a synthetic MLP, ResNet-18 and ViT-tiny on CIFAR-100, multiple random seeds, controls varying intervention strength, direction, and sensing, and a longer 200-epoch training experiment. The paper provides evidence about these interventions' limitations rather than a new optimizer or demonstrated accuracy improvements.

Critical strengths
  1. Direct experimental tests of curvature-based interventions. The paper goes beyond observing correlations between sharpness and training behaviour by changing gradient components and measuring the consequences. The full-batch experiments show responses that are not obvious from an optimiser-design point of view (damping increases sharpness while amplification reduces it, even though the threshold shift behind this is known), and quantitatively test the predicted oscillation location. This matters because it challenges a plausible optimiser-design intuition with direct evidence.

  2. Substantial effort to check and scope the negative results. The authors diagnose an inaccurate curvature sensor, verify that the corrected interventions actually engage, use paired seeds with CIs and equivalence tests with margins set in advance, and use SAM as a positive control. The ViT comparison and the 200-epoch run further probe the scope of the findings.

Critical weaknesses
  1. Missing controls and limited statistical power in the stochastic experiments. The wider-rank interventions mix damping and amplification because the k>1 rule amplifies below-target directions (Sec. 5.2). The rank-k damping arms use only three seeds, with CIs wider than SAM's whole effect (k=2: [-6.41, +7.51] in Table 9, while SAM moves lambda_ref by about 6.6). The edits are weak (1-6% of the gradient per step vs 28% for SAM, Table 10) with no strength sweep. Sensing also raises unresolved questions: the fixed-batch top eigenvector has a median overlap of only 0.26 with the current minibatch's top eigenvector (D.1), while the current-minibatch variant refreshes every 20 steps. A pure-damping version of the k>1 arms, a fixed-strength damping sweep, and equally strong random-subspace edits are needed to tell whether the interventions fail because there is no mechanism or because the edit is too weak and stale to do anything. The authors acknowledge most of these gaps in Appendix G. The stochastic null is therefore informative about the specific interventions tested, but provides limited evidence about why they fail or whether stronger or differently sensed interventions would succeed.

  2. Limited novelty and incomplete explanation of the new empirical findings. The authors acknowledge that the shifted stability threshold (2/eta to 2d/eta) is known preconditioning algebra (Cohen et al. 2022, Islamov et al. 2026), and Song et al. 2025 already showed that removing the sharp-direction component from the update lets sharpness grow. What is new on the full-batch side is the finding that training runs which settle into a steady oscillation land near 2d/eta, on one synthetic MLP with one data seed; the theory predicts the oscillation's location but not whether training reaches it or stays stable, and survival in the fixed-divisor grid is left unexplained. The stochastic failures are also unexplained (rank-20 amplification and linSAM raise sharpness there while the reduced model says amplification should lower it), the SAM comparison does not isolate a mechanism, and the findings depend on the model and the measurement protocol (SAM's lambda_ref moves from about 23 to 74 just by freezing BN running stats, D.6; the ViT and ResNet comparisons run at different batch sizes). So the paper offers little predictive guidance about when such interventions should help.

Questions for authors

Clarification requests:

  1. In the ResNet runs the edit is small (1-6% of the gradient per step) and the direction it acts on is mostly out of date by the time it is used (the fixed-batch top eigenvector overlaps the current minibatch's with |cos| 0.26, and the current-minibatch version is only refreshed every 20 steps). So the simplest explanation for the null is that the edit is too small and pointed at the wrong direction to do anything. The paper argues against this using the bs 2048 run, but that run is close to full-batch, where the sharp direction holds still, so it is a different situation. Is there anything at bs 128 itself that rules out the simple explanation?

  2. In the full-batch theory and the MLP experiments, amplifying the gradient along sharp directions lowers sharpness. In the ResNet runs it does the opposite: rank-20 amplification and linSAM both increase it. Do you have an explanation for this?

Experiments that would change my Soundness score:

  1. Rerun the damping arms with 5 seeds like the control arms, with the k>1 rule fixed so it only damps (keep the max(d_i, 1) floor), plus a random-direction edit of the same size as a baseline, and a sweep over the damping strength (e.g. d = 2, 5, 10, and removing the top-k directions completely as in Song et al.). This would show whether the null is because the edit is too weak, or because there is really no effect.

Not affecting my assessment: please rename the k>1 arms, since "damping" is misleading for a rule that also amplifies.

Reviewer_QXYyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The introduction is very hard to read.

  • "This accessibility motivates a natural intervention: modify the gradient along sharp Hessian directions in an attempt to control sharpnes." What is the "accessibility" here? Why is it a natural intervention? I could not get the motivation for studying this question here.
  • "We organize the experiments around a known piece of threshold algebra". What is "threshold algebra"?
  • "The edit acts but moves its own target" What does this mean? What is the edit? What is the target?
  • "At the reference stochastic operating point, the edited eigenspace carries little of the current minibatch gradient and the intervention produces no resolved sharpness reduction" What is the "reference stochastic operating point"? "no resolved sharpness reduction" What does it mean to say any "sharpness reduction" is resolved or not?

Since I could not understand the introduction, I have to refuse to summarize the whole paper by myself. After feeding the PDF into an AI, here is my understanding of the paper. Motivated by the EoS phenomenon, for some reason the authors seem to believe that modifying the gradient along the sharp directions of the Hessian may reduce sharpness, but both full-batch GD and minibatch SGD experiments produce negative results. So "Observation Does Not Imply Control".

Critical strengths

/

Critical weaknesses
  • The presentation of the paper is very bad and I doubt the authors have read the paper themselves. I refuse to spend more time reading this paper. Even though it is allowed to use AI for the paper writing, the authors should install proper skills for AI to improve its writing.
  • I cannot get the research question of this paper at all. Why can "observation" "imply control" in any sense? More specifically, my understanding of the EoS phenomenon is that there is something in the neural network training that induces the so-called progressive sharpening, where the sharpness gradually increases if no one stops it. This is likely due to the fact that some global minima lie in a sharp region. However, the sharpness cannot increase forever because if it reaches the stability boundary, then the optimizer has to prevent the sharpness from further increasing. If you manipulate the sharp directions of the Hessian, you are just changing the stability boundary of the optimizer in some way, and it is unclear to me whether the sharpness should be reduced or not. Therefore, it is also unclear to me why the previous EoS observations may imply that controlling the sharp direction can reduce sharpness.
Questions for authors

/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.