acceptodds
Under review as a conference paper at ICLR 2027

Optimizer Switching at the Edge of Stability: The Onset of Late-Phase SAM Dynamics

Abstract

Sharpness-aware minimization (SAM) is often applied throughout training, yet recent empirical evidence suggests that switching from standard gradient-based training to SAM only late in optimization can recover much of its benefit. Why can such a late optimizer switch have an immediate dynamical effect? We study this question through the interaction between optimizer switching and the edge of stability. We show that a point that remains stable under gradient descent can become unstable immediately after switching to normalized SAM: the two methods obey different local stability boundaries, and the discrepancy grows as the gradient norm decreases. Near a sharp valley, we further show that normalization converts this boundary crossing into a finite post-switch displacement of order \(O(\rho)\), rather than a perturbation that vanishes with the gradient. In the late-switch regime, the leading normal displacement is spectrally filtered by the local Hessian, preferentially concentrating motion toward dominant-curvature directions. For an exact frozen quadratic normal model, we characterize the resulting phase-alternating dynamics and its attracting principal shell. Controlled synthetic experiments verify the predicted switching boundary, finite-amplitude kick, and shell dynamics. Neural-network experiments on CIFAR-10 with ResNet-18 further reveal a structured post-switch regime boundary, with short phase-alternating transients separating rapid flat-side stabilization from sharp-side instability. Together, these results identify optimizer switching near the edge of stability as a mechanism that initiates the characteristic oscillatory dynamics of late-phase SAM.

Primary area:
optimization
Keywords:
sharpness-aware minimization, edge of stability, optimizer switching, optimization dynamics, stability analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_srmjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • Switch from GD to normalized SAM late in training, near edge of stability
  • Same-state window: SAM step can increase loss while GD decreases it
  • First SAM displacement bounded, weighted toward high curvature
  • Conditions for convergence to the known SAM two-cycle
  • CIFAR-10 ResNet checkpoint

UPDATE: Editing this down to a clear reject for me. I think the paper has changed too much since the original review and now has a pretty significant gap between the original setting, theorems, and experiments and the newly developed math and experiments. UPDATE 2: I got a chance to review some of the math more. I think I will maintain my score at a weak reject instead. I ran some small numerical tests to check out the new updated math since I was not a huge fan of the empirical tests done in the paper to connect the theory. Ordinary GD trajectories do seem to match the conditions that the authors have shown in their theoretical updates. These tests were done on synthetic data but made me feel a bit more confident in the new math presented in the rebuttals. However, I still think this paper needs a few more writing passes and needs to be revised heavily to make the writing and the experiments connect more. I would not mind updating my score if the authors can provide better experiments showcasing their new math holds in non synthetic conditions (small experiments are ok and if you can just run on your laptop, that would be great. Doesn't need to be anything requiring GPUs). I would like to see these: Start GD and momentum-GD from init, switch to SAM at different places, and report when the incoming states satisfy the capture conditions that are stated in the theory. Sweep alignment, residual magnitude, LR, and SAM radius so you can compare the certified region with the attraction base. Certificate based proofs truly need more work to showcase that the method lands in the certificate, even for a learning theory track paper. Can you also examine how these results change as LR approaches stability boundary including cases where sufficient conditions fail but capture still happens. I think with these and a rewrite, I would be willing to move this paper up to a weak accept.

Critical strengths
  • Proofs correc + claims accurate (basin result labelled conditional), side note- please don't make AI titles for theorems, it's confusing
Critical weaknesses
  • Experiments miss the theory's set up:
    • η\eta calibrated post hoc as (2−δ0)/λ1(2-\delta_0)/\lambda_1 (Appendix E only)
    • network trained with momentum SGD and cosine decay, not GD at the edge of stability??
    • with momentum, threshold is 2(1+β)/η2(1+\beta)/\eta
  • Incremental over Bartlett et al. (2023):
    • window x<ρ(2−δ)/δx < \rho(2-\delta)/\delta = two-cycle radius cc
    • Thm 1 = 1D SAM map pushes outward inside the known cycle
Questions for authors
  • (Clarification) results at the actual training step size or with momentum?
  • (Clarification) basin capture provable? (∥y1∥≤ηρλ1=δc\|y_1\| \le \eta\rho\lambda_1 = \delta c)
  • (Clarification) 5-state (main text) vs. 4-state (Appendix E) transient
  • (Clarification) ρ\rho-slopes 1.5581.558, 0.8190.819, 1.4281.428 vs. predicted 11; R2R^2 on 4 points uninformative
  • Suggestions: note window = cc; rate condition qfr2≤1−δq_{\rm fr}^2 \le 1-\delta needs λmin⁡/λ1≳δ/4\lambda_{\min}/\lambda_1 \gtrsim \delta/4; plain-language definitions; cite catapult dynamics (Lewkowycz et al., 2020)
  • Typos: "Section ??" (pg 30); incomplete references (pg 11); Fisher SAM missing co-author (pg 10)
13 comments
AuthorsOpenReview ↗

Our response to the weaknesses

We thank the reviewer for identifying two central weaknesses of the original submission: the mismatch between the theoretical optimizer and experimental protocol, and the limited novelty of the quadratic switching analysis relative to Bartlett et al. (2023). Both concerns motivated substantial new theoretical and empirical results in the revision.

1. Experimental–theoretical consistency: actual learning rates and momentum

The original near-edge interventions are now clearly distinguished from actual training. The reviewer is correct that the original experiments used momentum-SGD-trained checkpoints and post-hoc calibrated learning rates η=(2−δ0)/λ^1\eta=(2-\delta_0)/\widehat{\lambda}_1. These interventions establish controlled responses under prescribed diagnostic conditions, not naturally occurring edge-of-stability (EoS) training. We explicitly clarify this distinction in Sec. 7.1.

More importantly, we have conducted new experiments at the actual saved learning rates and with the corresponding optimizer states:

  • E5: 126 controlled saved-rate interventions across ResNet-18 and VGG-11-BN.
  • E6: 16,800 actual-rate intervention cells from 42 source trajectories, spanning GD, SGD, and momentum SGD (mSGD), seven seeds per architecture/regime, 20 checkpoints, and 20 perturbation radii.
  • Native SGD: 516/2,800 robust diagnostic reversals on ResNet-18 and 499/2,800 on VGG-11-BN; 751 of the combined 1,015 events survive a tenfold numerical margin (Table 1).
  • Long-run evaluation: 216 matched SGD/mSGD continuations through epoch 200, with 186 final test-accuracy improvements and lower test cross-entropy in all 216 comparisons (Table 2).

These results establish that the observed switching signatures are not restricted to artificially calibrated learning rates, and that late switching yields measurable held-out benefits under the tested practical training schedules.

We also report important distinctions: mSGD reversal events are substantially rarer, different source trajectories preclude attributing this difference solely to momentum, and the full-batch GD models are undertrained. Diagnostic reversals do not certify natural EoS or sustained neural oscillations.

The theoretical momentum mismatch is addressed by a new theorem, not by reinterpreting GD. Theorem 3 (Sec. 6; Appendix E) analyzes classical heavy-ball outer momentum, explicitly retaining the incoming optimizer buffer in an augmented dynamical state.

For a=ηλ1a=\eta\lambda_1 and momentum coefficient β∈[0,1)\beta\in[0,1), the quadratic stability range becomes

0<a<2(1+β),0<a<2(1+\beta),

as the reviewer correctly notes. We derive the corresponding attracting cycle radius

cβ=aρ2(1+β)−a,c_\beta=\frac{a\rho}{2(1+\beta)-a},

and establish its persistence as an exact attracting nonlinear period-two orbit. Crucially, we also prove conditional finite-time capture from an incoming augmented state under explicit residual, buffer, alignment, and spectral conditions (Theorems E.2–E.5).

The stronger capture condition (1+β)2<a<2(1+β)(1+\sqrt{\beta})^2<a<2(1+\beta) is sufficient, not necessary; its narrowing at large momentum is explicitly acknowledged. We further derive an exact shared-buffer reversal threshold, showing why the stability boundary alone cannot determine same-state loss signs.

Thus, the revised theory directly models the momentum mechanism raised by the reviewer, while the new experiments test switching under saved learning-rate schedules and optimizer states. We do not conflate deterministic heavy-ball guarantees with stochastic momentum training.

2. Novelty beyond Bartlett et al. (2023): from a known cycle to certified nonlinear capture

We agree that, on the aligned quadratic principal axis,

0<∣y∣<c,c=ηλ1ρ2−ηλ1,0<|y|<c,\qquad c=\frac{\eta\lambda_1\rho}{2-\eta\lambda_1},

is precisely the interior of the known SAM two-cycle radius. The revision states this explicitly (Sec. 3). We do not claim discovery of the quadratic cycle or its radius.

The substantive contribution is instead to resolve what the original quadratic calculation leaves open:

Does an incoming switching state actually enter an attracting oscillatory regime, rather than merely receive an outward perturbation?

Theorem 1 (Sec. 4; Appendix B) answers this question for the exact nonlinear SAM map. Near an isolated nondegenerate minimum with locally Lipschitz Hessian and a simple leading eigenvalue, sufficiently small residual and principal-gradient alignment guarantee:

  1. Strict same-state GD/SAM loss reversal;
  2. Finite-time entry into local contraction neighborhoods;
  3. Attraction to an exact nonlinear period-two orbit, with wρ±=w∗±[aρ/(2−a)]v1+O(ρ2)w_\rho^\pm=w_*\pm[a\rho/(2-a)]v_1+O(\rho^2).

Unlike the submitted invariant-basin result, the theorem proves basin entry without assuming prior membership in the original frozen basin or its restrictive spectral-rate condition. It does not prescribe the optimizer history.

Under additional GD-prefix spectral dominance, Eq. (1) provides an explicit sufficient switching-time certificate and conditional perturbation-radius guidance. Figure 1 verifies 120 controlled nonlinear cases, including 86 without the sufficient GD-prefix dominance condition. These are local guarantees, not universal capture claims.

The revision also moves beyond frozen quadratic dynamics: Theorem 2 establishes conditional oscillatory tracking under curved and approximate valley geometry with evolving eigendirections, and separates tracking from curvature-reducing drift. Theorems 3–4 add momentum-dependent nonlinear capture and an exact scalar stochastic separation between phase capture and loss reversal.

In conclusion, the revised manuscript therefore addresses both weaknesses with substantive advances: actual-rate and matched long-run neural evidence, momentum-consistent nonlinear theory, and a new finite-time switching-to-attractor guarantee that goes beyond the existence of Bartlett et al.'s quadratic cycle.

We believe these additions materially strengthen the paper's novelty, theoretical scope, and experimental relevance, and respectfully invite the reviewer to reassess the revised submission.

AuthorsOpenReview ↗

Official Response to Reviewer srmj - Questions for Authors

Thank you for these questions. The revision addresses each directly.

1. Actual training step sizes and momentum? Yes - both are now tested explicitly. We distinguish the original post-hoc calibrated η=(2−δ0)/λ^1\eta=(2-\delta_0)/\widehat\lambda_1 interventions (Sec. 7.1), which do not establish natural EoS, from the new studies. E5 adds 126 saved-rate controlled probes; E6 adds 16,800 actual-rate cells on ResNet-18/VGG-11-BN, across GD, SGD, and momentum SGD (mSGD), seven seeds, 20 checkpoints, and 20 radii per group. Native SGD yields 516/2,800 and 499/2,800 robust diagnostic reversals (Table 1). The 216 matched SGD/mSGD continuations preserve schedules and optimizer states; 186 improve final accuracy and all 216 lower test cross-entropy (Table 2). A 72-checkpoint audit verifies independent mSGD buffers on the tested CPU-to-CUDA loading path (App. M), though historical runtime identity is unarchived. Theorem 3 (Sec. 6; App. E) separately treats classical heavy-ball outer momentum, including the incoming buffer; its quadratic stability range is 0<a=ηλ1<2(1+β)0<a=\eta\lambda_1<2(1+\beta), for momentum coefficient β∈[0,1)\beta\in[0,1), with nonlinear cycle persistence and conditional augmented-state capture. Neither natural neural EoS nor sustained cycles are claimed.

2. Is basin capture provable, including ∣y1∣≤ηρλ1=δc|y_1|\le\eta\rho\lambda_1=\delta c? Yes, but radial smallness alone is insufficient. The new Theorem 1 (Sec. 4; App. B) proves finite-time entry into attraction neighborhoods of an exact nonlinear SAM two-cycle for a sufficiently small, principal-gradient-aligned incoming state, without assuming membership in the old frozen basin. With additional GD-prefix assumptions, Eq. (1) certifies a sufficient switching time; all 120 tests pass (Fig. 1).

For the suggested first-iterate condition, Appendix E.4 gives an exact test for the original invariant basin. Set a=ηλ1=2−δa=\eta\lambda_1=2-\delta, c=aρ/δc=a\rho/\delta, and let y1y_1 be the post-switch state. With top Hessian eigenvector v1v_1, P⊥=I−v1v1⊤P_\perp=I-v_1v_1^\top, α1=∣v1⊤y1∣>0\alpha_1=|v_1^\top y_1|>0, r1=∥P⊥y1∥/α1r_1=\|P_\perp y_1\|/\alpha_1, and κ=(λ2/λ1)2/2\kappa=(\lambda_2/\lambda_1)^2/2, the test is

α1/c+κr12≤1.\alpha_1/c+\kappa r_1^2\le1.

Thus ∣y1∣≤δc|y_1|\le\delta c cannot replace transverse control. Proposition E.7 gives a pre-switch certificate including α≤δc\alpha\le\delta c together with alignment and transverse margins. A nearly transverse state defeats radial-only entry, not eventual capture.

3. Five versus four controlled states? These are different cohorts: the 1,725-cell dense sweep reaches at most four consecutive controlled states; the separate 48-cell cross-seed replication reaches at most five (Sec. 7.1; App. L). We define a controlled state by 0<2−ηλ^1,t<10<2-\eta\widehat\lambda_{1,t}<1 and count states, not transitions: five states contain four transitions. Neither certifies a lasting neural cycle.

4. Why ρ\rho-slopes 1.558, 0.819, 1.428 rather than one? The claim is a fixed-state limit as ρ↓0\rho\downarrow0, not a four-point regression prediction. Let g=∇f(w)≠0g=\nabla f(w)\ne0, u=g/∥g∥u=g/\|g\|, H=∇2f(w)H=\nabla^2f(w), and Kρ=−η[∇f(w+ρu)−g]K_\rho=-\eta[\nabla f(w+\rho u)-g]. For an LHL_H-Lipschitz Hessian, Prop. E.9 proves

∥Kρ+ηρHu∥≤ηLHρ2/2.\|K_\rho+\eta\rho Hu\|\le\eta L_H\rho^2/2.

Hence ∥Kρ∥/ρ→η∥Hu∥\|K_\rho\|/\rho\to\eta\|Hu\|, giving limiting log–log slope one when Hu≠0Hu\ne0. Four-point R2R^2 does not verify this limit. E6 therefore reports six-smallest-radius fits over 140 checkpoints per architecture/regime, with median slopes .941/.875 for SGD and .533/.580 for mSGD (ResNet/VGG), and interquartile ranges (Table 1; App. M). Shared outer momentum cancels algebraically in the excess kick; different trained states and finite radii preclude inferring a causal momentum effect from the slope differences.

5. Spectral restriction and presentation. We explicitly identify the aligned reversal window 0<∣y∣<c=aρ/(2−a)0<|y|<c=a\rho/(2-a) as the interior of Bartlett et al.'s known quadratic cycle (Sec. 3). Proposition E.8 makes the old frozen-basin rate requirement qfr2≤1−δq_{\rm fr}^2\le1-\delta (with qfrq_{\rm fr} the transverse-rate bound) precise: each transverse eigenvalue ratio λi/λ1\lambda_i/\lambda_1 lies between δ/4+O(δ2)\delta/4+O(\delta^2) and 1−δ/4+O(δ2)1-\delta/4+O(\delta^2) as δ↓0\delta\downarrow0. Theorem 1 avoids that rate restriction. We clarified definitions and counting, simplified theorem names, cited catapult dynamics (Lewkowycz et al., 2020), corrected Fisher SAM authorship, and repaired the broken cross-reference and incomplete references.

Together, the actual-rate evidence, momentum-consistent theory, nonlinear capture certificate and quantified finite-radius bounds directly address these questions. We respectfully invite reassessment.

Reviewer_qKniOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the dynamical of switching from gradient descent to sharpness-aware minimization (SAM). The authors derive a switching condition under which GD decreases the loss while SAM increases it, and characterize a finite curvature-weighted displacement, and establish a conditional invariant basin leading to a known SAM two-cycle in a frozen quadratic model.

Critical strengths
  1. The paper investigates a relevant question regarding the immediate effects of switching optimizers contrasting existing studies of longer-term SAM dynamics.
  2. The theoretical analysis provides conditions for one-step loss reversal and convergence within a restricted quadratic model.
Critical weaknesses
  1. Insufficient Motivation for the Theoretical Model

The paper introduces the decomposition (w=(x,y)), where (x) and (y) denote tangential and normal coordinates, respectively. Although these variables are formally defined, their geometric motivation and connection to actual neural network optimization trajectories are insufficiently developed. In particular, the analysis assumes a specific sharp-valley loss decomposition without adequately explaining when this model is appropriate for studying late-stage optimizer switching near EoS. It would be helpful to clarify how these coordinates arise, what geometric assumptions justify the decomposition, and how these assumptions relate to realistic neural network loss landscapes.

  1. Limited Experimental Validation of the Theoretical Predictions

The experiments do not sufficiently establish whether the proposed theoretical mechanism explains the observed neural network dynamics. The controlled experiments recover the predicted behavior under relatively restrictive assumptions. However, the neural-network experiments provide primarily qualitative evidence for selected local signatures rather than quantitative validation of the full proposed mechanism. The results under normal or general training regime are still missing in this study. Furthermore, several experimental quantities and classification criteria are defined primarily in the appendix. The main experimental section would benefit from clearer definitions, explanations of how each quantity is calculated, and more direct comparisons between theoretical predictions and observed measurements.

  1. Insufficient Connection to Edge-of-Stability Dynamics

Although the theoretical analysis uses the classical GD stability threshold, it is unclear to what degree the observed phenomena are specific to EoS dynamics. In particular, the neural-network experiments calibrate the learning rate to obtain prescribed initial edge distances. Consequently, the experiments demonstrate behavior under constructed near-edge conditions rather than establishing that the same mechanism naturally emerges during EoS training.

  1. Limited Novelty and Scientific Significance

Several central phenomena investigated in this paper have already been explored in prior work, including the quadratic SAM two-cycle, the importance of normalization, SAM stability boundaries, and the effectiveness of late-stage optimizer switching. Although the paper provides additional local characterizations connecting these results, the contribution is relatively limited. In particular, the one-step switching condition and finite normalized displacement follow from the local SAM update under restrictive assumptions, while the invariant-basin analysis concerns an already-known quadratic two-cycle. More importantly, the paper does not establish that these local results explain the longer-term benefits of late-stage SAM switching or the behavior of realistic EoS optimization trajectories.

Questions for authors
  1. Under what geometric conditions can the neural network loss landscape be represented using the proposed local sharp valley model? How does this model relate to naturally occurring EoS trajectories?

  2. How does stochasticity involved in the training for SAM under the proposed mechanism as it is mini-batch sampling is more practical choice in normal training scenarios?

8 comments
AuthorsOpenReview ↗

Our responses to the weaknesses and questions

Thank you for identifying the need to justify the sharp-valley model rather than merely posit tangent and normal coordinates. The revised paper now derives this structure from explicit geometric hypotheses and proves a switching result with moving curvature and eigendirections.

Weakness 1 and Question 1 — Geometric motivation and relation to EoS

Where do the coordinates come from? Section 5 and Appendix C replace the assumed product-valley picture by a local Morse–Bott model. Let f0f_0 be a smooth nominal objective and M\mathcal M a smooth manifold of its minima. For a parameter vector ww in a tubular neighborhood, define p=πM(w)p=\pi_{\mathcal M}(w) as its nearest point on M\mathcal M and n=w−pn=w-p as the normal displacement. The Hessian vanishes in tangent directions, while its normal restriction HpH_p is uniformly positive definite. Thus

f0(p+n)=f∗+12⟨n,Hpn⟩+O(∥n∥3).f_0(p+n)=f_*+\tfrac12\langle n,H_pn\rangle+O(\|n\|^3).

The former (x,y)(x,y) coordinates are therefore local descriptions of motion along a manifold of minima and motion across its positively curved normal directions; they are not an arbitrary partition of parameters. Importantly, SAM is still analyzed in ambient Euclidean parameter space: the pullback metric of a curved chart is not assumed to be Euclidean.

When is this model justified? Appendix C gives a concrete class: a smooth zero-residual least-squares objective f0(w)=∥F(w)∥2/2f_0(w)=\|F(w)\|^2/2 whose residual map FF has locally constant Jacobian rank. Its zero-residual fiber is locally a manifold, with tangent space ker⁡DF(p)\ker DF(p) and Hessian DF(p)⊤DF(p)DF(p)^\top DF(p) at its minima. This is a sufficient example, not an assertion that general neural cross-entropy objectives, ReLU kink crossings, or rank-changing solution sets satisfy Morse–Bott regularity.

What survives when geometry changes? The new Theorem 2 (Sec. 5; Apps. C–D) admits curved manifolds, varying normal eigenvalues/eigenvectors, and an approximate actual objective f=f0+Rf=f_0+R. Let λ(p)\lambda(p) be the simple largest normal eigenvalue, with a uniform eigengap and fixed sub-edge margins 1+γ≤ηλ(p)≤2−δ1+\gamma\le\eta\lambda(p)\le2-\delta. If ∥∇R∥≤ε\|\nabla R\|\le\varepsilon, the incoming normal residual is sufficiently small relative to the SAM radius ρ\rho, and its gradient is sufficiently aligned with a leading normal eigenvector, finite-time nominal-shell entry follows. Until exit from the valid patch, the physical shell error satisfies

et+1≤ket+K(ρ2+ε),0<k<1.e_{t+1}\le k e_t+K(\rho^2+\varepsilon),\qquad 0<k<1.

For ε=O(ρ2)\varepsilon=O(\rho^2), refined tracking yields the conditional tangential drift

pt+1−pt=−ρ2A(pt)grad⁡Mλ(pt)+O(ρ3+ε),p_{t+1}-p_t=-\rho^2 A(p_t)\operatorname{grad}_{\mathcal M}\lambda(p_t)+O(\rho^3+\varepsilon),

where A(p)=η[1+ηλ(p)/(2−ηλ(p))]2/2A(p)=\eta[1+\eta\lambda(p)/(2-\eta\lambda(p))]^2/2. The stronger condition ε=o(ρ2)\varepsilon=o(\rho^2) preserves the leading drift direction away from curvature-stationary points. An explicit O(ρ2)O(\rho^2) tilted-valley counterexample retains oscillatory tracking while reversing nominal-curvature drift. Thus the revision establishes precisely what the geometry supports—and what it does not.

How does this relate to EoS trajectories? The local normal eigenvalue determines a deterministic subcritical parameter a(p)=ηλ(p)a(p)=\eta\lambda(p). Approaching a(p)=2a(p)=2 enlarges the reference shell, but neither this geometric derivation nor Theorem 2 proves that an SGD-trained network naturally reaches its stochastic edge. In particular, we do not infer Morse–Bott structure or a captured shell from an observed neural Hessian spectrum. The model is now an explicitly motivated, conditional local theory, not a universal reconstruction of neural training dynamics.

Weakness 2 — Quantitative neural validation under actual training protocols

The reviewer is right that controlled quadratic checks alone cannot establish a mechanism in a trained neural network. The revision therefore adds actual-rate interventions, matched long-run continuations, and direct geometry measurements, while distinguishing empirical signatures from unverified assumptions of the capture theorems.

(i) Explicit quantities and reproducible comparisons. Section 7 now defines the diagnostic objective D(w;b)D(w;b) as mean cross-entropy on a fixed subset of 2,048 CIFAR-10 training examples, with frozen BatchNorm (BN) evaluation buffers bb. For a matched baseline/SAM probe, let D0D_0 denote the initial diagnostic loss and ΔDbase,ΔDSAM\Delta D_{\rm base},\Delta D_{\rm SAM} the endpoint-minus-initial differences. A robust diagnostic reversal requires

ΔDbase<−τ,ΔDSAM>τ,τ=max⁡{10−6,10−4∣D0∣}.\Delta D_{\rm base}< -\tau,\quad \Delta D_{\rm SAM}>\tau,\quad \tau=\max\{10^{-6},10^{-4}|D_0|\}.

The definition, training/evaluation distinctions, and main results are now in Sec. 7 rather than only in an appendix. Training-mode BN can update auxiliary statistics during a probe, so the measured signs are not automatically parameter-only reversal signs on a fixed-buffer objective.

(ii) Actual saved learning rates and native minibatches. The original post-hoc edge-calibrated interventions are explicitly retained as constructed diagnostic tests (Sec. 7.1). E5 adds 126 saved-rate controlled cells. E6 adds 16,800 actual-rate cells from 42 source trajectories, spanning ResNet-18 and VGG-11-BN, source regimes GD/SGD/mSGD, seven seeds per architecture/regime, 20 checkpoints, and 20 radii. SGD/mSGD probes use minibatch gradients and saved optimizer settings. Native SGD produces 516/2,800 robust reversals on ResNet and 499/2,800 on VGG, including 751/1,015 at tenfold margin (Table 1); each architecture has events at all seven seeds. mSGD reversals are less common (2 and 36), but those groups begin from different trained states; this is not a controlled causal estimate of momentum's effect. Full-batch GD follows a different, undertrained recipe and is exploratory.

(iii) Long-term training outcomes. A separate six-seed cohort provides 624 matched short continuations and 216 long SGD/mSGD continuation pairs through epoch 200, sharing source checkpoints, incoming optimizer states, saved schedules, and matched augmented minibatches. In the long panel, 186/216 pairs improve final test accuracy and all 216 reduce final test cross-entropy (Table 2). For SGD alone, 93/108 improve accuracy, with mean gain +0.153 percentage points and descriptive six-seed 95% interval [0.118, 0.188]. The 216 configurations reuse source seeds; we do not treat them as 216 independent training replicates. Among 108 SGD configurations joined to separately executed short probes, 78/92 without an initial robust reversal still improve accuracy. Hence reversal is not a necessary observed signature of the gain in this grid; the replay association is not a causal mechanism test.

(iv) Direct post-switch geometry, including counterevidence. In a targeted 24-pair follow-up (not additional independent seeds), estimated finite-radius sharpness at evaluation radius .05.05 falls from 1.466 to .651 on ResNet and .236 to .128 on VGG (Table 3). All 24 native-buffer matched pairs show lower estimates; the direction holds in 96/96 post-switch comparisons over two epochs and two evaluation radii. However, the largest Hessian Ritz estimate increases in all 12 ResNet pairs (5.086 to 8.838 in group means), and VGG curvature changes are mixed. Substituting baseline BN buffers into SAM reverses all 96 sharpness-estimate comparisons. These are important negative controls: the numerical estimator is a lower bound on true worst-direction sharpness, and the swapped-buffer network is a hybrid model. We therefore do not equate estimated finite-radius sensitivity, infinitesimal curvature, or causal generalization.

(v) Discrepancy analysis rather than curve fitting. E5 compares frozen and refreshed-curvature response predictions: refreshing improves VGG error (2.620 to 2.117) but worsens ResNet error (1.410 to 1.891). Thus changing curvature alone does not explain the quantitative discrepancy. The revised experiments support native-rate switching signatures and measured long-run benefits, but do not establish that the neural trajectories enter the precise theorem-certified two-cycle basin or sustain neural period-two motion.

Weakness 3 — What is, and is not, specific to edge of stability?

The revision resolves this ambiguity at both the theoretical and experimental levels. The classical deterministic GD edge for a positive-definite quadratic is ηλmax⁡=2\eta\lambda_{\max}=2; a diagnostic fixed-buffer Hessian is not automatically the operator governing momentum-SGD's stochastic stability. We now distinguish three objects: (a) constructed near-edge diagnostic interventions, (b) actual-rate stochastic neural updates, and (c) conditional local theory with fixed spectral margins.

For the scalar quadratic, put a=ηλ∈(0,2)a=\eta\lambda\in(0,2). Strict same-state GD/SAM reversal occurs within the known two-cycle radius c=aρ/(2−a)c=a\rho/(2-a), and local principal-cycle attraction holds throughout the fixed subcritical interval—not only as a↑2a\uparrow2. Approaching the edge amplifies the shell radius and slows relaxation; it is not necessary for the switching mechanism. This resolves what is genuinely EoS-sensitive instead of conflating reversal with crossing a stability boundary (Sec. 3; App. C).

In the neural study, Sec. 7.1 clearly labels the original η=(2−δ0)/λ^1\eta=(2-\delta_0)/\widehat\lambda_1 probes as prescribed diagnostic-edge interventions. E5/E6 instead use actual saved learning rates. At the epoch-160 source checkpoints in the targeted geometry follow-up, diagnostic ηλ^max⁡\eta\widehat\lambda_{\max} ranges are [.0535, .1132] (ResNet) and [.0687, .3417] (VGG), far below 2 on that diagnostic objective. This does not locate the stochastic training edge; we make no claim of naturally occurring EoS or sustained neural cycles. Instead, it establishes that the observed native-rate diagnostic responses should not be presented as evidence requiring a constructed near-edge calibration. That sharper distinction is now part of the contribution.

Weakness 4 — Novelty and scientific significance beyond existing SAM theory

We explicitly credit Bartlett et al. for the quadratic SAM two-cycle, Wen et al. for prior curvature-drift analysis, and prior work on normalization, stability boundaries and late-SAM effectiveness. We do not claim that the aligned window, the known cycle radius, or the existence of late-SAM benefits is new. The revised paper instead answers an unanswered switching-state question: when can the actual incoming state be certified to enter an attracting nonlinear regime?

Theorem 1 (Sec. 4; App. B) establishes strict same-state reversal, finite-time entry into local attraction neighborhoods, and convergence to an exact nonlinear period-two SAM orbit near an isolated nondegenerate minimum. It assumes a sufficiently small residual and principal-gradient alignment, but does not assume prior membership in the old frozen invariant basin, its restrictive rate condition, or a particular incoming optimizer history. The new GD-prefix result (Eq. 1) gives sufficient, potentially conservative switching-time/radius guidance when additional spectral dominance and quantitative envelopes are available. Fig. 1 checks 120 nonlinear certificate cases, including 86 without the sufficient GD-prefix dominance condition. Those are controlled validation cases, not neural certification.

The remaining new theorems address distinct failure modes of that bridge: Theorem 2 proves conditional capture/tracking under curved and approximate valleys and separates tracking from curvature drift; Theorem 3 establishes momentum-dependent nonlinear cycles and conditional capture in augmented state space; Theorem 4 proves that scalar stochastic phase capture can persist even as strict reversal probability vanishes. These are not restatements of a quadratic switching inequality.

The experiments add a complementary, falsifiable perspective. Across 216 matched long continuations, late switching improves accuracy in 186 and lowers test cross-entropy in all 216; yet neural curvature, estimated finite-radius sharpness, and performance do not move as a single universal quantity. In particular, ResNet's estimated top curvature rises while finite-radius sharpness estimates fall, and diagnostic buffer swapping reverses those estimates. These observations rule out a simplistic parameter-only flatness interpretation of the measurements. They do not prove oscillation-mediated generalization.

The resulting scientific claim is deliberately precise: the revision establishes a conditional switching-to-attractor theorem beyond the frozen quadratic model, identifies robustness boundaries under geometry, momentum and stochastic noise, and quantifies both the benefits and limits of the neural signatures. This is a substantive advance beyond the original local switching calculation.

Reviewer_WSxFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the dynamical changes induced by switching from GD to SAM during the late stages of training, near the EoS. Through theoretical analysis, the authors show that switching optimizers can cause the Loss to transition from decreasing to increasing, while inducing finite perturbations with an H2H^2 spectral filtering effect. Under certain conditions, the parameters may also converge to a period-two oscillatory orbit along the direction of maximum curvature. Through synthetic experiments and experiments with ResNet-18 on CIFAR-10, the authors validate the post-switch Loss reversal, finite perturbations, and transient oscillatory behavior. These findings provide a theoretical explanation for the initial optimization dynamics associated with activating SAM during the late stages of training.

Critical strengths
  1. Clear theoretical contributions. The paper establishes three concrete theoretical results: the conditions under which GD and SAM produce opposite Loss changes from the same optimization state, the spectral structure of the O(ηρ)O(\eta\rho) finite perturbation induced by switching, and sufficient conditions for the local dynamics to converge to a period-two orbit.

  2. Further mechanistic insights into existing findings. Previous studies have investigated the Two-cycle behavior of SAM and the effectiveness of late-stage optimizer switching. This paper further explains how switching optimizers can immediately alter stability and trigger subsequent oscillatory dynamics.

  3. Good integration of theory and experiments. The authors progressively relax the theoretical assumptions through synthetic experiments and conduct controlled optimizer-switching interventions from identical parameter states on ResNet-18/CIFAR-10, validating phenomena such as Loss reversal, finite perturbations, and transient oscillations.

Critical weaknesses
  1. Strong theoretical assumptions. The main results rely on conditions such as local quadratic approximations, Hessian spectral gaps, and frozen curvature, making it difficult to directly generalize the findings to the complex nonlinear training dynamics of real neural networks.

  2. Lack of a complete theoretical connection between initial perturbations and long-term dynamics. Although the paper establishes convergence to a Two-cycle within a specific basin of attraction, it does not prove that switching to SAM necessarily drives the parameters into this basin. Moreover, the theory does not explain the dynamics when curvature and eigendirections continuously evolve during training.

  3. Limited theoretical validation on real neural networks. Although transient periodic oscillations are observed, the actual training trajectories exhibit substantial quantitative deviations from the predictions of the frozen quadratic model. Furthermore, long-term oscillatory behavior has not been validated.

  4. Insufficient connection to final generalization benefits. While the paper explains the initial dynamics following the switch to SAM, it does not establish how these phenomena subsequently lead to reduced Sharpness, the selection of flatter minima, or improved generalization. Therefore, the paper does not yet provide a complete explanation of why switching to SAM during late-stage training is effective.

Questions for authors
  1. How can the local theoretical analysis be extended to real neural networks?

The Two-cycle convergence theorem relies on the assumptions of a frozen Hessian and fixed eigendirections. Given that the curvature and Hessian eigenvectors of real neural networks continuously evolve during training, could the authors further analyze the dynamics under non-stationary curvature and establish corresponding theoretical guarantees?

  1. Does the finite perturbation induced by switching necessarily trigger a Two-cycle?

The paper establishes Two-cycle convergence within a specific basin of attraction, but does not prove that the O(ηρ)O(\eta\rho) perturbation induced by switching to SAM drives the parameters into this basin. Could the authors derive sufficient conditions under which the switching state enters the basin of attraction, thereby providing a more complete theoretical explanation?

  1. What is the relationship between the initial oscillations and the final generalization benefits of SAM?

The paper primarily investigates short-term dynamics after switching, but does not establish a direct connection between Loss reversal, spectral perturbations, or Two-cycle behavior and the eventual reduction in Sharpness or improvement in generalization. Could the authors provide theoretical analysis or experimental evidence explaining how these initial phenomena influence long-term training outcomes?

  1. Why do the dynamics observed in real neural networks deviate substantially from the theoretical Two-cycle predictions?

Although the ResNet-18 experiments exhibit transient alternating oscillations, the observed trajectories do not quantitatively match the predictions of the frozen quadratic model. Could the authors further investigate whether these discrepancies primarily arise from Hessian evolution, higher-order nonlinear effects, stochastic gradient noise, or violations of the theoretical basin-of-attraction conditions?

  1. What is the core novelty of this work compared to existing studies on SAM dynamics?

Previous studies have already investigated the stability boundaries of SAM, Two-cycle behavior, and the effectiveness of switching to SAM during late-stage training. Can the proposed Same-state Switching Window and Finite Spectral Kick predict new phenomena that cannot be explained by existing theories, or provide practical guidance for selecting the switching time and perturbation radius of SAM?

9 comments
AuthorsOpenReview ↗

Our responses to the weaknesses and questions

We appreciate your recognition of the original theoretical contributions and controlled theory--experiment connection. Your concern about the frozen-Hessian assumption identifies a key limitation of the submitted analysis. The revision now establishes a nonlinear, geometrically intrinsic extension that allows curvature, principal eigendirections, and the reference shell to evolve. We also specify precisely when this extension is valid.

W1 / Q1. Can the analysis handle nonstationary curvature and realistic loss geometry?

(a) The quadratic model is no longer the endpoint of the theory. The new Theorem 1 (Sec. 4; App. B) treats the exact nonlinear normalized-SAM map near an isolated nondegenerate minimum w∗w_* with locally Lipschitz Hessian H∗≻0H_*\succ0 and a simple leading eigenpair (λ1,v1)(\lambda_1,v_1). For fixed a=ηλ1∈(0,2)a=\eta\lambda_1\in(0,2) and sufficiently small SAM radius ρ\rho, the map has an attracting prime period-two orbit with

wρ±=w∗±aρ2−av1+O(ρ2).w_\rho^\pm=w_*\pm\frac{a\rho}{2-a}v_1+O(\rho^2).

Crucially, this is a nonlinear persistence and finite-time capture result, not the claim that the nonlinear objective equals its quadratic Taylor approximation. Its sufficient incoming-state conditions and capture proof are discussed separately in our response to W2/Q2.

(b) Theorem 2 removes the need for a fixed principal direction along a valley. Section 5 and Apps. C–D derive the tangential/normal decomposition from a smooth Morse--Bott manifold M\mathcal M of minima of a nominal objective f0f_0. For w=p+nw=p+n, pp is the nearest-point projection onto M\mathcal M and nn belongs to the normal space at pp. The normal Hessian HpH_p is uniformly positive definite, with a simple leading eigenvalue λ(p)\lambda(p) separated by a uniform gap. We analyze ambient Euclidean SAM, accounting for curved coordinates rather than silently treating the chart metric as Euclidean.

Let f=f0+Rf=f_0+R be the actual objective with ∥∇R∥≤ε\|\nabla R\|\le\varepsilon. Under fixed sub-edge margins 1+γ≤ηλ(p)≤2−δ1+\gamma\le\eta\lambda(p)\le2-\delta, sufficiently small/aligned incoming state, and valid tubular-patch margins, the trajectory enters an alternating moving nominal shell. Its physical tracking error satisfies

et+1≤ket+K(ρ2+ε),0<k<1,e_{t+1}\le k e_t+K(\rho^2+\varepsilon),\qquad 0<k<1,

until patch exit, provided forcing fits the invariant-tube margin. Here et=∥nt−stρC(pt)v1(pt)∥e_t=\|n_t-s_t\rho C(p_t)v_1(p_t)\|, st+1=−sts_{t+1}=-s_t, and C(p)=ηλ(p)/[2−ηλ(p)]C(p)=\eta\lambda(p)/[2-\eta\lambda(p)]. Thus the result explicitly tracks evolving eigenvalues, eigenvectors, and footpoints; it does not assert a globally fixed two-cycle along a moving manifold.

(c) Tracking and curvature reduction require different assumptions. When ε=O(ρ2)\varepsilon=O(\rho^2), refined normal error becomes O(ρ2)O(\rho^2) after logarithmic warm-up and the tangential motion obeys

pt+1−pt=−ρ2A(pt)grad⁡Mλ(pt)+O(ρ3+ε),p_{t+1}-p_t=-\rho^2 A(p_t)\operatorname{grad}_{\mathcal M}\lambda(p_t)+O(\rho^3+\varepsilon),

where A(p)=η[C(p)+1]2/2A(p)=\eta[C(p)+1]^2/2. Preserving the leading curvature-decreasing direction requires ε=o(ρ2)\varepsilon=o(\rho^2) away from curvature-stationary points. An explicit O(ρ2)O(\rho^2) tilted-valley counterexample preserves shell tracking yet reverses nominal-curvature drift (Sec. 5; App. D). This identifies a sharp limitation, rather than hiding model mismatch.

Neural scope. A smooth zero-residual least-squares objective with locally constant Jacobian rank provides a concrete Morse--Bott example (App. C). Generic ReLU/BatchNorm-equipped CIFAR-10 networks have not been shown to satisfy these assumptions; a positive top Hessian Ritz value cannot establish them. We therefore claim rigorous local robustness to evolving geometry under explicit conditions, not a theorem certifying arbitrary neural trajectories.

We agree with the central distinction in this question: a finite O(ηρ)O(\eta\rho) switching displacement is not itself a basin-entry proof. The submitted result established conditional convergence inside a restricted quadratic basin. The revised Theorem 1 establishes the missing implication from a certified incoming state to finite-time attraction of the exact nonlinear SAM map.

W2 / Q2. Does switching necessarily induce a two-cycle, and when is capture provable?

A new finite-time nonlinear capture theorem (Sec. 4; App. B). Let w∗w_* be an isolated nondegenerate minimum with a locally Lipschitz Hessian, simple largest eigenvalue λ1\lambda_1 and corresponding unit eigenvector v1v_1. Fix a=ηλ1∈(0,2)a=\eta\lambda_1\in(0,2). For sufficiently small ρ>0\rho>0, if the incoming gradient g=∇f(w)g=\nabla f(w) is nonzero and the state satisfies

∥w−w∗∥≤θρ,min⁡s=±1∥g∥g∥−sv1∥≤θ,\|w-w_*\|\le\theta\rho,\qquad \min_{s=\pm1}\left\|\frac{g}{\|g\|}-sv_1\right\|\le\theta,

for a sufficiently small geometric tolerance θ\theta, then (i) GD strictly decreases the same objective while normalized SAM strictly increases it at that state; (ii) after at most a finite N+1N+1 SAM steps, the orbit enters a pair of exchanged local contraction neighborhoods; and (iii) it converges to the unique prime nonlinear two-cycle within those neighborhoods:

wρ±=w∗±aρ2−av1+O(ρ2).w_\rho^\pm=w_*\pm\frac{a\rho}{2-a}v_1+O(\rho^2).

Why the proof answers the entry objection. After rescaling w−w∗=ρzw-w_*=\rho z, the exact nonlinear map is a controlled C1C^1 perturbation of the frozen map on annuli away from its normalization singularity. The first SAM step from the small aligned incoming state lies near the nonzero principal-axis kick. A finite reference rollout enters strict contraction balls; finite-time error bounds transfer that entry to the nonlinear trajectory. The proof then applies contraction to the exact map. It does not assume the incoming state is already in the submitted invariant basin, impose that basin's spectral-rate restriction, or require a specified pre-switch optimizer. The theorem nonetheless remains local and conditional: it does not guarantee capture from arbitrary states.

Switching-time and radius certificates. For the explicit-certificate regime 1<a<21<a<2, Sec. 4 gives the sufficient radius interval [∥w−w∗∥/θ,ρ0][\|w-w_*\|/\theta,\rho_0], provided the incoming angular test passes. When a GD prefix additionally has principal-mode dominance and certified residual/angular envelopes ∥wt−w∗∥≤R0ℓt\|w_t-w_*\|\le R_0\ell^t and angular error ≤Dangζt\le D_{\rm ang}\zeta^t, Eq. (1) gives a sufficient switching time:

\max\left( 0, \left\lceil \frac{\log(R_0/(\theta\rho))}{-\log\ell} \right\rceil, \left\lceil \frac{\log(D_{\mathrm{ang}}/\theta)}{-\log\zeta} \right\rceil \right).$$ Here, $\ell,\zeta\in(0,1)$; the requisite envelope and dominance hypotheses are spelled out in App. B. A separate finite-reference-rollout certificate also covers selected states outside the simple angular test. These are **sufficient, potentially conservative** criteria, not empirically optimized neural hyperparameters. **Verification and obstruction.** Figure 1(a) checks **120** controlled nonlinear switching states; all satisfy the tested reversal/entry conditions, including **86** without the sufficient GD-prefix dominance condition. Conversely, a lower Hessian eigenaxis may be invariant in a quadratic model: **small residual or loss reversal without principal alignment does not guarantee principal capture**. The correct answer to “necessarily?” is therefore **no in general, yes under the new explicit sufficient conditions**. This resolves the logical gap identified in the review without claiming global convergence.

We agree that a short alternating neural transient cannot be presented as quantitative confirmation of a frozen quadratic two-cycle. The revision now analyzes the discrepancy explicitly, tests refreshed versus frozen curvature on actual network checkpoints, and extends the protocol to native learning rates and stochastic updates. The results are informative precisely because they do not support a single-factor explanation.

W3 / Q4. What causes the mismatch between neural responses and the frozen reference?

(a) An exact response decomposition (App. F). At a common weight state wtw_t, let gt=∇L(wt)g_t=\nabla L(w_t), ut=gt/∥gt∥u_t=g_t/\|g_t\|, Ht=∇2L(wt)H_t=\nabla^2L(w_t) and H0H_0 an anchor Hessian. Compare the frozen-Hessian prediction qtfr=gt+ρH0utq_t^{\rm fr}=g_t+\rho H_0u_t, refreshed prediction qtre=gt+ρHtutq_t^{\rm re}=g_t+\rho H_tu_t, exact full-gradient SAM response qtex=∇L(wt+ρut)q_t^{\rm ex}=\nabla L(w_t+\rho u_t), and same-batch response qtBq_t^B. The difference between the minibatch and frozen-predicted one-step update vectors decomposes exactly into

ΔtB−Δtfr=Dtdrift+Dtnonlin+Dtbatch,\Delta_t^B-\Delta_t^{\rm fr}=D_t^{\rm drift}+D_t^{\rm nonlin}+D_t^{\rm batch},

where the terms are, respectively, curvature change −ηρ(Ht−H0)ut-\eta\rho(H_t-H_0)u_t, the nonlinear Taylor residual −η(qtex−gt−ρHtut)-\eta(q_t^{\rm ex}-g_t-\rho H_tu_t), and the batch/inner-direction discrepancy −η(qtB−qtex)-\eta(q_t^B-q_t^{\rm ex}). Under a Lipschitz Hessian along the perturbation segment, ∥Dtnonlin∥≤ηLHρ2/2\|D_t^{\rm nonlin}\|\le\eta L_H\rho^2/2. This is a vector identity, not a causal variance decomposition or a long-horizon shell-error bound: cross terms matter and basin violations are a separate possibility.

(b) Direct neural test of curvature refresh. In E5, refreshing the Hessian improves mean relative gradient-response error on VGG-11-BN 2.620 to 2.117, but makes it worse on ResNet-18 1.410 to 1.891 (Sec. 7.2; App. M). Those measurements come from short, dependent sequences, not independent neural cycle captures. Hence curvature evolution contributes a testable difference, but refreshing curvature is not a universal explanation of the mismatch. Nor are these errors physical distances to a two-cycle shell.

(c) Actual-rate stochastic probes and model mismatch. E6 includes 42 source trajectories / 16,800 native-rate cells from GD, SGD and momentum-SGD across both architectures. Native SGD gives 1,015/5,600 robust diagnostic reversals, observed across all seven seeds in each architecture (Table 1). The same-batch stochastic coupling bound (Sec. 6; App. C) exposes amplified directional noise when the incoming full gradient is small; Theorem 4 proves exact scalar phase-locking results under bounded noise, not universal neural phase locking. E6's checkpoint-level finite-radius slopes and negative minimum Hessian Ritz diagnostics further caution against interpreting the trained networks as satisfying the SPD shell model.

(d) BatchNorm/evaluation state is consequential. The 24-pair post-switch geometry analysis uses frozen native BN buffers for each trained model. Substituting baseline BN buffers into SAM parameters reverses all 96 post-switch estimated-sharpness comparisons (Sec. 7.4). This hybrid-buffer intervention demonstrates evaluation-state sensitivity, not that BN alone causally explains the frozen-shell discrepancy. Training-mode probes can also alter running statistics, so diagnostic signs are not automatically weight-only loss signs.

What the evidence establishes and what it does not. The revision goes substantially beyond qualitative comparisons with the frozen quadratic model. We derive an exact decomposition of the one-step response discrepancy into curvature evolution, higher-order nonlinear effects, and stochastic-gradient contributions, and evaluate competing explanations on actual neural checkpoints. Importantly, refreshing the curvature estimate improves prediction accuracy on VGG-11-BN but worsens it on ResNet-18, demonstrating that Hessian evolution alone cannot consistently explain the observed deviations. Although the available experiments do not uniquely identify the relative contributions of stochasticity, BatchNorm dynamics, nonlinear effects, and possible violations of the capture conditions, they establish a concrete framework for analyzing these discrepancies rather than treating them as unexplained exceptions. We distinguish short-lived neural oscillatory signatures from certified long-term attraction. The resulting contribution is a rigorous nonlinear switching-to-attractor theory, complemented by quantitative evidence that delineates its applicability and the limitations of frozen-curvature predictions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Author Response and Summary of Revisions

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We thank all three reviewers for their careful assessments of our submission. We especially appreciate the strengths they recognized: Reviewer WSxF highlighted clear theoretical contributions, mechanistic insights, and strong theory–experiment integration; Reviewer srmj found the proofs correct, conditional claims accurate, and presentation clear; and Reviewer qKni recognized the relevance of studying the immediate consequences of optimizer switching and its theoretical characterization.

The central theoretical advance: from switching to nonlinear attraction. Building on Bartlett et al. (2023), Theorem 1 (Sec. 4; App. B) establishes a key missing link between the initial optimizer switch and subsequent dynamics. Under explicit local residual and gradient-alignment conditions, it guarantees strict same-state loss reversal, finite-time basin entry, and attraction to an exact nonlinear period-two orbit, without assuming prior basin membership or a particular optimizer history. We further provide sufficient switching-time and radius certificates. All 120 controlled nonlinear tests satisfy the tested reversal and entry conditions, including 86 cases without sufficient GD-prefix dominance (Fig. 1). These results extend the established quadratic theory by rigorously connecting the incoming switching state to nonlinear oscillatory attraction.

Response to WSxF — Nonlinear capture and evolving geometry. We appreciate WSxF's recognition of the theoretical contributions, mechanistic insights, and theory–experiment integration. Addressing the central basin-entry concern, Theorem 1 now proves finite-time capture into an exact nonlinear attracting two-cycle without assuming prior membership in the original invariant basin. Theorem 2 extends the analysis to curved and approximate valleys with evolving eigendirections, while distinguishing oscillatory tracking from curvature reduction. Appendix F further analyzes neural discrepancies, showing that curvature refresh improves VGG predictions but not ResNet. Together with new long-run experiments, these advances substantially strengthen the theoretical and empirical picture.

Response to qKni — Geometric foundations and practical relevance. We appreciate qKni's recognition of the switching problem's relevance. The revision derives the valley model from Morse–Bott geometry (Sec. 5), extends tracking to evolving curvature (Theorem 2), and adds 126 saved-rate and 16,800 native-rate experimental cells across ResNet-18/VGG-11-BN, including 1,015 robust SGD reversals. These results demonstrate switching signatures beyond artificially calibrated rates. We further clarify that EoS amplifies, but is not required for, local attraction. Theorem 4 establishes an exact scalar stochastic phase-locking threshold, revealing that phase capture can persist even when strict loss-reversal probability vanishes.

Response to srmj — Momentum consistency and mathematical novelty. We appreciate srmj's recognition of the correctness of the proofs, precise conditional claims, and clear presentation. The revision directly addresses the learning-rate and momentum concerns through native-rate interventions and Theorem 3, which establishes momentum-dependent nonlinear cycles and conditional finite-time capture in augmented state space. We also provide first-step basin-entry criteria, clarify transient counts, strengthen finite-radius slope analysis, and correct the requested references. We explicitly credit Bartlett et al. for the quadratic cycle while establishing the distinct contribution of certified nonlinear capture from an incoming switching state.

New long-run evidence (Sec. 7; Tables 2–3, Fig. 2). Across 216 matched SGD/momentum-SGD continuations, late SAM improves final test accuracy in 186 cases and lowers test cross-entropy in all 216. In a targeted 24-pair follow-up, every pair exhibits lower estimated finite-radius sensitivity under native frozen BatchNorm buffers, despite increased leading Hessian Ritz estimates in all 12 ResNet pairs. Buffer substitution reverses the estimated-sharpness comparisons. These findings provide new evidence of late-switching benefits while clarifying why curvature, finite-radius sensitivity, and held-out performance cannot be treated as interchangeable or causally equivalent.

Finally, we sincerely thank the Program Chairs, Senior Area Chairs, Area Chairs, and all reviewers for their time, thoughtful feedback, and constructive guidance. Their comments have helped us substantially strengthen the paper's theoretical foundations, experimental evidence, and scientific positioning. We believe the revised manuscript now makes a clearer and more substantive contribution beyond the original submission. We respectfully invite reconsideration in light of these advances and remain grateful for the opportunity to engage in this valuable discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.