- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- Switch from GD to normalized SAM late in training, near edge of stability
- Same-state window: SAM step can increase loss while GD decreases it
- First SAM displacement bounded, weighted toward high curvature
- Conditions for convergence to the known SAM two-cycle
- CIFAR-10 ResNet checkpoint
UPDATE: Editing this down to a clear reject for me. I think the paper has changed too much since the original review and now has a pretty significant gap between the original setting, theorems, and experiments and the newly developed math and experiments. UPDATE 2: I got a chance to review some of the math more. I think I will maintain my score at a weak reject instead. I ran some small numerical tests to check out the new updated math since I was not a huge fan of the empirical tests done in the paper to connect the theory. Ordinary GD trajectories do seem to match the conditions that the authors have shown in their theoretical updates. These tests were done on synthetic data but made me feel a bit more confident in the new math presented in the rebuttals. However, I still think this paper needs a few more writing passes and needs to be revised heavily to make the writing and the experiments connect more. I would not mind updating my score if the authors can provide better experiments showcasing their new math holds in non synthetic conditions (small experiments are ok and if you can just run on your laptop, that would be great. Doesn't need to be anything requiring GPUs). I would like to see these: Start GD and momentum-GD from init, switch to SAM at different places, and report when the incoming states satisfy the capture conditions that are stated in the theory. Sweep alignment, residual magnitude, LR, and SAM radius so you can compare the certified region with the attraction base. Certificate based proofs truly need more work to showcase that the method lands in the certificate, even for a learning theory track paper. Can you also examine how these results change as LR approaches stability boundary including cases where sufficient conditions fail but capture still happens. I think with these and a rewrite, I would be willing to move this paper up to a weak accept.
Critical strengths
- Proofs correc + claims accurate (basin result labelled conditional), side note- please don't make AI titles for theorems, it's confusing
Critical weaknesses
- Experiments miss the theory's set up:
- calibrated post hoc as (Appendix E only)
- network trained with momentum SGD and cosine decay, not GD at the edge of stability??
- with momentum, threshold is
- Incremental over Bartlett et al. (2023):
- window = two-cycle radius
- Thm 1 = 1D SAM map pushes outward inside the known cycle
Questions for authors
- (Clarification) results at the actual training step size or with momentum?
- (Clarification) basin capture provable? ()
- (Clarification) 5-state (main text) vs. 4-state (Appendix E) transient
- (Clarification) -slopes , , vs. predicted ; on 4 points uninformative
- Suggestions: note window = ; rate condition needs ; plain-language definitions; cite catapult dynamics (Lewkowycz et al., 2020)
- Typos: "Section ??" (pg 30); incomplete references (pg 11); Fisher SAM missing co-author (pg 10)
13 comments
2. Novelty beyond Bartlett et al. (2023): from a known cycle to certified nonlinear capture
We agree that, on the aligned quadratic principal axis,
is precisely the interior of the known SAM two-cycle radius. The revision states this explicitly (Sec. 3). We do not claim discovery of the quadratic cycle or its radius.
The substantive contribution is instead to resolve what the original quadratic calculation leaves open:
Does an incoming switching state actually enter an attracting oscillatory regime, rather than merely receive an outward perturbation?
Theorem 1 (Sec. 4; Appendix B) answers this question for the exact nonlinear SAM map. Near an isolated nondegenerate minimum with locally Lipschitz Hessian and a simple leading eigenvalue, sufficiently small residual and principal-gradient alignment guarantee:
- Strict same-state GD/SAM loss reversal;
- Finite-time entry into local contraction neighborhoods;
- Attraction to an exact nonlinear period-two orbit, with .
Unlike the submitted invariant-basin result, the theorem proves basin entry without assuming prior membership in the original frozen basin or its restrictive spectral-rate condition. It does not prescribe the optimizer history.
Under additional GD-prefix spectral dominance, Eq. (1) provides an explicit sufficient switching-time certificate and conditional perturbation-radius guidance. Figure 1 verifies 120 controlled nonlinear cases, including 86 without the sufficient GD-prefix dominance condition. These are local guarantees, not universal capture claims.
The revision also moves beyond frozen quadratic dynamics: Theorem 2 establishes conditional oscillatory tracking under curved and approximate valley geometry with evolving eigendirections, and separates tracking from curvature-reducing drift. Theorems 3–4 add momentum-dependent nonlinear capture and an exact scalar stochastic separation between phase capture and loss reversal.
In conclusion, the revised manuscript therefore addresses both weaknesses with substantive advances: actual-rate and matched long-run neural evidence, momentum-consistent nonlinear theory, and a new finite-time switching-to-attractor guarantee that goes beyond the existence of Bartlett et al.'s quadratic cycle.
We believe these additions materially strengthen the paper's novelty, theoretical scope, and experimental relevance, and respectfully invite the reviewer to reassess the revised submission.
Official Response to Reviewer srmj - Questions for Authors
Thank you for these questions. The revision addresses each directly.
1. Actual training step sizes and momentum? Yes - both are now tested explicitly. We distinguish the original post-hoc calibrated interventions (Sec. 7.1), which do not establish natural EoS, from the new studies. E5 adds 126 saved-rate controlled probes; E6 adds 16,800 actual-rate cells on ResNet-18/VGG-11-BN, across GD, SGD, and momentum SGD (mSGD), seven seeds, 20 checkpoints, and 20 radii per group. Native SGD yields 516/2,800 and 499/2,800 robust diagnostic reversals (Table 1). The 216 matched SGD/mSGD continuations preserve schedules and optimizer states; 186 improve final accuracy and all 216 lower test cross-entropy (Table 2). A 72-checkpoint audit verifies independent mSGD buffers on the tested CPU-to-CUDA loading path (App. M), though historical runtime identity is unarchived. Theorem 3 (Sec. 6; App. E) separately treats classical heavy-ball outer momentum, including the incoming buffer; its quadratic stability range is , for momentum coefficient , with nonlinear cycle persistence and conditional augmented-state capture. Neither natural neural EoS nor sustained cycles are claimed.
2. Is basin capture provable, including ? Yes, but radial smallness alone is insufficient. The new Theorem 1 (Sec. 4; App. B) proves finite-time entry into attraction neighborhoods of an exact nonlinear SAM two-cycle for a sufficiently small, principal-gradient-aligned incoming state, without assuming membership in the old frozen basin. With additional GD-prefix assumptions, Eq. (1) certifies a sufficient switching time; all 120 tests pass (Fig. 1).
For the suggested first-iterate condition, Appendix E.4 gives an exact test for the original invariant basin. Set , , and let be the post-switch state. With top Hessian eigenvector , , , , and , the test is
Thus cannot replace transverse control. Proposition E.7 gives a pre-switch certificate including together with alignment and transverse margins. A nearly transverse state defeats radial-only entry, not eventual capture.
3. Five versus four controlled states? These are different cohorts: the 1,725-cell dense sweep reaches at most four consecutive controlled states; the separate 48-cell cross-seed replication reaches at most five (Sec. 7.1; App. L). We define a controlled state by and count states, not transitions: five states contain four transitions. Neither certifies a lasting neural cycle.
4. Why -slopes 1.558, 0.819, 1.428 rather than one? The claim is a fixed-state limit as , not a four-point regression prediction. Let , , , and . For an -Lipschitz Hessian, Prop. E.9 proves
Hence , giving limiting log–log slope one when . Four-point does not verify this limit. E6 therefore reports six-smallest-radius fits over 140 checkpoints per architecture/regime, with median slopes .941/.875 for SGD and .533/.580 for mSGD (ResNet/VGG), and interquartile ranges (Table 1; App. M). Shared outer momentum cancels algebraically in the excess kick; different trained states and finite radii preclude inferring a causal momentum effect from the slope differences.
5. Spectral restriction and presentation. We explicitly identify the aligned reversal window as the interior of Bartlett et al.'s known quadratic cycle (Sec. 3). Proposition E.8 makes the old frozen-basin rate requirement (with the transverse-rate bound) precise: each transverse eigenvalue ratio lies between and as . Theorem 1 avoids that rate restriction. We clarified definitions and counting, simplified theorem names, cited catapult dynamics (Lewkowycz et al., 2020), corrected Fisher SAM authorship, and repaired the broken cross-reference and incomplete references.
Together, the actual-rate evidence, momentum-consistent theory, nonlinear capture certificate and quantified finite-radius bounds directly address these questions. We respectfully invite reassessment.
Our response to the weaknesses
We thank the reviewer for identifying two central weaknesses of the original submission: the mismatch between the theoretical optimizer and experimental protocol, and the limited novelty of the quadratic switching analysis relative to Bartlett et al. (2023). Both concerns motivated substantial new theoretical and empirical results in the revision.
1. Experimental–theoretical consistency: actual learning rates and momentum
The original near-edge interventions are now clearly distinguished from actual training. The reviewer is correct that the original experiments used momentum-SGD-trained checkpoints and post-hoc calibrated learning rates η=(2−δ0)/λ1. These interventions establish controlled responses under prescribed diagnostic conditions, not naturally occurring edge-of-stability (EoS) training. We explicitly clarify this distinction in Sec. 7.1.
More importantly, we have conducted new experiments at the actual saved learning rates and with the corresponding optimizer states:
These results establish that the observed switching signatures are not restricted to artificially calibrated learning rates, and that late switching yields measurable held-out benefits under the tested practical training schedules.
We also report important distinctions: mSGD reversal events are substantially rarer, different source trajectories preclude attributing this difference solely to momentum, and the full-batch GD models are undertrained. Diagnostic reversals do not certify natural EoS or sustained neural oscillations.
The theoretical momentum mismatch is addressed by a new theorem, not by reinterpreting GD. Theorem 3 (Sec. 6; Appendix E) analyzes classical heavy-ball outer momentum, explicitly retaining the incoming optimizer buffer in an augmented dynamical state.
For a=ηλ1 and momentum coefficient β∈[0,1), the quadratic stability range becomes
0<a<2(1+β),
as the reviewer correctly notes. We derive the corresponding attracting cycle radius
cβ=2(1+β)−aaρ,
and establish its persistence as an exact attracting nonlinear period-two orbit. Crucially, we also prove conditional finite-time capture from an incoming augmented state under explicit residual, buffer, alignment, and spectral conditions (Theorems E.2–E.5).
The stronger capture condition (1+β)2<a<2(1+β) is sufficient, not necessary; its narrowing at large momentum is explicitly acknowledged. We further derive an exact shared-buffer reversal threshold, showing why the stability boundary alone cannot determine same-state loss signs.
Thus, the revised theory directly models the momentum mechanism raised by the reviewer, while the new experiments test switching under saved learning-rate schedules and optimizer states. We do not conflate deterministic heavy-ball guarantees with stochastic momentum training.