acceptodds
Under review as a conference paper at ICLR 2027

Beyond Quadratic Loss: The Stability Phase Diagram of Adam

Abstract

Loss spikes are recurrent instabilities in neural-network training and can arise from multiple mechanisms. For Adam in particular, macroscopic loss spikes have been linked to optimizer dynamics, yet how its two momentum timescales govern them remains unclear. We investigate this dependence by mapping training dynamics across the plane. Across a range of model–task settings, an approximately linear boundary, , separates spiky from non-spiky dynamics, whereas a one-dimensional quadratic loss produces approximately cubic slope. A one-dimensional superquadratic loss recovers the near-linear scaling and links the boundary coefficient to the effective loss exponent . We further show that confident cross-entropy losses develop a core–wall landscape comprising a narrow quadratic core followed by a steep wall, which produces effective superquadratic behavior at the scale of an optimizer update. Together, these results connect Adam loss spikes to both the mismatch between momentum timescales and finite-scale superquadratic loss geometry beyond the Hessian.

Primary area:
optimization
Keywords:
loss spikes, Adam, non-quadratic loss landscapes

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_vFXZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper studies how Adam's momentum parameters (β1,β2)(\beta_1, \beta_2) control macroscopic loss spikes. The authors scan the (β1,β2)(\beta_1, \beta_2) plane on six small full-batch settings: modular-arithmetic and character-level Transformers, a CIFAR-10 MLP and CNN, and an MNIST autoencoder. They find an approximately linear boundary 1−β2≈C(1−β1)1-\beta_2 \approx C(1-\beta_1) and spikes occur when β2\beta_2 is close to 1 relative to β1\beta_1. A 1D quadratic loss gives a cubic boundary instead, while a 1D superquadratic loss ∣x∣n|x|^n gives a wedge-shaped spiky region. To connect this to real networks, they show that just before a spike the loss along Adam's update direction has a narrow quadratic core followed by a steep superquadratic "wall" arising from confident cross-entropy. They plug the wall's measured exponent into the toy model's boundary formula to estimate each network's boundary coefficient.

Critical strengths
  1. The paper explains the shape of the spike boundary using the loss profile at the scale of an actual update, rather than the Hessian at a single point. The core–wall picture is intuitive and grounded in the adaptive edge-of-stability and loss-spike literature. It also makes simple, testable predictions. For example, the core width of about 1σa\frac{1}{\sigma_a} matches the measured transition in Fig. 5. Even if the quantitative link needs work, the idea that Adam's stability depends on geometry beyond the quadratic core is a useful lens for the community.

  2. The authors scan the full (β1,β1)(\beta_1, \beta_1) plane across six architectures with one consistent pipeline and find a similar near-linear boundary in all of them. This is a clean and reusable empirical finding. The work is also easy to follow and reproduce because:

a) 1D controls over the loss scale kk, learning rate η\eta, ϵ\epsilon and weight decay.

b) A rigorous right-boundary result for any real n>2n > 2 (Theorem 3).

c) A detailed appendix covering period detection, deterministic replay of pre-spike states, and bootstrap fitting.

Critical weaknesses
  1. The experimental scope does not support the paper's framing. All settings are small (at most 9.7M parameters) and trained to near-zero loss which the confident cross-entropy mechanism requires. Yet, the paper is motivated by LLM spikes where training loss stays far from zero. The boundary also lies far from practical settings (β2≤0.7\beta_2 \leq 0.7 at β1=0.9\beta_1 = 0.9) and spikes are not shown to hurt final performance so the practical takeaway is mostly directional.

  2. The theory covers the boundary that real networks do not exhibit. Theorem 3 derives the right boundary but networks show only the left boundary, which is an empirical fit even in 1D.

  3. The paper attributes the absence of the right boundary to the quadratic core but Appendix C.2 shows that weak weight decay alone removes it in the ∣x∣n|x|^n model and all networks use comparable or stronger per-step decay. A network scan without weight decay is needed to isolate the core–wall explanation.

  4. Robustness of the network phase diagrams is not established. Each scan uses a single seed, learning rate and weight decay, with no confidence bars reported on CC or pp. This is important to include because changing η\eta alone shifts the 1D quadratic boundary noticeably (Table 2).

Questions for authors

Q1 [Clarification]. Please report the fitted CC and pp next to the predicted CLC_L in Table 1 with confidence bars. How do you explain the reversed cross-task ordering where for example the CNN has the lowest nwalln_{\text{wall}} but the largest CC? Is there any setting where nwall≤5n_{\text{wall}} \leq 5 so that CLC_L is actually discriminative?

Q2 [Experiment]. Does a right boundary appear in at least one network scan without weight decay? If not, how do you separate the quadratic-core explanation from the weight-decay effect in Appendix C.2?

Q3 [Experiment]. Does the linear boundary persist with mini-batch training at non-zero loss and larger scale (e.g GPT-2-smal)l? Does the spiky phase actually hurt final loss or accuracy?

Q4 [Experiment]. How do CC and pp change across seeds and learning rates in the networks? Please repeat at least one scan with several seeds and two learning rates.

Q5 [Experiment]. Does linear logistic regression on separable data reproduce the linear boundary with the predicted CLC_L? Please also report core width against update size at pre-spike points for all six tasks(Fig 9)

5 comments
AuthorsOpenReview ↗

Response to Reviewer vFXZ

We thank the reviewer for the detailed assessment and constructive feedback. We appreciate the recognition of the intuitive finite-scale core–wall picture, the systematic phase-diagram study across six architectures, and the detailed controls and reproducibility procedures. Our point-by-point responses follow.

W1: The experimental scope limits the connection to practical LLM training.

We thank the reviewer for these comments on scope and practical relevance.

We acknowledge that LLM pretraining, where the training loss remains non-negligible, lies outside the confident, low-loss regime studied here. Nevertheless, training on datasets that are small relative to model capacity is relevant to practical applications where data acquisition or annotation is costly, including biomedical image segmentation (Ronneberger et al., 2015, U-Net) and industrial visual inspection (Batzner et al., 2024, EfficientAD). Understanding optimizer instabilities in these data-scarce settings is therefore valuable.

Our phase diagram also offers a qualitative perspective on practical Adam hyperparameter choices. Compared with the original default of (β1,β2)=(0.9,0.999)(\beta_1,\beta_2)=(0.9,0.999) (Kingma and Ba, 2015), LLM training recipes such as LLaMA use (0.9,0.95)(0.9,0.95) (Touvron et al., 2023). At fixed β1\beta_1, this reduction in β2\beta_2 corresponds to moving toward the non-spiky region in our phase diagram. This provides a directional connection between our findings and practical beta selection.


W2: The theory explains the right boundary, while networks exhibit only the empirical left boundary.

We thank the reviewer for highlighting the distinction between the two boundaries.

As explained in the Introduction, the right-boundary mechanism requires the loss to remain superquadratic as the trajectory approaches the minimum. In neural networks, the trajectory instead enters the quadratic core, cutting off this mechanism. We thank the reviewer for highlighting the lack of an analytical derivation for the left boundary. This remains an important direction for future work. The main contribution of this paper is to identify the left boundary in real neural networks and provide a semiquantitative explanation through the superquadratic toy model and the finite-scale core–wall geometry.


W3: A scan without weight decay is needed to isolate the quadratic-core explanation.

We thank the reviewer for suggesting a control without weight decay.

We have added modular-division phase diagrams with zero and weak weight decay to Appendix D, keeping the learning rate and all other parameters fixed at their baseline values. In both cases, the observed boundary retains an approximately unit slope on log–log axes, although its prefactor changes. These controls show that the approximately linear boundary observed in the neural-network experiments persists without weight decay.


W4: Boundary robustness across seeds, learning rates, and weight decay is not established.

We thank the reviewer for highlighting the need to assess robustness.

Using modular division as a representative task, we have verified the robustness of the phase boundary across three random seeds (Appendix D.2) and examined its sensitivity to learning rate and weight decay (Appendix D.1). The approximately linear scaling persists across these settings, although the prefactor can change.

Table 1 now includes error bars for the fitted CfreeC_{\rm free} and pp, given by their conditional regression standard errors. In addition, Appendix B.2.1 verifies that the overall boundary scaling is robust to changes in the period-detection parameters and low-loss eligibility threshold.

AuthorsOpenReview ↗

Response to Reviewer vFXZ

Q1: Can fitted and predicted coefficients be compared with uncertainties and their cross-task ordering explained?

We thank the reviewer for these suggestions on interpreting the boundary coefficients.

Table 1 now includes the fitted CfreeC_{\rm free} and pp, together with one-SD error bars. We acknowledge that predicting CC from neffn_{\rm eff} is less accurate for modular addition and Tiny Shakespeare. For the remaining four models, D⊥log⁡<0.1D_\perp^{\log}<0.1, as shown in the newly added Appendix F.2. We have not yet found a setting with nwall≤5n_{\rm wall}\leq5 among those examined so far.


Q2: Does a right boundary appear without weight decay?

We thank the reviewer for this question about the role of weight decay.

We have added modular-division phase diagrams with zero and weak weight decay to Appendix D, keeping the learning rate and all other parameters fixed at their baseline values. Without weight decay, the approximately linear left boundary persists, and no distinct right boundary is resolved over the scanned range. Thus, nonzero weight decay is not necessary for the right boundary to remain unresolved in this network, consistent with the quadratic-core explanation.


Q3: Does the boundary persist at larger scale and nonzero loss, and do spikes hurt performance?

We thank the reviewer for asking about broader applicability and optimization performance.

Our modular-addition and modular-division experiments already use mini-batch training with B=512B=512, so the observed approximately linear boundary is not restricted to full-batch training. However, our mechanism concerns confident, low-loss training and does not directly apply to LLM pretraining, where the loss remains non-negligible. We therefore do not establish the same boundary in the larger-scale, non-zero-loss regime asked about here.

Regarding performance, loss spikes are generally undesirable because they can impair training efficiency and final model performance, motivating efforts to mitigate them (Wortsman et al., 2023 and Huang et al., 2025, SPAM).


Q4: How do the boundary coefficient and scaling exponent vary across seeds and learning rates?

We thank the reviewer for suggesting checks across seeds and learning rates.

We have repeated the modular-division scan across three random seeds and two learning rates (Appendices D.1–D.2). The approximately linear scaling persists, with p≈0.88p\approx0.88–0.920.92, while the prefactor CC varies across settings. Tables 5 and 6 report the full fits.


Q5: Does separable logistic regression reproduce the boundary, and how do updates compare with core widths?

We thank the reviewer for suggesting the logistic regression comparison and geometric measurements.

Separable linear logistic regression reproduces a quadratic core and a superquadratic wall, but we do not observe a comparable linear phase boundary. At the examined low-loss state, the actual Adam update is nearly orthogonal to the transverse direction along which the core–wall structure occurs, so the update scarcely probes the wall (Appendix G). We also report the mean typical update and core width at pre-spike points for all six tasks, together with the ratio of these two means, in Appendix E.4, Table 8.

Thank you for addressing my questions. The WD control and multiple seed experiments specifically resolves a big concern of mine. I have increased soundness from 1 to 2 but am not ready to increase the overall score from a 2 to a 3 for the following reasons:

  1. The paper's central claim is still not supported: It says the measured wall exponent predicts the boundary coefficient, but in practice that prediction seems to be a constant. This is substantiated by CL(n)C_L(n) flattening out near 3.5–3.7 for every exponent observed and the authors confirmed that they found no setting with n_wall ≤5\leq 5. C also shifts with seed/LR/WD in the additional experiments without an associated change in n_wall.

  2. The scope might be a bit narrow: As mentioned previously and confirmed by the authors, this regime does not fit the LLM pretraining setting which was how the paper initially motivated itself by stating loss spikes are observed by (Chowdhery et al. 2026)/(Molybog et al. 2023)/(Thilak et al. 2022) which are LLM pretraining based papers as I understand it. The practical takeaway seems to be more directional, thus the main contribution of the paper appears to be mechanistic in nature. Although I acknowledge that it could be useful in settings where datasets are small relative to model capacity (e.g Molecular Discovery, Biomedical Image Segmentation). I would be inclined to move it from 2 to a 3 if even one such practical example could be demonstrated, even at a smaller scale.

Reviewer_aSzCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors investigate an interesting topic about the stability phase boundary of the Adam optimiser in the (β1,β2)(\beta_1,\beta_2) plane. Through experiments across six model–task settings, the authors observe that the phase boundary separating spiky from non-spiky training dynamics is approximately linear. They then study a one-dimensional superquadratic loss and propose a core–wall loss landscape, aiming to connect the finite-scale geometric features of the confidence cross-entropy loss with the empirically measured phase boundary. This work offers a potentially valuable perspective for understanding optimiser dynamics.

Critical strengths

S1: The paper systematically investigates Adam's optimization dynamics in the (β1,β2)(\beta_1,\beta_2) plane and observes a similar approximately linear phase-boundary trend across six model–task settings. S2: The analysis of one-dimensional superquadratic losses provides a useful framework for understanding the interaction between the momentum timescale and non-quadratic loss geometry. S3: The core–wall loss landscape analysis offers a potentially valuable finite-scale perspective that can complement traditional analyses based on the local Hessian.

Critical weaknesses

W1: The phase boundary in real models is primarily explained using the empirical relationship CL≃12(n−2)3n−4C_L \simeq \frac{12(n-2)}{3n-4}, which has yet to be theoretically derived. The authors substitute the empirically measured effective exponent in the wall region into this relationship to estimate the boundary coefficient. However, the current results primarily establish an empirical association and are insufficient to establish its predictive capability or the validity of its mechanistic interpretation. W2: Equation (9) constructs the loss slice along the preconditioned current gradient DtgtD_t g_t, whereas Adam's actual update depends on the first-moment estimate mtm_t. Since these two directions are not necessarily aligned, the current analysis does not sufficiently establish that the observed core–wall structure truly reflects the actual optimisation dynamics when spikes occur. The authors are encouraged to provide an analysis of directional alignment and verify whether the conclusions hold when measurements are taken along the actual update direction. W3: The AdamW weight decay coefficients used in the full-model experiments range from 0.10.1 to 0.50.5, whereas the one-dimensional controlled experiments test only a weak weight decay setting of 10−310^{-3}. The current experiments are insufficient to rule out the influence of weight decay on the phase boundary and loss landscape. The authors are encouraged to conduct controlled experiments on full models under different weight decay settings.

Questions for authors

Q1: For the six tasks, could the authors directly compare the linear boundaries predicted by Eq. (14) with those detected in Figure 2 over a common δ\delta range, and report the prediction errors along with the results accounting for uncertainty propagation from the effective exponent in the wall region? Furthermore, does the effective exponent in the wall region remain stable on both sides of the boundary and under different β\beta settings? Q2: If the effective exponent in the wall region at finite scales is changed by controlling the temperature or data conditions while keeping the momentum-scan protocol consistent, does the measured left boundary shift in the direction and by the magnitude predicted by Eq. (14)? This experiment would help distinguish whether the relationship has genuine predictive capability or merely reflects an empirical correlation observed retrospectively. Q3: Adam's actual updates depend on the first-moment estimate, whereas the paper constructs loss slices along the preconditioned current gradient. Could the authors report the degree of directional alignment between these two directions and verify whether the core–wall structure and its predictions for the phase boundary remain valid when measurements are taken along the actual Adam update direction? Q4: Theorem 3 claims that, under the specified conditions, trajectories will eventually cease to exhibit EoS-triggered spikes. However, the proof in Appendix A.2 relies on conditions such as the boundedness of the rescaled trajectory and entry into the local basin of attraction of a nonzero fixed point. Does this conclusion hold for all initial states within the parameter range specified by the theorem? If not, the authors should explicitly state the additional trajectory conditions required and explain how these conditions correspond to the original Adam dynamics.

2 comments
AuthorsOpenReview ↗

Response to Reviewer aSzC

We thank the reviewer for the careful reading and thoughtful comments. We appreciate the recognition of our systematic phase-diagram study across six model–task settings, the superquadratic model's perspective on momentum and loss geometry, and the core–wall analysis as a complement to local Hessian-based analyses. We respond to the comments and questions below.

W1: The empirical boundary relation lacks a derivation and sufficient predictive validation.

We thank the reviewer for raising this concern about the empirical relation.

We acknowledge that the left-boundary relation is empirical and that a complete analytical derivation remains open. Nevertheless, it fits the one-dimensional results in Fig. 3(h) well, with R2≈0.95R^2\approx0.95. Our superquadratic toy model captures the approximately linear boundary scaling and explains why spikes emerge at larger β2\beta_2 for fixed β1\beta_1.

To assess its predictive value for real models, Appendix F.2 now directly compares boundaries predicted from independently measured wall exponents with the detected neural-network boundary points. The comparison shows reasonably good agreement in the overall boundary location for most tasks. Together with the observed superquadratic geometry at the optimizer's update scale, these results support the proposed connection and the wall exponent's value as an empirical predictor of boundary location.


W2: Slices along the preconditioned gradient may not reflect actual Adam dynamics.

We thank the reviewer for suggesting that we examine the actual update direction.

We use the preconditioned gradient direction in the main text for consistency with the Edge-of-Stability (EoS) analysis. In Appendix E.3, we now quantify its alignment with the actual Adam update in the modular-division Transformer. At pre-spike points, the mean cosine similarity between the actual displacement Δθt=θt+1−θt\Delta\theta_t=\theta_{t+1}-\theta_t and the preconditioned descent direction −Dtgt-D_tg_t ranges from approximately 0.720.72 to 0.940.94 across the six beta settings, indicating substantial alignment.

Loss slices along the actual update retain the quadratic core, superquadratic wall, and outer rollover, with similar effective wall exponents. These results support the relevance of the core–wall geometry to the actual optimizer dynamics.


W3: Full-model controls are needed to isolate the influence of weight decay.

We thank the reviewer for suggesting further controls on weight decay.

We have added controlled modular-division experiments with weak and zero weight decay to Appendix D, keeping the learning rate and all other parameters fixed at their baseline values. The phase boundaries retain an approximately unit slope on log–log axes, although their prefactors change. These results show that weight decay can affect the boundary location, but the approximately linear scaling persists without weight decay.

AuthorsOpenReview ↗

Response to Reviewer aSzC

Q1: Do predicted boundaries match observations within uncertainty, and are wall exponents stable?

We thank the reviewer for suggesting a direct quantitative comparison of the boundaries.

We have added Appendix F to address both questions. Table 10 quantifies the distances between the boundaries predicted by Eq. (15) and the detected boundaries over a common δ\delta range. Modular addition and Tiny Shakespeare show larger discrepancies, whereas the remaining four tasks have D⊥log⁡<0.1D_\perp^{\log}<0.1, indicating good agreement between the boundaries predicted from neffn_{\rm eff} and those detected numerically.

In addition, for each model–task setting, we report the sample standard deviation sβs_\beta of neffn_{\rm eff} across different beta settings in Tables 1 and 9. The discussion above concerns the left boundary. The right boundary is not observed in real networks because of the core–wall structure.


Q2: Does changing the wall exponent shift the boundary as predicted?

We thank the reviewer for suggesting a more direct test of predictive capability.

Because neffn_{\rm eff} depends on a complex combination of factors, we are not yet able to tune it through changes to the data or temperature. This remains an interesting direction for future research.

In our current study, both neffn_{\rm eff} and the observed boundary coefficient CC vary across models and are positively correlated. This observation partially addresses your question.


Q3: Do alignment measurements and slices along actual updates support the geometry and boundary predictions?

We thank the reviewer for this question about directional alignment.

We use the preconditioned gradient direction in the main text for consistency with the Edge-of-Stability (EoS) analysis. Appendix E.3 now examines alignment and loss geometry along the actual Adam update in the modular-division Transformer. At pre-spike points, the mean cosine similarity between the actual update and the preconditioned descent direction −Dtgt-D_tg_t ranges from approximately 0.720.72 to 0.940.94 across the six beta settings. The actual-update slices retain the quadratic core, superquadratic wall, and outer rollover, with similar effective wall exponents. Substituting these exponents into Eq. (15) also gives comparable boundary estimates, supporting the robustness of the empirical prediction to the choice of slice direction in modular division.


Q4: Does Theorem 3 require additional trajectory conditions beyond its parameter inequality?

We thank the reviewer for requesting clarification of the theorem's assumptions.

We have revised Theorem 3 to explicitly state a local result for trajectories entering the attraction basin of the self-decay branch. The parameter inequality alone does not guarantee this for arbitrary initial states. Appendix A.2 now specifies the local-entry condition and derives bounded rescaling from it. In the original Adam variables, xt=qt(C+o(1))→0x_t=q^t(C+o(1))\to0, vt∼V∞β2tv_t\sim V_\infty\beta_2^t, and St→S∗<1S_t\to S_*<1, where q=β21/[2(n−2)]q=\beta_2^{1/[2(n-2)]}. Thus, the nonzero rescaled fixed points describe decay toward the minimum and eventual cessation of EoS-threshold crossings. This local scope is consistent with Bai et al. (2026), Remark 4.3.

Reviewer_8NrTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the mechanism of macroscopic or long-periodic loss spikes in Adam's practical training dynamics and the phase diagram for the existence of these spikes with respect to Adam's momentum parameters β1\beta_1 and β2\beta_2. The authors identify an approximately linear boundary 1−β2≲C(1−β1)1-\beta_2\lesssim C(1-\beta_1) across different tasks. This boundary resembles the upper boundary of superquadratic toy models in terms of scaling, while the spiky region extends toward small 1−β21-\beta_2, similar to quadratic losses. The authors propose a new geometric description, the core–wall landscape, to reconcile this difference. They partially validate this description through loss landscape measurements.

Critical strengths
  1. The paper presents an interesting empirical observation on the approximately linear beta-phase boundary of Adam across different models and tasks, and investigates it with some simple polynomial losses.

  2. The core–wall landscape provides a reasonable and measurable mental model to reconcile the discrepancy. The comparison between quadratic and superquadratic toy models provides useful insights into the role of non-quadratic loss geometry in Adam's stability.

Critical weaknesses
  1. The theoretical characterization of the phase boundary is incomplete. In particular, the left boundary relevant to the neural-network experiments is only characterized empirically, while the theoretical result concerns the other boundary of the pure superquadratic toy model. The proposed connection to neural-network stability is therefore not established theoretically.

  2. The quantitative connection between the wall exponent and the phase boundary is not sufficiently convincing. The measured wall exponents vary substantially across tasks, but the resulting predicted boundary coefficients lie in a narrow range (3.49–3.72). It is unclear how much explanatory power the measured exponent provides beyond a constant-coefficient baseline. Moreover, the fitted empirical boundaries do not always closely match the proposed linear scaling.

Questions for authors
  1. Learning-rate dependence. The comparison between the update scale and the core–wall width seems to be coupled with the learning rate. With a sufficiently small learning rate, Adam's updates might become much smaller, and the dynamics might be more restricted to the quadratic core. Meanwhile, Adam can still exhibit oscillatory dynamics even on quadratic losses. So my question is, do the authors still observe similar macroscopic spikes with smaller learning rates? If so, do they happen earlier or later in training? How does the beta-phase diagram change with different learning rates? In particular, could the boundary scaling approach the cubic scaling observed in the quadratic toy model? And does the magnitude comparison in Figure 9 hold across different learning rates?

  2. Long-term evolution of the loss geometry. Similar to the question above, assuming that the learning rate is fixed, do the authors expect the core–wall geometry to evolve over the course of training? For example, the oscillations might induce some form of sharpness regularization, which could potentially lead to a wider quadratic core. Have the authors tested how this geometry evolves over long training trajectories, rather than only around individual spikes? In particular, could the core become wider in later stages of training, so that the dynamics become more restricted to the quadratic region?

  3. A simpler model with separable logistic regression. Could the authors run a similar experiment using Adam on a linear logistic regression model with separable data? I think the loss landscape along directions perpendicular to the separating direction can naturally exhibit a quadratic core and superquadratic growth, which largely aligns with the proposed core–wall model. It would provide significant additional evidence for the proposed mechanism if the experiments on this simple loss reproduce what the authors observe on neural-network tasks. In particular, it would be interesting to see whether the same linear beta-phase boundary can be observed.

  4. Implications for practical optimization. In general, do the authors think that the near-stability behavior identified in this paper is helpful for optimization? In particular, how does the ((\beta_1,\beta_2)) phase diagram influence the long-term loss decay properties in LLM pretraining? Does being in the spiky or non-spiky region lead to meaningful differences in optimization efficiency? Also, do the authors expect the same phase transition to persist in stochastic mini-batch training?

  5. The role of weight decay. The authors argue that the quadratic core explains why the second phase boundary of the superquadratic toy model is absent in neural networks. However, Appendix C.2 shows that even weak weight decay substantially changes the phase diagram and makes the second boundary unresolved over the tested range. Since all the neural-network experiments use nonzero weight decay, could weight decay itself explain the absence of the second boundary? Have the authors tested the phase diagrams without weight decay?

  6. Actual oscillation directions and amplitudes. The geometry measurements are mainly conducted along the Adam-preconditioned gradient direction, which is not necessarily the actual update or oscillation direction due to momentum. Have the authors compared the actual oscillation amplitude with the quadratic-core width? In particular, do the oscillations really explore the superquadratic wall in the language-modeling experiment?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8NrT

We thank the reviewer for carefully reading our manuscript and for the constructive comments and suggestions. We appreciate the recognition of the approximately linear boundary across models and tasks, and of the core–wall landscape and toy-model comparisons as useful perspectives on non-quadratic loss geometry. Below, we address the comments and questions point by point.

W1: The theory characterizes the right boundary, while the relevant left boundary remains empirical.

We thank the reviewer for highlighting the limits of our theoretical analysis.

We acknowledge that the left-boundary relation is empirical. Nevertheless, it achieves R2=0.9497R^2=0.9497 for the data shown in Fig. 3(h), indicating a good fit. Regarding the use of a toy model for the right-boundary analysis, the complexity and fluctuations of Adam's dynamics in real networks make a direct analytical treatment impractical, motivating a minimal model. Our one-dimensional model captures the approximately unit boundary slope, explains why spikes emerge at larger β2\beta_2 for fixed β1\beta_1, and provides estimates of the boundary location. These results establish the proposed connection between the toy model and the observed neural-network phase diagrams.


W2: The wall exponent's explanatory power beyond a constant coefficient is unclear.

We thank the reviewer for raising this point about the quantitative connection.

The narrow range of predicted coefficients (3.493.49–3.723.72) follows from the form of CL(n)=12(n−2)/(3n−4)C_L(n)=12(n-2)/(3n-4), which approaches 44 as nn increases. The measured effective exponents still differ substantially across models.

We acknowledge the weaker agreement for Tiny Shakespeare and modular addition. However, Table 10 in Appendix F.2 shows that the boundaries predicted from the measured effective exponents have D⊥log⁡<0.1D_\perp^{\log}<0.1 for all four remaining models.

The uncertainty in this quantitative connection leaves our central contributions intact: identifying this boundary in real neural networks for the first time and explaining it through non-quadratic loss geometry, and establishing how this geometry arises from confident cross-entropy.


Q1: How does the learning rate affect spikes, boundary scaling, and core–wall geometry?

We thank the reviewer for these questions about learning-rate dependence.

We have added the phase boundary for modular division at a smaller learning rate in Appendix D.1 (Figure 10, left, and Table 5). Due to computational constraints, we have not tested additional learning rates or models. We still observe macroscopic spikes. At the smaller learning rate, the spikes appear at later training steps, while their period changes little. The reason is that the parameter displacement during each spike also depends on η\eta.

In the phase diagram, the boundary scaling exponent remains nearly unchanged. The prefactor CC changes, possibly because the effective exponent neffn_{\rm eff} changes. The finite-scale progressive sharpening illustrated in Figure 13, as well as the comparable scales of the update and the quadratic core at pre-spike points, also hold at the smaller learning rate.


Q2: Does the quadratic core widen over long training trajectories?

We thank the reviewer for suggesting a longer-term view of the loss geometry.

We have added Appendix E.4, "Long-term evolution of the core–wall structure," with a figure following the modular division Transformer at fixed (β1,β2)=(0.9,0.999)(\beta_1,\beta_2)=(0.9,0.999) over multiple spike cycles. We compare the total quadratic-core width with the typical projected update at pre-spike and ordinary checkpoints. The core is much wider than a typical update at ordinary checkpoints and contracts to a comparable scale before each spike. The sampled pre-spike widths do not systematically increase at later times, providing no evidence of progressive confinement to the quadratic core in this trajectory.

AuthorsOpenReview ↗

Response to Reviewer 8NrT

Q3: Does separable logistic regression reproduce the core–wall geometry and linear boundary?

We thank the reviewer for suggesting a comparison with logistic regression.

We have added Appendix G, "Separable linear logistic regression," with the requested linear model on strictly separable data. We observe a quadratic core and a superquadratic wall along the transverse direction, perpendicular to the separating direction. However, at the illustrated low-loss state, the absolute cosine between the actual Adam update and this transverse direction is only 1.64×10−61.64\times10^{-6}, so the update scarcely probes the wall. For comparison, the mean cosine between the actual update and the preconditioned descent direction is 0.71640.7164–0.93940.9394 across the six beta settings of modular division (Appendix E.3). Therefore, no linear boundary was observed.


Q4: What are the implications for optimization efficiency and stochastic mini-batch training?

We thank the reviewer for asking about the implications for practical optimization.

Our findings suggest a direction for optimizer design, as discussed in the second outlook point of our manuscript. They also help interpret the shift in Adam's beta settings from the original default of (β1,β2)=(0.9,0.999)(\beta_1,\beta_2)=(0.9,0.999) (Kingma and Ba, 2015) to settings such as (0.9,0.95)(0.9,0.95) in LLM pretraining (Touvron et al., 2023, LLaMA). In our phase diagram, this reduction in β2\beta_2 corresponds to moving toward the non-spiky region, consistent with the practical goal of avoiding loss spikes. Although there are exceptions, such as the Slingshot mechanism (Thilak et al., 2022), loss spikes generally degrade model performance and should be avoided (Wortsman et al., 2023).

As for stochastic mini-batch training, our modular-addition and modular-division experiments already use mini-batches with B=512B=512, and we expect the same transition to persist at smaller batch sizes.


Q5: Could weight decay explain the absence of the right boundary?

We thank the reviewer for drawing attention to the role of weight decay.

We have added phase boundaries for modular division with smaller weight decay and with no weight decay to Appendix D. In both cases, the boundaries retain an approximately unit slope on log–log axes.


Q6: Do actual update directions and oscillation amplitudes support the core–wall picture?

We thank the reviewer for asking about actual update directions and amplitudes.

We have added Appendix E.3, "Core–wall geometry along the actual Adam update," using the modular division Transformer. The new loss slices use the measured displacement Δθt=θt+1−θt\Delta\theta_t=\theta_{t+1}-\theta_t and show that the quadratic core, superquadratic wall, and outer rollover persist along the actual update direction.

The bottom panel of Figure 16 in Appendix E.4, "Long-term evolution of the core–wall structure," directly compares the total quadratic-core width with the typical projected single-step update at modular division pre-spike checkpoints. At five later checkpoints, this update exceeds the entire core span, with update-to-core-width ratios ranging from 1.0031.003 to 1.5561.556. Thus, a displacement at this projected scale already extends beyond the measured quadratic-core interval, supporting the relevance of the superquadratic wall and its large effective exponent at the update scale.

Thank you for the detailed rebuttal and substantial additional experiments. The analyses along the actual Adam updates, the comparison between core width and update scale, and the additional controls have addressed several of my concerns and strengthened the proposed core–wall interpretation.

Overall, I appreciate the authors' efforts and the improvements to the paper. I will raise my overall rating from 2 to 3.

Reviewer_FTw3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Summary: This submission analyzes Adam’s phase diagram in the (β1,β2)(\beta_1,\beta_2) space beyond quadratic loss functions. First, the authors empirically identify an approximately linear boundary, 1−β2≈C(1−β1)1-\beta_2 \approx C(1-\beta_1), separating spiky and non-spiky dynamics across six neural-network settings. At fixed β1\beta_1, the spiky regime occurs at larger values of β2\beta_2. Second, they show that a one-dimensional quadratic loss produces approximately cubic boundary scaling, whereas superquadratic losses recover approximately linear scaling. Finally, they propose a "core–wall" landscape for confident cross-entropy losses, consisting of a narrow quadratic core, a steep superquadratic wall, and an outer rollover. They support this picture with directional loss slices at pre-spike checkpoints and relate measured wall exponents to the empirical boundary coefficient.

Critical strengths
  1. Systematic empirical characterization of Adam’s instability regimes. The paper maps the dependence of macroscopic loss oscillations on both momentum parameters across six model–task settings and identifies an approximately linear phase boundary. The contrast with the approximately cubic boundary of the quadratic toy model provides an informative empirical target and motivates studying optimizer dynamics beyond quadratic approximations.

  2. A useful finite-scale perspective on loss geometry. The directional loss slices reveal a narrow quadratic core followed by a superquadratic wall, and compare the core width with the optimizer’s update scale. The confident cross-entropy analysis provides a plausible mechanism for this geometry and, using` a Gaussian slope model, an estimate of the core width. Together, these offer a concrete framework for investigating behavior that infinitesimal Hessian-based analysis may miss.

Critical weaknesses
  1. The connection between the proposed geometry and Adam’s phase boundary is not sufficiently established. The core–wall evidence comes from selected one-dimensional slices along the preconditioned gradient, whereas Adam’s actual updates depend on momentum and multidimensional dynamics. Moreover, the left-boundary formula relevant to the neural-network experiments is empirically fitted in the toy model, rather than analytically derived. The paper does not sufficiently validate why this formula should transfer to neural networks or whether the measured wall exponent accurately characterizes the geometry encountered by actual updates. This limits support for the central explanatory claim.

  2. The demonstrated applicability is restricted to small-data, low-loss training regimes. The experiments use small datasets or subsets, and most settings require a low training loss for an accepted period. The confident cross-entropy analysis also assumes predictions approach their targets, with bounded logit derivatives. These conditions are not generally satisfied in settings such as LLM pretraining, where training loss remains non-negligible. It is therefore unclear whether the observed phase boundary and proposed core–wall mechanism persist outside near-interpolation regimes, limiting the generality of the conclusions.

Questions for authors
  1. Does the linear beta-boundary and core–wall geometry persist when training loss remains non-negligible, as in LLM pretraining? What training regime is the proposed mechanism intended to explain?

  2. How well does the preconditioned gradient direction align with the actual momentum-based Adam update? Does the core–wall picture also hold along actual update directions?

  3. How sensitive are the phase boundaries to the period-detection criterion, low-loss eligibility condition, and training horizon?

  4. How well does the Gaussian model describe the loss-weighted margin slopes?

2 comments
AuthorsOpenReview ↗

Response to Reviewer FTw3

We thank the reviewer for the careful reading and constructive feedback. We appreciate the positive assessment of our systematic characterization of Adam's instability regimes across six model–task settings and of the finite-scale perspective offered by the core–wall geometry. Below, we address the reviewer's comments and questions point by point.

W1: The link between core–wall geometry and Adam's phase boundary is insufficiently established.

We thank the reviewer for highlighting the need for a stronger mechanistic connection.

We use the preconditioned gradient direction in the main text to align with the Edge-of-Stability (EoS) analysis. We have added Appendix E.3, "Core–wall geometry along the actual Adam update," which directly examines loss slices along the measured displacement Δθt=θt+1−θt\Delta\theta_t=\theta_{t+1}-\theta_t in the modular-division Transformer. The quiet, pre-spike, and peak slices and the pre-spike log–log profile show that the quadratic core, superquadratic wall, and outer rollover persist along this direction. Across the same 50 resolved events at six reference beta pairs, the unweighted mean wall exponent changes from 10.892310.8923 along DtgtD_tg_t to 9.92249.9224 along the actual update, a decrease of approximately 8.98.9\\%. This provides evidence that the core–wall geometry observed in modular division is retained along the actual optimizer update.

We agree that the left-boundary coefficient CL(n)=12(n−2)/(3n−4)C_L(n)=12(n-2)/(3n-4) is an empirical relation, and a complete analytical derivation remains open. Prior studies of Adam's stability phase boundaries have been limited. In particular, Bai et al. (2026) derived the right boundary for degenerate polynomial losses, which explains the suppression of spikes when β2\beta_2 becomes sufficiently large. It does not explain the boundary observed in our neural-network experiments, where spikes emerge as β2\beta_2 increases at fixed β1\beta_1. Our contribution is to systematically characterize this boundary across real models and identify a mechanism for the distinct left-boundary behavior through the superquadratic toy model and finite-scale core–wall geometry. Deriving the precise dependence of CLC_L on nn requires further analysis and remains an important direction for future work.

Nevertheless, over the 15 exponents shown in Fig. 3(h), the existing empirical relation achieves R2=0.9497R^2=0.9497. Table 10 further compares the predicted boundaries, obtained from the independently measured effective wall exponents, directly with the detected neural-network boundary points, showing good agreement. These results support the effective wall exponent as a useful empirical predictor of boundary location, while leaving the analytical derivation open.

The coupled, multidimensional dynamics of Adam make a direct analytical treatment of neural-network phase boundaries difficult. A controlled toy model allows us to isolate how superquadratic geometry changes the phase diagram and produces the left-boundary behavior observed in the full models. Its relevance is supported by the measured superquadratic walls at the scale explored by the optimizer and by the comparison between predicted and detected boundaries. The additional analysis in Appendix E.3 also accounts for momentum through the measured Adam displacement and retains the central core–wall structure in modular division. Together, these findings support the proposed connection.

AuthorsOpenReview ↗

Response to Reviewer FTw3

W2: The demonstrated applicability is limited to small-data, low-loss training.

We thank the reviewer for raising this concern about generality.

Our main-text analysis explicitly focuses on confident entropic losses. We acknowledge that LLM pretraining, where the training loss typically remains O(1)O(1), lies outside the confident, low-loss regime considered here. Nevertheless, training on datasets that are small relative to model capacity is relevant to many practical applications, particularly in domains where data acquisition or annotation is costly. Examples include biomedical image segmentation, such as cell and surgical-instrument segmentation (Ronneberger et al., 2015, U-Net and Ma et al., 2024, U-Mamba), and industrial visual inspection (Huang et al., 2020, A Compact Convolutional Neural Network for Surface Defect Inspection and Batzner et al., 2024, EfficientAD). Understanding optimizer instabilities during low-loss training in these data-scarce settings is therefore valuable for these applications.


Q1: Does the mechanism extend beyond confident, low-loss training?

We thank the reviewer for seeking clarification on the intended training regime.

Our mechanism concerns confident, low-loss training and does not directly apply to LLM pretraining, where the loss remains non-negligible. We therefore do not establish that the same phase boundary or core–wall geometry persists in that regime. As discussed in our response to W2, low-loss training remains relevant to data-scarce applications such as biomedical image segmentation and industrial visual inspection.


Q2: Do actual Adam updates align with the slice direction and retain core–wall geometry?

We thank the reviewer for this question about the actual update direction.

As reported in Table 7, the mean cosine similarity between the preconditioned descent direction −Dtgt-D_tg_t and the actual Adam update direction is approximately 0.720.72–0.940.94 across different beta settings in modular division, indicating substantial overlap between the two directions. As shown in Appendix E.3, the core–wall geometry also persists along the actual update direction.


Q3: How sensitive are the boundaries to detection, low-loss eligibility, and training horizon?

We thank the reviewer for raising the issue of boundary robustness.

We have added a sensitivity analysis to Appendix B.2.1, "Sensitivity to period detection and low-loss eligibility," using modular division as a representative example. The two main parameters are the low-pass cutoff fcf_c, which suppresses rapid fluctuations before spectral estimation, and the relative prominence threshold qq, which requires an accepted spectral peak to satisfy prominence⁡(fk)≥qmax⁡fP(f)\operatorname{prominence}(f_k)\geq q\max_f P(f). For modular division, we find that the phase boundary is robust to different choices of the period-detection parameters.

We also varied the low-loss eligibility threshold, with little effect on the overall boundary scaling (Figure 7 and Table 3).

For the training horizon, we begin with an initial step budget and return the detected period if one is found. Otherwise, we extend the trajectory in stages until reaching a task-specific cap. For example, the modular-division scan in Figure 2 uses a cap of 30,000 optimizer steps, while the MNIST autoencoder permits extensions up to 200,000 steps to accommodate longer periods (Appendix B.1). Manual inspection of selected trajectories that reached the cap also showed no discernible periodicity within the observation window.


Q4: How well does the Gaussian model describe the loss-weighted margin slopes?

We thank the reviewer for this question about the Gaussian approximation.

The Gaussian model is a convenient simplification that provides an intuitive picture. For more general distributions, σa\sigma_a can also characterize the core width. We have generalized Theorem 7 accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.