- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies how Adam's momentum parameters control macroscopic loss spikes. The authors scan the plane on six small full-batch settings: modular-arithmetic and character-level Transformers, a CIFAR-10 MLP and CNN, and an MNIST autoencoder. They find an approximately linear boundary and spikes occur when is close to 1 relative to . A 1D quadratic loss gives a cubic boundary instead, while a 1D superquadratic loss gives a wedge-shaped spiky region. To connect this to real networks, they show that just before a spike the loss along Adam's update direction has a narrow quadratic core followed by a steep superquadratic "wall" arising from confident cross-entropy. They plug the wall's measured exponent into the toy model's boundary formula to estimate each network's boundary coefficient.
Critical strengths
-
The paper explains the shape of the spike boundary using the loss profile at the scale of an actual update, rather than the Hessian at a single point. The core–wall picture is intuitive and grounded in the adaptive edge-of-stability and loss-spike literature. It also makes simple, testable predictions. For example, the core width of about matches the measured transition in Fig. 5. Even if the quantitative link needs work, the idea that Adam's stability depends on geometry beyond the quadratic core is a useful lens for the community.
-
The authors scan the full plane across six architectures with one consistent pipeline and find a similar near-linear boundary in all of them. This is a clean and reusable empirical finding. The work is also easy to follow and reproduce because:
a) 1D controls over the loss scale , learning rate , and weight decay.
b) A rigorous right-boundary result for any real (Theorem 3).
c) A detailed appendix covering period detection, deterministic replay of pre-spike states, and bootstrap fitting.
Critical weaknesses
-
The experimental scope does not support the paper's framing. All settings are small (at most 9.7M parameters) and trained to near-zero loss which the confident cross-entropy mechanism requires. Yet, the paper is motivated by LLM spikes where training loss stays far from zero. The boundary also lies far from practical settings ( at ) and spikes are not shown to hurt final performance so the practical takeaway is mostly directional.
-
The theory covers the boundary that real networks do not exhibit. Theorem 3 derives the right boundary but networks show only the left boundary, which is an empirical fit even in 1D.
-
The paper attributes the absence of the right boundary to the quadratic core but Appendix C.2 shows that weak weight decay alone removes it in the model and all networks use comparable or stronger per-step decay. A network scan without weight decay is needed to isolate the core–wall explanation.
-
Robustness of the network phase diagrams is not established. Each scan uses a single seed, learning rate and weight decay, with no confidence bars reported on or . This is important to include because changing alone shifts the 1D quadratic boundary noticeably (Table 2).
Questions for authors
Q1 [Clarification]. Please report the fitted and next to the predicted in Table 1 with confidence bars. How do you explain the reversed cross-task ordering where for example the CNN has the lowest but the largest ? Is there any setting where so that is actually discriminative?
Q2 [Experiment]. Does a right boundary appear in at least one network scan without weight decay? If not, how do you separate the quadratic-core explanation from the weight-decay effect in Appendix C.2?
Q3 [Experiment]. Does the linear boundary persist with mini-batch training at non-zero loss and larger scale (e.g GPT-2-smal)l? Does the spiky phase actually hurt final loss or accuracy?
Q4 [Experiment]. How do and change across seeds and learning rates in the networks? Please repeat at least one scan with several seeds and two learning rates.
Q5 [Experiment]. Does linear logistic regression on separable data reproduce the linear boundary with the predicted ? Please also report core width against update size at pre-spike points for all six tasks(Fig 9)
5 comments
Response to Reviewer vFXZ
Q1: Can fitted and predicted coefficients be compared with uncertainties and their cross-task ordering explained?
We thank the reviewer for these suggestions on interpreting the boundary coefficients.
Table 1 now includes the fitted and , together with one-SD error bars. We acknowledge that predicting from is less accurate for modular addition and Tiny Shakespeare. For the remaining four models, , as shown in the newly added Appendix F.2. We have not yet found a setting with among those examined so far.
Q2: Does a right boundary appear without weight decay?
We thank the reviewer for this question about the role of weight decay.
We have added modular-division phase diagrams with zero and weak weight decay to Appendix D, keeping the learning rate and all other parameters fixed at their baseline values. Without weight decay, the approximately linear left boundary persists, and no distinct right boundary is resolved over the scanned range. Thus, nonzero weight decay is not necessary for the right boundary to remain unresolved in this network, consistent with the quadratic-core explanation.
Q3: Does the boundary persist at larger scale and nonzero loss, and do spikes hurt performance?
We thank the reviewer for asking about broader applicability and optimization performance.
Our modular-addition and modular-division experiments already use mini-batch training with , so the observed approximately linear boundary is not restricted to full-batch training. However, our mechanism concerns confident, low-loss training and does not directly apply to LLM pretraining, where the loss remains non-negligible. We therefore do not establish the same boundary in the larger-scale, non-zero-loss regime asked about here.
Regarding performance, loss spikes are generally undesirable because they can impair training efficiency and final model performance, motivating efforts to mitigate them (Wortsman et al., 2023 and Huang et al., 2025, SPAM).
Q4: How do the boundary coefficient and scaling exponent vary across seeds and learning rates?
We thank the reviewer for suggesting checks across seeds and learning rates.
We have repeated the modular-division scan across three random seeds and two learning rates (Appendices D.1–D.2). The approximately linear scaling persists, with –, while the prefactor varies across settings. Tables 5 and 6 report the full fits.
Q5: Does separable logistic regression reproduce the boundary, and how do updates compare with core widths?
We thank the reviewer for suggesting the logistic regression comparison and geometric measurements.
Separable linear logistic regression reproduces a quadratic core and a superquadratic wall, but we do not observe a comparable linear phase boundary. At the examined low-loss state, the actual Adam update is nearly orthogonal to the transverse direction along which the core–wall structure occurs, so the update scarcely probes the wall (Appendix G). We also report the mean typical update and core width at pre-spike points for all six tasks, together with the ratio of these two means, in Appendix E.4, Table 8.
Thank you for addressing my questions. The WD control and multiple seed experiments specifically resolves a big concern of mine. I have increased soundness from 1 to 2 but am not ready to increase the overall score from a 2 to a 3 for the following reasons:
-
The paper's central claim is still not supported: It says the measured wall exponent predicts the boundary coefficient, but in practice that prediction seems to be a constant. This is substantiated by flattening out near 3.5–3.7 for every exponent observed and the authors confirmed that they found no setting with n_wall . C also shifts with seed/LR/WD in the additional experiments without an associated change in n_wall.
-
The scope might be a bit narrow: As mentioned previously and confirmed by the authors, this regime does not fit the LLM pretraining setting which was how the paper initially motivated itself by stating loss spikes are observed by (Chowdhery et al. 2026)/(Molybog et al. 2023)/(Thilak et al. 2022) which are LLM pretraining based papers as I understand it. The practical takeaway seems to be more directional, thus the main contribution of the paper appears to be mechanistic in nature. Although I acknowledge that it could be useful in settings where datasets are small relative to model capacity (e.g Molecular Discovery, Biomedical Image Segmentation). I would be inclined to move it from 2 to a 3 if even one such practical example could be demonstrated, even at a smaller scale.
Response to Reviewer vFXZ
We thank the reviewer for the detailed assessment and constructive feedback. We appreciate the recognition of the intuitive finite-scale core–wall picture, the systematic phase-diagram study across six architectures, and the detailed controls and reproducibility procedures. Our point-by-point responses follow.
W1: The experimental scope limits the connection to practical LLM training.
We thank the reviewer for these comments on scope and practical relevance.
We acknowledge that LLM pretraining, where the training loss remains non-negligible, lies outside the confident, low-loss regime studied here. Nevertheless, training on datasets that are small relative to model capacity is relevant to practical applications where data acquisition or annotation is costly, including biomedical image segmentation (Ronneberger et al., 2015, U-Net) and industrial visual inspection (Batzner et al., 2024, EfficientAD). Understanding optimizer instabilities in these data-scarce settings is therefore valuable.
Our phase diagram also offers a qualitative perspective on practical Adam hyperparameter choices. Compared with the original default of (β1,β2)=(0.9,0.999) (Kingma and Ba, 2015), LLM training recipes such as LLaMA use (0.9,0.95) (Touvron et al., 2023). At fixed β1, this reduction in β2 corresponds to moving toward the non-spiky region in our phase diagram. This provides a directional connection between our findings and practical beta selection.
W2: The theory explains the right boundary, while networks exhibit only the empirical left boundary.
We thank the reviewer for highlighting the distinction between the two boundaries.
As explained in the Introduction, the right-boundary mechanism requires the loss to remain superquadratic as the trajectory approaches the minimum. In neural networks, the trajectory instead enters the quadratic core, cutting off this mechanism. We thank the reviewer for highlighting the lack of an analytical derivation for the left boundary. This remains an important direction for future work. The main contribution of this paper is to identify the left boundary in real neural networks and provide a semiquantitative explanation through the superquadratic toy model and the finite-scale core–wall geometry.
W3: A scan without weight decay is needed to isolate the quadratic-core explanation.
We thank the reviewer for suggesting a control without weight decay.
We have added modular-division phase diagrams with zero and weak weight decay to Appendix D, keeping the learning rate and all other parameters fixed at their baseline values. In both cases, the observed boundary retains an approximately unit slope on log–log axes, although its prefactor changes. These controls show that the approximately linear boundary observed in the neural-network experiments persists without weight decay.
W4: Boundary robustness across seeds, learning rates, and weight decay is not established.
We thank the reviewer for highlighting the need to assess robustness.
Using modular division as a representative task, we have verified the robustness of the phase boundary across three random seeds (Appendix D.2) and examined its sensitivity to learning rate and weight decay (Appendix D.1). The approximately linear scaling persists across these settings, although the prefactor can change.
Table 1 now includes error bars for the fitted Cfree and p, given by their conditional regression standard errors. In addition, Appendix B.2.1 verifies that the overall boundary scaling is robust to changes in the period-detection parameters and low-loss eligibility threshold.