- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies the interaction between learning-rate schedules, weight decay, and the parameter norm in scale-invariant blocks induced by normalization layers. For SGD with coupled weight decay, the authors derive an exact discrete-time recurrence for the effective directional stepsize, which separates the dynamics into schedule forcing, captured entirely by a single scalar , and geometric self-quenching arising from norm growth. This yields a sharp boundary between contraction- and expansion-dominated regimes, and is extended to stochastic minibatch gradients. In a solved normalized-linear model, they show that the unique interior balance point lies exactly on this boundary and is an unstable spiral source, implying that constant learning rate with weight decay cannot maintain a stable equilibrium. They further generalize the recurrence to a homogeneous-optimizer framework, where a homogeneity exponent distinguishes SGD and SGDM () from Adam (), explaining why adaptive methods are more expansion-prone. Experiments on exact maps and neural networks (MLP, CNN, GPT-2) track the predicted quantities, and a schedule that enforces a constant achieves its best accuracy near .
Critical strengths
1. Clean and exact theoretical framework
The recurrence holds exactly in discrete time without linearization or continuous-time limits, and summarizing the effect of arbitrary schedules and weight decay into a single scalar provides an interpretable lens on training dynamics.
2. Insightful analysis of the solved model
The reduction to a two-dimensional map and the characterization of the balance point as an unstable spiral source give a concrete mechanism for recurrent behavior under constant schedules.
3. Unified view across optimizers
The homogeneous-optimizer framework connects SGD, SGDM, and Adam through a single exponent, and the radial amplification channel of momentum is an interesting optimizer-specific insight.
Critical weaknesses
1. Limited empirical validation
The main recurrence is an algebraic identity, so it always holds. The small residuals therefore only confirm the identity, not the predictive power of the framework. The more important claims, such as the instability explaining oscillations in real networks and Adam being less stable, are not directly tested. Similarly, the claim that controls training relies on a single CIFAR-10 experiment with a low best accuracy. Since changing also changes the shape of the learning-rate schedule, it is unclear whether the accuracy difference comes from itself, and no comparison with standard schedules such as cosine decay is provided.
2. Generality of the instability result
The instability is proven only for a simple linear model without minibatch noise. In practice, it is unclear whether it matters in real training.
Questions for authors
- Can the authors add experiments with standard models (e.g., ResNet, ViT), optimizers (SGDM, AdamW), and schedules?
- Do oscillations in real networks match the predicted mechanism, and do they remain when gradient noise is reduced?
- Is Adam actually less stable than SGD under normalization, and is this explained by ?
- How does a schedule compare with tuned standard schedules, and does stay optimal across settings?
- In Eqs. (20)–(21), the weight decay is applied outside the moment estimates, which looks like decoupled weight decay. Could the authors clarify why this is called coupled weight decay? Also, could the text in Figure 1a be enlarged for readability?
3 comments
Response for Reviewer Vz4S part 2
- Comparison with tuned schedules. How does compare with tuned standard schedules, and does remain the best choice across settings?
A5. With the initial learning rate and weight decay fixed, determines the entire learning-rate schedule and leaves fewer schedule parameters to tune than cosine decay. On full-data CIFAR-10, achieved the highest test accuracy among the five original fixed targets on all three architectures. It reached 78$%$ versus 73$%$ for validation-tuned cosine LR on the BN ConvNet. The corresponding three-run means were 90.70$%$ versus 91.74$%$ on ResNet-18 and 79.16$%$ versus 81.45$%$ on ViT with slightly worse performance. Thus, led the tested fixed-target sweep and was competitive with tuned cosine LR. Under the assumptions of Theorem 2.1, is the boundary for guaranteed effective-step non-expansion.
We also used the framework to make the effective step itself follow a cosine trajectory to increase the flexibility of the (see Figure 21 and Table 5). On the same 96/192/384-channel BN ConvNet, with the same data split and 50,000 SGD updates, cosine effective-step control achieved test accuracy versus for tuned cosine LR. It improved accuracy in all runs with a mean gain of 6.50 percentage points.
- Weight-decay terminology and figure readability. Clarify why Eqs. (20)–(21) call the decay coupled when it is applied outside Adam's moment estimates, and enlarge the text in Figure 1a.
A6: We thank the reviewer for identifying this terminology inconsistency. As explicitly clarified in Appendix F, Eqs. (20)–(21) employ AdamW-style decoupled weight decay, where the moment estimates are computed from the loss gradient and weight decay is applied through an external multiplicative factor . Importantly, Theorem 4.3 is derived for precisely this update formulation. Thus, the terminology correction does not affect the validity of the theoretical result. We will make the naming consistent throughout the manuscript and distinguish it from L2 regularization incorporated into Adam's moments. We will also enlarge the text in Figure 1a to improve readability.
Official comments by reviewer Vz4S
Thank you for the extensive rebuttal. Most of my concerns are resolved and I will maintain my score.
Response for Reviewer Vz4S part 1
A1: We thank the reviewer for the constructive comments. While the recurrence is an exact identity under scale invariance, our experiments further investigate whether the theoretical framework provides a meaningful approximation of optimization dynamics in neural networks. In full-batch SGD on a two-layer scale-invariant MLP trained on MNIST, both layers show repeated effective-step buildup and reset across five peak-to-peak loss windows (Figures 17 and 18). The phase portraits qualitatively resemble the synthetic dynamics inFigure 6. In a separate CIFAR-10 BN ConvNet experiment, coupled Adam exhibits recurring loss bursts on fixed training batches. Figures 19 and 20 show the burst traces and compare the loss, effective-step magnitude, and angular steps of coupled Adam and SGD around two bursts.
A2: The normalized-linear model is deliberately constructed to isolate the competition between schedule forcing and geometric self-quenching, allowing us to characterize the underlying instability mechanism analytically. The oscillations persist in a two-layer scale-invariant MLP trained with full-batch SGD on MNIST examples. Both layers exhibit repeated effective-step growth and abrupt contraction across peak-to-peak loss windows (Figures 17 and 18). Because every update uses the same training examples, minibatch sampling noise is absent. The trajectories qualitatively resemble the synthetic dynamics in Figure 6 and demonstrate the source of instability.
A3: We added full-data CIFAR-10 experiments with ResNet-18 trained using SGD and ViT trained using AdamW in Figure 22 and Table 6. Each experiment uses 100 epochs. We compare the original Bt targets from Figure 4 with a tuned cosine learning-rate baseline.
At Bt=1, ResNet-18 and ViT reached 90.70$%$ and 79.16$%$ mean test accuracy, respectively. This was the strongest of the original fixed targets for both models. All ViT runs with Bt>1 diverged.
A4. The loss oscillations persist under full-batch training. In a two-layer scale-invariant MLP trained with full-batch SGD on 512 MNIST examples, both layers exhibit repeated effective-step buildup and reset across five peak-to-peak loss windows (see Figure 17,18). This occurs without minibatch sampling noise and qualitatively resembles the synthetic dynamics in Figure 6.
On CIFAR-10, we compared SGD and coupled Adam from the same BN ConvNet checkpoint, using the same fixed training batch and matching the initial relative convolutional update in section H.3. Coupled Adam exhibited recurring loss bursts on all three fixed batches. Around the first burst, the median angular step in the second convolutional layer increased from 0.0025 to 0.0266 radians. At the burst update, the corresponding SGD angular step was 0.0066 radians. The supplement shows the validation loss, effective-step magnitude, and angular step around two bursts.
These observations identify a specific instability signature for coupled Adam in this setting: loss bursts accompanied by abrupt increases in directional motion. Our framework gives a common way to analyze effective-step dynamics across optimizers and specifies the assumptions under which the stability conditions apply. Establishing a general stability ordering between Adam and SGD, comparing burst magnitudes systematically, and isolating the role of Adam’s moment estimates are are promising future direction and is beyond the scope of current work but it remains a very promising future direction.