acceptodds
Under review as a conference paper at ICLR 2027

When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

Abstract

Normalization renders large parts of neural networks effectively scale invariant, inducing a hidden feedback loop in which learning-rate schedules and weight decay interact through the parameter norm to control the effective step taken by the optimizer. We show that this interaction is governed by an exact discrete-time law: a single scalar quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect. This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning rate regimes. To understand the underlying mechanism, we provide exact analysis of a fully solved normalized regression model where the dynamics reduce to two dimensions and show that the balance point is intrinsically unstable, implying that constant learning rate with weight decay cannot stably maintain an interior equilibrium and instead produces recurrent behavior driven by discrete-time Jacobian structure. We further extend this perspective across optimizers through unified homogeneous-optimizer framework that reveals a structural dichotomy in self-quenching strength, providing a first-principles explanation for why adaptive methods exhibit systematically weaker stabilization under normalization. Across dynamical systems and neural networks (MLP, CNN, GPT2 / MNIST, CIFAR, wikiText, OpenWebText), the predicted law holds with high precision and enables direct control of training via the identified scalar, with performance peaking sharply at the predicted boundary. Together, these results isolate a single governing quantity for scale-invariant optimization, providing a precise and actionable lens on training dynamics, optimizer behavior, and schedule design in modern deep learning.

Primary area:
optimization
Keywords:
stability analysis, optimization, scale invariant learning, deep learning, theory

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Vz4SOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the interaction between learning-rate schedules, weight decay, and the parameter norm in scale-invariant blocks induced by normalization layers. For SGD with coupled weight decay, the authors derive an exact discrete-time recurrence for the effective directional stepsize, which separates the dynamics into schedule forcing, captured entirely by a single scalar BtB_t, and geometric self-quenching arising from norm growth. This yields a sharp boundary between contraction- and expansion-dominated regimes, and is extended to stochastic minibatch gradients. In a solved normalized-linear model, they show that the unique interior balance point lies exactly on this boundary and is an unstable spiral source, implying that constant learning rate with weight decay cannot maintain a stable equilibrium. They further generalize the recurrence to a homogeneous-optimizer framework, where a homogeneity exponent ν\nu distinguishes SGD and SGDM (ν=1\nu = 1) from Adam (ν=0\nu = 0), explaining why adaptive methods are more expansion-prone. Experiments on exact maps and neural networks (MLP, CNN, GPT-2) track the predicted quantities, and a schedule that enforces a constant BtB_t achieves its best accuracy near B=1B = 1.

Critical strengths

1. Clean and exact theoretical framework

The recurrence holds exactly in discrete time without linearization or continuous-time limits, and summarizing the effect of arbitrary schedules and weight decay into a single scalar BtB_t provides an interpretable lens on training dynamics.

2. Insightful analysis of the solved model

The reduction to a two-dimensional map and the characterization of the balance point as an unstable spiral source give a concrete mechanism for recurrent behavior under constant schedules.

3. Unified view across optimizers

The homogeneous-optimizer framework connects SGD, SGDM, and Adam through a single exponent, and the radial amplification channel of momentum is an interesting optimizer-specific insight.

Critical weaknesses

1. Limited empirical validation

The main recurrence is an algebraic identity, so it always holds. The small residuals therefore only confirm the identity, not the predictive power of the framework. The more important claims, such as the instability explaining oscillations in real networks and Adam being less stable, are not directly tested. Similarly, the claim that BtB_t controls training relies on a single CIFAR-10 experiment with a low best accuracy. Since changing BB also changes the shape of the learning-rate schedule, it is unclear whether the accuracy difference comes from BtB_t itself, and no comparison with standard schedules such as cosine decay is provided.

2. Generality of the instability result

The instability is proven only for a simple linear model without minibatch noise. In practice, it is unclear whether it matters in real training.

Questions for authors
  1. Can the authors add experiments with standard models (e.g., ResNet, ViT), optimizers (SGDM, AdamW), and schedules?
  2. Do oscillations in real networks match the predicted mechanism, and do they remain when gradient noise is reduced?
  3. Is Adam actually less stable than SGD under normalization, and is this explained by ν\nu?
  4. How does a Bt≈1B_t \approx 1 schedule compare with tuned standard schedules, and does B=1B = 1 stay optimal across settings?
  5. In Eqs. (20)–(21), the weight decay is applied outside the moment estimates, which looks like decoupled weight decay. Could the authors clarify why this is called coupled weight decay? Also, could the text in Figure 1a be enlarged for readability?
3 comments
AuthorsOpenReview ↗

Response for Reviewer Vz4S part 1

  1. Empirical evidence beyond recurrence residuals. Test whether the predicted instability explains neural-network oscillations and whether Adam is less stable than SGD. Strengthen the schedule-control evidence by addressing the learning-rate-path confound and comparing with standard cosine decay.

A1: We thank the reviewer for the constructive comments. While the recurrence is an exact identity under scale invariance, our experiments further investigate whether the theoretical framework provides a meaningful approximation of optimization dynamics in neural networks. In full-batch SGD on a two-layer scale-invariant MLP trained on MNIST, both layers show repeated effective-step buildup and reset across five peak-to-peak loss windows (Figures 17 and 18). The phase portraits qualitatively resemble the synthetic dynamics inFigure 6. In a separate CIFAR-10 BN ConvNet experiment, coupled Adam exhibits recurring loss bursts on fixed training batches. Figures 19 and 20 show the burst traces and compare the loss, effective-step magnitude, and angular steps of coupled Adam and SGD around two bursts.

  1. Generality of the instability result. Does the instability proved in the simplified deterministic linear model also matter in neural-network training?

A2: The normalized-linear model is deliberately constructed to isolate the competition between schedule forcing and geometric self-quenching, allowing us to characterize the underlying instability mechanism analytically. The oscillations persist in a two-layer scale-invariant MLP trained with full-batch SGD on MNIST examples. Both layers exhibit repeated effective-step growth and abrupt contraction across peak-to-peak loss windows (Figures 17 and 18). Because every update uses the same training examples, minibatch sampling noise is absent. The trajectories qualitatively resemble the synthetic dynamics in Figure 6 and demonstrate the source of instability.

  1. Standard architectures, optimizers, and schedules. Add experiments with models such as ResNet and ViT, optimizers such as SGDM and AdamW, and standard schedules.

A3: We added full-data CIFAR-10 experiments with ResNet-18 trained using SGD and ViT trained using AdamW in Figure 22 and Table 6. Each experiment uses 100 epochs. We compare the original BtB_t targets from Figure 4 with a tuned cosine learning-rate baseline.

At Bt=1B_t=1, ResNet-18 and ViT reached 90.70$%$ and 79.16$%$ mean test accuracy, respectively. This was the strongest of the original fixed targets for both models. All ViT runs with Bt>1B_t>1 diverged.

  1. Oscillations under reduced noise and Adam stability. Do the observed oscillations match the predicted mechanism and persist when gradient noise is reduced? Is Adam less stable than SGD under normalization, and what explains the difference?

A4. The loss oscillations persist under full-batch training. In a two-layer scale-invariant MLP trained with full-batch SGD on 512 MNIST examples, both layers exhibit repeated effective-step buildup and reset across five peak-to-peak loss windows (see Figure 17,18). This occurs without minibatch sampling noise and qualitatively resembles the synthetic dynamics in Figure 6.

On CIFAR-10, we compared SGD and coupled Adam from the same BN ConvNet checkpoint, using the same fixed training batch and matching the initial relative convolutional update in section H.3. Coupled Adam exhibited recurring loss bursts on all three fixed batches. Around the first burst, the median angular step in the second convolutional layer increased from 0.00250.0025 to 0.02660.0266 radians. At the burst update, the corresponding SGD angular step was 0.00660.0066 radians. The supplement shows the validation loss, effective-step magnitude, and angular step around two bursts.

These observations identify a specific instability signature for coupled Adam in this setting: loss bursts accompanied by abrupt increases in directional motion. Our framework gives a common way to analyze effective-step dynamics across optimizers and specifies the assumptions under which the stability conditions apply. Establishing a general stability ordering between Adam and SGD, comparing burst magnitudes systematically, and isolating the role of Adam’s moment estimates are are promising future direction and is beyond the scope of current work but it remains a very promising future direction.

AuthorsOpenReview ↗

Response for Reviewer Vz4S part 2

  1. Comparison with tuned schedules. How does Bt≈1B_t\approx1 compare with tuned standard schedules, and does Bt=1B_t=1 remain the best choice across settings?

A5. With the initial learning rate and weight decay fixed, Bt=1B_t=1 determines the entire learning-rate schedule and leaves fewer schedule parameters to tune than cosine decay. On full-data CIFAR-10, Bt=1B_t=1 achieved the highest test accuracy among the five original fixed targets on all three architectures. It reached 78$%$ versus 73$%$ for validation-tuned cosine LR on the BN ConvNet. The corresponding three-run means were 90.70$%$ versus 91.74$%$ on ResNet-18 and 79.16$%$ versus 81.45$%$ on ViT with slightly worse performance. Thus, Bt=1B_t=1 led the tested fixed-target sweep and was competitive with tuned cosine LR. Under the assumptions of Theorem 2.1, Bt=1B_t=1 is the boundary for guaranteed effective-step non-expansion.

We also used the framework to make the effective step itself follow a cosine trajectory to increase the flexibility of the BtB_t (see Figure 21 and Table 5). On the same 96/192/384-channel BN ConvNet, with the same data split and 50,000 SGD updates, cosine effective-step control achieved 80.14±0.28%80.14 \pm 0.28\% test accuracy versus 73.64±0.21%73.64 \pm 0.21\% for tuned cosine LR. It improved accuracy in all runs with a mean gain of 6.50 percentage points.

  1. Weight-decay terminology and figure readability. Clarify why Eqs. (20)–(21) call the decay coupled when it is applied outside Adam's moment estimates, and enlarge the text in Figure 1a.

A6: We thank the reviewer for identifying this terminology inconsistency. As explicitly clarified in Appendix F, Eqs. (20)–(21) employ AdamW-style decoupled weight decay, where the moment estimates are computed from the loss gradient and weight decay is applied through an external multiplicative factor ata_t. Importantly, Theorem 4.3 is derived for precisely this update formulation. Thus, the terminology correction does not affect the validity of the theoretical result. We will make the naming consistent throughout the manuscript and distinguish it from L2 regularization incorporated into Adam's moments. We will also enlarge the text in Figure 1a to improve readability.

Reviewer_Vz4SOpenReview ↗

Official comments by reviewer Vz4S

Thank you for the extensive rebuttal. Most of my concerns are resolved and I will maintain my score.

Reviewer_sNwuOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the dynamics of scale-invariant optimization using coupled weight decay.

It shows that under a scale invariant loss function and SGD with coupled weight decay, the effective directional step size Φt\Phi_t is governed by a single one-scalar law. The results imply that the schedule factor BtB_t determines directly governs whether the effective directional step size Φt\Phi_t decreases over time unconditionally or only under so-called geometric self-quenching.

The results are analyzed exactly on a normalized-linear model and extended beyond SGD via a homogeneous-optimizer template to SGD + momentum. Optimizers are classified by the homogeneity degree ν\nu of their update direction, which indicates how much norm growth regulates future updates.

The paper is complemented with empirical results to validate the results across different architectures, LR schedules and optimizers.

Critical strengths

The paper shows that the dynamics of scale-invariant optimization can be fully captured through a single one-scalar low, which is purely algebraic and does not require additional assumptions beyond scale invariance of the loss.

Critical weaknesses
  1. I think the main limitation of this work is that it requires the assumption that the loss function is positive scale invariant, i.e. (αw)=(\w)\mathcal(\alpha w) = \mathcal(\w) for the parameter block considered. As the authors also mention, this does not hold in general, but mainly for the layer immediately before a normalization layer. As such, this work only analyzes the dynamics of a subset of parameters and ignores the interaction with non-scale-invariant parameters of the model

  2. In some cases the figure captions are not really consistent with what is visible in the figures. For instance in Figure 1b) the cosine schedule on BN ConvNet on Cifar-10 is not so different from the constant schedule (BtB_t is also larger than 1), yet the caption claims that "cosine decay suppresses expansion in subcritical phases" and that this holds "across architectures".

Questions for authors
  • What is the best achievable performance on Cifar-10 given the BN ConvNet that you considered? The best achieved accuracy of 65% at Bt=1B_t=1 are still far below what is possible. Optimized CNNs should reach >80% easily.

  • It is unclear to me what Figure 1b) is supposed to show. The caption describes this as "BtB_t organizes expansion across architectures and schedules.", however only tracking BtB_t under different LR schedules does not allow this conclusion. It requires a separate plot of the corresponding weight norms and how they are controlled by BtB_t. Also what are the effective step sizes for each of those cases?

  • It is also unclear to me what Figure 2 is supposed to show. I understand that you want to the causal effect of BtB_t by directly controlling for it instead of computing it from the resulting LR schedule. However I think the first result which is needed is to show that in this case the weight norm neither collapses nor diverges. It is also unclear to me from the theoretical results in the paper, why one would expect the performance to peak at B=1B=1 and sharply decrease for slightly larger or smaller values. Can the authors elaborate on this perhaps?

3 comments
AuthorsOpenReview ↗

Response for Reviewer sNwu part 1

  1. Scope of positive scale invariance. The assumption covers only certain parameter blocks and this limits the analysis, particularly when those blocks interact with non-scale-invariant parameters?

A1: We thank the reviewer for pointing this out. However, we want to clafrify that this assumption is deliberately introduced to isolate and rigorously study normalization-induced optimization dynamics, following an established approach in prior works (see introduction and appendix A). While our blockwise analysis does not fully characterize interactions with non-scale-invariant parameters, this modeling restriction enables a precise understanding of normalization-induced dynamics, which play an important role in neural-network optimization. Our empirical results further support the usefulness of this framework. Section 5 reports small recurrence residuals and effective-step expansion/contraction patterns consistent with the theoretical criteria across tested architectures, demonstrating that the blockwise framework meaningfully describes effective-step dynamics within larger networks. Therefore, we regard positive scale invariance as a well-motivated modeling assumption rather than a restriction that undermines the relevance of our results.

  1. Evidence for cosine suppression. In Figure 1b, the constant and cosine schedules look similar for the CIFAR-10 BN ConvNet. Quantify the difference and qualify the claim across architectures.

A2: We thank the reviewer for pointing out. In newly added section H.1, we show the convolutional layer of the BN ConvNet indeed show reduction compared to Constant learning with Cosine learning schedule. The smoothed expansion fraction drops from 76.12$%$ under constant LR to 69.22$%$ under cosine LR, a reduction of 6.90 percentage. Despite that the visual decrease is not obvious, the general reduction exist across different setting. We will make this clear and add numerical value for further justification.

  1. Achievable CIFAR-10 accuracy. What accuracy can the BN ConvNet achieve? The reported best of about 65 is below the reviewer's expectation of over 80 for an optimized CNN.

A3. We additionally add the Figure 16 constant-BtB_t sweep on the full CIFAR-10 training split using a widened version of the same three-block BN ConvNet (96/192/384 channels). With the five original per-update targets and 50,000 SGD updates, Bt=1B_t=1 gave the highest test accuracy in the sweep, 78.73$%$. At the same width and update budget, cosine effective-step control reached 80.14±0.28%80.14 \pm 0.28\% test accuracy (Figure 21) compared with 73.64±0.21%73.64 \pm 0.21\% for tuned cosine LR. We also show tuned results with ResNet-18 and ViT in Figure 22 in which the effect of BtB_t is further demonstrated.

  1. Interpretation of Figure 1b. Meaning of the figure measures and demonstration of the corresponding weight norms and absolute effective step sizes for each schedule.

A4: We would like to clarify that Figure 3b does not merely track BtB_t under different learning-rate schedules. It measures the effective-step expansion fraction as stated in the main content, defined as the proportion of monitored parameter blocks satisfying Φt+1/Φt>1\Phi_{t+1}/\Phi_t>1, and compares it with BtB_t across architectures and schedules. The theoretical motivation follows directly from Theorem 2.1: Φt+1/Φt=Bt/(1+Φt2∥gˉt∥2)\Phi_{t+1}/\Phi_t=B_t/(1+\Phi_t^2\|\bar g_t\|^2). For a fixed geometric self-quenching term, larger BtB_t increases the effective-step ratio and makes expansion more likely. Equivalently, expansion occurs when Bt>1+Φt2∥gˉt∥2B_t>1+\Phi_t^2\|\bar g_t\|^2. Figure 3b examines whether this schedule-forcing effect is reflected in observed expansion fractions. Thus, Figure 3b directly evaluates the relationship between schedule forcing and realized effective-step expansion. To further illustrate the underlying geometric feedback, we additionally add Figure 13, 14 for parameter norms ∥wt∥\|w_t\| and absolute effective steps Φt\Phi_t of the corresponding experimental settings. These complement the expansion-fraction analysis and make the schedule–geometry or control more transparent.

AuthorsOpenReview ↗

Response for Reviewer sNwu part 2

  1. Interpretation of Figure 2. Show whether the weight norms collapse or diverge under the fixed-BtB_t interventions, and explain why accuracy peaks near Bt=1B_t=1 and drops on either side.

A5: We thank the reviewer for this question. Figure 4 aims to demonstrate that even small changes in the schedule-forcing factor BtB_t can produce substantial differences in optimization dynamics and training performance. From the exact recurrence, for constant Bt=BB_t=B, we obtain

ΦT=Φ0BT∏t<T(1+Φt2∥gˉt∥2)−1.\Phi_T=\Phi_0 B^T\prod_{t<T}(1+\Phi_t^2\|\bar g_t\|^2)^{-1}.

This reveals how small deviations in BB accumulate multiplicatively over training. For example, B=0.99B=0.99 guarantees Φ100≤0.99100Φ0≈0.366Φ0\Phi_{100}\leq 0.99^{100}\Phi_0\approx0.366\Phi_0, whereas B=1.01B=1.01 introduces an opposing expansion pressure regulated by geometric self-quenching. Thus, seemingly small differences in BB can lead to markedly different effective-step trajectories. Importantly, B=1B=1 is the theoretical boundary between unconditional contraction and conditional expansion as stated in the main theory in section 2 and this provides a natural reference for our controlled experiment. Figure 4 demonstrates that performance is highly sensitive to this choice with the observed accuracy peak at B=1B=1 in the tested setting, rather than claiming universal optimality. We also provide parameter-norm trajectories in Figure 15 for this plot to clarify the interaction between schedule forcing and geometric self-quenching and rule out potential norm collapse or divergence in the reported experiments.

I thank the authors for the revision and their response, and the additional figures in the appendix.

Let's leave out the first point for now and discuss 2.-5. Unfortunately, I am not sure they help to clarify my questions or I am unable to follow the explanation of the authors:

  1. Thank you for providing additional results in Table 3 in the appendix. Still, I'm not sure whether Figure 3b) helps to make the point because Bt>1B_t > 1 on the ConvNet on Cifar-10 and your results only guarantee non-expansion for Bt<1B_t < 1. Therefore I still think that the figure caption on "cosine decay suppresses expansion in subcritical phases" is incorrect and misleading.

  2. Thank you for your explanation. I see now that the plot both BtB_t and the expansion fraction in the y-axis, although it is not clear to me what the dotted lines in the figure correspond to. Also in the bottom row, the y-label is simply called "middle conv2" which should be corrected. Also, I find it quite misleading that the y-label on the left corresponds to the dashed lines and that the solid lines correspond to the expansion fraction, which is only shown on the right. The additionally provided figures 13 and 14 show quite distinct behavior in the MLP and ConvNet: The effective step size for a constant LR schedule grows continuously in the BN MLP, but remains constant for the BN ConvNet. The direct benefit of tracking BtB_t is also not directly evident to me from these illustrations. For instance, how does it predict the increasing and then decreasing effective step size of the cosine schedule on BN MLP and the almost monotonically decreasing curve for BN ConvNet?

  3. Thank you for providing Figure 15. What do the shaded regions correspond to? The way it is currently plotted, I do not see a clear connection between the value of BB and how well it controls the weight norm.

However, my main concerns remain. Overall, the presentation of this work makes it hard for me to grasp the main contributions as well its theoretical valid scope, and I believe it will benefit from another round of revision. I will retain my score.

Reviewer_z45FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the training dynamics of neural network weights when the loss depends only on the direction of the weights and not their size, for example when these weights are followed by normalization. In this setting, the learning rate, weight decay, and the growing norm of the weights interact to determine how far the optimizer moves at each step. The authors show that this interaction can be written exactly as a single per-step relation. In it, one scalar summarizes the effects of the learning-rate and weight-decay schedules, while weight-norm growth pushes in the opposite direction. This relation yields a boundary between regimes where the effective step grows and where it shrinks. In a simplified, exactly solvable regression model with isotropic data, the authors show that the balance point between these forces is unstable, so training under a constant learning rate with weight decay oscillates. They further extend the analysis to momentum and Adam, arguing that the optimizer type determines how strongly norm growth brakes the effective step. Empirically, the authors report that the relation holds numerically on MLPs, CNNs, and GPT-2 models. They also report that learning-rate schedules constructed to hold the proposed scalar fixed perform best at the predicted boundary value.

Critical strengths
  1. The interplay between learning-rate schedules, weight decay, and normalization layers is an active research topic, and a precise account of how these factors jointly control the effective step size is of clear practical interest.
  2. The proposed per-step relation provides a compact rule of the schedule and weight-decay effects on the effective step size. The derivations appear correct and require few assumptions beyond scale invariance.
  3. The analysis goes beyond plain SGD, extending the same relation to momentum and Adam. It also provides an end-to-end result in a simplified setting (an exact characterization of the dynamics in an isotropic normalized regression model).
  4. The experiments span several architectures and modalities, from MLPs and CNNs on image classification to GPT-2 models trained on real language data.
Critical weaknesses

The main concern I have with the paper is its presentation, which in my opinion makes the contributions difficult to follow and evaluate. I also found several inconsistencies between the paper’s claims and its own results.

  1. The paper lacks a clear structure and a proper setup. The main text moves between the exact relation, its stochastic extension, the solved regression model, the optimizer extensions, and the schedule-control experiments with no clear central question. It is not clear which result the authors consider the main contribution. The motivation is also underdeveloped. The introduction states that prior work lacks an exact per-step law but does not explain what such a law enables beyond existing analyses of how the weight norm evolves.
  2. The mathematical setting is never introduced before it is used. Definitions are scattered across theorem statements, footnotes, and a notation table in Appendix B. Theoretical claims in the main text are presented without a direct link to their proofs (the mapping only appears in Appendix B). Notation is also reused for different quantities (e.g., q,β,λq, \beta, \lambda).
  3. Important material is deferred to the appendix. The intuition in Appendix G should appear early in the main text. The scope restrictions in Section F substantially narrow the claims made in the abstract and introduction, so they should be stated alongside those claims.
  4. The manuscript requires substantial polishing:
  • Several figures are difficult to read or have unclear captions (e.g., Figures 1, 2, and 3).
  • Several figures are never referenced in the paper (e.g., Figures 3, 5, 6, 8, and 9).
  • GPT-2 is attributed to Cheng et al. (2024) rather than Radford et al. (2019).
  • The paper contains numerous typos and grammatical errors throughout.
  1. The paper is internally inconsistent in several places:
  • Appendix D.2 states that the tracked c_attn and c_proj blocks are exactly scale-invariant, while Section F states that such blocks are generally not.
  • Section 4.2 and the experiments describe Adam with coupled weight decay, while Section F describes Theorem 4.3 as the decoupled form and places L2 regularization inside Adam’s moments outside the result.
  • The abstract and introduction present, as a general property, that a constant learning rate with weight decay cannot maintain a stable equilibrium. This is proven only in the simplified model with isotropic data, and Figure 6 shows the dynamics converging for condition numbers κ≥5\kappa\geq 5.
Questions for authors
  1. What are the practical implications of the results? Can the proposed relation inform the design of learning-rate and weight-decay schedules in practice?
2 comments
AuthorsOpenReview ↗

Response for Reviewer z45F

  1. Manuscript organization, mathematical setup, and presentation and clarification of the central question and contribution.

A1: We revised the introduction around a single question: how do the learning-rate schedule and weight decay interact with the parameter norm to control the effective directional step? We now explain what the exact per-step law adds to a norm-based analysis. It separates schedule forcing from geometric feedback and gives a criterion for when the effective step expands or contracts. We also moved the scale-invariant loss example and its figures into the introduction to show why the loss depends on direction while the rate of directional motion depends on the norm.

We clarify the scope where we state the results. The schedule law applies to parameter blocks satisfying positive scale invariance, and the spiral-source and period-two results concern the isotropic regression model. We added direct proof pointers to the theorem and proposition statements, including separate references for the descent and PL parts of Theorem 3.1.

We revised figure captions to explain what is plotted and how to read it, added in-text callouts for the previously uncited figures flagged by the reviewer, and corrected cross-references affected by renumbering. We also corrected the GPT-2 citation to Radford et al. (2019).

  1. Consistency of technical claims. Reconcile the statements about GPT blockwise scale invariance and Adam weight decay, and clarify how the anisotropic trajectories relate to the isotropic instability result.

A2. We clarify three distinctions in the revised manuscript.

GPT blockwise scale invariance. The exact recurrence requires positive scale invariance of the tracked parameter block with the rest of the model held fixed. We use the GPT experiments as architecture and modality stress tests. The small recurrence residuals show that the recurrence describes the measured effective-step evolution along the tested trajectories. Establishing exact blockwise scale invariance for the tracked attention projections would require a separate symmetry argument.

Adam weight decay. In Eqs. (20)–(21), Adam’s moments are formed from the loss gradient, and weight decay is applied through the external shrinkage factor ata_t. This is AdamW-style decoupled weight decay and is the update analyzed in Theorem 4.3. We distinguish this formulation from the coupled Adam experiments, where the L2 term enters the moment estimates.

Isotropic instability and anisotropic trajectories. Propositions 3.5–3.6 establish an exact period-two orbit and local spiral-source instability in the isotropic model. The effective-step recurrence applies more broadly to blocks satisfying the scale-invariance assumption. Figure 8 shows that changing the covariance changes the trajectories, with stronger damping at larger condition numbers. This behavior is consistent with the scope of the theorem, whose instability conclusion is specific to isotropic covariance.

  1. Practical implications. Explain how the recurrence can guide the joint design of learning-rate and weight-decay schedules.

A3: The practical implication of our work is to understand and control the effective directional step which is not reflected faithfully by naive learning rate. Under scale invariance, the effective step depends jointly on the learning rate, weight decay, and evolving parameter norm. Thus, conventional schedules such as cosine decay do not directly create the desired effective-step trajectory. Our exact recurrence identifies BtB_t as a controllable schedule-forcing factor, separating its effect from geometric self-quenching. This enables (i) diagnosing effective-step expansion or contraction under existing schedules and (ii) constructing learning-rate schedules from prescribed BtB_t trajectories. Section 5 demonstrates this construction through Equation (23) and Figure 4 shows that controlling BtB_t can substantially affect training performance. These results provide a concrete framework for schedule analysis and design with more flexible BtB_t-based schedules offering a promising direction for further investigation. In our newly added experiment in Figure 21, we show that by direct controlling effective step size instead of the naive learning rate to match the desired schedule, we can improve the final performance. This offer a new insight in schedule design compared to prior works and identify the importantance of effective step size in optimization behavior shaping.

I thank the authors for their response and revision. Some of my comments were addressed, such as the proof pointers, the GPT-2 citation, and the added Limitations paragraph.

However, my main concerns remain. The paper still lacks a clear structure and central contribution, the figures remain hard to read, and the notation is still reused for different quantities. In addition, the abstract and introduction still overstate the scope of the results relative to what is proven, and some inconsistencies between sections remain (e.g., regarding coupled vs. decoupled weight decay).

Overall, I believe the paper requires another round revisions, and I therefore retain my score.

Reviewer_wLLiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how learning-rate schedules and weight decay affect the effective directional step size through parameter norms in scale-invariant optimization. The authors derive an exact discrete-time recurrence that separates schedule forcing, represented by a scalar BtB_t, from geometric self-quenching caused by parameter-norm growth. The authors make three main contributions:

  1. They derive an exact schedule law for SGD with coupled weight decay, characterize the conditions for contraction or expansion of the effective step size, and analyze how stochastic gradient noise affects its expected evolution.
  2. They reduce a normalized-linear regression model to an exact two-dimensional system, show that its interior balance point is unstable under constant learning rate and weight decay, and use a homogeneous-optimizer framework to analyze differences in self-quenching across optimizers.
  3. They test the law in controlled dynamical systems and neural-network experiments, and construct learning-rate schedules that control BtB_t to examine its effects on training dynamics and performance.
Critical strengths
  1. The authors provide a careful treatment of stochastic gradients. Theorem 2.3 retains the nonlinear stochastic term in the effective-step update, establishes an exact conditional log-drift relation, and separates the mean drift from random fluctuations through a martingale decomposition. This allows the analysis to capture how gradient noise affects effective-step dynamics and, under explicit assumptions, quantify deviations of the realized trajectory from the accumulated conditional drift.
  2. The extension to SGDM provides a concrete mechanistic explanation. The authors identify that the effective step alone cannot form a closed description of momentum dynamics and restore a closed recurrence by introducing a scale-free momentum state. The radial–tangential decomposition further reveals a radial amplification channel specific to momentum, showing that its effect on the effective step depends not only on the magnitude of the update direction but also on its alignment with the current parameters.
Critical weaknesses
  1. The correspondence between the Transformer experiments and the theoretical assumptions remains unclear. Appendix D.2 treats the attention projection matrices in GPT as exactly scale-invariant blocks, whereas Appendix F states that these matrices generally do not satisfy this property in standard pre-LayerNorm Transformers, since scaling them can change attention logits or the magnitude of a residual branch. It is therefore unclear whether the specific architectural modifications restore the required symmetry, and to what extent the reported low recurrence residual supports the cross-architecture applicability claimed in the main text.
  2. The interpretation of long-term dynamics in the main text goes beyond the established local results. Propositions 3.5–3.6 establish the existence of an exact period-two orbit and the local instability of the interior fixed point, but these results alone do not determine the long-term behavior of trajectories from general initializations. Although Appendix F acknowledges that global convergence to a limit cycle is not proved, the main text still interprets these results as implying unavoidable persistent recurrence.
Questions for authors
  1. Which specific architectural modifications in Appendix D.2 make c_attn and c_proj satisfy blockwise scale invariance? How is this description consistent with the statement in Appendix F that these matrices generally do not satisfy this property?
  2. What is the precise mathematical meaning of persistent recurrence in this paper? Beyond local fixed-point instability and the existence of a particular period-two orbit, are there results constraining long-term behavior from general initializations?
2 comments
AuthorsOpenReview ↗

Response for Reviewer wLLi part 1

  1. Scale invariance in the Transformer experiments. Do the GPT modifications make 'c_attn' and 'c_proj' exactly scale-invariant? Reconcile Appendices D.2 and F and what the small recurrence residuals establish.

A1: We would like to clarify that the Transformer experiments are intended as architecture and modality stress tests of our theoretical framework, rather than as claims that standard Transformer architectures satisfy exact blockwise scale invariance. Our empirical results further investigate whether the recurrence remains a useful approximation beyond its exact assumptions. The small recurrence residuals reported across the tested Transformer configurations suggest that the framework can describe the measured effective-step evolution along these training trajectories, even without exact blockwise symmetry. We will clarify the wording in Appendix D.2 to distinguish architectural modifications from mathematically guaranteed scale invariance and make the stress-test interpretation more explicit.

  1. Scope of the long-term claims. The exact period-two orbit and local fixed-point instability do not establish persistent recurrence from general initializations. The main text appears to make a stronger claim than Propositions 3.5–3.6 support.

A2: As explicitly stated in Appendix F and mentioned by reviewer, we do not claim or prove global convergence to a limit cycle. However, this distinction is intentional: our objective is to identify a deterministic mechanism through which the competition between schedule forcing and geometric self-quenching generates instability rather than to characterize the asymptotic convergence behavior of arbitrary initializations for the two propositions. More specifically, Proposition 3.5 establishes an exact period-two orbit, satisfying ut+2=utu_{t+2}=u_t for all iterations along the constructed trajectory. This provides an explicit example of persistent, nonvanishing oscillatory behavior over an infinite time step. Separately, Proposition 3.6 proves that the interior balance point is a locally unstable spiral source, with Jacobian eigenvalues satisfying ∣λ±∣=1+a−a2>1|\lambda_\pm|=\sqrt{1+a-a^2}>1 for all a∈(0,1)a\in(0,1). Hence, the balance between schedule forcing and geometric self-quenching does not produce a locally attracting equilibrium in the analyzed model. These propositions establish two complementary properties: the existence of exact recurrent trajectories and the intrinsic local instability of the balance point. Neither result requires global convergence to a periodic attractor for which we regard as different objective. We therefore emphasize that the absence of a global convergence theorem is not prerequisite for our theoretical conclusions. It is an explicitly acknowledged distinction between the instability mechanism established in this work and the substantially stronger problem of classifying long-term dynamics from arbitrary initializations with instability mechanism. We recognize that some phrasing in the main text could be interpreted more broadly than intended, and we will clarify that the established recurrence is an existence result rather than a universal statement about all initializations. However, this clarification does not alter the validity or significance of Propositions 3.5–3.6. Furthermore, experimentally, we show that the instability oscillation consistently occur throughout the training (see Figure 17,18) in newly added experiments. The experiment demonstrate same pattern shown in the analytical setting and its influence over the optimization dynamics.

AuthorsOpenReview ↗

Response for Reviewer wLLi part 2

  1. Meaning of persistent recurrence. Definition of persistent recurrence precisely and long-term trajectories from general initializations.

A3: By persistent recurrence, we refer to the existence of nontrivial periodic trajectories whose oscillation amplitude remains nonvanishing over an long time window. Our central motivation is to explain why optimization trajectories under constant learning rate and weight decay can exhibit sustained oscillations rather than settling into a stable equilibrium. Our results provide three complementary pieces of evidence: First, Proposition 3.5 establishes an exact period-two orbit satisfying ut+2=utu_{t+2}=u_t, with nonvanishing amplitude ∥u+−u−∥=2(1−a)>0\|u_+-u_-\|=\sqrt{2(1-a)}>0, demonstrating persistent oscillations over an infinite time horizon. Second, Proposition 3.6 identifies the instability mechanism: the interior balance point is a locally unstable spiral source, with Jacobian eigenvalues satisfying ∣λ±∣=1+a−a2>1|\lambda_\pm|=\sqrt{1+a-a^2}>1. Thus, the competition between schedule forcing and geometric self-quenching does not yield a locally attracting equilibrium. Third, Figure 6, Appendix E.1 and newly added experiments in Figure 17,18 demonstrate sustained oscillations over the simulated setting from off-orbit initializations and show that the observed behavior is not restricted to the analytically constructed periodic orbit.

Together, these results establish a deterministic mechanism capable of generating persistent oscillations without stochastic noise. Characterizing global convergence from arbitrary initializations is a distinct mathematical objective, explicitly acknowledged in Appendix F as beyond our results. We will clarify the main-text wording to avoid implying universal recurrence, without altering our theoretical conclusions.

Comments on the paper

1 comment

Dear Reviewers,

Thank you to those who have already followed up with the authors. As we approach the end of the discussion period, if you have not yet responded to the authors' rebuttal, please do so soon.

Thank you for your time and effort in contributing to the ICLR community.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.