acceptodds
Under review as a conference paper at ICLR 2027

Reachability and Control of Directional Steps in Scale-Invariant SGD

Abstract

In scale-invariant models, the learning rate controls optimization only indirectly. Weight decay changes the parameter norm and thus also influences gradient updates in directions that subsequently affect prediction. We study this coupling through directional forcing , which is the input that learning rate and decay jointly supply to the effective-step dynamics. This viewpoint turns schedule design into an inverse path-control problem, requiring a choice of a `forcing path' and a characterization of how bounded weight decay can realize it. We characterize exactly the paths permitted by bounded decay multipliers, derive their optimal approximation, and solve exactly a noisy alignment problem whose optimal policy is shared by every finite horizon. We then establish a sharp stochastic classification of directional steps and prove almost-sure stationarity on the sphere under weak moment assumptions. The classification reveals that weight decay can change the decay exponent and summability class of effective steps even under a fixed harmonic learning-rate shape. The resulting design perspective has practical consequences. Across datasets, we find that a narrow forcing ramp improves mean accuracy in different tested settings, even over popular cosine learning rate setup. In a separate exact-symmetry experiment with signed decay, merely reversing an identical forcing multiset results in a massive accuracy drop of percentage points. Together, the results set foundations for making directional motion a tractable object of schedule design.

Primary area:
optimization
Keywords:
Effective learning rate, Optimization, Stochasticity, Deep learning, scale invariant learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_N25yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how learning rates and weight decay jointly control directional steps in scale-invariant SGD. It characterizes feasible forcing paths, solves a noisy spherical alignment benchmark, and establishes stochastic step asymptotics and stationarity conditions. Synthetic experiments test the theory, while image-classification and language-modeling experiments evaluate forcing-based schedules and temporal-order effects.

Critical strengths
  1. Quantitative theoretical characterization. The reachability conditions and stochastic step classification clarify how learning rates and weight decay jointly govern directional motion. The explicit square-summability boundary and stationarity result provide insight beyond nominal learning-rate analysis.
  2. Complementary empirical evidence. Synthetic experiments directly test predicted exponents, constants, and heavy-tailed regimes. Paired neural-network experiments report improvements across 14 settings, while the controlled reversal study demonstrates order dependence within its stated scope.
Critical weaknesses
  1. Limited connection to practical schedule design. The optimal policy assumes a specialized spherical objective and noise geometry, whereas the main experiments use an empirical linear request with sequential clipping. They do not isolate the training benefit of the proposed global projection.
  2. Insufficient decay-schedule controls. The long-horizon ramp uses zero decay for 61.66% of updates, but comparisons lack tuned constant decay or simple late-stage decay removal. Consequently, the gains do not isolate forcing-path design from changes in decay strength and duration.
Questions for authors

Please refer to the Critical Weaknesses.

1 comment
AuthorsOpenReview ↗

Response for Reviewer N25y

1. Weak practical connection and unclear benefit of global projection

The optimal reference problem motivates controlled deviations around B=1B=1 through

Bt⋆=1−t−1+O(t−2).B_t^\star=1-t^{-1}+O(t^{-2}).

The fixed-amplitude narrow ramp tests a simple forcing family around this reference, with early positive and later negative log forcing. Reachability and approximation then determine how native decay can implement the request.

We conducted the requested matched comparison between sequential clipping and global projection on CIFAR-100/ResNet-50 at 58,650 updates. Both share the forcing request, cosine LR 0.5−0.050.5-0.05, initial WD 0.0010.001, decay bounds [0,0.1][0,0.1], whole-network scope, initialization, sampling randomness, and validation split.

Their log-forcing RMSEs are approximately 7.831529617803×10−47.831529617803\times10^{-4} and 7.831529615095×10−47.831529615095\times10^{-4}, respectively. Sequential clipping is already nearly optimal for this request, leaving negligible approximation error to remove. Validation accuracies are 77.40%\% and 77.14%\%. Its guarantee concerns optimal feasible approximation under the stated objective.

2. Missing tuned constant-decay and late-removal controls

We have newly added both the constant-decay tuning sweep and an independent decay-removal timing sweep in new section G.6 and G.7 and use the tuned cosine for this part of experiments. The timing experiment separately tests how early or late removal affects performance at a fixed LR/WD configuration (Section G.6, Table 10, and Figure 10).

The late removal affects experiments are done with

λt=0.001 1{t<⌊fT⌋},f∈{0,0.1,…,1},T=58,650.\lambda_t=0.001\,\mathbf{1}\{t<\lfloor fT\rfloor\}, \qquad f\in\{0,0.1,\ldots,1\},\quad T=58{,}650.

The cosine learning rate remains positive and unchanged. The focused early/midpoint/late comparison is:

Removal timeValidation accuracy (%)
10% — early75.74
50% — requested-ramp midpoint77.46
90% — late75.76
Sequential ramp77.40

Removing decay very early or retaining it until late training is less competitive: the 10%\% and 90%\% cutoffs trail the midpoint by 1.72 and 1.70 percentage points. The 50%\% candidate follows directly from the prescribed request's midpoint crossing of B=1B=1, without a separate cutoff search to define that candidate. The results support the forcing construction as a source of competitive candidate timings. However, one can see that without prior knowledge on the BtB_t, it is hard to directly pinpoint the optimum decay removal timing.

Reviewer_D7eUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper analyzes the directional forcing determined by the learning rate and weight decay in scale-invariant SGD, and investigates the inverse problem of implementing a forcing path through bounded weight decay. Given a fixed learning rate path and an initial decay multiplier, the authors provide the necessary and sufficient conditions for a path to be realizable via bounded weight decay. In a spherical alignment with a specific noise geometry, this paper demonstrates that the same policy remains optimal across all finite horizons. Furthermore, the authors classify the almost-sure asymptotics of the effective step into three regimes based on the decay exponent of the cumulative forcing product, and establish stationarity under additional conditions. Experiments on CIFAR and WikiText datasets show that forcing-based scheduling improves average performance and highlight the impact of temporal ordering.

Critical strengths
  1. Inverse schedule design. The authors establish that the coupling, in which adjacent forcing values share a decay multiplier, serves as a necessary and sufficient condition for the feasibility of an entire path. A clear contribution is the characterization of paths under bounded decay with a fixed LR.

  2. Stochastic classification. Analyze three distinct regimes, determined by the exponent of the cumulative forcing product with the associated logarithmic corrections. Also, the authors distinguish between the moment assumptions required for effective-step asymptotics and those needed for staionarity, providing theory that hold even in the absence of a fourth gradient moment.

  3. Interesting framework. The paper provides an interesting perspective on learning rate and weight decay scheduling by treating directional forcing as an explicit control variable.

Critical weaknesses

I have some concerns about the paper.

  1. Limited novelty over prior work.

The main contribution of this paper is the analysis through the effective directional-step recurrence and forcing formulation, which was already introduced in Amin et al. (2026). In this sense, the reachability characterization seems useful, but it essentially reduces to an inverse problem of an existing recurrence combined with a box-constrained quadratic optimization. Although the stochastic asymptotic analysis gives meaningful and independent contribution, the overall novelty of the proposed framework appears quite limited. Given that existing work has already addressed schedule control via "forcing", it is necessary to more clearly distinguish the contribution of the proposed approach in terms of practical methodology.

  1. Lack of a principled and practically useful schedule design methodology.

The primary limitation of this paper is that it fails to provide general principles for choosing a specific forcing sequence that produces superior optimization performance. The optimal policy derived in Theorem 2 holds only under a highly restrictive stochastic alignment, and the asymptotic characterization in Theorem 3 does not directly translate to the choosing of schedules in practice. In particular, it remains unclear how the linear forcing ramp used in the experiments could be justified by the theoretical claims.

  1. Consequently, the proposed method appears to introduce some possible design rather than replacing or improving upon existing techniques such as tuning learning rates or weight decay. The authors should demonstrate what advantages their approach offers -- given a fixed tuning budget -- compared to conventional LR or WD tuning, and whether the proposed method effectively facilitates the hyperparameter search for optimal schedules empirically.
Questions for authors

Please refer to weaknesses.

2 comments
AuthorsOpenReview ↗

Response for Reviewer D7eU

1. Limited novelty beyond the existing recurrence

We explicitly take the recurrence of Amin et al. (2026) as our starting point. Our contributions address three questions beyond that forward description:

  1. Reachability and synthesis: Which requested forcing paths can bounded native decay realize? Theorem 1 gives necessary and sufficient conditions, unique realization given the initial multiplier, and global approximation of infeasible requests. Adjacent decay multipliers couple consecutive requests, making feasibility a joint path constraint.
  2. An optimal reference: Theorem 2 derives a policy optimal at every finite horizon for a specified stochastic alignment problem, providing an analytical reference for forcing design.
  3. Stochastic asymptotics and stationarity: Theorem 3 establishes the almost-sure effective-step regimes induced by cumulative forcing and their consequences for stationarity under the stated assumptions. Our experiment results also distinguish our contribution from prior works with additional experiment on exponent tails, cumulated forcing, different request testing and comprehensive/systematic experiments conducted. The additional experiments assess practical consequences through additional tuning runs, a matched two-arm projection comparison, and an independent point cutoff sweep in newly added section G with Table 7,9,10,11-15. The contribution therefore extends from describing directional dynamics to determining feasible schedules, deriving an optimal reference, and establishing their long-run optimization consequences.

2. Insufficient theoretical guidance for the practical ramp design

The optimal reference path supplies an optimization-based reference for schedule design. Theorem 2 establishes

Bt⋆=1−t−1+O(t−2),B_t^\star=1-t^{-1}+O(t^{-2}),

connecting B=1B=1 to optimal late-stage behavior in the reference problem. Without this result, we would lack this direct optimization-based justification for centering the design around one. It motivates a concrete principle: explore controlled deviations around B=1B=1, rather than choose an unrestricted forcing trajectory.

We instantiate this principle through

Btreq=1+ε(1−2tN−1),ε=0.002.B_t^{\mathrm{req}} =1+\varepsilon\left(1-\frac{2t}{N-1}\right), \qquad \varepsilon=0.002.

The request offsets effective-step decay early and reinforces it later, without estimating alignment or noise. Temporal-order analysis motivates studying this allocation, while reachability and approximation determine its feasible implementation through native decay. The reference problem motivates the design center; the ramp tests a simple finite-horizon intervention with fixed amplitude, rather than claiming to transfer the optimal reference policy unchanged to conventional networks.

3. Unclear advantage over conventional tuning under a fixed budget

Our framework structures schedule search through the coupled variable

Bt=ηt+1ηt(1−ηtλt)(1−ηt+1λt+1).B_t= \frac{\eta_{t+1}} {\eta_t(1-\eta_t\lambda_t)(1-\eta_{t+1}\lambda_{t+1})}.

Under the stated scale-invariance assumptions, matched initial directional states, and matched randomness, schedules with the same forcing path generate identical directional trajectories. Independent searches over LR and WD can therefore revisit equivalent directional behavior. Designing a forcing family and then determining its feasible implementation provides a more structured search, analogous in spirit to using learning-rate–batch-size scaling to guide joint hyperparameter choices.

The whole tuning results can be seen in Table 11-15. Besides higher selected accuracy, the ramp achieves lower variance across the tested setting:

SettingConstant: mean ± SDRamp: mean ± SDConstant varianceRamp variance
CIFAR-10 / ResNet-1890.34 ± 1.5693.39 ± 0.512.430.26
CIFAR-10 / ResNet-3490.61 ± 1.4093.75 ± 0.321.960.10
CIFAR-100 / ResNet-1867.34 ± 3.6471.85 ± 0.7913.250.63
CIFAR-100 / ResNet-3467.28 ± 3.0171.65 ± 1.099.071.19
CIFAR-100 / ResNet-5071.31 ± 5.5577.11 ± 0.5530.780.30

The ramp ranges from 0.32–1.09 %\%, compared with 1.40–5.55 %\% for constant decay. Strong performance therefore extends across a broader portion of the tested grid rather than depending only on one favorable configuration. These statistics describe hyperparameter sensitivity across different configurations. Therefore, by focusing on the BtB_t design, we can achieve simultaneously better performance and lower hyperparameter sensitivity. This supports a practical advantage within the tested grid.

While I appreciate the authors' rebuttal, my concerns have been partially resolved. Concretely, the new experiments seem to improve the empirical studies, but the novelty of the proposed method and the principled schedule selection remain unclear. Therefore, I decide to keep my current score.

Reviewer_N9w4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a weight-decay scheduler. It analyzes its effectiveness on scale-invariant loss functions that are unchanged under positive rescaling. This scheduler is derived from a recurrence that captures the coupled effects of the learning rate and weight decay. Theorem 1 suggests when this weight decay is feasible and proposes an alternative when it is not. Theorem 2 provides optimality for a simple optimization problem under the unit sphere. Theorem 3 explains the behavior of the directional step, like a polynomially decreasing schedule. The paper also provides empirical results. It runs experiments in a synthetic setup to verify Theorem 3 and compares its proposed weight-decay scheduler with constant weight decay on CIFAR-10, CIFAR-100, and WikiText-103 using ResNet and GPT-2.

Critical strengths
  • The empirical evaluation covers several architectures and datasets, and shows an advantage of the suggested approach.
  • The paper provides theoretical motivation for the proposed approach.
Critical weaknesses
  • The analysis appears to provide useful intuition, but its significance as an optimization result is less clear since the theoretical results are limited to specialized optimization problems. I therefore view the paper as more empirically motivated than theoretically driven.

  • My main empirical concern is the fairness of the baseline comparison. The proposed approach is more involved than constant weight decay, and it is unclear whether this additional complexity is justified. The experiments appear to tune the proposed approach's hyperparameters, while the constant-decay baseline reports no corresponding learning-rate or weight-decay sweep. Since both choices can significantly affect final performance, a comparably tuned baseline would make the empirical contribution more convincing.

  • An additional baseline with no weight decay can also be useful here.

Questions for authors

See weaknesses above.

1 comment
AuthorsOpenReview ↗

Response for Reviewer N9w4

1. Limited theoretical significance and generality

We want to clarify that the specialized stochastic alignment setup is required only for Theorem 2, where it provides an exactly solvable optimization benchmark and an optimal reference trajectory. It is not required for Theorems 1 or 3.

Theorem 1 characterizes which forcing sequences are realizable under bounded weight decay, provides their unique realization given the initial multiplier, and gives a globally optimal approximation for infeasible requests. Theorem 3 characterizes almost-sure effective-step behavior under the stated stochastic gradient assumptions, without requiring the alignment objective or noise geometry of Theorem 2. It identifies the transition at forcing exponent β=1/2\beta=1/2 and establishes almost-sure stationarity for 1/2<β≤11/2<\beta\leq1. The accompanying corollary shows that, even under the same harmonic nominal learning rate, changing weight decay can change the effective-step exponent and its square-summability regime.

These results concern optimization properties rather than only empirical intuition. In scale-invariant optimization, conditions imposed on nominal learning rates do not automatically transfer to effective directional steps: their behavior depends jointly on learning rate, weight decay, and parameter norm. Our contributions establish reachability, an optimal reference under specified geometry, and stochastic convergence properties of the resulting directional dynamics. We hope that the reviewer can further take into account of the theoretical contribution with our explanation.

2. Fairness of baseline tuning

We directly addressed this concern through a matched LR/WD search. We completed tuning runs across five settings: CIFAR-10/100 with ResNet-18/34 at 10,000 updates, and CIFAR-100 with ResNet-50 at 58,650 updates. Constant decay and the narrow ramp each receive the same 16-trial grid:

ηhi∈{0.05,0.1,0.25,0.5},λ0∈{0.0001,0.0003,0.001,0.003}.\eta_{\mathrm{hi}}\in\{0.05,0.1,0.25,0.5\}, \qquad \lambda_0\in\{0.0001,0.0003,0.001,0.003\}.

The ramp amplitude remains fixed at 0.0020.002. All methods use cosine LR ending at one-tenth of its initial value, batch size 128, SGD without momentum, whole-network decay, and the same update budget within each setting.

SettingConstant decayNarrow rampZero decay
CIFAR-10 / ResNet-1892.8493.8689.98
CIFAR-10 / ResNet-3492.7494.2889.94
CIFAR-100 / ResNet-1871.7273.0468.78
CIFAR-100 / ResNet-3471.2873.5867.18
CIFAR-100 / ResNet-5077.7277.8270.46

After tuning, the ramp exceeds constant decay by 1.02, 1.54, 1.32, 2.30, and 0.10 percentage points. (see Table 7 and the selected hyperparameter in Table 8) Thus, the validation improvements persist when constant decay receives the same tuning budget.

3. Missing zero-weight-decay baseline

We added zero decay with its own search over the same four initial learning rates. Its best validation accuracies are 89.98%\%, 89.94%\%, 68.78%\%, 67.18%\%, and 70.46%\%, respectively, below the corresponding tuned constant-decay and ramp results. All results can be seen in Table 7, 11, 12, 13, 14, 15. Zero decay receives four LR trials as it has no WD parameter. Constant decay and the ramp each receive 16 LR/WD trials. The selected zero-decay configurations are also included in the completed fresh-seed final evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.