acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Tuning of Weight Decay: Stability Analysis and Empirical Scheduling

Abstract

Weight decay is often held constant while the learning rate varies during training, although both affect optimization dynamics. In this paper, we investigate this interaction on two levels: static hyperparameter coupling between training configurations, and the potential performance gains from dynamic weight decay scheduling. Specifically, under uniform stability analysis, we characterize how weight decay contracts perturbations in stochastic gradient descent (SGD) and changes the learning-rate conditions for stability. Based on this, matching stability bounds under a heuristic regularization-equivalence hypothesis suggests the coupling , relating weight decay , learning rate , and the number of updates . For a fixed number of training epochs, this coupling further suggests that the product scales with batch size . Motivated by this inverse relationship, we examine two weight-decay schedules, linear-up and iso, that increase as decays while leaving the learning-rate schedule unchanged. In our experiments, hyperparameter sweeps on image-classification tasks and in Qwen3-0.6B for LoRA fine-tuning show trends consistent with the proposed coupling. Scheduling experiments across architectures and tasks with SGD show higher reported accuracy with increasing weight-decay schedules than with fixed weight decay in several settings. These findings support considering weight decay jointly with the learning rate when choosing its magnitude and scheduling it during training.

Primary area:
interpretability and explainable AI
Keywords:
Weight Decay, Stability, Coupling of Hyperparameters

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WzVQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript studies the interaction between weight decay, learning rate, batch size, and training duration from the perspective of uniform stability. It shows a stability bound for SGD with coupled weight decay. It uses a bound-matching heuristic with SWA to motivate the relations λη≈C/T and λη \propto B, and thus two weight-decay schedules. Experiments on CIFAR-100 and Qwen3 show trends consistent with the proposed coupling.

Critical strengths

The manuscript focuses on an interesting problem: how weight decay should be tuned jointly with learning rate, batch size, and training duration rather than treated as a fixed constant.

They said the proposed dynamic schedules are easy to implement, and evaluated under a shared learning-rate trajectory.

Critical weaknesses

a) Theorem 4.6 closely follows a known result. With the L2 term, the objective is λ-strongly convex. Hardt et al. (2016, Sec. 3.4, Thm. 3.9) already give the same O(1/(λn)) stability bound for this setting, and their Sec. 4 (Def. 4.1) defines the same weight-decay update. The proof in App. F.1 (Eqs. 27-28) follows the same recursion. What is the difference? This manuscript does not discuss this result.

b) The main coupling is a heuristic, not a result. Eq. 11 does not depend on η or T, while Eq. 12 grows with ηT. Setting them equal only shows where two upper bounds happen to coincide. It does not imply equivalent stability or regularization, as the authors adimit this (e.g., Abstract: it calls the basis of the coupling a heuristic regularization-equivalence hypothesis. Introduction, about lines 091-093: This comparison concerns stability upper bounds and does not establish equal test performance.)

c) It is not explained why SWA is the right reference algorithm, or how the relation would change with a different algorithm or bound. d) This manuscript treats C as adjustable (line 307). It does not check whether the bound-derived value holds across settings, e.g., different batch sizes or the LoRA experiments.

e) The theoretical novelty over prior work is unclear. The manuscript's own related work already covers the ηλ timescale (Li et al., 2020; Wang & Aitchison, 2025), near-constant ηλ behavior (DAngelo et al., 2024), and batch-dependent weight-decay scaling (Bergsma et al., 2025).

Since these coupling forms already exist, the authors should explain what the stability-matching argument predicts beyond them. They should also explain when it gives more useful tuning guidance than the existing rules, for example a setting where it makes a different or more accurate prediction.

Questions for authors
  1. Why is SWA the appropriate reference for the regularization equivalence argument? How would the proposed relation change if a different algorithm or constant were used, and can C (this constant) be predicted rather than empirically calibrated?

  2. Can the stability analysis be extended to time-varying η_t and λ_t, and to SGD with momentum?

  3. How do the proposed schedules compare with decreasing weight decay, AutoL2-style, adaptive weight decay, and gradient-norm-aware schedules under the same matched cumulative decay strength?

  4. If η is already scaled linearly with batch size, does Eq. (14) imply that λ should remain roughly constant rather than increase? How do the conclusions change under square-root or empirically tuned learning-rate scaling?

16 comments
AuthorsOpenReview ↗

Response to Weaknesses a.

Thank you for point this. We agree that Theorem 4.6 uses the classical analysis framework of Hardt et al. (2016), and that the O(1/(λn))O(1/(\lambda n)) stability rate is established by their strongly convex analysis. The distinction in our formulation concerns the assumptions, the loss being evaluated, and the explicit learning-rate condition.

The distinction is in the explicit treatment of the original loss. We impose convexity, GG-Lipschitz continuity, and LL-smoothness on FF, and evaluate stability using FF. Theorem 3.9, as stated, considers projected updates and uses a gradient bound for the analyzed strongly convex loss. In our formulation, for example, the shared regularizer cancels in the sample-replacement term at a common parameter: [∇F(θ;z)+λθ]−[∇F(θ;z′)+λθ]=∇F(θ;z)−∇F(θ;z′)\big[\nabla F(\theta;z)+\lambda\theta\big]-\big[\nabla F(\theta;z')+\lambda\theta\big]=\nabla F(\theta;z)-\nabla F(\theta;z').

Thus, this perturbation is bounded by 2ηG2\eta G, yielding 2G2/(λn)2G^2/(\lambda n) for the original loss without a separate gradient-bound assumption on the regularized objective. We will also clarify the domain assumptions for the unprojected iterates.

For the learning rate, direct substitution into Theorem 3.9 gives η≤1/(L+λ)\eta\le1/(L+\lambda), while our argument uses η≤2/(L+2λ)\eta\le2/(L+2\lambda). The latter threshold already appears through their Lemma 3.7, so we do not claim a new step-size principle. We will revise the contribution statement and retain this specialization to make the base-loss constants and contraction factor explicit for the subsequent analysis.

Accordingly, we will revise the introduction to Theorem 4.6, Appendix F.1, Table 1, and the contribution statement to provide direct attribution and present the result as a WD-specific specialization. We retain it to aid in understanding the establishment of the η\eta–λ\lambda heuristic relationship, providing consistent notation for the subsequent coupling discussion. They do not constitute an independent theoretical contribution.

AuthorsOpenReview ↗

Response to Weaknesses c

Thank you for point this. We selected the full-trajectory average in Eq. (7) as a transparent reference for comparing explicit weight decay with iterate averaging. It has an explicit, $lambda$-free stability bound under comparable convexity, Lipschitz, smoothness, and sampling assumptions, without introducing additional averaging hyperparameters. This makes the comparison analytically convenient, but does not make SWA uniquely correct or imply equivalence between SWA and SGD-WD.

The dependence on the reference can be seen from a general reference bound Uref=aηG2TnU_{\mathrm{ref}}=a\frac{\eta G^{2}T}{n}. Matching it with the displayed WD bound 2G2/(λn)2G^{2}/(\lambda n) gives λη=2aT\lambda\eta=\frac{2}{aT}.

The SWA bound used in the manuscript corresponds to a=1a=1, yielding C=ληT=2C=\lambda\eta T=2. For comparison, the standard last-iterate SGD bound has a=2a=2, yielding C=1C=1 [Hardt et al, 2016]. Thus, even within closely related SGD references, the coefficient depends on the selected bound and its tightness.

The coefficient also depends on the tightness of the bounds, including whether the finite-horizon WD factor discussed above is retained. More generally, a reference bound with a different dependence on η\eta or TT need not preserve the same scaling. We will clarify these dependencies in Section 5.1: C=2C=2 is specific to the displayed comparison, rather than an algorithm-independent or risk-optimal constant.

A more distinct reference is the Gibbs learner, a randomized learning algorithm defined through an optimization problem over probability distributions. Let S=(z1,…,zn)S=(z_1,\ldots,z_n) be the training dataset, F(θ;z)F(\theta;z) the loss of model θ\theta on example zz, and RS(θ)=1n∑i=1nF(θ;zi)R_S(\theta)=\frac1n\sum_{i=1}^n F(\theta;z_i) the empirical risk. Given a data-independent probability prior π\pi and a fixed temperature τ>0\tau>0, the algorithm solves QS=arg⁡min⁡Q≪π{Eθ∼Q[RS(θ)]+τDKL(Q∥π)}Q_S=\arg\min_{Q\ll\pi}\{\mathbb{E}_{\theta\sim Q} [R_{S}(\theta)]+\tau D_{\mathrm{KL}}(Q\Vert\pi)\}, where DKLD_{\mathrm{KL}} denotes Kullback–Leibler divergence. Its solution is QS(dθ)=exp⁡[−RS(θ)/τ]ZS π(dθ),ZS=∫exp⁡[−RS(v)/τ] π(dv)Q_S(d\theta)=\frac{\exp[-R_S(\theta)/\tau]}{Z_S}\,\pi(d\theta),\qquad Z_S=\int \exp[-R_S(v)/\tau]\,\pi(dv).

The learner then outputs a single sample θ∼QS\theta\sim Q_S. Thus, unlike SWA, this algorithm does not average an optimization trajectory. It is specified by its output distribution rather than by a particular gradient optimizer. The comparison below concerns exact Gibbs sampling; a finite-step sampling implementation would require additional approximation-error analysis.

Under the additional assumption 0≤F(θ;z)≤M0\le F(\theta;z)\le M, with the same π\pi and τ\tau for neighboring datasets, its replace-one uniform stability in expected loss satisfies sup⁡S∼S′, z∣Eθ∼QSF(θ;z)−Eθ′∼QS′F(θ′;z)∣≤M22τn\sup_{S\sim S',\,z}\left|\mathbb E_{\theta\sim Q_S}F(\theta;z)-\mathbb E_{\theta'\sim Q_{S'}}F(\theta';z)\right|\le \frac{M^2}{2\tau n}. Here S∼S′S\sim S' means that the datasets differ in one example, and the expectations are over the learner’s random output. This follows from the total-variation bound TV⁡(QS,QS′)≤M/(2τn)\operatorname{TV}(Q_S,Q_{S'})\le M/(2\tau n).

For comparison within a common bounded-loss setting, one can use a compact convex parameter domain and projected SGD-WD; the nonexpansive projection preserves the WD contraction argument. Retaining the original convexity, GG-Lipschitz, smoothness, and step-size conditions, matching the displayed bounds gives 2G2λn=M22τn⟹λ=4τG2M2\frac{2G^2}{\lambda n}=\frac{M^2}{2\tau n}\Longrightarrow\lambda=\frac{4\tau G^2}{M^2}, where λ\lambda is the WD coefficient and GG is the Lipschitz constant of the data loss.

For fixed τ,G,M\tau,G,M, this relation contains neither the WD learning rate η\eta nor its update count TT. Consequently, C=ληT=4τG2M2ηTC=\lambda\eta T=\frac{4\tau G^2}{M^2}\eta T is not constant across training horizons. This example shows that changing the reference algorithm and its assumptions can change the functional form of the matched relationship, not merely its coefficient. Therefore, such an algorithm is unsuitable for our study of the coupling relationship between learning rate and weight decay.

AuthorsOpenReview ↗

Response to Weaknesses d and Question 1 (Part 1)

We thank the reviewer for asking whether CC can be predicted and transferred. The displayed bound comparison specifies C=2C=2 without empirical calibration, but it does not establish that C=2C=2 is universally risk-optimal. We therefore distinguish the bound-derived reference coefficient from an empirically calibrated coefficient. The former is a falsifiable prediction of the stability-matching heuristic; the latter may depend on the model, optimizer, data, and training protocol.

Unless stated otherwise, the experiments use CIFAR-100, SGD without momentum, an initial learning rate of 0.10.1, and cosine learning-rate decay for 100 epochs. We define C=η0λˉT,λˉ=Cη0TC=\eta_0\bar\lambda T,\qquad \bar\lambda=\frac{C}{\eta_0T}, where TT is the exact number of optimizer updates. Fixed, Iso, and L-up share the same learning-rate trajectory and are matched in cumulative decay I=∑tηtλtI=\sum_t\eta_t\lambda_t within each setting. Under the discrete cosine convention, I=0.505CI=0.505C for the 100-epoch experiments. Errors are sample standard deviations; bold denotes the highest displayed mean.

Sensitivity to CC and batch size. On ResNet-18, Iso or L-up gives the highest mean final-test accuracy in all eight configurations below, covering C∈{1,1.5,2,2.5}C\in\{1,1.5,2,2.5\} and B∈{32,64}B\in\{32,64\}, with three paired seeds per configuration. Thus, the observed schedule benefit is not confined to a single tested value of CC in this range.

BatchCCFixedIsoL-up
32173.51±0.3373.52±0.2873.47±0.11
321.573.69±0.3173.78±0.3574.22±0.35
32273.95±0.3174.29±0.1274.11±0.07
322.574.57±0.4874.64±0.4074.63±0.10
64173.27±0.1773.25±0.6773.29±0.25
641.573.69±0.2573.68±0.3773.89±0.27
64274.27±0.3674.23±0.2874.38±0.17
642.574.58±0.0574.60±0.2074.55±0.31

Exploration at larger CC. A separate ResNet-18/Batch size=128 search gives the following single-seed validation accuracies. Iso or L-up is best among the four static schedules at each tested strength. These results are exploratory validation estimates, not multi-seed test estimates.

CCFixedIsoL-upLinear-down
475.6074.8675.6874.78
876.2476.2277.1676.58
1677.6278.1278.5277.04
3277.5278.5678.0277.50

Transfer across architectures with C=2C=2. Without recalibrating CC, Iso or L-up has the highest mean in all six ResNet-18/ResNet-34/VGG-16 comparisons, with gains of 0.11–0.40 percentage points over Fixed.

ArchitectureBatchFixedIsoL-up
ResNet-183273.92±0.3174.30±0.2673.95±0.20
ResNet-186474.27±0.3674.23±0.2874.38±0.17
ResNet-343273.93±0.3174.31±0.4674.28±0.34
ResNet-346473.98±0.4274.38±0.4374.38±0.25
VGG-163269.56±0.3769.57±0.0770.05±0.35
VGG-166469.98±1.1070.01±0.2169.93±0.47

Expanded CC sensitivity on VGG-16. At B=128B=128, the complete six-seed grid shows a higher mean for Iso or L-up at five of six tested values, while Fixed remains best at C=2C=2. This supports schedule usefulness over a range of strengths, but also shows that the preferred strength and schedule depend on the setting.

CCFixedIsoL-up
1.569.48±0.2969.68±0.1669.51±0.43
270.37±0.2870.19±0.3970.11±0.56
2.570.81±0.3070.70±0.3470.83±0.33
370.86±0.4571.01±0.3271.14±0.35
3.571.12±0.4671.13±0.4971.34±0.28
471.28±0.4971.47±0.2271.33±0.22
Reviewer_7vSwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that the learning rate step (η\eta), the weight decay parameter (λ\lambda), and the number of parameter updates (TT) in the learning are related as ηλ≈C/T\eta \lambda \approx C/T, where C is a problem-specific constant. The relation is derived from two generalization bounds on convex loss functions and a reasoning on stability, using the "Regularization Equivalence Hypothesis". The regularization equivalence hypothesis is, as I understand it, that the effect of SWA (Stochastic Weight Averaging) is very similar to SGD-WD (Stochastic Gradient Descent with Weight Decay), i.e., they are both a form of regularization and therefore can the scaling behavior of their generalization bounds be equaled. The relationship is supported with some experiments.

Critical strengths

The relation ηλ≈C/T\eta \lambda \approx C/T, where C is a problem-specific constant, is new. This is interesting and certainly worth exploring further. The empirical evidence points to that this may indeed be a valid relation.

Critical weaknesses

The argumentation builds on a "Regularization Equivalence Hypothesis", and I think this hypothesis should have been motivated more clearly. I am not convinced that the theory holds and that the generalization bounds of SWA and SGD-WD can be equaled. I don't see how the two provided cases (Case 1 and Case 2) clarify the "Regularization Equivalence Hypothesis". That a function can be expanded using different basis functions is true, the function is the same but the basis is different, but how does that motivate that the SWA and SDG-WD generalization bounds can be equaled? Also, that a one-dimensional symmetric loss function complemented with a regularization term centered on zero leads to a minimum that "coincides exactly with the average" (zero) is also true, but going from there to concluding that SWA (which is averaging) and SGD-WD (which is regularization) are very similar is quite a stretch.

Questions for authors

I think the paper should focus more on the relation ηλ≈C/T\eta \lambda \approx C/T. I recommend moving some of the experiments on this that are in the supplementary material into the main paper and remove the dynamic weight decay experiments from the main paper.

4 comments
AuthorsOpenReview ↗

Response to Weaknesses

Thank you for point this. We agree that, absent assumptions, equality of stability upper bounds does not establish equivalent generalization or optimal hyperparameters. The logic we will state more explicitly in the manuscript is how a common criterion can be used to establish a coupling relationship among hyperparameters, so that the motivation is presented more directly. In the revised version, we strictly restrict this hyperparameter coupling study to a stability-based heuristic relationship.

First, the individual stability bounds do not depend on the "Regularization Equivalence Hypothesis": they follow from their respective analyses under the stated assumptions. Under comparable assumptions, SWA provides an explicit, λ\lambda-free stability bound, making it a convenient reference for this comparison. Under the same assumptions, the weight decay (WD) update provides additional intuition. Under the conditions of Theorem 4.6, a previously introduced perturbation is attenuated by subsequent factors of 1−ηλ1-\eta\lambda.

For the displayed expressions, 2G2λn=ηG2Tn⟹λη=2T\frac{2G^2}{\lambda n}=\frac{\eta G^2T}{n}\Longrightarrow\lambda\eta=\frac{2}{T}. Here, equality is imposed as a comparison criterion, rather than inferred as a property of the algorithms. It identifies where the two upper bounds coincide; it does not establish equal actual stability, equal generalization, or optimal regularization.

Accordingly, C=2C=2 belongs to this particular bound comparison. Population risk also depends on training fit, which changes with weight decay, so stability alone cannot determine the test-optimal coefficient. It is worth noting, however, that in our ablation experiments on the constant CC, this particular constant factor obtained by matching SWA and SGD-WD does not affect the effectiveness of the hyperparameter dynamic coupling framework proposed in this paper. As shown in the table below:

Transfer across architectures with C=2C=2. Without recalibrating CC, Iso or L-up has the highest mean in all six ResNet-18/ResNet-34/VGG-16 comparisons, with gains of 0.11–0.40 percentage points over Fixed.

ArchitectureBatchFixedIsoL-up
ResNet-183273.92±0.3174.30±0.2673.95±0.20
ResNet-186474.27±0.3674.23±0.2874.38±0.17
ResNet-343273.93±0.3174.31±0.4674.28±0.34
ResNet-346473.98±0.4274.38±0.4374.38±0.25
VGG-163269.56±0.3769.57±0.0770.05±0.35
VGG-166469.98±1.1070.01±0.2169.93±0.47

Sensitivity to CC and batch size. On ResNet-18, Iso or L-up gives the highest mean final-test accuracy in all eight configurations below, covering C∈{1,1.5,2,2.5}C\in\{1,1.5,2,2.5\} and B∈{32,64}B\in\{32,64\}, with three paired seeds per configuration. Thus, the observed schedule benefit is not confined to a single tested value of CC in this range.

BatchCCFixedIsoL-up
32173.51±0.3373.52±0.2873.47±0.11
321.573.69±0.3173.78±0.3574.22±0.35
32273.95±0.3174.29±0.1274.11±0.07
322.574.57±0.4874.64±0.4074.63±0.10
64173.27±0.1773.25±0.6773.29±0.25
641.573.69±0.2573.68±0.3773.89±0.27
64274.27±0.3674.23±0.2874.38±0.17
642.574.58±0.0574.60±0.2074.55±0.31

Expanded CC sensitivity on VGG-16. At B=128B=128, the complete six-seed grid shows a higher mean for Iso or L-up at five of six tested values, while Fixed remains best at C=2C=2. This supports schedule usefulness over a range of strengths, but also shows that the preferred strength and schedule depend on the setting.

CCFixedIsoL-up
1.569.48±0.2969.68±0.1669.51±0.43
270.37±0.2870.19±0.3970.11±0.56
2.570.81±0.3070.70±0.3470.83±0.33
370.86±0.4571.01±0.3271.14±0.35
3.571.12±0.4671.13±0.4971.34±0.28
471.28±0.4971.47±0.2271.33±0.22

When C=2C=2, the results under different settings remain advantageous; when CC varies, these results still demonstrate the superiority of our proposed method; of course, when C=2C=2, Fixed may also yield good performance, so the matched CC is not necessarily optimal. Therefore, we will clarify this logical distinction around Eq. (13) and remove wording that equates bound matching with identical generalization.

In addition, we agree with the reviewer that these two examples are insufficient to give readers an understanding that would justify an equivalence between averaging and weight decay within the algorithmic stability framework. We will therefore remove both examples from the revised manuscript.

AuthorsOpenReview ↗

Response to Question

Thank you for your constructive suggestions. We have removed the relevant content, including Cases 1 and 2, from the main text and revised the statements about the regularization assumption to ensure that the motivation is clear. In addition, we have moved the experimental results validating the hyperparameter coupling relationship, including Figure 5, to the main text to ensure the completeness of the main body of the paper.

However, with regard to the experimental results on the dynamic strategies, we prefer to retain them in the main text. This is because the proposal of these two dynamic schedules is one of the important contributions of this paper and also serves as a validation of the coupling relationship in practical applications.

Reviewer_7vSwOpenReview ↗

The significance of ISO and L-up results.

Thank you for the extra results. Still, I don't see significant differences between the "fixed" schedule and the two dynamic schedules, so I'm still confused why you want to include them. However, this is your choice to make.

AuthorsOpenReview ↗

Regarding Content Adjustments

Thank you for your careful feedback and emphasis on the quality and focus of the paper. We agree that the main text should give greater attention to the coupling relationship. We will replace Table 3 and its detailed analysis with a focused discussion of CC, supported by the existing sensitivity and transfer experiments. We will also bring the existing cross-architecture learning-rate–weight-decay results into the main text, showing how the high-performing regions change across learning rates and architectures.

Our original reason for retaining Table 3 was to provide experimental support for the two proposed scheduling frameworks. We agree that this purpose can be served more concisely. We will retain only a brief empirical summary in the main text and move the complete schedule comparisons and detailed analysis to the appendix. This will preserve support for the frameworks while keeping the main discussion centered on the coupling relationship, its empirical usefulness, and its limitations.

Reviewer_XTL1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper provides an uniform stability based analysis on how weight decay, learning rate and training duration jointly affect the generalization bounds for SGD. They also propose two schedules, linear-up and iso and present improved performance over fixed weight decay in several evaluated settings.

Critical strengths

They studied a pratical problem in training about dynamic weight decay selection and the coupling of it and learning rate. Their emprirical finding shows that linear-up and iso are easy to implement and introduce negligible computational overhead. And they experiment over various model architectures.

Critical weaknesses
  1. The relationship λη≈C/T\lambda\eta\approx C/T is obtained by equating two upper bounds on uniform stability under a heuristic regularization-equivalence assumption. Equality of upper bounds does not imply equivalent generalization or optimal hyperparameter choices. Consequently, the paper's central coupling rule is not theoretically established.
  2. The experiments focus mainly on image classification and Qwen3-0.6B fine-tuning. There is no large-scale language-model pretraining study, despite the relevance of weight-decay scaling to this setting.
  3. Coupling between weight decay, learning rate, and batch size has been investigated in prior work. Please provide literature review on relevant topics.
Questions for authors

Do the proposed coupling rules and increasing schedules remain effective for AdamW, particularly in large-scale language-model pretraining?

3 comments
AuthorsOpenReview ↗

Response to Weaknesses 1

Thank you for point this. We agree that, absent assumptions, equality of stability upper bounds does not establish equivalent generalization or optimal hyperparameters. The logic we will state more explicitly in the manuscript is how a common criterion can be used to establish a coupling relationship among hyperparameters, so that the motivation is presented more directly. In the revised version, we strictly restrict this hyperparameter coupling study to a stability-based heuristic relationship.

First, the individual stability bounds do not depend on the "Regularization Equivalence Hypothesis": they follow from their respective analyses under the stated assumptions. Under comparable assumptions, SWA provides an explicit, λ\lambda-free stability bound, making it a convenient reference for this comparison. Under the same assumptions, the weight decay (WD) update provides additional intuition. Under the conditions of Theorem 4.6, a previously introduced perturbation is attenuated by subsequent factors of 1−ηλ1-\eta\lambda.

For the displayed expressions, 2G2λn=ηG2Tn⟹λη=2T\frac{2G^2}{\lambda n}=\frac{\eta G^2T}{n}\Longrightarrow\lambda\eta=\frac{2}{T}. Here, equality is imposed as a comparison criterion, rather than inferred as a property of the algorithms. It identifies where the two upper bounds coincide; it does not establish equal actual stability, equal generalization, or optimal regularization.

Accordingly, C=2C=2 belongs to this particular bound comparison. Population risk also depends on training fit, which changes with weight decay, so stability alone cannot determine the test-optimal coefficient. It is worth noting, however, that in our ablation experiments on the constant CC, this particular constant factor obtained by matching SWA and SGD-WD does not affect the effectiveness of the hyperparameter dynamic coupling framework proposed in this paper. As shown in the table below:

Transfer across architectures with C=2C=2. Without recalibrating CC, Iso or L-up has the highest mean in all six ResNet-18/ResNet-34/VGG-16 comparisons, with gains of 0.11–0.40 percentage points over Fixed.

ArchitectureBatchFixedIsoL-up
ResNet-183273.92±0.3174.30±0.2673.95±0.20
ResNet-186474.27±0.3674.23±0.2874.38±0.17
ResNet-343273.93±0.3174.31±0.4674.28±0.34
ResNet-346473.98±0.4274.38±0.4374.38±0.25
VGG-163269.56±0.3769.57±0.0770.05±0.35
VGG-166469.98±1.1070.01±0.2169.93±0.47

Sensitivity to CC and batch size. On ResNet-18, Iso or L-up gives the highest mean final-test accuracy in all eight configurations below, covering C∈{1,1.5,2,2.5}C\in\{1,1.5,2,2.5\} and B∈{32,64}B\in\{32,64\}, with three paired seeds per configuration. Thus, the observed schedule benefit is not confined to a single tested value of CC in this range.

BatchCCFixedIsoL-up
32173.51±0.3373.52±0.2873.47±0.11
321.573.69±0.3173.78±0.3574.22±0.35
32273.95±0.3174.29±0.1274.11±0.07
322.574.57±0.4874.64±0.4074.63±0.10
64173.27±0.1773.25±0.6773.29±0.25
641.573.69±0.2573.68±0.3773.89±0.27
64274.27±0.3674.23±0.2874.38±0.17
642.574.58±0.0574.60±0.2074.55±0.31

Expanded CC sensitivity on VGG-16. At B=128B=128, the complete six-seed grid shows a higher mean for Iso or L-up at five of six tested values, while Fixed remains best at C=2C=2. This supports schedule usefulness over a range of strengths, but also shows that the preferred strength and schedule depend on the setting.

CCFixedIsoL-up
1.569.48±0.2969.68±0.1669.51±0.43
270.37±0.2870.19±0.3970.11±0.56
2.570.81±0.3070.70±0.3470.83±0.33
370.86±0.4571.01±0.3271.14±0.35
3.571.12±0.4671.13±0.4971.34±0.28
471.28±0.4971.47±0.2271.33±0.22

When C=2C=2, the results under different settings remain advantageous; when CC varies, these results still demonstrate the superiority of our proposed method; of course, when C=2C=2, Fixed may also yield good performance, so the matched CC is not necessarily optimal. Therefore, we will clarify this logical distinction around Eq. (13) and remove wording that equates bound matching with identical generalization.

AuthorsOpenReview ↗

Response to Weaknesses 2 and Question 1

R1-W2: Full-parameter pretraining

Thank you for this suggestion. We have added full-parameter language-model pretraining experiments on C4 using a randomly initialized Llama60M decoder-only Transformer, covering 75M and 150M training tokens and different effective batch sizes.

For SGD, we use no momentum, an initial learning rate of 0.3, and cosine learning-rate decay. Fixed, Iso, and L-up share the same learning-rate trajectory and matched cumulative decay strength I=∑tηtλt=0.005I=\sum_t\eta_t\lambda_t=0.005. Within each seed, initialization and data order are paired across methods. The following results report final validation perplexity as mean ± sample standard deviation; lower is better.

TokensBatchFixedIsoL-up
75M16416.834±1.605416.621±1.651416.558±1.055
75M1281029.692±9.1501029.385±7.3231030.933±8.993
150M16284.745±3.590284.165±3.534283.859±2.954
150M128752.358±4.425751.682±5.055752.787±4.429

Iso or L-up has the lowest mean PPL among these three schedules in each reported setting, although the improvements are modest relative to seed variability.

R1-Q1: Does the approach remain useful with AdamW?

We have added controlled AdamW pretraining experiments using the same Llama60M/C4 setup. AdamW uses an initial learning rate of 10−310^{-3}, β1=0.9\beta_1=0.9, β2=0.95\beta_2=0.95, and ϵ=10−8\epsilon=10^{-8}. All parameters receive weight decay. The comparisons use sequence length 1024, cosine learning-rate decay without warmup or a learning-rate floor, gradient clipping at 1.0, and matched I=0.005I=0.005. All prespecified seeds are included.

TokensBatchFixedIsoL-up
75M1673.522±0.48873.282±0.52272.875±0.531
75M64111.761±4.031111.130±4.068111.362±4.945
75M128181.881±6.713180.718±4.643181.648±5.513
150M1654.725±0.29954.646±0.49354.532±0.441
150M6461.932±0.88162.183±1.13462.179±1.120
150M12887.205±1.59087.201±2.08386.223±2.360

These results show that the schedules can provide empirical benefits with AdamW: across two different training budgets, our proposed method achieved superior performance in 5 out of 6 experimental settings. However, when B=64, although the "fixed" strategy performed best, the margin by which it outperformed the others was not significant.

And, we conducted randomly initialized, full-parameter pretraining of SmolLM2-135M on C4 using AdamW. All runs use approximately 150M training tokens, sequence length 1024, initial learning rate 10−310^{-3}, cosine learning-rate decay, AdamW betas (0.9,0.95)(0.9,0.95), ϵ=10−8\epsilon=10^{-8}, and gradient clipping at 1.

Batchstrength IFixedIsoL-up
640.00565.075±0.22864.802±0.32264.903±0.424
640.0164.723±0.36864.883±0.32864.970±0.410
1280.00591.430±1.46991.139±2.89791.754±1.755
1280.0190.985±2.84990.331±0.12490.198±0.495
2560.005156.121±2.701155.019±0.450155.284±0.810
2560.01156.115±1.115155.538±0.898155.164±0.907

Iso or L-up achieves the lowest mean perplexity in five of six settings. The reductions relative to Fixed range from 0.273 to 1.102 PPL; Fixed remains best at B=64, I=0.01B=64,\ I=0.01. These observations suggest that temporal allocation of weight decay can also be useful in AdamW pretraining. In summary, the proposed method maintains a certain advantage even when using pre-training and the AdamW optimizer.

AuthorsOpenReview ↗

Response to Weaknesses 3

Thank you for point this. We agree that the relationships among weight decay, learning rate, batch size, and training duration have substantial precedents. Although several of these studies are discussed in Appendix A, their connections to our formulation should be more explicit in the main related-work section.

The coupling between learning rate and weight decay has been studied through training dynamics. Li et al. (2020) identify ηλ\eta\lambda as an intrinsic learning rate in normalized networks and investigate its relationship to equilibration. Lewkowycz & Gur-Ari (2020) observe that the time to peak performance scales approximately as 1/(ηλ)1/(\eta\lambda) for vanilla SGD and use this relationship to predict regularization for a given training budget. D’Angelo et al. (2024) further report approximately constant-ηλ\eta\lambda contours of comparable performance in multipass SGD experiments. These findings establish important precedents for both the product coupling and its connection to training duration.

For adaptive optimization, Loshchilov & Hutter (2019) distinguish decoupled weight decay from an L2 penalty, making the optimizer and decay parameterization essential to any comparison. Wang & Aitchison (2025) interpret AdamW parameters as an exponential moving average of updates, with an iteration timescale τiter≈1/(ηλ)\tau_{\mathrm{iter}}\approx1/(\eta\lambda), and study transfer through the epoch timescale τepoch≈B/(nηλ)\tau_{\mathrm{epoch}}\approx B/(n\eta\lambda). Bergsma et al. (2025) extend this perspective to LLM pretraining, finding approximately linear scaling of the preferred weight decay with batch size below the critical batch size, at fixed model size, data budget, and learning rate. They also show that the preferred training-normalized timescale depends on tokens per parameter.

Our analysis connects these relationships to sensitivity to changes in the training dataset through algorithmic stability. Accordingly, we will position the static coupling as a stability-motivated interpretation of established relationships. Our experiments additionally investigate how allocating decay over time affects performance under an unchanged learning-rate trajectory and matched cumulative strength. We will expand Section 2 to explain these connections, the overlap in the resulting formulas, and the different assumptions underlying the respective analyses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.