- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This manuscript studies the interaction between weight decay, learning rate, batch size, and training duration from the perspective of uniform stability. It shows a stability bound for SGD with coupled weight decay. It uses a bound-matching heuristic with SWA to motivate the relations λη≈C/T and λη \propto B, and thus two weight-decay schedules. Experiments on CIFAR-100 and Qwen3 show trends consistent with the proposed coupling.
Critical strengths
The manuscript focuses on an interesting problem: how weight decay should be tuned jointly with learning rate, batch size, and training duration rather than treated as a fixed constant.
They said the proposed dynamic schedules are easy to implement, and evaluated under a shared learning-rate trajectory.
Critical weaknesses
a) Theorem 4.6 closely follows a known result. With the L2 term, the objective is λ-strongly convex. Hardt et al. (2016, Sec. 3.4, Thm. 3.9) already give the same O(1/(λn)) stability bound for this setting, and their Sec. 4 (Def. 4.1) defines the same weight-decay update. The proof in App. F.1 (Eqs. 27-28) follows the same recursion. What is the difference? This manuscript does not discuss this result.
b) The main coupling is a heuristic, not a result. Eq. 11 does not depend on η or T, while Eq. 12 grows with ηT. Setting them equal only shows where two upper bounds happen to coincide. It does not imply equivalent stability or regularization, as the authors adimit this (e.g., Abstract: it calls the basis of the coupling a heuristic regularization-equivalence hypothesis. Introduction, about lines 091-093: This comparison concerns stability upper bounds and does not establish equal test performance.)
c) It is not explained why SWA is the right reference algorithm, or how the relation would change with a different algorithm or bound. d) This manuscript treats C as adjustable (line 307). It does not check whether the bound-derived value holds across settings, e.g., different batch sizes or the LoRA experiments.
e) The theoretical novelty over prior work is unclear. The manuscript's own related work already covers the ηλ timescale (Li et al., 2020; Wang & Aitchison, 2025), near-constant ηλ behavior (DAngelo et al., 2024), and batch-dependent weight-decay scaling (Bergsma et al., 2025).
Since these coupling forms already exist, the authors should explain what the stability-matching argument predicts beyond them. They should also explain when it gives more useful tuning guidance than the existing rules, for example a setting where it makes a different or more accurate prediction.
Questions for authors
-
Why is SWA the appropriate reference for the regularization equivalence argument? How would the proposed relation change if a different algorithm or constant were used, and can C (this constant) be predicted rather than empirically calibrated?
-
Can the stability analysis be extended to time-varying η_t and λ_t, and to SGD with momentum?
-
How do the proposed schedules compare with decreasing weight decay, AutoL2-style, adaptive weight decay, and gradient-norm-aware schedules under the same matched cumulative decay strength?
-
If η is already scaled linearly with batch size, does Eq. (14) imply that λ should remain roughly constant rather than increase? How do the conclusions change under square-root or empirically tuned learning-rate scaling?
16 comments
Response to Weaknesses c
Thank you for point this. We selected the full-trajectory average in Eq. (7) as a transparent reference for comparing explicit weight decay with iterate averaging. It has an explicit, $lambda$-free stability bound under comparable convexity, Lipschitz, smoothness, and sampling assumptions, without introducing additional averaging hyperparameters. This makes the comparison analytically convenient, but does not make SWA uniquely correct or imply equivalence between SWA and SGD-WD.
The dependence on the reference can be seen from a general reference bound . Matching it with the displayed WD bound gives .
The SWA bound used in the manuscript corresponds to , yielding . For comparison, the standard last-iterate SGD bound has , yielding [Hardt et al, 2016]. Thus, even within closely related SGD references, the coefficient depends on the selected bound and its tightness.
The coefficient also depends on the tightness of the bounds, including whether the finite-horizon WD factor discussed above is retained. More generally, a reference bound with a different dependence on or need not preserve the same scaling. We will clarify these dependencies in Section 5.1: is specific to the displayed comparison, rather than an algorithm-independent or risk-optimal constant.
A more distinct reference is the Gibbs learner, a randomized learning algorithm defined through an optimization problem over probability distributions. Let be the training dataset, the loss of model on example , and the empirical risk. Given a data-independent probability prior and a fixed temperature , the algorithm solves , where denotes Kullback–Leibler divergence. Its solution is .
The learner then outputs a single sample . Thus, unlike SWA, this algorithm does not average an optimization trajectory. It is specified by its output distribution rather than by a particular gradient optimizer. The comparison below concerns exact Gibbs sampling; a finite-step sampling implementation would require additional approximation-error analysis.
Under the additional assumption , with the same and for neighboring datasets, its replace-one uniform stability in expected loss satisfies . Here means that the datasets differ in one example, and the expectations are over the learner’s random output. This follows from the total-variation bound .
For comparison within a common bounded-loss setting, one can use a compact convex parameter domain and projected SGD-WD; the nonexpansive projection preserves the WD contraction argument. Retaining the original convexity, -Lipschitz, smoothness, and step-size conditions, matching the displayed bounds gives , where is the WD coefficient and is the Lipschitz constant of the data loss.
For fixed , this relation contains neither the WD learning rate nor its update count . Consequently, is not constant across training horizons. This example shows that changing the reference algorithm and its assumptions can change the functional form of the matched relationship, not merely its coefficient. Therefore, such an algorithm is unsuitable for our study of the coupling relationship between learning rate and weight decay.
Response to Weaknesses d and Question 1 (Part 1)
We thank the reviewer for asking whether can be predicted and transferred. The displayed bound comparison specifies without empirical calibration, but it does not establish that is universally risk-optimal. We therefore distinguish the bound-derived reference coefficient from an empirically calibrated coefficient. The former is a falsifiable prediction of the stability-matching heuristic; the latter may depend on the model, optimizer, data, and training protocol.
Unless stated otherwise, the experiments use CIFAR-100, SGD without momentum, an initial learning rate of , and cosine learning-rate decay for 100 epochs. We define , where is the exact number of optimizer updates. Fixed, Iso, and L-up share the same learning-rate trajectory and are matched in cumulative decay within each setting. Under the discrete cosine convention, for the 100-epoch experiments. Errors are sample standard deviations; bold denotes the highest displayed mean.
Sensitivity to and batch size. On ResNet-18, Iso or L-up gives the highest mean final-test accuracy in all eight configurations below, covering and , with three paired seeds per configuration. Thus, the observed schedule benefit is not confined to a single tested value of in this range.
| Batch | Fixed | Iso | L-up | |
|---|---|---|---|---|
| 32 | 1 | 73.51±0.33 | 73.52±0.28 | 73.47±0.11 |
| 32 | 1.5 | 73.69±0.31 | 73.78±0.35 | 74.22±0.35 |
| 32 | 2 | 73.95±0.31 | 74.29±0.12 | 74.11±0.07 |
| 32 | 2.5 | 74.57±0.48 | 74.64±0.40 | 74.63±0.10 |
| 64 | 1 | 73.27±0.17 | 73.25±0.67 | 73.29±0.25 |
| 64 | 1.5 | 73.69±0.25 | 73.68±0.37 | 73.89±0.27 |
| 64 | 2 | 74.27±0.36 | 74.23±0.28 | 74.38±0.17 |
| 64 | 2.5 | 74.58±0.05 | 74.60±0.20 | 74.55±0.31 |
Exploration at larger . A separate ResNet-18/Batch size=128 search gives the following single-seed validation accuracies. Iso or L-up is best among the four static schedules at each tested strength. These results are exploratory validation estimates, not multi-seed test estimates.
| Fixed | Iso | L-up | Linear-down | |
|---|---|---|---|---|
| 4 | 75.60 | 74.86 | 75.68 | 74.78 |
| 8 | 76.24 | 76.22 | 77.16 | 76.58 |
| 16 | 77.62 | 78.12 | 78.52 | 77.04 |
| 32 | 77.52 | 78.56 | 78.02 | 77.50 |
Transfer across architectures with . Without recalibrating , Iso or L-up has the highest mean in all six ResNet-18/ResNet-34/VGG-16 comparisons, with gains of 0.11–0.40 percentage points over Fixed.
| Architecture | Batch | Fixed | Iso | L-up |
|---|---|---|---|---|
| ResNet-18 | 32 | 73.92±0.31 | 74.30±0.26 | 73.95±0.20 |
| ResNet-18 | 64 | 74.27±0.36 | 74.23±0.28 | 74.38±0.17 |
| ResNet-34 | 32 | 73.93±0.31 | 74.31±0.46 | 74.28±0.34 |
| ResNet-34 | 64 | 73.98±0.42 | 74.38±0.43 | 74.38±0.25 |
| VGG-16 | 32 | 69.56±0.37 | 69.57±0.07 | 70.05±0.35 |
| VGG-16 | 64 | 69.98±1.10 | 70.01±0.21 | 69.93±0.47 |
Expanded sensitivity on VGG-16. At , the complete six-seed grid shows a higher mean for Iso or L-up at five of six tested values, while Fixed remains best at . This supports schedule usefulness over a range of strengths, but also shows that the preferred strength and schedule depend on the setting.
| Fixed | Iso | L-up | |
|---|---|---|---|
| 1.5 | 69.48±0.29 | 69.68±0.16 | 69.51±0.43 |
| 2 | 70.37±0.28 | 70.19±0.39 | 70.11±0.56 |
| 2.5 | 70.81±0.30 | 70.70±0.34 | 70.83±0.33 |
| 3 | 70.86±0.45 | 71.01±0.32 | 71.14±0.35 |
| 3.5 | 71.12±0.46 | 71.13±0.49 | 71.34±0.28 |
| 4 | 71.28±0.49 | 71.47±0.22 | 71.33±0.22 |
Response to Weaknesses a.
Thank you for point this. We agree that Theorem 4.6 uses the classical analysis framework of Hardt et al. (2016), and that the O(1/(λn)) stability rate is established by their strongly convex analysis. The distinction in our formulation concerns the assumptions, the loss being evaluated, and the explicit learning-rate condition.
The distinction is in the explicit treatment of the original loss. We impose convexity, G-Lipschitz continuity, and L-smoothness on F, and evaluate stability using F. Theorem 3.9, as stated, considers projected updates and uses a gradient bound for the analyzed strongly convex loss. In our formulation, for example, the shared regularizer cancels in the sample-replacement term at a common parameter: [∇F(θ;z)+λθ]−[∇F(θ;z′)+λθ]=∇F(θ;z)−∇F(θ;z′).
Thus, this perturbation is bounded by 2ηG, yielding 2G2/(λn) for the original loss without a separate gradient-bound assumption on the regularized objective. We will also clarify the domain assumptions for the unprojected iterates.
For the learning rate, direct substitution into Theorem 3.9 gives η≤1/(L+λ), while our argument uses η≤2/(L+2λ). The latter threshold already appears through their Lemma 3.7, so we do not claim a new step-size principle. We will revise the contribution statement and retain this specialization to make the base-loss constants and contraction factor explicit for the subsequent analysis.
Accordingly, we will revise the introduction to Theorem 4.6, Appendix F.1, Table 1, and the contribution statement to provide direct attribution and present the result as a WD-specific specialization. We retain it to aid in understanding the establishment of the η–λ heuristic relationship, providing consistent notation for the subsequent coupling discussion. They do not constitute an independent theoretical contribution.