acceptodds
Under review as a conference paper at ICLR 2027

EqWD: Bounded Weight Decay from Standardized Ratio Deviations

Abstract

Weight decay controls parameter scale in modern optimizers, and decoupled implementations expose it as a separate update. Most recipes keep its coefficient fixed while loss-driven update size changes during training. The gradient-to-weight norm ratio tracks that size, but absolute values vary across tensors and mix persistent scale with temporal change. We measure each tensor's deviation from its moving ratio mean and normalize it by a moving residual scale. We use this standardized deviation to drive adaptive decoupled decay, which we call Equilibrium-Driven Weight Decay (EqWD). EqWD increases the base coefficient in proportion to the deviation, changes only radial decay, and leaves the base optimizer's loss update unchanged. Including the current residual in the scale bounds the per-step coefficient; the exponential moving-average (EMA) dynamics also bound accumulated squared modulation over a finite horizon and characterize when the signal vanishes. Across three paired 90-epoch ImageNet-1K ResNet-50 runs, EqWD improves best-epoch Top-1 by points over the reference fixed-decay recipe. The exposure-matched fixed-decay control reduces this margin to 0.255 points, showing that effective decay strength explains much of the reference gain. On 300-epoch ViT-Tiny/ImageNet-1K, EqWD reaches final Top-1 versus for fixed decay. On COCO2017 DINO (DETR with Improved DeNoising Anchor Boxes) detection, EqWD raises average precision (AP) by 0.6 points with ResNet-50 and 0.1 points with Swin-T.

Primary area:
optimization
Keywords:
optimizer

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rJoaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes EqWD, which increases decoupled weight decay according to the standardized deviation of each tensor's gradient-to-weight norm ratio from its moving mean. It establishes bounds on instantaneous and accumulated modulation, characterizes recovery to base decay, and proves local convergence for a scalar quadratic. Experiments evaluate image classification and object detection against fixed-decay baselines.

Critical strengths

The algorithm is simple and easy to follow, requiring only two moving-average statistics per eligible parameter tensor.

Critical weaknesses
  1. Limited methodological and theoretical contribution. The method combines a gradient-to-weight norm ratio, moving-average statistics, and a correction to weight decay. I find the resulting innovation limited. Section 3 mentions related approaches such as Scheduled Weight Decay, Cautious Weight Decay, and Constrained Parameter Regularization, but provides little comparison explaining the advantage of this design. Although the contributions are explicitly listed, the presentation does not sufficiently establish the significance of the mathematical results for optimization. The pointwise coefficient bound follows directly from including the current residual in the denominator's scale estimate. The recovery and cumulative bounds further characterize the controller, but do not establish better optimization or generalization. The only optimization-convergence result, Proposition 3, assumes deterministic gradient descent on a one-dimensional quadratic, with local initialization and additional learning-rate and gain restrictions. This is too narrow to provide theoretical support for the practical settings studied.

  2. Insufficient evidence of practical benefit. The gains are mixed and often disappear against stronger fixed-decay controls. On CIFAR-100/ResNet-20, searched fixed decay achieves 69.180% best-epoch accuracy versus 67.627% for EqWD. ImageNet ViT-Tiny is essentially tied (+0.021 points). Comparisons against other adaptive-decay methods are also absent. These results do not convincingly isolate a benefit from adaptation beyond stronger regularization. The evaluation is restricted to vision models. If practical utility is the main contribution, I would expect convincing evidence at a larger scale, for example language models with several hundred million parameters, against well-tuned baselines. Otherwise, more substantive theory is needed to justify the contribution. The measured step-time overhead on ResNet-50 further raises the question of whether the small residual gains justify the added cost.

Questions for authors
  1. What evidence distinguishes the benefit of adaptive timing and tensor-specific decay from simply increasing fixed decay, given the searched and exposure-matched results?
  2. What technical insight distinguishes EqWD from the adaptive-decay methods discussed in Section 3, beyond its particular choice of normalization?
3 comments

We thank Reviewer rJoa for recognizing EqWD's simplicity and two-scalar per-tensor state, and for clearly specifying the evidence needed to substantiate the paper's contribution: mechanistic and empirical comparisons with SWD, CWD, and CPR; optimization theory beyond 1D gradient descent; evaluation on 100M+ language models against tuned baselines; and negligible runtime overhead. All four are included in the revised manuscript. Part 1 addresses W1 and Q2; Part 2 addresses W2 and Q1.


W1 & Q2: What technical insight distinguishes EqWD from the adaptive-decay methods discussed in Section 3 (SWD, CWD, CPR), beyond its particular choice of normalization? And how do the mathematical results connect to optimization beyond 1D gradient descent?

Appendix A.8 (Table 7) formalizes the structural differences among the four adaptive approaches:

SWD (Xie et al., 2023) scales weight decay globally by the inverse square root of the Adam/SGDS second moment (∝1/vˉt\propto 1/\sqrt{\bar v_t}). Because it does not reference parameter norm ∥wtl∥\Vert w_t^l\Vert and has no intrinsic upper bound, sharp drops in gradient norm late in training can inflate effective decay substantially (reaching λ^=4.8×10−3\hat{\lambda} = 4.8\times 10^{-3} on ImageNet-1K — 3.5× higher than EqWD's 1.4×10−31.4\times 10^{-3} — and trailing EqWD at 75.36% vs. 76.01%). CWD (Chen et al., 2026) multiplies decay coordinate-wise by the sign-alignment mask I(ut,iwt,i≥0)\mathbb{I}(u_{t,i}w_{t,i} \ge 0); this alters update directions (breaking radial isotropy) without tracking layer norm equilibrium, reaching 75.52% on ImageNet SGD. CPR (Franke et al., 2024) replaces fixed decay with per-tensor norm-bound constraints ∥wl∥≤κl\Vert w^l\Vert \le \kappa^l updated via augmented-Lagrangian dual multipliers; this performs strongly when explicit norm caps suit AdamW language modeling (59.35 PPL on GPT-2 350M) or CIFAR-100 (68.09%), but is sensitive to the warm-start heuristic for κl\kappa^l under step-drop SGD on ImageNet-1K (71.60%).

By contrast, EqWD separates persistent architectural scale from transient non-stationarity by measuring departures from each tensor's own moving ratio μt−1l\mu_{t-1}^l. Across GPT-2's 48–96 linear layers, raw gradient-to-weight ratios rtlr_t^l span 322.6×–1039.4× simultaneously (Figure 3); subtracting μt−1l\mu_{t-1}^l factors out this persistent cross-layer spread while preserving pure radial isotropy, guaranteeing a tuning-free instantaneous ceiling qtl≤(1−αv)−1/2q_t^l \le (1-\alpha_v)^{-1/2} via same-step residual inclusion (Proposition 1), and automatically returning to λ0\lambda_0 once ratios stabilize (Proposition 2) so asymptotic regularization strength is unperturbed.

To connect this controller to optimization dynamics beyond 1D gradient descent, Section 5 (Appendices A.9–A.10) adds two closed-loop theorems. Theorem 2 extends Proposition 3 to pp-dimensional SPD quadratics f(w)=12(w−c)⊤H(w−c)f(w) = \frac12(w-c)^{\top} H(w-c) (H≻0,c≠0H \succ 0, c \neq 0) under preconditioned heavy-ball momentum (P≻0,m∈[0,1)P \succ 0, m \in [0,1)), proving that the (2p+2)(2p+2)-state system preserves the exact fixed point w∗=(H+(1−m)λ0P)−1Hcw_{\ast} = (H+(1-m)\lambda_0 P)^{-1}Hc of fixed decoupled decay and converges locally exponentially under 2aheff<1−a2a h_{\text{eff}} < 1 - a. Theorem 3 (Figure 1) analyzes orthogonal loss gradients ut=gtu_t = g_t (G0>0G_0 > 0) in normalized layers where xt=∥wt∥2x_t = \Vert w_t\Vert^2 follows xt=(1−γλt)2xt−1+γ2Gt2x_t = (1-\gamma\lambda_t)^2 x_{t-1} + \gamma^2 G_t^2, proving conditional single-step overshoot damping on spikes G1>G0G_1 > G_0 (Δdamp<2Δspike\Delta_{\text{damp}} < 2\Delta_{\text{spike}}), strict contraction ∥wtEqWD∥<∥wtfixed∥\Vert w_t^{\text{EqWD}}\Vert < \Vert w_t^{\text{fixed}}\Vert for all t≥1t \ge 1 after a scale drop G0→G1∈(0,G0)G_0 \to G_1 \in (0,G_0), and a mandatory finite-step residual sign flip (e1<0→et∗>0e_1 < 0 \to e_{t_{\ast}} > 0 when w∗(G1)>a1−aεw_{\ast}(G_1) > \frac{a}{1-a}\varepsilon) before local exponential recovery.

W2 & Q1: What evidence distinguishes the benefit of adaptive timing and tensor-specific decay from simply increasing fixed decay, given the searched and exposure-matched results — including comparisons against adaptive baselines and on language models with several hundred million parameters?

Following your recommendation, we pretrained GPT-2 Small (124M, 48 decayed matrices, 734M tokens, 3 seeds) and GPT-2 Medium (350M, 96 matrices, 524M tokens, 2 seeds) on disjoint FineWeb-Edu train/val/test shards (Section 6.4, Tables 4, 22, Figure 3). Hyperparameters were selected on GPT-2 124M via a 35-run 600-step screen on val.bin (9 FixedWD, 12 EqWD, 6 CWD, 4 CPR, and 4 SWD trials, where EqWD ranked first overall at val_loss = 4.6053) and transferred zero-shot to GPT-2 350M without retuning:

Method (FineWeb-Edu GPT-2)124M Test PPL (3 seeds)Zero-Shot 350M PPL (2 seeds)Global λ^\hat{\lambda} (124M / 350M)
Reference AdamW (λ=0.10\lambda = 0.10)43.87±0.0960.98±0.200.1000 / 0.1000
Tuned FixedWD (λ=0.01\lambda = 0.01)43.76±0.0561.10±0.080.0100 / 0.0100
Per-Tensor Matched AdamW (2-pass)43.92±0.2260.88±0.460.0807 / 0.0806
Global Exposure-Matched AdamW43.82±0.13N/A0.0807 / N/A
Nominal Global EqWD43.83±0.27N/A0.0835 / N/A
Strength-Matched Global EqWD43.74±0.1260.82±0.180.0822 / 0.0786
Tuned SWD (λ=10−5\lambda = 10^{-5})43.81±0.11N/AN/A
Tuned CWD (λ=0.003\lambda = 0.003)43.74±0.0760.60±0.24N/A
Tuned CPR (warm-start 140 / 100)43.17±0.1559.35±0.36N/A
Tuned EqWD (λ0=0.07,β=0.5,αμ=0.95\lambda_0 = 0.07, \beta = 0.5, \alpha_{\mu} = 0.95)43.80±0.0860.94±0.160.0807 / 0.0806

Three findings across our LLM and vision suites directly address Q1:

  1. Zero-shot LLM scale transfer and variance stability (Table 4, Figure 3). On GPT-2, EqWD assigns stronger modulation to residual-write projections (c_proj: 0.0885–0.0899 on 350M) than input projections (c_attn/c_fc: 0.0767/0.0726). On 124M, EqWD beats reference AdamW on 3/3 seeds (−0.34 PPL at matched λ0=0.10\lambda_0 = 0.10, Table 22a) and lowers cross-seed PPL SD by 3.3× vs. uncalibrated global EqWD (0.266) and by 2.8× vs. two-pass per-tensor matching (0.222). When transferred zero-shot to GPT-2 350M, 124M-tuned FixedWD (λ=0.01\lambda = 0.01) degrades to 61.10±0.08 PPL, whereas both EqWD variants (60.94±0.16 per-tensor and 60.82±0.18 strength-matched global) outperform tuned FixedWD and reference AdamW (60.98±0.20), with per-tensor EqWD self-preserving global exposure (0.0807→0.08060.0807 \to 0.0806) automatically in one pass. While CWD (60.60) and norm-constrained CPR (59.35) reach lower AdamW LM perplexity, they trail EqWD by 0.49–4.41 percentage points on 90-epoch ImageNet SGD (75.52% and 71.60% vs. 76.01%), making EqWD consistently reliable across both optimizer families.

  2. Held-out ImageNet-1K and CIFAR-100 controls (Table 2, Figure 2a). On 90-epoch ImageNet-1K with properly scaled baselines, single-pass online EqWD (76.01%) beats scalar global exposure matching (75.75%, +0.26%), tuned decoupled FixedWD (75.92%), and coupled L2L_2 (75.77%), matching the two-pass 53-tensor oracle (76.05%) in one pass. On CIFAR-100, auditing selection.json showed archived EqWD had been capped at λ0≤10−3\lambda_0 \le 10^{-3} (10× below FixedWD's 10−210^{-2}); removing that cap brings 5-seed EqWD (67.51±0.35%) slightly ahead of tuned FixedWD (67.44±0.21%) while winning 6 of 7 base rates across a 200× λ0\lambda_0 sweep.

  3. Strength-matched gain calibration (Table 18). Calibrating β∗\beta^{\ast} within ±1.83% exposure across 5 seeds shows unnormalized ∣et∣|e_t| trailing full EqWD in both regimes (65.72% vs. 66.04%; 67.29% vs. 67.51%) with a 32× shift in β∗\beta^{\ast} (1825→56.31825 \to 56.3), confirming DtlD_t^l's role beyond scalar exposure.


W2 (Overhead): The measured step-time overhead on ResNet-50 further raises the question of whether the small residual gains justify the added cost.

Our initial overhead came from per-tensor Python .item() GPU–CPU syncs in the prototype. Vectorizing EqWD with fused torch._foreach norms (Section 4, Tables 6, 23, Figure 2b) cuts median step-time overhead to +0.55% on ResNet-50, +0.63% on ViT-Tiny, +0.54% on GPT-2 124M, +1.23% on GPT-2 350M, and +3.57% on ResNet-20, with zero GPU–CPU synchronizations and no increase in reported peak VRAM.

Thank you for the detailed response and additional experiments. However, the new results do not show a clear benefit: on GPT-2, EqWD trails CWD and CPR and is within noise of tuned fixed decay, and the vision gains remain small. The added theory also does not establish improved optimization or generalization. I will keep my score.

Reviewer_VoK5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Equilibrium-Driven Weight Decay (EqWD), a per-tensor controller for decoupled weight decay. For each weight tensor it tracks a moving average of the gradient-to-weight norm ratio and a moving residual scale. It standardizes the current deviation from that average and increases the decay coefficient in proportion to it, so that departures in either direction raise decay. The theory shows that the coefficient stays bounded, characterizes when the modulation vanishes, bounds accumulated modulation, and gives a local stability condition on a scalar quadratic. Validation covers CIFAR-100 and ImageNet with ResNets, ViT-Tiny with AdamW, and DINO detection on COCO. EqWD is compared with reference, searched and exposure-matched fixed decay. It improves on the reference recipe for ResNet-50 on ImageNet, but the margin largely disappears against exposure-matched decay. It is on par with fixed decay for ViT and gives small AP gains for DINO.

Critical strengths

The paper includes the controls needed to test its own claim. Exposure-matched fixed decay, built from the logged EqWD coefficients, and a searched fixed coefficient on CIFAR separate adaptive timing from overall decay strength, and per-seed results are reported. This matters to the assessment because it makes the evidence interpretable, although what the controls show argues against acceptance.

Critical weaknesses

The experiments do not show a benefit from the adaptive signal beyond the decay strength it induces, and the paper's own discussion acknowledges this. On CIFAR-100, both searched and exposure-matched fixed decay outperform EqWD. On ImageNet, the margin over exposure-matched decay is small and not consistent across seeds, and no searched fixed coefficient is reported. Fixed decay is ahead under every recipe change in Table 9, and the ViT results are within noise of fixed decay. The premise of the rule is not argued either: the text explains why the sign of the ratio change is discarded, but not why a departure in either direction should call for more shrinkage. The theory establishes that the controller is bounded but gives no reason it should improve training. This is the main reason for rejection: a decay controller that adds several hyperparameters and noticeable step-time overhead needs to outperform a tuned fixed coefficient, and it does not.

The baseline and selection protocol favor the method. The reference ResNet-50 accuracy is well below what is usually reported for this 90-epoch recipe. One plausible reason is that applying the reference coefficient as decoupled decay outside the momentum buffer is considerably weaker than the coupled L2 decay of the standard recipe. If so, any rule that raises the average coefficient would help, which would explain the gain over the reference. Hyperparameters are also selected on the CIFAR test split and the ImageNet validation set, with no held-out split (Appendix B.3). This is consequential because the remaining margins are small enough that an under-regularized reference and selection on the evaluation split could account for them.

Questions for authors
  • What accuracy does the reference ResNet-50 recipe reach with a tuned fixed decoupled coefficient, or with standard coupled L2 decay, and what remains of the EqWD gain against it?
  • What is the argument for increasing decay symmetrically for upward and downward ratio departures? The one-sided variants in Table 2 perform about the same as the full rule.
  • Were any results re-run with hyperparameters selected on a held-out split of the training data?
  • Per-tensor exposure matching, which the discussion names as the missing control, and comparisons with existing adaptive-decay methods such as SWD and CPR would isolate adaptive timing and show where EqWD stands relative to prior work.
  • The abstract leads with the gain over the reference recipe. Leading with the exposure-matched comparison, which the paper treats as the decisive control, would describe the result more accurately.
4 comments

We thank Reviewer VoK5 for an incisive and constructive review. Your diagnoses regarding decoupled-vs-coupled momentum scaling on ResNet-50, evaluation-split hyperparameter selection, and the need for an optimization argument for symmetric modulation were spot-on. When auditing our archived CIFAR-100 search records (selection.json), we also found a symmetric grid cap that had penalized EqWD in the opposite direction: FixedWD searched up to λ=10−2\lambda = 10^{-2}, whereas EqWD had been capped at λ0≤10−3\lambda_0 \le 10^{-3} (ten times lower). Below we address Q1 in Part 1, Q2 in Part 2, and Q3–Q5 in Part 3.


Q1: What accuracy does the reference ResNet-50 recipe reach with a tuned fixed decoupled coefficient, or with standard coupled L2L_2 decay, and what remains of the EqWD gain against it?

Theorem 2 (Section 5, Appendix A.9) formalizes your exact observation: on f(w)=12(w−c)⊤H(w−c)f(w) = \frac12(w-c)^{\top} H(w-c) under heavy-ball momentum mm (P=IP = I), decoupled weight decay λ0\lambda_0 has fixed point w∗=(H+(1−m)λ0I)−1Hcw_{\ast} = (H+(1-m)\lambda_0 I)^{-1}Hc, matching coupled L2L_2 regularization of strength (1−m)λ0(1-m)\lambda_0. At m=0.9m = 0.9, applying λ=10−4\lambda = 10^{-4} outside the momentum buffer imposed a 10× weaker stationary penalty (10−510^{-5}) than standard coupled L2L_2 (10−410^{-4}), explaining why our archived reference baseline reached only 72.78%.

To evaluate what remains of EqWD's gain against properly scaled baselines without validation leakage, we split the 1.281M ImageNet-1K training set into 1.231M sub-train and 50k train-val, ran 17 15-epoch screening runs on train-val (Table 20), and evaluated full 90-epoch runs (seed 42) on the untouched 50k official-val set (Section 6.2, Table 2):

Method (ImageNet-1K / ResNet-50, 90 epochs)train-val Bestofficial-val @ BestFinal (Ep 90)Cumulative λ^\hat{\lambda}
Standard Coupled L2L_2 (λ=10−4\lambda = 10^{-4})78.70%75.77%75.77%1.0×10−41.0\times 10^{-4} (coupled)
Tuned Decoupled FixedWD (λ=1.5×10−3\lambda = 1.5\times 10^{-3})78.53%75.92%75.94%1.5×10−31.5\times 10^{-3}
Global Exposure-Matched FixedWD78.69%75.75%75.74%1.4×10−31.4\times 10^{-3}
Per-Tensor Static Oracle (2-pass replay)78.75%76.05%76.05%1.4×10−31.4\times 10^{-3}
Tuned CWD (Chen et al., 2026)78.11%75.52%75.48%1.5×10−31.5\times 10^{-3}
Tuned SWD (Xie et al., 2023)77.97%75.36%75.29%4.8×10−34.8\times 10^{-3}
Tuned CPR (Franke et al., 2024)74.41%71.60%71.69%5.6×10−55.6\times 10^{-5}
Tuned EqWD (λ0=10−3,β=0.5\lambda_0 = 10^{-3}, \beta = 0.5)78.86%76.01%75.99%1.4×10−31.4\times 10^{-3}

Once baselines are brought to the standard 76% regime, the artificial 1.05% gap over under-regularized FixedWD disappears as expected, yet single-pass online EqWD still ranks first among all six online families on both train-val (78.86%) and untouched official-val (76.01%) — edging coupled L2L_2 (+0.25%), tuned decoupled FixedWD (+0.09%), and scalar global exposure matching (75.75%, +0.26%), while matching the two-pass 53-tensor static oracle (76.05%) in one online pass.

Q2: What is the argument for increasing decay symmetrically for upward and downward ratio departures? The one-sided variants in Table 2 perform about the same as the full rule.

Section 5 (Theorem 3, Figure 1, Appendix A.10) provides the radial-dynamics argument for orthogonal loss gradients ut=gtu_t = g_t (G0>0G_0 > 0), where xt=∥wt∥2x_t = \Vert w_t\Vert^2 evolves via xt=(1−γλt)2xt−1+γ2Gt2x_t = (1-\gamma\lambda_t)^2 x_{t-1} + \gamma^2 G_t^2 with constant-scale equilibrium radius w∗(G)∝G/λ0w_{\ast}(G) \propto G/\sqrt{\lambda_0}:

On an upward spike G1>G0G_1 > G_0 (e1>0e_1 > 0), the gradient inflow γ2G12\gamma^2 G_1^2 pushes ∥w1∥\Vert w_1\Vert above w∗(G0)w_{\ast}(G_0); raising λ1>λ0\lambda_1 > \lambda_0 damps single-step norm overshoot whenever Δdamp<2Δspike\Delta_{\text{damp}} < 2\Delta_{\text{spike}} (Theorem 3(1)). Conversely, after a downward gradient drop G0→G1∈(0,G0)G_0 \to G_1 \in (0,G_0) (such as after a learning-rate drop or entering a flatter basin, giving e1<0e_1 < 0), the historical weight norm ∥w0∥=w∗(G0)\Vert w_0\Vert = w_{\ast}(G_0) is now too large relative to the new target radius w∗(G1)<w∗(G0)w_{\ast}(G_1) < w_{\ast}(G_0) (stale norm excess). Raising λt>λ0\lambda_t > \lambda_0 accelerates contraction of this stale norm, enforcing ∥wtEqWD∥<∥wtfixed∥\Vert w_t^{\text{EqWD}}\Vert < \Vert w_t^{\text{fixed}}\Vert for all t≥1t \ge 1.

Crucially, Theorem 3(2) proves that whenever w∗(G1)>a1−aεw_{\ast}(G_1) > \frac{a}{1-a}\varepsilon, the initial negative residual e1<0e_1 < 0 (Phase I) must flip sign to et∗>0e_{t_{\ast}} > 0 at a finite step t∗≥2t_{\ast} \ge 2 (Phase II, Figure 1b) as ∥wt∥\Vert w_t\Vert shrinks past the lagging EMA μt−1\mu_{t-1}. Because relaxation after a single drop spans both signs of ete_t, a high-side-only rule misses Phase I (q1=0q_1 = 0), whereas signed rules (±et/Dt\pm e_t / D_t) either weaken decay below λ0\lambda_0 or reverse direction mid-relaxation; symmetric ∣et∣|e_t| keeps λt≥λ0\lambda_t \ge \lambda_0 across both phases without adding a sign hyperparameter, and returns to λ0\lambda_0 once ratios equilibrate.

Empirically, archived Table 2 (now Table 3) fixed β=5\beta = 5 across unequal exposures. In Table 18 (5 seeds, held-out CIFAR-100), we calibrated β∗\beta^{\ast} within ±1.83% exposure for all 8 variants (including signed ±et/Dt\pm e_t / D_t and one-sided rules). Unnormalized ∣et∣|e_t| trails full EqWD in both low- and tuned-decay regimes (65.72% vs. 66.04%; 67.29% vs. 67.51%, β∗:1825→56.3\beta^{\ast}: 1825 \to 56.3), while on smooth-cosine ResNet-20 all bounded DtD_t-normalized variants cluster within seed noise (full 67.51±0.35%, signed 67.44–67.50%, one-sided 67.70–67.91%); symmetric ∣et∣|e_t| provides schedule-general response under step-decay LR drops (Figure 4) and steep early LM ratio drops (Table 22a) without risking λt<λ0\lambda_t < \lambda_0.

Q3: Were any results re-run with hyperparameters selected on a held-out split of the training data?

Yes — all new CIFAR-100, ViT-Tiny, ImageNet-1K, and FineWeb-Edu GPT-2 experiments select hyperparameters and best checkpoints strictly on held-out validation splits carved out of the training data, evaluating official test/validation splits purely for final reporting (Appendix E.1).

On CIFAR-100/ResNet-20 (45k train / 5k val / 10k untouched test, 12 screening runs per main family on 5k val, 5 confirmation seeds, Table 2), removing the archived λ0≤10−3\lambda_0 \le 10^{-3} search cap reverses the 1.55% deficit from the initial submission: validation-selected EqWD (λ0=0.007,β=0.25\lambda_0 = 0.007, \beta = 0.25) reaches 67.51±0.35% Test@Val (67.63±0.47% final, λ^=0.0084\hat{\lambda} = 0.0084) at 16% lower cumulative exposure than tuned FixedWD (λ=0.01\lambda = 0.01, 67.44±0.21%, final 67.44±0.24%) and ahead of coupled L2L_2 (67.24±0.50%), clustering within 0.19% of global exposure matching (67.70±0.31%). Moreover, across a 200× sweep of λ0∈[10−4,0.02]\lambda_0 \in [10^{-4}, 0.02] (Figure 2a, 3 seeds), EqWD outperforms FixedWD at 6 of 7 base coefficients (peaking at 67.88±0.24%), and under zero-retuning recipe transfer (Table 19a, replacing archived Table 9) EqWD improves over tuned FixedWD at batch size 64 (+0.45±0.20% across all 3 seeds) and 5-epoch linear warmup (+0.25±0.44%), ties on the standard recipe, and stays within 0.20% noise on batch size 256 and step decay.


Q4: Per-tensor exposure matching, which the discussion names as the missing control, and comparisons with existing adaptive-decay methods such as SWD and CPR would isolate adaptive timing and show where EqWD stands relative to prior work.

We added both controls across vision and language modeling (Tables 2, 4, 7, 17, 19b):

On 4-level isolation (global scalar matching →\to per-tensor static matching →\to shuffled-time replay →\to online EqWD), homogeneous ResNet-20 shows small differences across levels (67.36%–67.70% in Table 17), whereas on 90-epoch ImageNet-1K per-tensor allocation accounts for the +0.26% gain over scalar global matching (76.01% online EqWD and 76.05% two-pass static oracle vs. 75.75% global matching), and on GPT-2 124M and zero-shot 350M (Table 4) online EqWD reduces cross-seed PPL standard deviation by 2.8×–2.9× relative to two-pass per-tensor static matching (43.80±0.08 vs. 43.92±0.22 on 124M; 60.94±0.16 vs. 60.88±0.46 on 350M) while beating both zero-shot tuned FixedWD (61.10±0.08 on 2/2 seeds) and reference AdamW (60.98±0.20).

Against audited published implementations of SWD (Xie et al., 2023), CWD (Chen et al., 2026), and primal-dual augmented-Lagrangian CPR (Franke et al., 2024), EqWD is consistently competitive across both SGD and AdamW: on 90-epoch ImageNet SGD (Table 2), EqWD (76.01%) outperforms CWD (75.52%), SWD (75.36%), and CPR (71.60%); under AdamW on GPT-2 (Table 4) and small-scale CIFAR/ViT, CWD (60.60 PPL on 350M) and norm-constrained CPR (59.35 PPL on 350M; 68.09% on CIFAR) achieve strong results when explicit coordinate masks or hard norm bounds suit the task, which we report in full alongside a mechanistic taxonomy in Appendix A.8 (Table 7).

Finally, regarding runtime and hyperparameter overhead: replacing our prototype Python loop with fused torch._foreach kernels (Section 4, Tables 6, 23) cuts median step-time overhead from 6.7%–34.8% down to +3.57% (ResNet-20), +0.55% (ResNet-50; +1.83% under channels_last end-to-end training), +0.63% (ViT-Tiny), +0.54% (GPT-2 124M), and +1.23% (GPT-2 350M) with no increase in reported peak VRAM, and fixed default (αμ,αv)(\alpha_{\mu}, \alpha_v) works across the entire 200× λ0\lambda_0 sweep without per-rate retuning.


Q5: The abstract leads with the gain over the reference recipe. Leading with the exposure-matched comparison, which the paper treats as the decisive control, would describe the result more accurately.

We adopted this recommendation in full. The revised Abstract (Page 1) removes the +1.049% lead over the under-regularized reference recipe and leads directly with our held-out tuned FixedWD, coupled L2L_2, exposure-matched controls, and zero-shot GPT-2 transfer results.

Thank you for the response. With properly scaled baselines, the gain over the reference recipe disappears, and EqWD is within noise of tuned fixed decay on ImageNet (a single seed) and on CIFAR-100, where global exposure matching is ahead of it. The adaptive signal therefore still shows no benefit beyond the decay strength it induces, while adding hyperparameters and overhead. I will keep my score.

Reviewer_kNkiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EqWD, an adaptive decoupled weight decay method that adjusts the regularization coefficient based on standardized deviations of the gradient-to-weight norm ratio.

The method maintains exponential moving averages of the ratio and its residual variance for each parameter tensor, using the normalized deviation to modulate the decay coefficient.

The authors provide theoretical analyses establishing bounds on the instantaneous decay coefficient, recovery conditions, and accumulated modulation.

The results show improvements over reference fixed-decay configurations in some settings.

Critical strengths
  1. well-defined adaptive regularization mechanism with theoretical analysis The proposed method provides a relatively simple and interpretable mechanism for adapting weight decay based on training dynamics. The mathematical analysis establishes explicit bounds on the decay coefficient and characterizes the behavior of the modulation signal over time.

  2. transparent experimental evaluation with relevant control experiments the authors evaluate the method across multiple architectures and tasks, including image classification and object detection, while they also also acknowledge limitations and report settings where EqWD does not outperform fixed decay

Critical weaknesses
  1. the benefist of adaptive decay are not convincingly isolated: Although the theoretical results establish bounded decay coefficients and characterize the modulation dynamics, they do not demonstrate improved convergence rates, optimization efficiency, or generalization compared with fixed weight decay. The analysis of the coupled optimization dynamics is limited to a scalar quadratic objective under plain gradient descent, making its implications for large-scale neural networks and adaptive optimizers unclear.

  2. insufficient evidence that adaptive weight decay provides meaningful improvements over well-tuned fixed decay: On ImageNet-1K/ResNet-50, EqWD improves accuracy by 1.049 percentage points over the reference fixed-decay configuration. However, this advantage decreases to only 0.255 percentage points against the exposure-matched baseline, with two of the three paired runs showing negative improvements.

Questions for authors
  1. What is the empirical benefit specifically attributable to tensor-wise standardized deviations?

  2. Given the nearly identical performance of global statistics and full EqWD in Table 2, can the authors explain which component is essential and provide strength-matched evidence?

  3. Can the authors provide additional runs or uncertainty estimates for the COCO detection experiments? Given the small improvements, particularly the 0.1 AP gain with Swin-T, it is difficult to determine whether the differences are meaningful.

3 comments

We thank Reviewer kNki for the thoughtful review and for recognizing EqWD's interpretable mechanism, explicit bounds, and transparent control experiments. Below we address W1 in Part 1, W2 and Q1 in Part 2, and Q2–Q3 in Part 3.


W1: Although the theoretical results establish bounded decay coefficients and characterize the modulation dynamics, they do not demonstrate improved convergence rates, optimization efficiency, or generalization compared with fixed weight decay. The analysis of the coupled optimization dynamics is limited to a scalar quadratic objective under plain gradient descent, making its implications for large-scale neural networks and adaptive optimizers unclear.

To bridge controller bounds and optimization dynamics beyond 1D gradient descent, we added two closed-loop theorems in Section 5 (Figure 1; full proofs in Appendices A.9–A.10).

First, Theorem 2 analyzes EqWD coupled to a pp-dimensional quadratic f(w)=12(w−c)⊤H(w−c)f(w) = \frac12(w-c)^{\top} H(w-c) (H≻0,c≠0H \succ 0, c \neq 0) optimized with preconditioned heavy-ball momentum ut=mut−1+P−1∇f(wt)u_t = m u_{t-1} + P^{-1}\nabla f(w_t) (P≻0,m∈[0,1)P \succ 0, m \in [0,1)) and decoupled update wt+1=(1−ηλt)wt−ηutw_{t+1} = (1-\eta\lambda_t)w_t - \eta u_t. We prove that the (2p+2)(2p+2)-state coupled loop preserves the exact stationary point w∗=(H+(1−m)λ0P)−1Hcw_{\ast} = (H+(1-m)\lambda_0 P)^{-1}Hc of fixed decoupled decay (with q∗=0q_{\ast} = 0 and λ∗=λ0\lambda_{\ast} = \lambda_0, introducing zero stationary bias), and achieves local exponential convergence whenever the small-gain condition 2aheff<1−a2a h_{\text{eff}} < 1 - a holds (reducing cleanly to Proposition 3 when p=1,m=0,P=1p = 1, m = 0, P = 1).

Second, to model scale-invariant and normalized layers in modern networks where effective updates are approximately orthogonal to weights, Theorem 3 analyzes orthogonal loss gradients ut=gtu_t = g_t (⟨wt−1,gt⟩=0,∥gt∥=Gt,G0>0\langle w_{t-1}, g_t\rangle = 0, \Vert g_t\Vert = G_t, G_0 > 0), under which xt=∥wt∥2x_t = \Vert w_t\Vert^2 evolves as xt=(1−γλt)2xt−1+γ2Gt2x_t = (1-\gamma\lambda_t)^2 x_{t-1} + \gamma^2 G_t^2 around the constant-scale equilibrium x∗(G)=γ2G2/[1−(1−γλ0)2]x_{\ast}(G) = \gamma^2 G^2 / [1-(1-\gamma\lambda_0)^2]. Starting from (x∗(G0),r∗(G0),0)(x_{\ast}(G_0), r_{\ast}(G_0), 0), Theorem 3 establishes three concrete behaviors: (i) on an upward gradient spike G1>G0G_1 > G_0, EqWD raises λ1>λ0\lambda_1 > \lambda_0 and reduces single-step norm overshoot (∣x1EqWD−x∗(G0)∣<∣x1fixed−x∗(G0)∣|x_1^{\text{EqWD}} - x_{\ast}(G_0)| < |x_1^{\text{fixed}} - x_{\ast}(G_0)|) whenever Δdamp<2Δspike\Delta_{\text{damp}} < 2\Delta_{\text{spike}}; (ii) after a downward scale drop G0→G1∈(0,G0)G_0 \to G_1 \in (0,G_0), EqWD enforces strict radial contraction ∥wtEqWD∥<∥wtfixed∥\Vert w_t^{\text{EqWD}}\Vert < \Vert w_t^{\text{fixed}}\Vert for every step t≥1t \ge 1, and whenever w∗(G1)>a1−aεw_{\ast}(G_1) > \frac{a}{1-a}\varepsilon, the residual flips sign from e1<0e_1 < 0 to et∗>0e_{t_{\ast}} > 0 at a finite step t∗≥2t_{\ast} \ge 2; and (iii) (x∗(G1),r∗(G1),0)(x_{\ast}(G_1), r_{\ast}(G_1), 0) is locally exponentially stable under 2ahrad<1−a2a h_{\text{rad}} < 1 - a.

W2: Insufficient evidence that adaptive weight decay provides meaningful improvements over well-tuned fixed decay: On ImageNet-1K/ResNet-50, EqWD improves accuracy by 1.049 percentage points over the reference fixed-decay configuration. However, this advantage decreases to only 0.255 percentage points against the exposure-matched baseline, with two of the three paired runs showing negative improvements.

Theorem 2 clarifies why the archived ResNet-50 reference baseline (decoupled λ=10−4\lambda = 10^{-4}, 72.78%) was under-regularized: at m=0.9m = 0.9 (P=IP = I), decoupled decay outside momentum imposes a stationary penalty (1−m)λ0=10−5(1-m)\lambda_0 = 10^{-5}, ten times weaker than standard coupled L2L_2 at 10−410^{-4}. To test EqWD against properly scaled baselines without evaluation-split leakage, we split the 1.281M ImageNet-1K training images into 1.231M sub-train and 50k train-val, ran 17 15-epoch screening runs on train-val across six method families (Table 20), and trained full 90-epoch models (seed 42) evaluated on the untouched 50k official-val split (Section 6.2, Table 2):

Method (ImageNet-1K / ResNet-50, 90 epochs)train-val Bestofficial-val @ BestFinal (Ep 90)Cumulative λ^\hat{\lambda}
Standard Coupled L2L_2 (λ=10−4\lambda = 10^{-4})78.70%75.77%75.77%1.0×10−41.0\times 10^{-4} (coupled)
Tuned Decoupled FixedWD (λ=1.5×10−3\lambda = 1.5\times 10^{-3})78.53%75.92%75.94%1.5×10−31.5\times 10^{-3}
Global Exposure-Matched FixedWD78.69%75.75%75.74%1.4×10−31.4\times 10^{-3}
Per-Tensor Static Oracle (2-pass replay)78.75%76.05%76.05%1.4×10−31.4\times 10^{-3}
Tuned CWD (Chen et al., 2026)78.11%75.52%75.48%1.5×10−31.5\times 10^{-3}
Tuned SWD (Xie et al., 2023)77.97%75.36%75.29%4.8×10−34.8\times 10^{-3}
Tuned CPR (Franke et al., 2024)74.41%71.60%71.69%5.6×10−55.6\times 10^{-5}
Tuned EqWD (λ0=10−3,β=0.5\lambda_0 = 10^{-3}, \beta = 0.5)78.86%76.01%75.99%1.4×10−31.4\times 10^{-3}

At the ~76% ResNet-50 accuracy level, tuned EqWD ranks first among all six online families on both train-val and untouched official-val (76.01%), outperforming tuned FixedWD (+0.09%), coupled L2L_2 (+0.25%), and scalar global exposure matching (75.75%, +0.26%). On CIFAR-100 (45k/5k/10k split, 5 seeds, Table 2), removing an archived λ0≤10−3\lambda_0 \le 10^{-3} search cap likewise brings tuned EqWD (67.51±0.35%, final 67.63±0.47%, λ^=0.0084\hat{\lambda} = 0.0084) ahead of tuned FixedWD (67.44±0.21%, λ^=0.0100\hat{\lambda} = 0.0100) and within 0.19% of global matching (67.70%), CWD (67.53%), and SWD (67.66%), while winning 6 of 7 base rates across a 200× λ0\lambda_0 sweep (Figure 2a). Fused torch._foreach kernels also cut ResNet-50 step overhead to +0.55% (Table 6).


Q1: What is the empirical benefit specifically attributable to tensor-wise standardized deviations?

First, in the ImageNet-1K table above, matching EqWD's global average exposure with a uniform scalar coefficient (75.75%) gives no improvement over coupled L2L_2 (75.77%), whereas EqWD discovers a 53-tensor decay allocation online in a single pass (76.01%) that matches the two-pass Per-Tensor Static Oracle (76.05%).

Second, on heterogeneous autoregressive Transformers — GPT-2 Small (124M, 48 decayed matrices, 734M tokens, 3 seeds) and zero-shot GPT-2 Medium (350M, 96 matrices, 524M tokens, 2 seeds) on FineWeb-Edu (Section 6.4, Table 4, Figure 3, Table 22b) — simultaneous raw ratios rtlr_t^l span 322.6× on 124M and 1039.4× on 350M, and EqWD assigns stronger decay to residual-write projections (attn.c_proj and mlp.c_proj: 0.0885–0.0899 on 350M) than input projections (attn.c_attn and mlp.c_fc: 0.0767 and 0.0726). On 124M, EqWD (43.80±0.08 test PPL) beats reference AdamW (43.87±0.09 on 3/3 seeds) and static per-tensor matching (43.92±0.22), lowering PPL SD by 3.3× vs. uncalibrated global EqWD (43.83±0.27) and by 2.8× vs. static per-tensor matching. Transferred zero-shot to GPT-2 350M, per-tensor EqWD self-preserves global exposure within 0.2% (0.0807→0.08060.0807 \to 0.0806) and reaches 60.94±0.16 test PPL, beating zero-shot tuned FixedWD (61.10±0.08 on 2/2 seeds) and reference AdamW (60.98±0.20) with 2.9× smaller PPL SD than two-pass per-tensor matching (60.88±0.46).

Q2: Given the nearly identical performance of global statistics and full EqWD in Table 2, can the authors explain which component is essential and provide strength-matched evidence?

Two factors explained the small gap between global and per-tensor statistics in archived Table 2 (now Table 3): ResNet-20 is composed almost entirely of scale-homogeneous 3×33\times 3 convolutions (unlike GPT-2's 323×–1039× cross-layer spread), and archived Table 2 held nominal β=5\beta = 5 fixed across variants that induced very different cumulative exposures.

To test each component at equal decay strength, Appendix Table 18 calibrates β∗\beta^{\ast} for all 8 signal variants on the held-out CIFAR-100 split across 5 seeds so cumulative exposure matches full EqWD within ±1.83% in both Regime A (low base decay, λ^≈2.47×10−3\hat{\lambda} \approx 2.47\times 10^{-3}) and Regime B (validation-selected decay, λ^≈8.37×10−3\hat{\lambda} \approx 8.37\times 10^{-3}). Even with total exposure equalized, unnormalized ∣et∣|e_t| trails full EqWD in both regimes (65.72±0.49% vs. 66.04±0.72% in Regime A; 67.29±0.72% vs. 67.51±0.35% in Regime B) and requires a 32× jump in calibrated gain across base scales (β∗:1825→56.3\beta^{\ast}: 1825 \to 56.3, whereas full EqWD stays order-unity at 5.0→0.255.0 \to 0.25). This confirms that same-step residual normalization DtlD_t^l is essential both for boundedness (Proposition 1) and for keeping β∗\beta^{\ast} stable across decay regimes. Between global and per-tensor statistics at matched exposure, full EqWD leads global EqWD by +0.36% in Regime A (66.04% vs. 65.68%), ties on homogeneous ResNet-20 in Regime B (67.51% vs. 67.52%), and eliminates global EqWD's 3.3× seed-variance inflation and post-hoc β\beta recalibration on heterogeneous GPT-2 (Table 4).


Q3: Can the authors provide additional runs or uncertainty estimates for the COCO detection experiments? Given the small improvements, particularly the 0.1 AP gain with Swin-T, it is difficult to determine whether the differences are meaningful.

We ran a new multi-seed COCO2017 four-scale DINO suite in MMDetection 3.3.0 under a unified bfloat16 protocol (effective batch size 16, 12 epochs, float32 deformable attention and focal loss, zero skipped optimizer steps; Section 6.4, Appendix E.4, Table 21). Across three paired seeds (42, 123, 456) on ResNet-50, EqWD reaches 48.30±0.12 AP versus 48.26±0.06 AP for FixedWD at 1.0165× cumulative exposure (paired difference +0.04±0.08+0.04 \pm 0.08 AP; per-seed deltas +0.12,+0.03,−0.03+0.12, +0.03, -0.03 AP), with consistent gains on AP50 (+0.16, 66.24±0.12 vs. 66.08±0.08), AP75 (+0.09), APs (+0.24, 30.50±0.45 vs. 30.26±0.52), and APm (+0.12). Due to the roughly 21 GPU-hour cost per 12-epoch Swin-T run, we completed one full paired 12-epoch Swin-T run (seed 42) under the same unified protocol, where EqWD improves overall AP from 50.44 to 50.62 (+0.18 AP, +0.32 AP75, +0.36 APl, +0.20 AR100, and −1.16 single-seed APs) at 1.0144× exposure. Taken together, the 3-seed ResNet-50 uncertainty bounds show that under tightly matched exposure (about 1.015×), EqWD preserves or slightly edges overall DINO AP while remaining within run-to-run detection variance on ResNet-50.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.