- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes EqWD, which increases decoupled weight decay according to the standardized deviation of each tensor's gradient-to-weight norm ratio from its moving mean. It establishes bounds on instantaneous and accumulated modulation, characterizes recovery to base decay, and proves local convergence for a scalar quadratic. Experiments evaluate image classification and object detection against fixed-decay baselines.
Critical strengths
The algorithm is simple and easy to follow, requiring only two moving-average statistics per eligible parameter tensor.
Critical weaknesses
-
Limited methodological and theoretical contribution. The method combines a gradient-to-weight norm ratio, moving-average statistics, and a correction to weight decay. I find the resulting innovation limited. Section 3 mentions related approaches such as Scheduled Weight Decay, Cautious Weight Decay, and Constrained Parameter Regularization, but provides little comparison explaining the advantage of this design. Although the contributions are explicitly listed, the presentation does not sufficiently establish the significance of the mathematical results for optimization. The pointwise coefficient bound follows directly from including the current residual in the denominator's scale estimate. The recovery and cumulative bounds further characterize the controller, but do not establish better optimization or generalization. The only optimization-convergence result, Proposition 3, assumes deterministic gradient descent on a one-dimensional quadratic, with local initialization and additional learning-rate and gain restrictions. This is too narrow to provide theoretical support for the practical settings studied.
-
Insufficient evidence of practical benefit. The gains are mixed and often disappear against stronger fixed-decay controls. On CIFAR-100/ResNet-20, searched fixed decay achieves 69.180% best-epoch accuracy versus 67.627% for EqWD. ImageNet ViT-Tiny is essentially tied (+0.021 points). Comparisons against other adaptive-decay methods are also absent. These results do not convincingly isolate a benefit from adaptation beyond stronger regularization. The evaluation is restricted to vision models. If practical utility is the main contribution, I would expect convincing evidence at a larger scale, for example language models with several hundred million parameters, against well-tuned baselines. Otherwise, more substantive theory is needed to justify the contribution. The measured step-time overhead on ResNet-50 further raises the question of whether the small residual gains justify the added cost.
Questions for authors
- What evidence distinguishes the benefit of adaptive timing and tensor-specific decay from simply increasing fixed decay, given the searched and exposure-matched results?
- What technical insight distinguishes EqWD from the adaptive-decay methods discussed in Section 3, beyond its particular choice of normalization?
3 comments
W2 & Q1: What evidence distinguishes the benefit of adaptive timing and tensor-specific decay from simply increasing fixed decay, given the searched and exposure-matched results — including comparisons against adaptive baselines and on language models with several hundred million parameters?
Following your recommendation, we pretrained GPT-2 Small (124M, 48 decayed matrices, 734M tokens, 3 seeds) and GPT-2 Medium (350M, 96 matrices, 524M tokens, 2 seeds) on disjoint FineWeb-Edu train/val/test shards (Section 6.4, Tables 4, 22, Figure 3). Hyperparameters were selected on GPT-2 124M via a 35-run 600-step screen on val.bin (9 FixedWD, 12 EqWD, 6 CWD, 4 CPR, and 4 SWD trials, where EqWD ranked first overall at val_loss = 4.6053) and transferred zero-shot to GPT-2 350M without retuning:
| Method (FineWeb-Edu GPT-2) | 124M Test PPL (3 seeds) | Zero-Shot 350M PPL (2 seeds) | Global (124M / 350M) |
|---|---|---|---|
| Reference AdamW () | 43.87±0.09 | 60.98±0.20 | 0.1000 / 0.1000 |
| Tuned FixedWD () | 43.76±0.05 | 61.10±0.08 | 0.0100 / 0.0100 |
| Per-Tensor Matched AdamW (2-pass) | 43.92±0.22 | 60.88±0.46 | 0.0807 / 0.0806 |
| Global Exposure-Matched AdamW | 43.82±0.13 | N/A | 0.0807 / N/A |
| Nominal Global EqWD | 43.83±0.27 | N/A | 0.0835 / N/A |
| Strength-Matched Global EqWD | 43.74±0.12 | 60.82±0.18 | 0.0822 / 0.0786 |
| Tuned SWD () | 43.81±0.11 | N/A | N/A |
| Tuned CWD () | 43.74±0.07 | 60.60±0.24 | N/A |
| Tuned CPR (warm-start 140 / 100) | 43.17±0.15 | 59.35±0.36 | N/A |
| Tuned EqWD () | 43.80±0.08 | 60.94±0.16 | 0.0807 / 0.0806 |
Three findings across our LLM and vision suites directly address Q1:
-
Zero-shot LLM scale transfer and variance stability (Table 4, Figure 3). On GPT-2, EqWD assigns stronger modulation to residual-write projections (
c_proj: 0.0885–0.0899 on 350M) than input projections (c_attn/c_fc: 0.0767/0.0726). On 124M, EqWD beats reference AdamW on 3/3 seeds (−0.34 PPL at matched , Table 22a) and lowers cross-seed PPL SD by 3.3× vs. uncalibrated global EqWD (0.266) and by 2.8× vs. two-pass per-tensor matching (0.222). When transferred zero-shot to GPT-2 350M, 124M-tuned FixedWD () degrades to 61.10±0.08 PPL, whereas both EqWD variants (60.94±0.16 per-tensor and 60.82±0.18 strength-matched global) outperform tuned FixedWD and reference AdamW (60.98±0.20), with per-tensor EqWD self-preserving global exposure () automatically in one pass. While CWD (60.60) and norm-constrained CPR (59.35) reach lower AdamW LM perplexity, they trail EqWD by 0.49–4.41 percentage points on 90-epoch ImageNet SGD (75.52% and 71.60% vs. 76.01%), making EqWD consistently reliable across both optimizer families. -
Held-out ImageNet-1K and CIFAR-100 controls (Table 2, Figure 2a). On 90-epoch ImageNet-1K with properly scaled baselines, single-pass online EqWD (76.01%) beats scalar global exposure matching (75.75%, +0.26%), tuned decoupled FixedWD (75.92%), and coupled (75.77%), matching the two-pass 53-tensor oracle (76.05%) in one pass. On CIFAR-100, auditing
selection.jsonshowed archived EqWD had been capped at (10× below FixedWD's ); removing that cap brings 5-seed EqWD (67.51±0.35%) slightly ahead of tuned FixedWD (67.44±0.21%) while winning 6 of 7 base rates across a 200× sweep. -
Strength-matched gain calibration (Table 18). Calibrating within ±1.83% exposure across 5 seeds shows unnormalized trailing full EqWD in both regimes (65.72% vs. 66.04%; 67.29% vs. 67.51%) with a 32× shift in (), confirming 's role beyond scalar exposure.
W2 (Overhead): The measured step-time overhead on ResNet-50 further raises the question of whether the small residual gains justify the added cost.
Our initial overhead came from per-tensor Python .item() GPU–CPU syncs in the prototype. Vectorizing EqWD with fused torch._foreach norms (Section 4, Tables 6, 23, Figure 2b) cuts median step-time overhead to +0.55% on ResNet-50, +0.63% on ViT-Tiny, +0.54% on GPT-2 124M, +1.23% on GPT-2 350M, and +3.57% on ResNet-20, with zero GPU–CPU synchronizations and no increase in reported peak VRAM.
Thank you for the detailed response and additional experiments. However, the new results do not show a clear benefit: on GPT-2, EqWD trails CWD and CPR and is within noise of tuned fixed decay, and the vision gains remain small. The added theory also does not establish improved optimization or generalization. I will keep my score.
We thank Reviewer rJoa for recognizing EqWD's simplicity and two-scalar per-tensor state, and for clearly specifying the evidence needed to substantiate the paper's contribution: mechanistic and empirical comparisons with SWD, CWD, and CPR; optimization theory beyond 1D gradient descent; evaluation on 100M+ language models against tuned baselines; and negligible runtime overhead. All four are included in the revised manuscript. Part 1 addresses W1 and Q2; Part 2 addresses W2 and Q1.
Appendix A.8 (Table 7) formalizes the structural differences among the four adaptive approaches:
SWD (Xie et al., 2023) scales weight decay globally by the inverse square root of the Adam/SGDS second moment (∝1/vˉt). Because it does not reference parameter norm ∥wtl∥ and has no intrinsic upper bound, sharp drops in gradient norm late in training can inflate effective decay substantially (reaching λ^=4.8×10−3 on ImageNet-1K — 3.5× higher than EqWD's 1.4×10−3 — and trailing EqWD at 75.36% vs. 76.01%). CWD (Chen et al., 2026) multiplies decay coordinate-wise by the sign-alignment mask I(ut,iwt,i≥0); this alters update directions (breaking radial isotropy) without tracking layer norm equilibrium, reaching 75.52% on ImageNet SGD. CPR (Franke et al., 2024) replaces fixed decay with per-tensor norm-bound constraints ∥wl∥≤κl updated via augmented-Lagrangian dual multipliers; this performs strongly when explicit norm caps suit AdamW language modeling (59.35 PPL on GPT-2 350M) or CIFAR-100 (68.09%), but is sensitive to the warm-start heuristic for κl under step-drop SGD on ImageNet-1K (71.60%).
By contrast, EqWD separates persistent architectural scale from transient non-stationarity by measuring departures from each tensor's own moving ratio μt−1l. Across GPT-2's 48–96 linear layers, raw gradient-to-weight ratios rtl span 322.6×–1039.4× simultaneously (Figure 3); subtracting μt−1l factors out this persistent cross-layer spread while preserving pure radial isotropy, guaranteeing a tuning-free instantaneous ceiling qtl≤(1−αv)−1/2 via same-step residual inclusion (Proposition 1), and automatically returning to λ0 once ratios stabilize (Proposition 2) so asymptotic regularization strength is unperturbed.
To connect this controller to optimization dynamics beyond 1D gradient descent, Section 5 (Appendices A.9–A.10) adds two closed-loop theorems. Theorem 2 extends Proposition 3 to p-dimensional SPD quadratics f(w)=21(w−c)⊤H(w−c) (H≻0,c=0) under preconditioned heavy-ball momentum (P≻0,m∈[0,1)), proving that the (2p+2)-state system preserves the exact fixed point w∗=(H+(1−m)λ0P)−1Hc of fixed decoupled decay and converges locally exponentially under 2aheff<1−a. Theorem 3 (Figure 1) analyzes orthogonal loss gradients ut=gt (G0>0) in normalized layers where xt=∥wt∥2 follows xt=(1−γλt)2xt−1+γ2Gt2, proving conditional single-step overshoot damping on spikes G1>G0 (Δdamp<2Δspike), strict contraction ∥wtEqWD∥<∥wtfixed∥ for all t≥1 after a scale drop G0→G1∈(0,G0), and a mandatory finite-step residual sign flip (e1<0→et∗>0 when w∗(G1)>1−aaε) before local exponential recovery.