acceptodds
Under review as a conference paper at ICLR 2027

UTA: Universal Training Adaptive Learning Rate Decay

Abstract

Learning rate decay (LRD) is critical for training high-performance models, yet most designs remain empirically driven. Although the Universal Budget Aware (UBA) LRD offers a theoretical framework, its reliance on SGD dynamics limits its applicability to modern Adam-type optimizers and leaves its key `optimization difficulty parameter' opaque. To bridge this gap, we propose the Universal Training Adaptive (UTA) LRD. Unlike UBA's fixed offline trajectory anchored solely to iterations, UTA treats LRD as a closed-loop shape-and-scale problem. By continuous-time formulation, UTA is training-state adaptive with segment-wise optimal shape. This adaptivity links external iterations with internal training states, enabling a near-optimal path in both progression shape and learning rate (LR) scale, a property unattainable by other LRDs, including UBA. Empirically, we demonstrate consistent improvements across language models (LLaMA 60M to 1B) and vision models (ResNet50, ViT-B) on ImageNet with plug-and-play ease and negligible overhead. Extensive ablation studies further confirm UTA's robustness across dimensions, including peak LR, final decay ratio, its parameter , component efficacy and generalization to fine-tuning scenarios, underscoring its practical implementability.

Primary area:
optimization
Keywords:
learning rate decay, networks training, optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eWUROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors generalize the Universal Budget Aware (UBA) learning rate decay (LRD) framework to the Adam optimizer. They divide the training horizon into segments and analyze the continuous-time dynamics to derive the optimal intrinsic time Uk∗U_k^* in closed form. They then introduce Adam-specific approximations to obtain a computationally efficient LRD applicable to the entire training horizon. The proposed LRD is evaluated on LLaMA models ranging from 60M to 1B parameters, as well as ResNet-50 and ViT-B on ImageNet, demonstrating better performance than existing LR schedulers. The authors also conduct various ablation studies to demonstrate the robustness of the proposed method and its applicability to other scenarios, such as fine-tuning.

Critical strengths

The paper provides extensive experimental evaluations, including not only performance comparisons but also various ablation studies, such as sensitivity analyses on the peak learning rate and φ0\varphi_0, as well as fine-tuning experiments. The proposed method shows positive results across these experiments.

Critical weaknesses
  • The paper proposes UTA by extending UBA, which was originally developed for SGD dynamics, to adaptive optimizers such as Adam. However, the theoretical analysis focuses almost exclusively on Adam. This is closely related to my main concern. Although Theorem 3.3 and Corollaries 3.4 and 3.5 derive solutions for the intrinsic-time selection problem under certain assumptions, the authors subsequently approximate the key quantity νℓ(k)E[E(Tk)]/α\nu_\ell^{(k)}\mathbb{E}[E(T_k)]/\alpha using the inverse squared norm of Adam's preconditioned update direction, 1/∥m^t/(v^t+ϵ)∥21/\lVert\hat m_t/(\sqrt{\hat v_t}+\epsilon)\rVert^2, and then replace it with a surrogate of the form (1+c∥⋅∥)−1(1+c\lVert\cdot\rVert)^{-1}. Furthermore, in Section 3.4, the authors extend the segment-wise analysis to the entire training horizon without sufficiently justifying how the theoretical guarantees are preserved in the final global learning rate schedule. Most of these steps appear heuristic and specific to Adam. Therefore, I find the claimed theoretically grounded generalization of UBA insufficiently justified.
  • From an empirical perspective, the performance improvements of UTA do not appear particularly substantial. Moreover, its applicability to other commonly used optimizers, such as Muon, has not been demonstrated, which further limits the evidence for its claimed generality. UTA also appears highly sensitive to the peak learning rate. For example, in the LLaMA-130M results in Table 3, UTA performs worse than all other learning rate schedulers at every tested peak learning rate except 1×10−31\times10^{-3}. Regarding φ0\varphi_0, it also remains unclear whether the fixed value of 0.15 would be universally effective across different tasks, datasets, and model architectures.
Questions for authors
  1. In Appendix I.3 (line 2036), the authors mention that they evaluated φ0=0.1\varphi_0=0.1 for LLaMA-130M, but the corresponding result is missing from Table 8. Could the authors provide this result? Additionally, how robust is UTA to different choices of φ0\varphi_0 on other tasks, such as ImageNet and other datasets?
  2. How could UTA be extended to other optimizers, such as Muon? Would a linearization approximation similar to the one used in [1] be applicable?

Minor Requests

  • The Preliminary section could benefit from a more detailed explanation. In particular, the discussion of UBA introduces many notations and definitions rather abruptly, making it difficult to follow.
  • In Figures 1(a) and 1(c), the plotted values appear to represent loss, although the y-axis is labeled "Perplexity." The y-axis labels or the plotted values should be corrected accordingly.

Reference

[1] Wang et al., OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-Training in Every Iteration, ICML 2026 Oral.

Reviewer_E6dEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UTA, a learning rate decay that extends the UBA robust schedule family from SGD to Adam. The curvature parameter phi of the UBA family is interpreted through intrinsic time U = integral of eta dt and is calibrated online from a bias variance bound on the energy evolution. The final schedule is a global UBA shaped curve eta(t) = g(phi_t, pi t / T), where phi_t depends on the norm of the Adam preconditioned update and a single hyperparameter phi_0 = 0.15 is used for all tasks. The theory develops in three steps. Theorem 3.2 states that the optimal step size shape on a short segment belongs to the UBA family. Theorem 3.3 with Corollaries 3.4 and 3.5 derives the segment scale by minimizing an energy upper bound. Theorem F.1 matches the intrinsic time of the UBA shape to the energy optimal value and solves for phi_k in closed form. Experiments cover LLaMA 60M to 1B on C4 and ImageNet with ResNet50 and ViT B, with gains over cosine, linear decay, WSD and UBA, plus comparisons with D-Adaptation and Prodigy and ablations on the peak LR, phi_0, the final LR ratio, the dynamic mechanism, curvature alignment and LoRA fine tuning. I checked the main derivations step by step. Lemmas D.1 and D.3, Corollaries 3.4 and 3.5, the final bound of Theorem 3.3 and the closed form of Theorem F.1 are all correct as far as I can tell. My concerns are of three kinds: the theory does not cover the deployed algorithm, the algorithm is not specified precisely enough to reproduce from the text, and several statements in the text do not match the paper's own tables. Details follow.

Critical strengths
  1. Clean problem positioning and a practical final design. Recasting the static UBA curvature as a scale of intrinsic time and calibrating it online from the training state is a clear and appealing idea. The method is plug and play, inherits the torch lr_scheduler API, fixes phi_0 = 0.15 for all tasks and architectures, and costs roughly one norm computation per step. This has real value for a tuning free LRD.
  2. The experimental breadth and the ablation design are strong for an LRD paper. Four model scales with multiple budgets, two vision architectures, peak LR sweeps, and in particular the curvature alignment experiment in Table 11, where UBA is aligned to phi approximately phi_0 x 0.2 and UTA still wins. This experiment makes a genuine attempt to separate the effect of the dynamic fluctuations from a pure hyperparameter scaling, which is rare in this line of work.
Critical weaknesses
  1. The theory does not cover the deployed algorithm, and two connections in the derivation need clarification. Theorem 3.2 shows that an approximate problem has the same optimization form as UBA. The original problem in Eq. 10, however, contains the momentum tracking error epsilon_g and a time varying preconditioner P(t), and the paper does not show that its optimal shape remains in the UBA family. This is not a detail. The error factor beta_1 / (1 - beta_1) equals 9 at beta_1 = 0.9 and does not shrink at Delta T_k = 1, which is exactly the value the deployed method uses, so it is unclear whether the short segment approximation is in force at the operating point of the algorithm. There is also a gap between the segment construction in Appendix F and the final curve. With Delta T_k = 1 the per segment phase spans [(k-1) pi, k pi], and the boundary continuity condition stated in the appendix would make the learning rate alternate between near eta_max and near eta_min from step to step. The deployed curve in Eq. 98 is a single global phase and is not the object this construction produces. The paper should clarify which object Theorem 3.2 characterizes and how the global curve is obtained from the segment wise result. Finally, the plug-in estimate in Eq. 7 is derived through a chain of approximate steps, including a relation between T_k and the energy decrease that is dimensionally unclear, and the replacement of an inverse square dependence on the update norm by a reciprocal linear form, which does not have the same asymptotic behavior. As stated, the claim of near optimal shape and energy calibrated scale is not defined precisely enough to be evaluated.
  2. The algorithm cannot be reproduced from the text alone. The quantity ||m_hat / sqrt(v_hat + eps)|| in Eqs. 7 and 86 is not defined. It is not stated whether it is the full L2 norm over all parameters or an RMS norm. The two conventions differ by a factor of sqrt(N), and with the L2 norm on a 60M model the schedule would behave qualitatively differently from what Figure 5 shows. The paper should state the convention explicitly and report the range of phi_t during training. This is also needed to interpret the curvature alignment experiment in Table 11.
  3. Several statements in the text do not match the paper's own tables, and the statistical evidence is thin. The text claims that UTA has the lowest perplexity at the 30 percent checkpoint across all model sizes, but Table 1 shows CS at 22.03 below UTA at 22.14 for 350M at 7k. The text describes UTA as consistently better throughout training, but at 350M 14k UTA sits at 19.58 while CS reaches 18.24, and at 1B 16k UTA sits at 16.97 while CS reaches 16.32. The mid training gap is large and closes only near the end. The average reductions over D-Adaptation and Prodigy are stated as 1.2 and 0.9, while Table 10 itself gives 0.32 and 1.16. The Ours-Full numbers in Table 9, 26.78 on LLaMA 60M and 78.55 on ImageNet ResNet50, do not match the main tables, 25.80 and 78.73, and the static variant is not fully specified. In addition, the LLaMA results appear to be single runs, and the claims of statistical significance have no test behind them. With the smallest gain being 0.08 perplexity at 350M, variance information is needed to judge these numbers.
Questions for authors
  1. Eq. 86 gives phi_t = phi_0 ||d||^2, which is quadratic in the preconditioned update norm, while the curvature alignment experiment in Table 11 and Appendix I.3 uses phi approximately phi_0 x 0.2, which is linear in the norm. With the reported RMS value of about 0.2, the two readings give 0.006 and 0.03, a factor of 5 apart. What is the exact expression and norm convention behind phi_t, and what justifies the linear conversion used in the alignment?
  2. Which object does Theorem 3.2 characterize, given that the segment concatenation at Delta T_k = 1 appears to produce a step to step alternation of the learning rate under the stated continuity condition, while Eq. 98 uses a single global phase? Also, Appendix H.3 refers to a phase drift mechanism that keeps an ascending phase early on, but theta(t) = pi t / T starts at eta_max and decreases monotonically. Where is this mechanism defined?
  3. What are the exact settings behind Table 9, the final decay ratio and the fixed phi of the static variant, so that its numbers can be reconciled with the main tables? Can seed variance or significance tests be provided for the LLaMA results?
Reviewer_SDwbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes UTA, a state-adaptive learning-rate schedule for Adam-type optimizers. UTA takes the existing UBA schedule and replaces its fixed curvature parameter with a function of the Adam update norm, namely phi_t = phi_0 ||m_hat_t / sqrt(v_hat_t + epsilon)||^2. The method is motivated through segment-wise quadratic approximations, an intrinsic-time quantity defined as cumulative learning rate, and a bias-variance-style energy bound. Experiments report modest but generally consistent improvements on LLaMA models from 60M to 1B, ImageNet classification, and LoRA fine-tuning.

Critical strengths

The method is simple to integrate into Adam-based training and is evaluated across both language and vision tasks.

The paper includes peak-learning-rate sweeps, hyperparameter sensitivity studies, a static ablation, comparisons with automatic learning-rate methods, and experiments up to a 1B-parameter language model.

Critical weaknesses
  1. The theory does not derive the implemented UTA rule or support the claimed “near-optimality.” The energy analysis establishes only a generic trade-off between cumulative optimization and stochastic noise, while the transition from the unobservable optimal intrinsic time to 1/(1+c||d_t||) is heuristic. Appendix E drops the learning-rate factor from the actual Adam step Delta W_t = -eta_t d_t and replaces an inverse-quadratic dependence with a reciprocal-linear function while claiming preserved asymptotic behavior, although the two decay at different rates. The segment-wise reduction is also questionable: it treats the time-varying preconditioned Hessian P_t H as effectively fixed and assumes a common eigensystem, although these matrices need not commute or share eigenvectors. Setting the segment length to one avoids some approximation error but makes the claimed within-segment shape optimality vacuous. No bound connects the practical plug-in rule to the true optimal intrinsic time or to the globally optimal schedule.

  2. The algorithmic contribution is limited, and the experiments do not isolate which part of the construction drives the gains. UTA modulates the curvature parameter of the existing UBA family using the Adam update norm. The paper does include a UBA parameter-alignment comparison, but it neither tests whether the same feedback improves simpler cosine/WSD schedules nor compares against average-learning-rate-matched controls. Thus, the evidence does not establish that the UBA shape, intrinsic-time interpretation, or energy matching is necessary. This concern matters because some LLM gains are small (0.08 perplexity) and no multi-seed LLM results are reported. The implementation also leaves ambiguity about the norm definition and differs from the mathematical description in bias correction and norm computation. Finally, the negligible-overhead claim is not backed by wall-clock measurements.

  3. The evaluation is not broad enough to support the paper's stronger claims about generation quality and reliability. The paper reports held-out C4 perplexity, ImageNet accuracy, and LoRA fine-tuning results, but no language-model downstream, zero-shot, out-of-distribution, or direct generation-quality evaluation. It nevertheless claims that small perplexity reductions imply statistically significant gains in generation quality and reliability; no multi-seed LLM uncertainty or statistical test is provided for that claim. Separately, the theory analyzes local training energy and substitutes training loss for the unobservable energy, without establishing a link to validation performance or generalization. The proposed implicit-regularization explanation is also speculative without direct regularization or train-validation-gap analyses.

  4. The training-budget description is internally unclear, and the experiments do not establish robustness across budgets at a fixed model scale. The text says that each model is evaluated under four budgets (10K, 15K, 20K, and 25K), but the main table instead presents one apparent total horizon per scale (10K for 60M, 15K for 130M, 20K for 350M, and 25K for 1B), with intermediate 30% and 60% checkpoints. These checkpoints are not equivalent to independently configured shorter-budget runs because each schedule depends on its planned horizon T. The authors should clarify the actual run matrix; as reported, the results cannot establish budget robustness for any fixed model size.

  5. The static-versus-dynamic ablation does not cleanly establish the value of state adaptation, and its full-method results are inconsistent with the main table. The main table reports 25.80 perplexity for LLaMA-60M and 78.73 accuracy for ImageNet-ResNet50, whereas “Ours-Full” reports 26.78 and 78.55, respectively, despite stating that other settings are identical. The LLaMA ablation value is close to the separately reported zero-final-LR result (26.77), raising the possibility of a configuration mismatch; the paper does not explain this discrepancy. Further, the static variant is only described as removing “the dynamic part,” without enough detail to reproduce it. A matched and fully specified ablation is needed before attributing the gains to state adaptation.

Questions for authors
  1. Can the authors provide an end-to-end guarantee connecting the practical plug-in rule to the theoretical optimum, while including the omitted eta_t factor and handling the time-varying, noncommuting Adam preconditioner?

  2. How does UTA compare with cosine/WSD using the same update-norm feedback and with average-LR-matched controls? What exact norm is used, and what are the multi-seed LLM results and measured runtime overhead?

  3. Do the C4 perplexity gains translate to downstream or generation-quality improvements at matched compute, and what evidence connects the training-energy analysis to validation generalization or the claimed implicit regularization effect?

  4. What exact model-size-by-budget run matrix was used? Can the authors clarify why the text describes four budgets per model while the main table appears to assign one horizon to each size, and report independently configured budgets at fixed model scale?

  5. Why do the Ours-Full results in the dynamic ablation differ from the main UTA results despite the claim that other settings are identical? What precisely is the static variant, and can the authors provide a reproducible, matched comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.