- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors generalize the Universal Budget Aware (UBA) learning rate decay (LRD) framework to the Adam optimizer. They divide the training horizon into segments and analyze the continuous-time dynamics to derive the optimal intrinsic time in closed form. They then introduce Adam-specific approximations to obtain a computationally efficient LRD applicable to the entire training horizon. The proposed LRD is evaluated on LLaMA models ranging from 60M to 1B parameters, as well as ResNet-50 and ViT-B on ImageNet, demonstrating better performance than existing LR schedulers. The authors also conduct various ablation studies to demonstrate the robustness of the proposed method and its applicability to other scenarios, such as fine-tuning.
Critical strengths
The paper provides extensive experimental evaluations, including not only performance comparisons but also various ablation studies, such as sensitivity analyses on the peak learning rate and , as well as fine-tuning experiments. The proposed method shows positive results across these experiments.
Critical weaknesses
- The paper proposes UTA by extending UBA, which was originally developed for SGD dynamics, to adaptive optimizers such as Adam. However, the theoretical analysis focuses almost exclusively on Adam. This is closely related to my main concern. Although Theorem 3.3 and Corollaries 3.4 and 3.5 derive solutions for the intrinsic-time selection problem under certain assumptions, the authors subsequently approximate the key quantity using the inverse squared norm of Adam's preconditioned update direction, , and then replace it with a surrogate of the form . Furthermore, in Section 3.4, the authors extend the segment-wise analysis to the entire training horizon without sufficiently justifying how the theoretical guarantees are preserved in the final global learning rate schedule. Most of these steps appear heuristic and specific to Adam. Therefore, I find the claimed theoretically grounded generalization of UBA insufficiently justified.
- From an empirical perspective, the performance improvements of UTA do not appear particularly substantial. Moreover, its applicability to other commonly used optimizers, such as Muon, has not been demonstrated, which further limits the evidence for its claimed generality. UTA also appears highly sensitive to the peak learning rate. For example, in the LLaMA-130M results in Table 3, UTA performs worse than all other learning rate schedulers at every tested peak learning rate except . Regarding , it also remains unclear whether the fixed value of 0.15 would be universally effective across different tasks, datasets, and model architectures.
Questions for authors
- In Appendix I.3 (line 2036), the authors mention that they evaluated for LLaMA-130M, but the corresponding result is missing from Table 8. Could the authors provide this result? Additionally, how robust is UTA to different choices of on other tasks, such as ImageNet and other datasets?
- How could UTA be extended to other optimizers, such as Muon? Would a linearization approximation similar to the one used in [1] be applicable?
Minor Requests
- The Preliminary section could benefit from a more detailed explanation. In particular, the discussion of UBA introduces many notations and definitions rather abruptly, making it difficult to follow.
- In Figures 1(a) and 1(c), the plotted values appear to represent loss, although the y-axis is labeled "Perplexity." The y-axis labels or the plotted values should be corrected accordingly.
Reference
[1] Wang et al., OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-Training in Every Iteration, ICML 2026 Oral.