- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose an interesting way to automatically adjust the learning rate during the training of a neural network, it helps to reduce the need to conduct multiple experiments to find the optimal LR and scheduler. The idea is to replace part of the manual selection of the learning rate with brief searches performed during the training process itself.
I found it very useful that the method combines Armijo conditions during warmup with held-out loss comparisons during decay. It allows to reduce the cost of conventional learning rate and scheduler tuning in an interesting way.
The authors experimented using SFT and DPO, LoRA, and full fine-tuning. The models used are the Qwen2.5 with 0.5B and 1.5B parameters which are not too big. The paper shows experiments with performance within 5% of the best-tuned baselines when they use four trials and approximately 19% additional runtime per trial.
Critical strengths
In my point of view, the problem investigated is practical and important because the computational cost associated with learning rate and scheduler selection is critical and essential for the success of the training model.
When approaching the problem, the authors used an experimental protocol carefully designed around tuning cost. I like that the comparison performed accounts for trial duration, divergence, and the number of trials required to achieve competitive performance is investigated.
The experimental protocol and evaluation are clear, with individual runs, hyperparameter values, and divergent configurations are provided, allowing most of the main results to be independently checked and making the findings clear to the reader. The evaluation process includes different fine-tuning objectives and parameterization strategies. I also like the assessment across post-training configurations.
One important aspect is that the algorithm is relatively simple and introduces a controlled number of learning rate adjustments, which in a practical way makes the computational overhead predictable for the user.
Critical weaknesses
Besides the interesting approach, I have some concerns. As discussed before, the authors argue that their approach reduces the problem of choosing a combination of LR and scheduler to the tuning of a single hyperparameter that is the Armijo constant.
However, if I understand correctly, the method still relies on several fixed decisions, such as the number of search points, the duration of the warmup phase, and the LR reduction factor.
In my view, its most concrete contribution is the combination of infrequent searches, automatic LR adjustment, and a warmup–decay structure imposed on the training process. The authors should argue more in this direction.
Another point to note is that comparisons in experiments are not fully controlled. For instance, during decay, the proposed method uses held-out data to adjust the learning rate, and using it while the principal baselines do not receive an equivalent validation signal what does not seem fair to all methods. Considering that the method only outperforms the best-tuned baseline on Math-Step, what in my understanding is the task exhibiting substantial overfitting in the process.
Considering the related literature, the novelty of the method as presented is incremental because related methods already explore periodic learning rate adjustment, online warmup adaptation, and validation-guided scheduling. Considering the related literature such as AutoWU, AutoLRS, and two-way backtracking that inclusive are not adequately discussed in the paper and need to be considered by the authors as comparison basis.
I think the comparison with Prodigy is incomplete because it is reported on only nine of sixteen configurations, why that? Another point to investigate is that a single Prodigy run matches or outperforms the best of the four proposed trials in six of those nine cases.
Questions for authors
I have some questions that may help the authors to explore more the paper:
-
Does the approach remain competitive on larger models and longer training schedules, and when compared with a single well-selected value of c1?
-
Please comment on the claim that only one hyperparameter requires tuning since several design choices are fixed without sufficient explanation in the paper. For instance, an additional value of c1 = 0.9 appears outside the declared experimental sampling protocol. Why that?
Why is Armijo applied only during warmup? the decay phase is based on greedy held-out comparisons rather than an Armijo line search.
Seems to me that the experimental evidence relies on a single random seed. In this kind of experiment it is important to have variability associated with training, hyperparameter sampling, and optimization and those are not quantified. Please comment on some results that suggest sensitivity larger than the 5% tolerance used to establish competitiveness.
Please verify a possible inconsistency between the main text and Appendix F regarding the number of tasks solved at a tuning budget of beta = 3.5.