acceptodds
Under review as a conference paper at ICLR 2027

THE RENAISSANCE OF LINE SEARCH FOR DEEP LEARNING?

Abstract

The learning rate (LR) is a critical hyperparameter in deep neural network (DNN) training. In practice, predefined LR schedules must be tuned through repeated trial runs, which incurs substantial human effort and hidden computational cost. Classical optimization determines the LR in a more principled way. One such way is line search, which adaptively selects the LR by checking whether a candidate step sufficiently decreases the objective. Line search is considered impractical for DNN training because it relies on accurate, and therefore expensive, evaluations of the gradient and the objective. However, line search need not be performed at every step. Established DNN schedules provide a useful prior on the LR trajectory: the LR follows a warmup–decay shape and varies slowly within each stage. Based on this insight, we propose a line-search-based LR scheduler that runs searches only periodically on accumulated batches and reuses each selected LR until the next search. During warmup, a search may only increase the LR. During decay, a search may only decrease it, by at most one step. The resulting scheduler has a single tuned hyperparameter, the Armijo constant, and adds only about wall-clock overhead on average. We evaluate it on LLM post-training with Qwen2.5-0.5B and 1.5B, covering SFT and DPO with both LoRA and full fine-tuning. With only four trials, it comes within 5% of the best of sixteen tuned (base LR, schedule) runs on every task, substantially reducing the need for manual LR-schedule tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
learning rate schedule, line search, optimization for deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_q9YZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose an interesting way to automatically adjust the learning rate during the training of a neural network, it helps to reduce the need to conduct multiple experiments to find the optimal LR and scheduler. The idea is to replace part of the manual selection of the learning rate with brief searches performed during the training process itself.

I found it very useful that the method combines Armijo conditions during warmup with held-out loss comparisons during decay. It allows to reduce the cost of conventional learning rate and scheduler tuning in an interesting way.

The authors experimented using SFT and DPO, LoRA, and full fine-tuning. The models used are the Qwen2.5 with 0.5B and 1.5B parameters which are not too big. The paper shows experiments with performance within 5% of the best-tuned baselines when they use four trials and approximately 19% additional runtime per trial.

Critical strengths

In my point of view, the problem investigated is practical and important because the computational cost associated with learning rate and scheduler selection is critical and essential for the success of the training model.

When approaching the problem, the authors used an experimental protocol carefully designed around tuning cost. I like that the comparison performed accounts for trial duration, divergence, and the number of trials required to achieve competitive performance is investigated.

The experimental protocol and evaluation are clear, with individual runs, hyperparameter values, and divergent configurations are provided, allowing most of the main results to be independently checked and making the findings clear to the reader. The evaluation process includes different fine-tuning objectives and parameterization strategies. I also like the assessment across post-training configurations.

One important aspect is that the algorithm is relatively simple and introduces a controlled number of learning rate adjustments, which in a practical way makes the computational overhead predictable for the user.

Critical weaknesses

Besides the interesting approach, I have some concerns. As discussed before, the authors argue that their approach reduces the problem of choosing a combination of LR and scheduler to the tuning of a single hyperparameter that is the Armijo constant.

However, if I understand correctly, the method still relies on several fixed decisions, such as the number of search points, the duration of the warmup phase, and the LR reduction factor.

In my view, its most concrete contribution is the combination of infrequent searches, automatic LR adjustment, and a warmup–decay structure imposed on the training process. The authors should argue more in this direction.

Another point to note is that comparisons in experiments are not fully controlled. For instance, during decay, the proposed method uses held-out data to adjust the learning rate, and using it while the principal baselines do not receive an equivalent validation signal what does not seem fair to all methods. Considering that the method only outperforms the best-tuned baseline on Math-Step, what in my understanding is the task exhibiting substantial overfitting in the process.

Considering the related literature, the novelty of the method as presented is incremental because related methods already explore periodic learning rate adjustment, online warmup adaptation, and validation-guided scheduling. Considering the related literature such as AutoWU, AutoLRS, and two-way backtracking that inclusive are not adequately discussed in the paper and need to be considered by the authors as comparison basis.

I think the comparison with Prodigy is incomplete because it is reported on only nine of sixteen configurations, why that? Another point to investigate is that a single Prodigy run matches or outperforms the best of the four proposed trials in six of those nine cases.

Questions for authors

I have some questions that may help the authors to explore more the paper:

  • Does the approach remain competitive on larger models and longer training schedules, and when compared with a single well-selected value of c1?

  • Please comment on the claim that only one hyperparameter requires tuning since several design choices are fixed without sufficient explanation in the paper. For instance, an additional value of c1 = 0.9 appears outside the declared experimental sampling protocol. Why that?

Why is Armijo applied only during warmup? the decay phase is based on greedy held-out comparisons rather than an Armijo line search.

Seems to me that the experimental evidence relies on a single random seed. In this kind of experiment it is important to have variability associated with training, hyperparameter sampling, and optimization and those are not quantified. Please comment on some results that suggest sensitivity larger than the 5% tolerance used to establish competitiveness.

Please verify a possible inconsistency between the main text and Appendix F regarding the number of tasks solved at a tuning budget of beta = 3.5.

Reviewer_hJ5tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a line-search-based learning rate scheduler that combines a warmup-decay prior with periodic line searches. The method selects a learning rate using accumulated batches and reuses it until the next search. During warmup, the learning rate can only increase or remain unchanged. During decay, it can only decrease by at most one multiplicative step per search. Experiments on LLM post-training, including SFT and DPO with LoRA and full fine-tuning, show evaluation losses close to those of carefully tuned baselines under a limited tuning budget.

Critical strengths

The paper presents a simple and intuitive approach that integrates line search into the widely used warmup-decay scheduling framework. Periodic searches provide a practical way to limit the additional computation associated with line search. Evaluating performance against cumulative wall-clock time also addresses the practical cost of learning rate tuning.

Critical weaknesses
  1. Limited novelty relative to closely related prior work.

The proposed method is closely related to Zhang et al. (2020), Statistical Adaptive Stochastic Gradient Methods [1], which introduced SALSA. That method combines smoothed stochastic line search for warmup with subsequent learning rate reductions triggered by statistical stationarity tests.

The two methods share the goal of combining an initial line-search phase with adaptive learning rate decay to reduce manual tuning. However, their adaptation rules differ. SALSA (2020) performs smoothed line search at each warmup step and automatically determines when to switch to statistical scheduling. The proposed method instead fixes the warmup duration, performs searches at predefined intervals, and uses a single comparison between the current and reduced learning rates during decay. Despite these similarities, SALSA (2020) is neither discussed nor experimentally compared in the manuscript. The SALSA baseline attributed to Kenneweg et al. (2024) [2] is a different algorithm. This omission makes it difficult to assess the methodological novelty and practical advantages of the proposed approach.

  1. Training instability and the dependence of tuning efficiency on early termination.

The proposed method has a substantial divergence rate: 9 of 32 trials for the 0.5B model and 6 of 32 for the 1.5B model diverge, whereas SALSA (2024) has no divergent trials in the reported experiments. The paper evaluates cumulative wall-clock time and argues that divergent trials are inexpensive because they are terminated early. Nevertheless, the practical advantage depends partly on detecting and stopping unsuccessful runs promptly. Given the average 19% overhead per training run, a breakdown of the computation spent on successful trials, failed trials, and line-search evaluations would make the cost-benefit trade-off clearer. It would also help establish how sensitive the reported advantage is to the early-termination rules.

  1. Limited evidence of downstream benefits over inexpensive learning rate heuristics.

The main experiments compare against a broad random search over learning rates and scheduler types. Appendix F partially addresses practical learning rate priors by restricting the baseline search to a narrower range around reference values. However, this remains a search-based comparison rather than a direct evaluation of a fixed, inexpensive recipe, such as a peak learning rate of 10−410^{-4} with a standard scheduler for LoRA SFT. Moreover, the primary metric is evaluation loss. Appendix G examines DPO preference accuracy, but it remains unclear whether the method improves downstream task performance enough to justify its additional computation. This distinction matters for practitioners who already have a reasonable fine-tuning recipe and may have little need for an extensive learning rate search.

Questions for authors
  1. Comparison with prior work: Could the authors clarify the methodological differences from SALSA by Zhang et al. (2020) and compare the two methods under matched training conditions and tuning budgets?

  2. Total tuning cost: How does the proposed method compare with conventional schedulers and SALSA (2024) in terms of total GPU hours required to achieve a target performance, explicitly including failed or divergent runs and the additional line-search overhead?

  3. Practical baselines and downstream performance: Beyond the narrower search in Appendix F, how does the method compare with fixed learning rate recipes, such as 10−410^{-4} for LoRA SFT, on downstream task performance? Do the resulting benefits justify the additional computation when a reasonable baseline recipe is already available?

Reviewer_ePNJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a learning-rate scheduler that combines periodic line searches with a prescribed warmup–decay structure. Searches operate on accumulated batches, and the selected learning rate is reused between search anchors. The Armijo constant is the main tuned parameter, while other settings are fixed. The evaluation covers SFT and DPO using Qwen2.5 models, with LoRA and full fine-tuning. Comparisons include randomly tuned conventional schedulers and SALSA, with additional Prodigy results in the appendix. The results show that a small number of trials can approach the best observed evaluation loss at lower cumulative tuning cost, despite approximately 19% average per-run overhead.

Critical strengths
  1. Addressing the tuning cost of the learning rate is a practically meaningful problem for deep learning.
  2. Anchoring searches at prescribed points and constraining their outputs to follow warmup–decay behavior offers a simple way to combine local loss information with a global trajectory prior. This is a useful practical idea because unrestricted local improvement need not produce a good learning-rate trajectory over an entire training run. The component ablations provide some supporting evidence for these restrictions.
Critical weaknesses
  1. The evaluation is limited. All main experiments use two relatively small models from the same family, and the component ablations cover one task and one seed. The main outcome is evaluation loss. The authors acknowledge that this does not necessarily imply improved downstream quality, particularly for DPO. Appendix G provides useful preference-accuracy evidence, but does not fully establish broader model-quality benefits. The results therefore support an effective scheduler for the tested post-training settings more strongly than a general conclusion about deep learning.

  2. The improvement over the closest baseline, SALSA, seems not sufficiently compelling. Both methods achieve 38 of 64 trials within 5% of the target and incur nearly identical average runtime overhead (1.19× versus 1.20×), while the proposed method diverges in 15 trials and SALSA in none. The budget profiles show advantages in some regimes, but these depend on fixed hyperparameter draws and early termination of unsuccessful runs. With only one training seed per configuration, the evaluation does not establish a sufficiently consistent benefit to justify the additional scheduling rules. This leaves the practical contribution promising but insufficiently demonstrated.

Questions for authors
  1. Can you identify the settings in which the proposed method consistently improves over SALSA, accounting jointly for achieved loss, total tuning time, and divergence?
  2. How stable is that conclusion across independent training seeds or independently sampled Armijo constants?
  3. How sensitive is the budget-based advantage over SALSA to the early-termination thresholds? Does the relative advantage persist when divergent trials are detected less quickly?
  4. Appendix D isolates the effects of the scheduling rules on UltraFeedback with the 0.5B model and DPO LoRA. Have you performed the same component ablations in any of the SFT or full fine-tuning settings evaluated elsewhere in the paper? This would help determine whether these rules' benefits generalize across training settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.