THE RENAISSANCE OF LINE SEARCH FOR DEEP LEARNING?

The learning rate (LR) is a critical hyperparameter in deep neural network (DNN) training. In practice, predefined LR schedules must be tuned through repeated trial runs, which incurs substantial human effort and hidden computational cost. Classical optimization determines the LR in a more principled way. One such way is line search, which adaptively selects the LR by checking whether a candidate step sufficiently decreases the objective. Line search is considered impractical for DNN training because it relies on accurate, and therefore expensive, evaluations of the gradient and the objective. However, line search need not be performed at every step. Established DNN schedules provide a useful prior on the LR trajectory: the LR follows a warmup–decay shape and varies slowly within each stage. Based on this insight, we propose a line-search-based LR scheduler that runs searches only periodically on accumulated batches and reuses each selected LR until the next search. During warmup, a search may only increase the LR. During decay, a search may only decrease it, by at most one step. The resulting scheduler has a single tuned hyperparameter, the Armijo constant, and adds only about $19\%$ wall-clock overhead on average. We evaluate it on LLM post-training with Qwen2.5-0.5B and 1.5B, covering SFT and DPO with both LoRA and full fine-tuning. With only four trials, it comes within 5% of the best of sixteen tuned (base LR, schedule) runs on every task, substantially reducing the need for manual LR-schedule tuning.