- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes SOLAR, an online learning rate scheduler for LLM pre-training. SOLAR maintains a predefined base schedule and uses a controller trained with PPO to adjust the learning rate for each parameter group based on the training state. The adjustment magnitude is capped and applied to the current base learning rate at each step. The method also includes a reward based on training progress and stability, as well as a circuit breaker to recover from severe loss spikes. Experiments demonstrate a reduction in validation perplexity across multiple model sizes, optimizers, and dense and MoE models.
Critical strengths
- SOLAR learns an LR controller during pre-training, improving final perplexity with minimal runtime overhead. Comparison results demonstrate the benefits of using a base schedule, limiting the action range, and controlling LR per parameter group.
- Experiments cover multiple model sizes, AdamW and Muon, and dense and MoE models. The paper also includes several ablation analyses to validate design choices. It is useful that policies trained on small models can be reused on large models without further PPO updates.
Critical weaknesses
- The evaluation of model quality is limited to validation perplexity. It is unclear whether the improvements also lead to better performance on downstream tasks.
- The dense models are at most 1B parameters, and the additional 3B MoE experiment has only one run for each method. More evidence is needed to assess the effectiveness at larger scales.
- The benefit of the Circuit-Breaker is less clear than the other components. The stress test shows successful recovery, but the Circuit-Breaker is not triggered in the main experiments. It is also unclear whether the failure penalty and PPO update help beyond simply restarting from a safe checkpoint.
Questions for authors
- Do the improvements in validation perplexity also lead to better performance on downstream language tasks, even after post-training?
- In the Circuit-Breaker experiment, does the failure penalty and PPO update improve recovery compared with restarting from the same safe checkpoint without updating the policy?
Dear Reviewers,
The authors have finished uploading their responses to reviewers, and the discussion phase is now underway. If you haven't already, please carefully review the authors' responses to understand their perspectives. Please engage in thoughtful and constructive discussions with the authors, and update your review or rating if necessary.
The discussion period lasts until November 18th AoE, which means you have less than five days left to interact with the authors. Please start engaging well before the period ends, so that the authors will have sufficient time to respond to any follow-up questions/comments.
Your active participation and contribution to the ongoing discussion are highly encouraged. Thank you very much for your contribution to ICLR.
Best regards,
AC