acceptodds
Under review as a conference paper at ICLR 2027

SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining

Abstract

Learning-rate (LR) scheduling plays a central role in large language model (LLM) pretraining, yet current practice still relies heavily on hand-crafted heuristics such as Warmup-Cosine-Decay and Warmup-Stable-Decay. Because these schedules are fixed in advance, they cannot adapt to evolving optimization dynamics. Online learned scheduling within the Learning to Optimize (L2O) framework offers a dynamic alternative, but remains brittle at LLM scale due to noisy signals, delayed feedback, and the risk of catastrophic divergence. We propose SOLAR, a stabilized framework for reliable online LR adaptation. SOLAR uses a base schedule as a reference and learns bounded, state-dependent residual corrections for individual parameter groups. Each correction re-anchors to the base at every step, allowing the policy to adapt the LR without relearning the warmup-decay profile. A lightweight state representation and progress-aware reward guide online learning, while a Circuit-Breaker restores training after rare unsafe actions. Across autoregressive language-model pretraining, SOLAR improves final perplexity over tuned static schedules and automatic LR tuners for dense models from 60M to 1B, AdamW and Muon, and two MoE settings up to 3B. Matched 130M controls show that adding base anchoring and action bounds improves a global PPO controller from 27.09 to 23.74 final PPL, while group-wise control reaches 22.87 on the same two seeds. A residual policy trained on a 60M proxy can also be frozen and reused at larger dense scales without target PPO updates, remaining effective across a fourfold base-LR range. These results establish SOLAR as a practical learned LR controller for LLM pretraining.

Primary area:
optimization
Keywords:
large language model pretraining, deep learning optimization, adaptive training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5G63OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SOLAR, an online learning rate scheduler for LLM pre-training. SOLAR maintains a predefined base schedule and uses a controller trained with PPO to adjust the learning rate for each parameter group based on the training state. The adjustment magnitude is capped and applied to the current base learning rate at each step. The method also includes a reward based on training progress and stability, as well as a circuit breaker to recover from severe loss spikes. Experiments demonstrate a reduction in validation perplexity across multiple model sizes, optimizers, and dense and MoE models.

Critical strengths
  • SOLAR learns an LR controller during pre-training, improving final perplexity with minimal runtime overhead. Comparison results demonstrate the benefits of using a base schedule, limiting the action range, and controlling LR per parameter group.
  • Experiments cover multiple model sizes, AdamW and Muon, and dense and MoE models. The paper also includes several ablation analyses to validate design choices. It is useful that policies trained on small models can be reused on large models without further PPO updates.
Critical weaknesses
  • The evaluation of model quality is limited to validation perplexity. It is unclear whether the improvements also lead to better performance on downstream tasks.
  • The dense models are at most 1B parameters, and the additional 3B MoE experiment has only one run for each method. More evidence is needed to assess the effectiveness at larger scales.
  • The benefit of the Circuit-Breaker is less clear than the other components. The stress test shows successful recovery, but the Circuit-Breaker is not triggered in the main experiments. It is also unclear whether the failure penalty and PPO update help beyond simply restarting from a safe checkpoint.
Questions for authors
  • Do the improvements in validation perplexity also lead to better performance on downstream language tasks, even after post-training?
  • In the Circuit-Breaker experiment, does the failure penalty and PPO update improve recovery compared with restarting from the same safe checkpoint without updating the policy?
Reviewer_TYriOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SOLAR is an RL-based online learning-rate scheduler for LLM pretraining. A hand-crafted base schedule (Cosine or WSD) gives the warmup and decay profile. A small PPO-trained MLP policy outputs a bounded multiplicative correction per parameter tensor, ηt,g=ηt,gbaseexp⁡(αtat,g)\eta_{t,g} = \eta^{\text{base}}_{t,g}\exp(\alpha_t a_{t,g}) with at,g∈(−1,1)a_{t,g}\in(-1,1). The correction is applied to the current base LR at every step, so it cannot compound. The state is a few scalar features: loss statistics, per-tensor gradient and parameter norms, depth, and the previous action. The reward combines one-step loss decrease, an EMA trend term and a per-tensor gradient-norm stability term. A "Circuit-Breaker" rolls back to a checkpoint on a severe loss spike. Two modes are evaluated: SOLAR-online trains the policy inside the target run, SOLAR-frozen trains it on a few small-model runs and reuses it without updates.

Experiments follow the GaLore benchmark (Llama-style 60M to 1B on C4, sequence length 256) with AdamW and Muon, against Cosine, WSD, CLR, Blockwise LR, AutoLRS, MECHANIC, Prodigy and Schedule-Free. There is also a Qwen2-MoE 1B run on The Pile and a single 3B MoE run. The appendix has many controls (anchored vs. unanchored global PPO, group-wise vs. global, a GANNO adaptation, hypergradient, untrained / deterministic / state-blind policies, replay controls, reward leave-one-out, base-LR sweeps, μ\muP transfer, corpus shift) and a runtime accounting.

Critical strengths

S1. The control experiments actually isolate the design choices. The paper does not just show that SOLAR beats Cosine. It shows that an unanchored global RL controller fails, that anchoring and bounding fixes it, and that group-wise actions add more on top. Untrained, near-deterministic, state-blind and open-loop replay variants all fall back toward the static baselines. The reward leave-one-out, the base-LR sweep and the μ\muP transfer directly address the question "is the baseline LR just wrong". This makes the claims testable, and it is the main reason I think the paper can become publishable.

S2. The method is cheap at run time. No extra forward or backward passes, about one percent wall-clock overhead, a one-time small search on a 60M proxy, and the frozen mode removes all target-scale RL work. The RL constants, PPO settings and baseline search grids are reported in detail, which is rare for RL-in-the-loop methods.

Critical weaknesses

W1. The 1B AdamW baseline looks under-tuned, and the headline claim rests on it. The setup is explicitly the GaLore benchmark. GaLore reports a full-rank Adam result at 1B on the same architecture, data, sequence length and token budget (13.1B tokens) of 15.56 PPL, about one point better than this paper's tuned Cosine baseline of 16.52. At 60M to 350M the paper's baselines are at or better than GaLore's, but there the paper trains on more tokens than GaLore (1.4B/2.6B/7.8B against 1.1B/2.2B/6.4B). The gap appears exactly at the one scale where the budgets match. If a properly tuned 1B baseline lands where the literature puts it, most of the headline gain goes away. The relative gain grows with scale in this paper, which fits a baseline getting weaker with scale better than a method getting stronger. The 1B results also use only two seeds.

W2. The mechanistic claim is not closed, and simple competitors are missing. The paper's story is that the gain comes from closed-loop, state-conditioned control and not from the LR trajectory itself. The control that would settle this is to replay the exact recorded per-tensor LR sequence on the same seed and base LR it was recorded from. This is not in Table 17. The time-varying replay is only reported cross-seed and under shifted base LRs (Table 18), so it is unclear whether the difference comes from the closed loop or from the seed change. The paper's own analysis says SOLAR mostly does two things: it runs a higher average LR than Cosine, and it gives embeddings and the LM head larger multipliers than internal projections. These are tested one at a time (a level-corrected scalar replay, and a static group profile normalized to mean one) but never together, and never against the obvious hand-written version, Cosine with a higher peak and a separate larger LR for embeddings and head, which is common practice. Also, the local state is log⁡∥∇t,g∥\log\|\nabla_{t,g}\| and log⁡∥wt,g∥\log\|w_{t,g}\| and the output is a per-tensor LR multiplier. This is close to what LARS/LAMB compute with a closed-form rule, and neither is cited or compared. Without these controls it is unclear how much of the gain actually needs learned closed-loop control. Right now the paper does not show much beyond a per-tensor schedule that beats the baselines it tried.

W3. Secondary points. The dense benchmark is dated for a pretraining paper: sequence length 256, T5 tokenizer, and a sub-Chinchilla token budget at 1B. Only the single 3B MoE run uses a longer context. Selection is on seed 42 and reporting on seed 52, 1B has two seeds, and the 3B MoE has one run per method. The Circuit-Breaker is part of the method but never fires in any main run. Apart from one rollback in the reward ablation without the stability term, its only validation is a stress test with a larger action bound. Evaluation is perplexity only. No code release is mentioned. Novelty is modest: RL-based LR control, layer-wise RL agents, learned global multipliers and spike rollback all exist and are cited, the contribution is the combination and the LLM-scale evidence. The paper's own novelty claim is narrow and seems honest.

Questions for authors

These are clarification requests, answers could change my score.

On W1. (1) How do you reconcile your 1B AdamW Cosine baseline with GaLore's full-rank Adam result on the same setup, which is about one PPL better? Was the LR/WD grid re-run at 1B, and what did the sweep look like? (2) Can you add a third seed at 1B? (3) Were the baseline schedulers' method-specific hyperparameters re-tuned at each scale and under Muon, or inherited from the 130M/AdamW search? The Prodigy result at 60M looks broken rather than tuned.

On W2. (4) Please report the time-varying group replay, the exact recorded ηt,g\eta_{t,g} for every step and tensor, on seed 52 at the selected base LR, i.e. on the same run it was recorded from. Does it reproduce SOLAR-online? If not, what differs (sampled vs. mean action, nondeterminism)? (5) Two simple controls: (a) the static per-tensor profile without normalizing its mean to one, (b) Cosine with a higher peak LR and a separate larger LR for embeddings and LM head. How much of the gap do they close? A LARS/LAMB-style per-tensor trust ratio would be the natural non-learned baseline given your state features. (6) Why does the frozen 60M policy beat the online one at the larger dense scales? If in-run PPO updates do not help at scale, what is the online mode for? (7) How do the two warmups interact? During the base warmup the action scale is also warming up, so the policy is nearly muted. After that, with σ\sigma initialized at 1 and a non-trivial entropy bonus, the per-step multiplier is very noisy (Fig. 3d, 4b). Is the policy effectively learning a noise distribution per tensor rather than a schedule? The near-deterministic and state-agnostic controls hint at this but do not resolve it.

On W3. (8) The 1B run uses roughly 13B tokens, below the Chinchilla-optimal budget, at context 256. Does the gain hold at a compute-optimal or over-trained budget, or at a longer context? Even one 124M FineWeb-style run with 1024 context would help. (9) Will the code be released? With this many fixed constants and a rollback pipeline, the appendix alone is not enough to reproduce the method. (10) Missing settings: Qwen2-MoE 1B peak LR / WD / warmup / β\beta's, 3B MoE WD and tokenizer, the group partition for both MoE runs, seeds of the five source runs for the frozen policy and whether state-normalization statistics are transferred, PPO minibatch count. For Muon, whether the LM head goes to Muon or to its internal AdamW, what LR the AdamW-handled tensors use, and whether SOLAR modulates both the same way.

Suggestions, not affecting the score: a downstream evaluation, a nanoGPT / FineWeb-style run.

Reviewer_Xg4dOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SOLAR, an online learning rate scheduler for LLM pretraining using PPO. It learns group-wise learning rate adjustments on top of an existing schedule based on loss and gradient statistics. The paper provides experiments on LLMs up to 1B and MoE models up to 3B, showing improvements over existing LR schedules. It also shows that the learned policy can be transferred from a 60M model to larger models.

Critical strengths
  1. The idea proposed by the paper is easy to understand and the paper is presented clearly.

  2. The experiments show promising results across different model sizes and optimizers. The ablations are also reasonably detailed.

  3. The transfer of the learned policy from a 60M model to larger models is interesting. The additional training overhead is also low.

Critical weaknesses
  1. I am not convinced that the baselines are sufficiently tuned. For example, the 1B AdamW + Cosine baseline has a PPL of 16.52, whereas GaLore reports 15.56 for full-rank AdamW at the same token budget. This makes the improvement of the proposed method less significant. It is unclear what causes this difference.

  2. It is not clear how much of the improvement comes from learning the LR online. Figure 4 shows that the embedding and LM head receive much larger learning rates. I think a simpler method with different learning rates for these modules could also give similar improvements. This comparison is missing.

  3. The individual components of the method are not very novel. RL-based learning rate scheduling and group-wise learning rates have been studied before. The experiments are also limited to relatively small LLMs.

Questions for authors
  1. Why is the 1B AdamW + Cosine baseline worse than the result reported in GaLore? Are there differences in the experimental setup?

  2. Can the authors compare against a Cosine schedule with separately tuned learning rates for the embedding and LM head?

  3. Would the improvements remain similar if the Cosine baseline were tuned more carefully, including decay to zero?

Comments on the paper

1 comment

Dear Reviewers,

The authors have finished uploading their responses to reviewers, and the discussion phase is now underway. If you haven't already, please carefully review the authors' responses to understand their perspectives. Please engage in thoughtful and constructive discussions with the authors, and update your review or rating if necessary.

The discussion period lasts until November 18th AoE, which means you have less than five days left to interact with the authors. Please start engaging well before the period ends, so that the authors will have sufficient time to respond to any follow-up questions/comments.

Your active participation and contribution to the ongoing discussion are highly encouraged. Thank you very much for your contribution to ICLR.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.