acceptodds
Withdrawn from ICLR 2027

Hyperball may not be a free lunch

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Norm-constrained optimizers perform well in large language model pretraining, but the source of their gains and their relationship with learning-rate scheduling remain unclear. We study their mechanisms and learning-rate schedules from a scale-invariance perspective. First, we derive the relationship between the scale and angular effective learning rates in preconditioned optimizers. Numerical analyses show that radial updates have a limited direct effect on one-step angular displacement. By adjusting the nominal learning rate to match the angular effective learning rate, we can largely reproduce the phase-dependent loss trajectories of Hyperball and non-Hyperball optimizers in both directions. Scale effective learning-rate alignment also yields training outcomes close to those of angular alignment. Theoretically, we establish a local dynamic regret bound under strict scale invariance, local geodesic convexity, and bounded directional gradients. Combining this analysis with a quotient-space optimization perspective, we propose Effective Learning Rate Stable–Decay (ELR-SD), a heuristic schedule that holds the scale effective learning rate constant before linearly decaying it once the gradient norm before preconditioning stabilizes. Experiments on dense, mixture-of-experts (MoE), and long-horizon pretraining tasks show that ELR-SD improves validation loss. These results further suggest that the effectiveness of long-horizon linear decay in Hyperball is linked to its control of angular step sizes, motivating the use of step-size design principles from convex optimization in large language model pretraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Deep learning optimization, Hyperball optimization, Muon, Effective learning rate, Learning rate scheduling, Training dynamics, Language model pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ki5dOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies heavy-ball's effective angular learning rate and suggests that there is no free lunch. It also proposes a two phase leanring rate schedule, where the first phase holds lr constant and second phase linearly decays leanring rate. The switching trigger is based on stability of gradient norm.

Critical strengths
  1. The insights about the angular effective learning rate is quite interesting.
Critical weaknesses
  1. I believe there is a gap in motivation of ELR-SD. it's either not stated clearly or it's not been established why we need to keep angular effective lr the consistent with outside scheduling. Especially,

"Thus, the scale effective learning rate can guide learning-rate scheduling for loss reduction nearly as effectively as the angular effective learning rate. Together, these results suggest that Hyperball alone may not be a free lunch: improving performance requires a more effective learning-rate schedule."

Frankly, I do not see how these experiments lead to the said conclusion. It also has not established why angular changes of weights are the more important aspect in loss reduction.

  1. The empirical evidence feels inconclusive, for example figure 5, why was it not run longer, if the claims sustained it should be a clear win?

  2. The gradient instability trigger thresholds seem arbitrary and there is neither ablation nor explanation in the paper nor in the appendix. Why is it defined based on gradient norm, but not the angular changes?

  3. The theorem 7.1 provides no justification for any of the optimizers: The example in Appendix C constructs multiple projected Riemannian gradient learners, each with a fixed step size, and combines them through an exponential weighting rule. It does not establish this guarantee for MuonH, AdamH, or ELR-SD.

Questions for authors

I am a bit confused by what it means by angular effective learning rate at j-th stage in equation 12? what is j and where was it defined?

Reviewer_28dBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper analyzes Hyperball optimization and its effect on training trajectories. In particular, Hyperball variants train slow in earlier phase but fast in later phases. It decouples the effect of Hyperball into radial updates and an implicit learning rate schedule. They find that the latter has a major effect on the slow-then-fast training dynamics. Then, they propose a learning rate schedule called ELR-SD.

Critical strengths
  1. They start from first principles and analyze how the alignment between parameters and updates affect the effective learning rate.
  2. They show that transferring the implicit learning rate from Hyperball to WD largely recovers the Hyperball dynamics and the reverse is true as well.
Critical weaknesses

Major weaknesses

  1. The analysis is not novel and quite straightforward. What would you describe as the main novelty and contribution of the paper?
  2. ELR-SD is not very well motivated and explained. The last paragraph of Section 5 introduces it very abruptly. Can you elaborate more? What does ELR-SD stand for?
  3. Experiment (Section 6) is quite weak. Table 1 changes both optimizer and learning rate schedules, so it's impossible to decouple the effect of the learning rate schedule alone. I suggest the author changes what are shown in Table 1. Figure 5 controls the optimizer, but the benefit of ELR-SD is quite small. Do you have more controlled experiments to compare ELR-SD and linear decay?
  4. The experiment results don't support the title. It is somewhat expected Hyperball's main effect on training dynamics is the implicit learning rate schedule for scale invariant loss function. The result in Figure 3 supports this. Radial changes won't affect the output for scale invariant functions, so what else do we expect from Hyperball?

Others

  1. Presentation requires work. In particular, the figures are very hard to understand. The captions are short and it's hard to understand what is plotted even after reading the main text. Figure 1 is especially underspecified. What does subscript jj in Ij(t)I_j(t) mean what what specific quantities are plotted in Figure 1?
  2. Section 5 should specify which effective learning rate is used as there are two definitions.
Questions for authors

I listed the questions in the Weaknesses section.

Reviewer_SwJfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies why Hyperball-wrapped optimizers can outperform their standard counterparts. The authors start from a scale-invariant loss, derive the angular effective learning rate (ELR), and show that matching ELR can largely reproduce the loss trajectories of Hyperball and non-Hyperball optimizers. They further propose ELR-SD, a stable-decay scheduler based on ELR which is demonstrated to outperform baselines with common LR scheduler.

Critical strengths
  1. The derivation of angular effective learning rate is simple, clean and sensible.
  2. The bidirectional ELR-matching experiments are interesting and provide useful evidence for the proposed explanation of Hyperball's performance.
Critical weaknesses
  1. The proposed mechanism does not seem very new. In particular, the explanation through angular learning-rate control was already brought up in the original Hyperball paper.
  2. The comparative experiments are not sufficiently controlled to be fully convincing. Some of the algorithmic and experiment details seemed missing.
Questions for authors
  1. For ELR-SD, why is it mainly applied to Hyperball optimizers instead of standard optimizers? According to Figure 2(c/d), MuonH's ELR seems to follow its nominal LR up to a constant throughout training, which makes sense because Hyperball keeps the weight norm fixed. In contrast, MuonWD's ELR decays even under a nominal SD schedule. So shouldn't ELR scheduling be more useful for standard optimizers instead? I would suggest that the authors provide a clean algorithmic box for ELR-SD to clarify exactly how it is applied to different optimizers.

  2. Following the last question, is the main benefit of ELR-SD on Hyperball optimizers more about the adaptive decay-point decision, with the switching point given by Algorithm 1? Could the authors provide more intuition for this algorithm? I currently don't get a very clear sense of what it is detecting here — is it something like edge of stability? That I can buy as a motivation, but I suppose there might be simpler alternatives.

  3. The comparison in Table 1 is very confusing to me. The three optimizers have different weight-decay configurations and schedulers. Also, I cannot tell whether WSD performs worse because of the warmup or because of the actual difference in scheduling. The authors should clarify these comparisons with more details somewhere. Same confusion for Table 2: why compare a stable-decay scheduler with linear decay instead of a nominal-LR stable-decay scheduler? I think the authors should at least plot the effective LR and nominal LR, or otherwise the comparison is difficult to interpret. More controlled studies would help identify the true factors driving the improvement — is it the adaptive switch, the schedule shape, or differences in LR tuning? The authors may already have tuned these carefully, but it would be helpful to clarify how the peak LRs and other schedule parameters were selected.

  4. A more controlled comparison, in my understanding, would be something like this: suppose we run AdamH with WSD, and Adam/AdamW with an ELR-WSD schedule that tries to match the scale ELR trajectory of AdamH, up to some layerwise constants. In that case, do we see any marginal improvement from ELR-WSD? More generally, can effective-LR scheduling recover the benefits of Hyperball without explicitly constraining the weight norms?

  5. Suppose that, with some appropriate manipulation of ELR, we find a scheduler for standard normalized optimizers that matches or even beats Hyperball optimizers. Do we expect the stable LR-transfer property of Hyperball to generalize to this approach as well?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.