acceptodds
Under review as a conference paper at ICLR 2027

Hyperball Optimization for LLM Pretraining

27 views

Abstract

Matrix-based optimizers such as Muon can substantially speed up language model pretraining, but their gains over AdamW are observed to shrink as model size and data scale grow when using standard constant decoupled weight decay. We propose **Hyperball**, a simple optimizer wrapper that addresses this issue. The method is motivated by prior theory showing that weight decay drives the weight norm toward an equilibrium, eventually making the weights move only on a hypersphere of fixed radius. With Hyperball, we explicitly control this behavior: given a base optimizer such as Adam or Muon, Hyperball sets the Frobenius norms of weight matrices and their corresponding optimizer updates to fixed constants. On Qwen3 style models up to B parameters, Muon Hyperball achieves – token equivalent speedup over weight decay baselines. Hyperball also improves learning rate transfer across widths and depths compared to decoupled weight decay.

Primary area:
optimization
Keywords:
optimization, language model, pretraining

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NLQVOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Hyperball, an optimizer wrapper that normalizes the base optimizer update and projects each constrained weight matrix onto a Frobenius sphere whose radius is determined at initialization. This provides explicit control over the relative update magnitude while eliminating weight-norm drift. The wrapper is evaluated with Adam, Muon, and additional optimizers across language-model pretraining benchmarks, training horizons, and model widths and depths. The authors report approximately 20–30% token-equivalent speedup over AdamW and reduced drift in the optimal learning rate across tested scales. A stationary stochastic-gradient analysis characterizes update autocorrelation, equilibrium weight norms, and angular-step estimates to motivate the design.

Critical strengths
  1. The method provides a simple and practical way to control optimization scale. Equation (1) and Algorithm 1 introduce only update normalization and radial weight rescaling, while reusing the base optimizer’s update direction. The distinction between constrained attention/MLP matrices and unconstrained embeddings and normalization gains is clearly specified in §2. The empirical evaluation provides encouraging evidence across multiple training conditions. The experiments cover several base optimizers, model scales, training budgets, and public optimization benchmarks. Particularly useful is the 130M experiment, which separately sweeps learning rates at each token budget from 1B to 128B and reports consistently lower best validation loss for MuonH. Together with the width/depth sweeps, these results support the method’s practical promise beyond an isolated training configuration.

  2. The analysis offers a useful account of the interaction between weight decay, momentum, and directional motion. Incorporating temporal update autocorrelation through (C_\alpha), and treating SVD Muon through its matrix-sign correlation kernel, makes the equilibrium analysis more informative than a qualitative argument based solely on weight norms (§4.2; Appendix B). The explicit distinction between effective learning rate and actual angular displacement in §4.2 and Appendix A.1 is also valuable. These results provide an interpretable framework for understanding why identical nominal learning-rate schedules can produce different optimization dynamics.

Critical weaknesses
  1. The experiments do not yet sufficiently isolate the contribution of Hyperball from changes in the effective learning-rate schedule. Section 3.2 attributes the improvement to the effective schedule and acknowledges that “matching effective learning rates can largely align the loss trajectories” (p. 3, lines 155–158). However, Figure 10 only compares schedules after normalizing each by its peak; the paper does not provide a corresponding loss comparison under matched absolute effective schedules in its principal training setup. Such a control is particularly relevant given the trajectory-alignment experiments reported by Xiao et al., Hyperball May Not Be a Free Lunch, §4.2. Component ablations would further clarify the roles of weight projection and update normalization, especially because idealized SVD Muon already has a fixed update Frobenius norm for a given matrix shape (Lemma 4.3). A representative comparison with explicit rotational-update control would also help establish the incremental benefit over closely related approaches such as Rotational Equilibrium (Kosson et al., 2024). Hyperball could remain valuable as a convenient schedule-control mechanism, but the current evidence does not clearly distinguish improved optimization quality from reduced tuning effort.
  2. Incomplete reporting of tuning and uncertainty makes the strength and fairness of some empirical conclusions difficult to assess. Although §3.1 identifies the architecture family and data sources, the manuscript does not systematically specify the principal experiments’ complete configurations, data-mixture proportions, optimizer settings, search ranges, and tuning budgets. The 8B comparison uses “manually chosen hyperparameters” and compares MuonH against AdamW (§3.2), so it provides exploratory evidence rather than an isolated estimate of the wrapper’s contribution. Repeated-run statistics are also important for the public benchmark differences of 0.0008 between MuonH and MuonW and 0.0011 between NorMuonH and NorMuonW (§3.2). The token-equivalent speedup and fitted long-horizon asymptotes would benefit from uncertainty estimates. Finally, the approximately 1.4× optimal-learning-rate drift is close to one interval of the (\sqrt{2})-spaced search grid, and the transfer experiments use a HybridNorm variant (§3.1, §3.3). Reporting the loss penalty from directly transferring a learning rate selected on a smaller model would more clearly establish practical transferability. A consolidated configuration table, representative repeated runs, and this direct transfer evaluation would substantially strengthen the empirical claims.
Questions for authors
  1. Can the authors provide a matched-effective-schedule control and targeted component ablations?
  2. I would be more confident in acceptance if there would be proper responses to the questions raised in the weaknesses.
3 comments

We thank the reviewer for the review.

1. Does Hyperball provide a benefit beyond controlling the effective learning rate schedule?

Our claim is that Hyperball makes the effective learning rate schedule easier to choose and control directly. We do not claim any additional optimization advantage once the absolute effective learning rate schedules are matched; in that case, we expect very similar performance. Consistent with this expectation, Xiao et al. (2026, §4.2, Figure 3), as cited in our paper, show that adjusting the learning rate schedule substantially aligns the validation loss trajectories of MuonH and Muon with weight decay in their experiments.

2. Roles of weight projection and update normalization.

Update normalization in hyperball serves a conceptual purpose instead of an empirical purpose: it gives the effective learning rate a common parameterization across matrices and base optimizers. Weight projection fixes each matrix norm at RiR_i, while update normalization makes the relative magnitude of the step before projection equal to the shared Hyperball learning rate ηt\eta_t. For a nonzero base optimizer update ui,tu_{i,t} of matrix ii, retaining the same projection makes update normalization exactly equivalent to using the matrix-specific learning rate η~i,t=ηtRi∥ui,t∥F.\widetilde\eta_{i,t}=\eta_t\frac{R_i}{\|u_{i,t}\|_F}. For idealized SVD Muon in Lemma 4.3, the update norm is constant for a fixed matrix shape, so this reduces to a fixed rescaling of the learning-rate schedule for each matrix.

3. What is the incremental benefit over explicit rotational-update control?

Hyperball offers a conceptually simpler way to control the effective learning rate: the update norm relative to the weight norm. The rotational wrapper of Kosson et al. (2024) removes the radial component of each constrained weight update and maintains an exponential moving average (EMA) of the projected update’s squared norm to control average rotation. Hyperball normalizes the current matrix update and rescales the updated matrix to its initialization Frobenius norm. It requires neither the radial projection nor an additional EMA for rotation control. The base optimizer retains its usual state, and Hyperball stores the fixed target norms.

4. Are the configurations and tuning budgets reported sufficiently to assess the comparisons?

We thank the reviewer for the suggestion. We add an appendix E listing implementation details, hyperparameter search ranges, and the rule used to select the reported run.

5. The 8B result provides exploratory evidence rather than an isolated estimate of the wrapper’s contribution

We agree with the reviewer and would clarify in the paper.

6. How uncertain are the small public benchmark differences?

These comparisons already include repeated runs: 10 each for MuonH and NorMuonH (the Hyperball variants), and 20 each for MuonW and NorMuonW (the weight decay baselines). Using these runs, we computed two-sided 95% Welch confidence intervals for the difference in mean validation loss, treating the runs as unpaired and allowing different variances. We define improvement as the baseline mean loss minus the Hyperball mean loss, so positive values favor Hyperball. We note that between NorMuonH and NormuonW, we observe a statiscally significant difference.

ComparisonTraining stepsMean improvement95% confidence interval
MuonH vs MuonW33250.000837[−0.000166, 0.001840]
NorMuonH vs NorMuonW32500.000898[0.000227, 0.001569]

7. Uncertainty in token-equivalent speedups and fitted asymptotes.

We computed approximate 95% profile-F intervals for the fitted asymptotes. For the long-horizon experiment, we fit L(N)=L∞+AN−αL(N)=L_\infty+AN^{-\alpha} to the best tested C4 validation loss at each of the 13 token budgets from 1B to 128B.

OptimizerFitted limiting loss L∞L_\inftyApproximate 95% fit interval
MuonW3.0787[3.0693, 3.0875]
MuonH3.0654[3.0593, 3.0713]

For the token-equivalent speedup, we compute the speedup mostly using interpolation or mild extrapolation between different level of AdamW's loss at different token count.

8. What loss penalty results from directly transferring a learning rate?

The reported 1.4×1.4\times drift is approximately one step on our 2\sqrt{2}-spaced learning-rate grid. We therefore also quantify the validation-loss cost of transferring the learning rate.

We analyzed 60 MuonH runs using four-layer Qwen3 models with QK normalization enabled without HybridNorm. At each width (128, 256, 512, and 1024), we tested 15 learning rates, training each model for approximately 1B tokens with a cosine schedule. We selected the rate with the lowest final C4 validation loss at width 128 and transferred it unchanged to the larger widths.

Target widthC4 loss at transferred rateBest C4 loss in target gridLoss increase
2564.278764.278760.00000
5124.001823.997950.00387
10243.814963.812720.00224

The loss penalty is below 0.004 relative to each target’s best tested rate, supporting useful width transfer even without HybridNorm.

The author has addressed my questions. I will keep my rating unchanged and raise my confidence for this paper.

Reviewer_5BZMOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Hyperball, an optimizer wrapper that normalizes each matrix update and projects the updated weights onto a Frobenius sphere whose radius is fixed at initialization.
The authors report improved token efficiency and learning-rate transfer across model widths and depths, including ~20–30% token-equivalent speedup relative to an AdamW scaling-law baseline at 1.2B parameters.
A stationary stochastic model motivates the method by relating weight decay to equilibrium weight norms and angular updates.

Critical strengths
  1. Novelty. I strongly believe this paper should be accepted for its algorithmic novelty. While prior and recent work has extensively discussed effective learning rates, this paper puts these ideas into practice by using fixed parameter norms to replace weight decay. However, constraining norm has already appeared in nGPT, SSO, SCION, and RotationalOptimizer.

  2. Empirical breadth. The evaluation covers multiple optimizers, model sizes, training budgets, and width/depth sweeps, supporting the method’s applicability across different training settings.

Critical weaknesses

Although I support acceptance, I am very skeptical that the paper reaches the spotlight level.

  1. Fixing norm may hurt performance. Beyond using the initialization norm, the paper provides neither a heuristic nor a methodology for choosing an appropriate fixed norm. Even within a single run, choosing the wrong norm can hurt performance. Based on my own observations, fixing parameter norms can also harm optimization when scaling up. This definitely limits the practical applicability, and this limitation deserves more discussion and experimental investigation.

  2. Minor theoretical concern. The main theorem overstates what is established: stationary second-moment results and approximate angular-step calculations are presented as stronger convergence claims.

Questions for authors

Already addressed above.

3 comments

We thank the reviewer for the review.

1. How should the fixed weight norm be chosen, and can fixing it hurt performance?

Our default is to fix each constrained matrix at its initialization Frobenius norm, and we initialize every entry in a weight matrix W0∈Rdout,dinW_0 \in \mathbb{R}^{d_{out}, d_{in}} from a zero-mean normal distribution with standard deviation 1/din1/\sqrt{d_{in}}. The experiments provide evidence that this choice can work well. We agree with the reviewer that choosing the norm can potentially influences the performance and we add the following experiments to the paper to show that Θ(1/din)\Theta(1/\sqrt{d_{in}}) is reasonable initialization norm to choose.

To assess sensitivity to the norm choice, we additionally trained a model with 154M parameters with QK Norm and hybrid norm for approximately 2.6B tokens across four norm multipliers and three learning rates. For each constrained matrix, we multiplied its initialized weights by 0.5, 1, 2, or 4 and fixed its norm at that rescaled value throughout training. A multiplier of 1 is our default. We observe a consistent optimal learning rate and a relatively stable performance within this range.

Norm multiplierLearning rate 0.01Learning rate 0.02Learning rate 0.04
0.53.806893.786123.80968
13.803563.783413.80854
23.805303.783773.80825
43.807533.787703.81643

2. Implication on theory on convergence

We use an infinite-history formulation to simplify the analysia and avoid explicitly tracking the transient evolution toward the stationary weight scale and angular-step estimate. While our arguments can be easily extended to a finite weight initialization under the same stationary optimizer update model, with the influence of the initial weights decaying geometrically, we will simplify the theorem statement to make clear that it characterizes the stationary regime, rather than the transition to it.

Then author addressed my partial concern: I'm particularly concerned about fixing norm during scaling, as well as degenerated performance compared to effective learning rate schedule. As a result, I will keep my rating unchanged.

We thank the reviewer for the prompt reply.

Regarding the concern of scaling up, public results support Hyperball’s applicability at larger scales. Delphi reports lower loss for its AdamH recipe than its previous recipe with AdamC across matched compute budgets, and successfully trains a 25B dense model on 600B tokens.

Marin 535B-A23B further reports completing 18T-token pretraining using MuonH for its main weight matrices and AdamH for its output head. These results demonstrate that fixed weight norms can work at substantial scale.

Reviewer_T23nOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Hyperball wraps a base optimizer to directly constrain relative step sizes by normalizing updates and projecting weight matrices back to their initial Frobenius norms, bypassing the need for weight decay. The authors demonstrate 20–30% token-equivalent speedups over AdamW baselines at 1.2B parameters, robust width/depth learning-rate transfer, and exploratory gains at 8B scale. Additionally, a stationary noise model is introduced to explain optimizer equilibrium norms and angular step behaviors.

Critical strengths
  1. The motivation connects to an actual training choice. I like the move from a weight-decay coefficient to explicit control of the weight norm and relative update size. The wrapper is simple, and the discussion explains why the nominal learning-rate schedule can differ from the effective one.

  2. The evidence covers several ways the method might be useful. The 1.2B duration sweep, the 130M token-budget sweep, and the depth/width transfer experiments support the idea that the benefit is not restricted to one setup. The transfer result is particularly relevant: the reported best-learning-rate drift is around one grid step for Hyperball, versus several steps for the weight-decay baselines.

Critical weaknesses

Disentangle schedule tuning from intrinsic gains: The headline speedup appears heavily conflated with a better effective learning-rate schedule, especially since Section 3.2 notes that trajectories align when effective rates match. While Figure 10 compares normalized schedules, it does not test performance under equalized step sizes. How much of the 1.2B gain persists if MuonW is tuned to match Hyperball’s measured angular step sizes? Please clarify whether Hyperball offers an intrinsic algorithmic advantage, or simply a convenient parameterization for a better schedule.

Questions for authors

Does the same learning-rate transfer hold in the standard Qwen3-style architecture, without the hybrid normalization used for the transfer sweeps? A smaller representative check would be enough to clarify the scope.

1 comment

We thank the reviewer for the review.

1. Schedule tuning versus intrinsic gains

Our claim is that Hyperball makes the effective learning rate schedule easier to choose and control directly. We do not claim any additional optimization advantage once the absolute effective learning rate schedules are matched; in that case, we expect very similar performance. Consistent with this expectation, Xiao et al. (2026, §4.2, Figure 3), as cited in our paper, show that adjusting the learning rate schedule substantially aligns the validation loss trajectories of MuonH and Muon with weight decay in their experiments.

2. Does transfer of learning rates hold in Qwen3 without HybridNorm?

Useful width transfer also holds without HybridNorm in this Qwen3 setting. We evaluated this using the MuonH optimizer in Qwen3 models without hybrid normalization (HybridNorm) disabled. These models have four layers and hidden widths 128, 256, 512, and 1024, each trained for approximately 1B tokens. We tested 15 learning rates per width with a cosine schedule, totaling 60 runs.

We selected the learning rate with the lowest final C4 validation loss at width 128 and transferred it unchanged to larger widths. The table compares the resulting loss with the best loss among the 15 learning rates tested at each target width.

Target widthC4 loss at transferred LRBest C4 loss in target gridLoss increase
2564.278764.278760.00000
5124.001823.997950.00387
10243.814963.812720.00224

Across these target widths, transferring the learning rate increases validation loss by less than 0.004 relative to tuning it separately at each width.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.