- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Hyperball, an optimizer wrapper that normalizes the base optimizer update and projects each constrained weight matrix onto a Frobenius sphere whose radius is determined at initialization. This provides explicit control over the relative update magnitude while eliminating weight-norm drift. The wrapper is evaluated with Adam, Muon, and additional optimizers across language-model pretraining benchmarks, training horizons, and model widths and depths. The authors report approximately 20–30% token-equivalent speedup over AdamW and reduced drift in the optimal learning rate across tested scales. A stationary stochastic-gradient analysis characterizes update autocorrelation, equilibrium weight norms, and angular-step estimates to motivate the design.
Critical strengths
-
The method provides a simple and practical way to control optimization scale. Equation (1) and Algorithm 1 introduce only update normalization and radial weight rescaling, while reusing the base optimizer’s update direction. The distinction between constrained attention/MLP matrices and unconstrained embeddings and normalization gains is clearly specified in §2. The empirical evaluation provides encouraging evidence across multiple training conditions. The experiments cover several base optimizers, model scales, training budgets, and public optimization benchmarks. Particularly useful is the 130M experiment, which separately sweeps learning rates at each token budget from 1B to 128B and reports consistently lower best validation loss for MuonH. Together with the width/depth sweeps, these results support the method’s practical promise beyond an isolated training configuration.
-
The analysis offers a useful account of the interaction between weight decay, momentum, and directional motion. Incorporating temporal update autocorrelation through (C_\alpha), and treating SVD Muon through its matrix-sign correlation kernel, makes the equilibrium analysis more informative than a qualitative argument based solely on weight norms (§4.2; Appendix B). The explicit distinction between effective learning rate and actual angular displacement in §4.2 and Appendix A.1 is also valuable. These results provide an interpretable framework for understanding why identical nominal learning-rate schedules can produce different optimization dynamics.
Critical weaknesses
- The experiments do not yet sufficiently isolate the contribution of Hyperball from changes in the effective learning-rate schedule. Section 3.2 attributes the improvement to the effective schedule and acknowledges that “matching effective learning rates can largely align the loss trajectories” (p. 3, lines 155–158). However, Figure 10 only compares schedules after normalizing each by its peak; the paper does not provide a corresponding loss comparison under matched absolute effective schedules in its principal training setup. Such a control is particularly relevant given the trajectory-alignment experiments reported by Xiao et al., Hyperball May Not Be a Free Lunch, §4.2. Component ablations would further clarify the roles of weight projection and update normalization, especially because idealized SVD Muon already has a fixed update Frobenius norm for a given matrix shape (Lemma 4.3). A representative comparison with explicit rotational-update control would also help establish the incremental benefit over closely related approaches such as Rotational Equilibrium (Kosson et al., 2024). Hyperball could remain valuable as a convenient schedule-control mechanism, but the current evidence does not clearly distinguish improved optimization quality from reduced tuning effort.
- Incomplete reporting of tuning and uncertainty makes the strength and fairness of some empirical conclusions difficult to assess. Although §3.1 identifies the architecture family and data sources, the manuscript does not systematically specify the principal experiments’ complete configurations, data-mixture proportions, optimizer settings, search ranges, and tuning budgets. The 8B comparison uses “manually chosen hyperparameters” and compares MuonH against AdamW (§3.2), so it provides exploratory evidence rather than an isolated estimate of the wrapper’s contribution. Repeated-run statistics are also important for the public benchmark differences of 0.0008 between MuonH and MuonW and 0.0011 between NorMuonH and NorMuonW (§3.2). The token-equivalent speedup and fitted long-horizon asymptotes would benefit from uncertainty estimates. Finally, the approximately 1.4× optimal-learning-rate drift is close to one interval of the (\sqrt{2})-spaced search grid, and the transfer experiments use a HybridNorm variant (§3.1, §3.3). Reporting the loss penalty from directly transferring a learning rate selected on a smaller model would more clearly establish practical transferability. A consolidated configuration table, representative repeated runs, and this direct transfer evaluation would substantially strengthen the empirical claims.
Questions for authors
- Can the authors provide a matched-effective-schedule control and targeted component ablations?
- I would be more confident in acceptance if there would be proper responses to the questions raised in the weaknesses.
3 comments
Rebuttal
7. Uncertainty in token-equivalent speedups and fitted asymptotes.
We computed approximate 95% profile-F intervals for the fitted asymptotes. For the long-horizon experiment, we fit to the best tested C4 validation loss at each of the 13 token budgets from 1B to 128B.
| Optimizer | Fitted limiting loss | Approximate 95% fit interval |
|---|---|---|
| MuonW | 3.0787 | [3.0693, 3.0875] |
| MuonH | 3.0654 | [3.0593, 3.0713] |
For the token-equivalent speedup, we compute the speedup mostly using interpolation or mild extrapolation between different level of AdamW's loss at different token count.
8. What loss penalty results from directly transferring a learning rate?
The reported drift is approximately one step on our -spaced learning-rate grid. We therefore also quantify the validation-loss cost of transferring the learning rate.
We analyzed 60 MuonH runs using four-layer Qwen3 models with QK normalization enabled without HybridNorm. At each width (128, 256, 512, and 1024), we tested 15 learning rates, training each model for approximately 1B tokens with a cosine schedule. We selected the rate with the lowest final C4 validation loss at width 128 and transferred it unchanged to the larger widths.
| Target width | C4 loss at transferred rate | Best C4 loss in target grid | Loss increase |
|---|---|---|---|
| 256 | 4.27876 | 4.27876 | 0.00000 |
| 512 | 4.00182 | 3.99795 | 0.00387 |
| 1024 | 3.81496 | 3.81272 | 0.00224 |
The loss penalty is below 0.004 relative to each target’s best tested rate, supporting useful width transfer even without HybridNorm.
The author has addressed my questions. I will keep my rating unchanged and raise my confidence for this paper.
Rebuttal
We thank the reviewer for the review.
1. Does Hyperball provide a benefit beyond controlling the effective learning rate schedule?
Our claim is that Hyperball makes the effective learning rate schedule easier to choose and control directly. We do not claim any additional optimization advantage once the absolute effective learning rate schedules are matched; in that case, we expect very similar performance. Consistent with this expectation, Xiao et al. (2026, §4.2, Figure 3), as cited in our paper, show that adjusting the learning rate schedule substantially aligns the validation loss trajectories of MuonH and Muon with weight decay in their experiments.
2. Roles of weight projection and update normalization.
Update normalization in hyperball serves a conceptual purpose instead of an empirical purpose: it gives the effective learning rate a common parameterization across matrices and base optimizers. Weight projection fixes each matrix norm at Ri, while update normalization makes the relative magnitude of the step before projection equal to the shared Hyperball learning rate ηt. For a nonzero base optimizer update ui,t of matrix i, retaining the same projection makes update normalization exactly equivalent to using the matrix-specific learning rate ηi,t=ηt∥ui,t∥FRi. For idealized SVD Muon in Lemma 4.3, the update norm is constant for a fixed matrix shape, so this reduces to a fixed rescaling of the learning-rate schedule for each matrix.
3. What is the incremental benefit over explicit rotational-update control?
Hyperball offers a conceptually simpler way to control the effective learning rate: the update norm relative to the weight norm. The rotational wrapper of Kosson et al. (2024) removes the radial component of each constrained weight update and maintains an exponential moving average (EMA) of the projected update’s squared norm to control average rotation. Hyperball normalizes the current matrix update and rescales the updated matrix to its initialization Frobenius norm. It requires neither the radial projection nor an additional EMA for rotation control. The base optimizer retains its usual state, and Hyperball stores the fixed target norms.
4. Are the configurations and tuning budgets reported sufficiently to assess the comparisons?
We thank the reviewer for the suggestion. We add an appendix E listing implementation details, hyperparameter search ranges, and the rule used to select the reported run.
5. The 8B result provides exploratory evidence rather than an isolated estimate of the wrapper’s contribution
We agree with the reviewer and would clarify in the paper.
6. How uncertain are the small public benchmark differences?
These comparisons already include repeated runs: 10 each for MuonH and NorMuonH (the Hyperball variants), and 20 each for MuonW and NorMuonW (the weight decay baselines). Using these runs, we computed two-sided 95% Welch confidence intervals for the difference in mean validation loss, treating the runs as unpaired and allowing different variances. We define improvement as the baseline mean loss minus the Hyperball mean loss, so positive values favor Hyperball. We note that between NorMuonH and NormuonW, we observe a statiscally significant difference.