acceptodds
Under review as a conference paper at ICLR 2027

Distance-Aware Muon: Adaptive Step Scaling for Normalized Optimization

Abstract

Muon and related normalized optimizers determine a geometry-aware update direction but leave its magnitude to be tuned. Existing stochastic guarantees for star-convex Muon set this magnitude using the unknown distance to a minimizer. We formulate the missing scale as a one-dimensional online-learning problem and introduce Distance-Free Muon, which uses weighted follow-the-regularized-leader to learn the corresponding radius from past stochastic estimates of directional derivatives. For smooth star-convex objectives in general norm geometries, it achieves expected last-iterate suboptimality, matching the known-distance method in its dependence on . The method requires neither the distance to a minimizer nor the noise level and assumes neither a bounded domain nor a bounded trajectory. A complementary predictable stepsize rule gives an bound on the expected gradient norm for general smooth non-convex objectives. Controlled matrix experiments quantify both the finite-horizon cost of learning the radius and its benefit when a fixed radius is transferred across distance scales. For practical additive variants beyond the analyzed updates, separate GPT-124M/WikiText-103 studies show that FTRL rules match tuned Muon, while a current-gradient stepsize-search proxy with a Nesterov-style momentum direction improves an otherwise identical fixed-scale baseline tuned on a disjoint seed.

Primary area:
optimization
Keywords:
Muon, gradient orthogonalization, adaptive stepsizes, distance adaptation, online learning, stochastic optimization, non-Euclidean optimization, star-convex optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1AtCOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces Distance-Free Muon, a Muon-style algorithm that uses a weighted FTRL scheme to adaptively select the radius along the LMO direction. The authors provide convergence guarantees for star-convex objectives, which match the rate of prior work without requiring knowledge of the distance to the minimizer or the noise level. Furthermore, the authors introduce an additional algorithm with a FTRL scheme that tunes the step size for the case of non-convex objectives and provide corresponding convergence guarantees that match the standard non-convex convergence rate of regular Muon. Finally, the authors provide some synthetic and real data experiments to evaluate their FTRL variants.

Critical strengths
  • Unlike prior work (Kovalev (2025)) in the star-convex case, the convergence guarantees in this work do not require knowledge of an upper bound on the norm of the minimizer and the stochastic gradient noise level, while the Distance-Free Muon algorithm (Algorithm 1) achieves the same last-iterate convergence rate.
  • The idea of adjusting the radius or step-size using FTRL scheme in Muon-style algorithms is interesting and well-motivated.
Critical weaknesses
  • Some of the algorithmic design choices, particularly in the extension to the non-convex setting, are not adequately explained. For example, Algorithm 1 uses a Nesterov-style momentum direction and updates the iterates through a convex combination involving a fixed reference point, whereas Algorithm 2 uses standard EMA momentum and an additive update. The motivation for these different choices, and whether they are essential for the respective convergence guarantees and objectives, is not sufficiently discussed.
  • In the non-convex setup, it is unclear whether the theoretical result shows any advantage of using the FTRL-based stepsize as opposed to any fixed stepsize, beyond matching the convergence rate of regular Muon.
  • In the GPT-2 experiments, the authors compare regular Muon with two algorithmic variants (Practical weighted-FTRL Muon and Non-convex DF Muon), none of which exactly correspond to the algorithms presented in the theoretical results. Specifically, Practical weighted-FTRL Muon uses an additive update with weight decay and applies gradient clipping and rectangular adjustment, whereas Non-convex DF Muon uses Nesterov-style momentum and also applies weight decay, gradient clipping and rectangular scaling. Thus, the theoretical guarantees provided in the paper do not apply to these algorithmic variants.
  • From the results in Table 3 it seems like both of the comparison methods perform very similarly to or slightly worse than regular Muon, and they seem to also be slower in terms of wall-clock time. Thus, there is not enough evidence that the online adaptation provides any benefit in real data experiments.
  • In the current-gradient stepsize-search study, the stepsize is selected using the current stochastic gradient, a choice that is not covered by the theoretical analysis and its justification and implications are not sufficiently discussed in the paper.
  • More generally, the experiments section could be presented more clearly. While the authors provide some tables and figures, I found it difficult to identify the specific experimental setups, understand which baselines were used, and how the comparison methods relate to the theoretically analyzed algorithms.
Questions for authors
  • Could the authors clarify the role of the reference point cc in the theoretical setup and dynamics of the Distance-Free Muon? In particular, is the reference point only motivated by the theoretical analysis? Beyond the choices c=0c=0 and c=x0c=x_0 discussed in the paper, would other choices of cc be meaningful?
Reviewer_3VVAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies scale selection for Muon-type normalized optimizers. Its main algorithm uses weighted FTRL to learn a radius from past directional derivatives, with delayed momentum ensuring that the direction and radius are chosen before observing the fresh stochastic gradient. Under smoothness and star-convexity, it removes the solution-distance and noise-level inputs while retaining the known-distance method’s last-iterate convergence exponent. A separate extension provides a stationarity guarantee for general non-convex objectives. Experiments cover synthetic matrix problems and practical variants on GPT-124M.

Critical strengths
  1. A concrete theoretical improvement over known-distance Muon. The method removes unavailable distance and noise inputs while preserving the last-iterate convergence exponent, without assuming bounded iterates or a bounded domain. Controlling the interaction between the learned radius and momentum error is a substantive part of the contribution.

  2. The matched known-distance baseline isolates the cost of radius learning, while the transfer experiments expose both its benefits and limitations. The schedule-replay experiment and the stepsize-only comparison in Table 4 are also useful controls. The authors report results that limit, rather than uniformly favor, their approach.

Critical weaknesses
  1. The practical value of the main FTRL mechanism remains uncertain. On GPT-124M, the FTRL variants match tuned Muon, and replaying a schedule from another seed performs similarly to online adaptation. The practical implementations also differ from the analyzed updates. Table 4 supports the separate proxy, but does not establish a benefit for the main theoretical algorithm.
  2. The reduction in tuning burden is not demonstrated. Requiring L does not invalidate the distance-free result, but the practical variants replace it with a tuned empirical coefficient and use additional regularization choices. Sensitivity and tuning-budget comparisons are needed to determine whether this is easier to use than tuning Muon’s learning rate. For the proxy, the main text also leaves its search procedure and computational overhead insufficiently specified.
Questions for authors
  1. How were the empirical smoothness and regularization coefficients selected, and how does the total tuning budget compare with Muon’s? Do the available runs show how sensitive performance is to these choices?
  2. For the Table 4 proxy, what are the candidate stepsizes, scoring procedure, and additional computation? How is this cost counted in the gradient-evaluation budget, and was the selected fixed stepsize inside the tuning grid or at its boundary?
  3. Does replaying the proxy’s stepsize sequence from the tuning seed preserve its advantage in Table 4? This would help distinguish the benefit of a transferable schedule from that of online feedback.
Reviewer_vWPuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Distance-Aware Muon, an adaptive step-size optimization framework that uses FTRL to learn update scales from historical gradient feedback. The main contribution is removing the dependence on the unknown distance to the optimal solution while maintaining theoretical convergence guarantees.

I find adaptive step-size selection for Muon an interesting direction. However, I have several concerns regarding the practical applicability of the method and the strength of the experimental evidence.

Critical strengths
  1. Interesting direction. Automatically selecting update step sizes for Muon is a promising direction that could potentially reduce manual learning-rate tuning in large-scale training.

  2. Large part of the paper is clear. The paper is mostly well organized, and the motivation and high-level theoretical contributions are clearly explained.

Critical weaknesses

1. Algorithm descriptions are difficult to follow.

I found Algorithm 1 and Algorithm 2 unnecessarily difficult to understand. In particular, it took considerable effort to distinguish between scalars, vectors, matrices, and accumulated statistics. The roles of different quantities, such as RkR_k, ηk\eta_k, momentum variables, and FTRL coefficients, are not sufficiently intuitive. I strongly recommend adding a notation table specifying the dimensions and meanings of the key variables, together with simplified implementation-oriented pseudocode.

2. The requirement of the smoothness constant LL limits practical applicability.

Both algorithms require the smoothness constant LL, which is generally unknown and difficult to estimate for modern neural networks. Its value may differ significantly across architectures and training configurations. Although the method removes dependence on the unknown distance to the optimum, it still requires a potentially difficult hyperparameter to be specified. In the practical experiments, the authors use empirically calibrated smoothness-related coefficients. I would like to see sensitivity experiments showing how performance changes under different choices of LL, and a clearer discussion of how practitioners should obtain this value.

3. Unclear robustness to the training horizon TT.

In standard LLM training, learning-rate schedules are typically designed according to the total number of training steps, often using warmup and cosine decay. These manually designed schedules are relatively well established and stable in practice. The proposed algorithms also depend on TT, but their robustness to substantially longer training horizons remains unclear. Although the paper includes horizon-dependent matrix experiments, I would like to see similar evaluations for LLM training, especially when TT is very large or the actual training duration changes. Moreover, the practical GPT experiments still use a conventional warmup-cosine schedule, making it difficult to isolate the benefit of adaptive step-size selection.

4. Limited experimental scale and no clear improvement over standard Muon.

The language-model experiments are limited to GPT-124M with 10,000 gradient evaluations, which is relatively small in both model scale and training duration. More importantly, the primary FTRL-based methods achieve validation losses similar to, but not better than, standard Muon (Table 3). Although the paper reports improvements from a stepsize-search proxy, these improvements also involve additional design choices such as Nesterov-style momentum, rectangular scaling, and weight decay. Therefore, the experiments do not convincingly demonstrate that the main proposed FTRL mechanism provides practical improvements over Muon.

5. The strength of the Muon baseline is difficult to assess.

I find it difficult to contextualize the reported validation losses against existing long-horizon Muon experiments. For example, nanoGPT-style training experiments often run for 100K or 200K steps, where the behavior of learning-rate schedules can differ substantially. Although the authors tune the baseline on a separate seed, I am not convinced that the current experiments sufficiently establish its competitiveness. Longer training runs, more extensive baseline tuning, and comparisons on established training benchmarks would help strengthen the empirical claims.

Questions for authors

The paper explores an interesting problem and provides meaningful theoretical results. However, I am concerned about the gap between the theoretical formulation and practical applicability, particularly the requirement of the unknown smoothness constant LL. Additionally, the current experiments are relatively limited and do not demonstrate clear improvements over a well-tuned standard Muon optimizer.

I would be more convinced by stronger evidence of robustness to LL and TT, along with larger-scale and longer-horizon language-model experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.