- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work introduces Distance-Free Muon, a Muon-style algorithm that uses a weighted FTRL scheme to adaptively select the radius along the LMO direction. The authors provide convergence guarantees for star-convex objectives, which match the rate of prior work without requiring knowledge of the distance to the minimizer or the noise level. Furthermore, the authors introduce an additional algorithm with a FTRL scheme that tunes the step size for the case of non-convex objectives and provide corresponding convergence guarantees that match the standard non-convex convergence rate of regular Muon. Finally, the authors provide some synthetic and real data experiments to evaluate their FTRL variants.
Critical strengths
- Unlike prior work (Kovalev (2025)) in the star-convex case, the convergence guarantees in this work do not require knowledge of an upper bound on the norm of the minimizer and the stochastic gradient noise level, while the Distance-Free Muon algorithm (Algorithm 1) achieves the same last-iterate convergence rate.
- The idea of adjusting the radius or step-size using FTRL scheme in Muon-style algorithms is interesting and well-motivated.
Critical weaknesses
- Some of the algorithmic design choices, particularly in the extension to the non-convex setting, are not adequately explained. For example, Algorithm 1 uses a Nesterov-style momentum direction and updates the iterates through a convex combination involving a fixed reference point, whereas Algorithm 2 uses standard EMA momentum and an additive update. The motivation for these different choices, and whether they are essential for the respective convergence guarantees and objectives, is not sufficiently discussed.
- In the non-convex setup, it is unclear whether the theoretical result shows any advantage of using the FTRL-based stepsize as opposed to any fixed stepsize, beyond matching the convergence rate of regular Muon.
- In the GPT-2 experiments, the authors compare regular Muon with two algorithmic variants (Practical weighted-FTRL Muon and Non-convex DF Muon), none of which exactly correspond to the algorithms presented in the theoretical results. Specifically, Practical weighted-FTRL Muon uses an additive update with weight decay and applies gradient clipping and rectangular adjustment, whereas Non-convex DF Muon uses Nesterov-style momentum and also applies weight decay, gradient clipping and rectangular scaling. Thus, the theoretical guarantees provided in the paper do not apply to these algorithmic variants.
- From the results in Table 3 it seems like both of the comparison methods perform very similarly to or slightly worse than regular Muon, and they seem to also be slower in terms of wall-clock time. Thus, there is not enough evidence that the online adaptation provides any benefit in real data experiments.
- In the current-gradient stepsize-search study, the stepsize is selected using the current stochastic gradient, a choice that is not covered by the theoretical analysis and its justification and implications are not sufficiently discussed in the paper.
- More generally, the experiments section could be presented more clearly. While the authors provide some tables and figures, I found it difficult to identify the specific experimental setups, understand which baselines were used, and how the comparison methods relate to the theoretically analyzed algorithms.
Questions for authors
- Could the authors clarify the role of the reference point in the theoretical setup and dynamics of the Distance-Free Muon? In particular, is the reference point only motivated by the theoretical analysis? Beyond the choices and discussed in the paper, would other choices of be meaningful?