- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Drop-Muon, a randomized progressive training framework for non-Euclidean layer-wise optimizers. At each step, the method samples a cutoff and updates only a suffix of the network. The paper provides convergence guarantees under layer-wise smoothness and generalized -smoothness , including stochastic optimization with momentum, and derives a cost analysis characterizing when full-network updates are suboptimal. Experiments on CNNs, ResNet50, and GPT-style language models show reduced training cost and improved wall-clock efficiency relative to full-network Muon.
Critical strengths
- The theoretical extension of randomized progressive training to non-Euclidean LMO-based optimization, particularly in the stochastic generalized-smooth setting, is technically meaningful.
- The method is simple and practically implementable, and the language-model profiling provides convincing evidence that skipping layer updates reduces backward, communication, and optimizer costs.
Critical weaknesses
- The practical algorithm is only partially connected to the compute-optimality theory. The latter is derived under a deterministic cost model with smoothness-dependent optimal sampling, while the experiments use a heuristic sampling distribution and a tuned wake-up phase.
- Some claims are broader than the evidence warrants. In particular, the empirical smoothness estimates are trajectory-dependent proxies rather than verification of the global theoretical assumptions, and the largest-scale experiments have limited statistical repetition.
Questions for authors
- How does the theoretically optimal or smoothness-informed sampling distribution compare empirically with the heuristic linear schedule used in the main experiments?
- How should the empirical smoothness estimates be interpreted relative to the global constants required by the theory?
5 comments
The largest-scale experiments have limited statistical repetition
This is a fair point. However, the CNN results do average several runs. Moreover, our component profiling repeated over three paired seeds shows a consistent 12.9% reduction in per-step computation time. These results confirm that Drop-Muon reduces the cost of backward computation, orthogonalization, and gradient communication.
We hope this clarifies how the theory and the experiments relate. We thank the reviewer again for the positive assessment of our work.
Please engage with the authors during the discussion period.
We thank the reviewer for the constructive comments, and for recognizing both the technical contribution and the simplicity and practicality of Drop-Muon. Below we address both weaknesses and the two questions.
The paper contains two kinds of theoretical results, and they relate to the experiments differently:
Convergence result (Theorem 4.4) analyzes Algorithm 1 itself, with stochastic gradients and momentum. It holds for any RPT distribution with p1>0, including rule (9) used in our experiments, and is a special case of Theorem G.1, which allows any sampling scheme under which every layer is updated with positive probability. As is usual, the experiments depart from it in the momentum and learning-rate schedules.
Compute optimality (Theorem 4.3, and Theorem F.1 for the generalized-smooth case) is a deterministic result, derived for Algorithm 2 with subset-dependent stepsizes γik. We focus on the deterministic setting because the stochastic complexity bound of Theorem 4.4 contains several interacting terms involving the sampling probabilities, the momentum parameter and the variance constants, which preclude a clean characterization of the optimal sampling distribution (see the discussion after Theorem 4.4). The deterministic setting isolates the trade-off between optimization progress and computational cost, and allows an exact characterization of when full-network updates are compute-optimal. As is common in the analysis of optimization complexity, practical components such as stochasticity, momentum and learning-rate schedules are left out of this part of the analysis.
To answer the questions, we computed the smoothness-informed optimal RPT distribution for NanoGPT-162M by solving (38) with the constants of Table 2. Since L0≈0 there, the O(1/ε) regime (37) is the relevant one. We calibrated cov, ci and ci♯ from the profiling in Section H.7 and tried several values for the unmeasured embedding cost. The solution is p1⋆≈0.7, in line with the smooth case where the optimal probabilities depend only on the smoothness constants. The resulting bound on the total cost is about 8-9% below the corresponding bound for full-network updates. Rule (9) drops layers more aggressively (p1≈0.14), so in the worst-layer criterion of (37) its bound is limited by the rarely updated embedding layer. The difference comes from the stepsizes. The gain of p⋆ relies on the larger subset-dependent stepsizes of Algorithm 2, while our implementation applies the same learning rate to every active subset, so its main source of savings is a cheaper step. The late wake-up partly compensates, raising the overall fraction of full-network steps.
We agree that these are trajectory-dependent estimates rather than a verification of a global assumption. They are nevertheless the relevant quantity for our theory.
First, although Assumptions 4.2-4.4 are stated globally for simplicity, the proofs of Theorems 4.1 and 4.2 use them only at consecutive iterates (X=Xk and Γ=Xk+1−Xk). Our empirical estimates probe the same kind of pairs. At each Xk, Section H.6.1 applies the update to every suffix Sj and fits the constants to the resulting gradient-difference ratios. Second, global constants would be ideal, but they cannot be estimated in practice. Trajectory estimates are therefore standard in the literature. For example, [1] motivated (L0,L1)-smoothness by smoothness estimated along training trajectories, [2] tracks sharpness during training, and [3] estimates layer-wise (L0,L1) constants from consecutive NanoGPT iterates. The approach is imperfect but is a good proxy. Lastly, the cost conclusions depend only on an ordering. Theorems 4.3 and F.1 ask whether L1,[b]=maxiLi,[b]. The estimates show a clear margin (35.2 versus 52.3).
[1] Zhang, J., He, T., Sra, S. and Jadbabaie, A., 2019. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv:1905.11881.
[2] Cohen, J.M., Kaur, S., Li, Y., Kolter, J.Z. and Talwalkar, A., 2021. Gradient descent on neural networks typically occurs at the edge of stability. arXiv:2103.00065.
[3] Riabinin, A., Shulgin, E., Gruntkowska, K. and Richtárik, P., 2025. Gluon: Making muon & scion great again!(bridging theory and practice of LMO-based optimizers for LLMs). arXiv:2505.13416.