acceptodds
Under review as a conference paper at ICLR 2027

Drop-Muon: Update Less, Converge Faster

Abstract

Conventional wisdom in deep learning optimization dictates updating all layers at every step – a principle followed by all recent state-of-the-art optimizers such as Muon. In this work, we challenge this assumption, showing that full-network updates can be fundamentally suboptimal, both in theory and in practice. We introduce a non-Euclidean Randomized Progressive Training method – Drop-Muon – a simple yet powerful framework that updates only a subset of layers per step according to a randomized schedule, combining the efficiency of progressive training with layer-specific non-Euclidean updates for top-tier performance. We provide rigorous convergence guarantees under both layer-wise smoothness and layer-wise -smoothness, covering deterministic and stochastic gradient settings, marking the first such results for progressive training in the stochastic and non-smooth regime. Our cost analysis further reveals that full-network updates are not optimal unless a very specific relationship between layer smoothness constants holds. Through experiments on NanoGPT and small controlled CNN studies, we empirically demonstrate that Drop-Muon consistently outperforms full-network Muon, achieving the same accuracy up to faster in wall-clock time. Together, our results suggest a shift in how large-scale models can be efficiently trained, challenging the status quo and offering a highly efficient, theoretically grounded alternative to full-network updates.

Primary area:
optimization
Keywords:
optimization, deep learning, Muon, progressive training

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_fRbpOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Drop-Muon, a randomized progressive training framework for non-Euclidean layer-wise optimizers. At each step, the method samples a cutoff and updates only a suffix of the network. The paper provides convergence guarantees under layer-wise smoothness and generalized (L0,L1)(L_0,L_1)-smoothness , including stochastic optimization with momentum, and derives a cost analysis characterizing when full-network updates are suboptimal. Experiments on CNNs, ResNet50, and GPT-style language models show reduced training cost and improved wall-clock efficiency relative to full-network Muon.

Critical strengths
  • The theoretical extension of randomized progressive training to non-Euclidean LMO-based optimization, particularly in the stochastic generalized-smooth setting, is technically meaningful.
  • The method is simple and practically implementable, and the language-model profiling provides convincing evidence that skipping layer updates reduces backward, communication, and optimizer costs.
Critical weaknesses
  • The practical algorithm is only partially connected to the compute-optimality theory. The latter is derived under a deterministic cost model with smoothness-dependent optimal sampling, while the experiments use a heuristic sampling distribution and a tuned wake-up phase.
  • Some claims are broader than the evidence warrants. In particular, the empirical smoothness estimates are trajectory-dependent proxies rather than verification of the global theoretical assumptions, and the largest-scale experiments have limited statistical repetition.
Questions for authors
  • How does the theoretically optimal or smoothness-informed sampling distribution compare empirically with the heuristic linear schedule used in the main experiments?
  • How should the empirical smoothness estimates be interpreted relative to the global constants required by the theory?
5 comments

We thank the reviewer for the constructive comments, and for recognizing both the technical contribution and the simplicity and practicality of Drop-Muon. Below we address both weaknesses and the two questions.


The practical algorithm is only partially connected to the compute-optimality theory...

The paper contains two kinds of theoretical results, and they relate to the experiments differently:

  • Convergence result (Theorem 4.4) analyzes Algorithm 1 itself, with stochastic gradients and momentum. It holds for any RPT distribution with p1>0p_1 > 0, including rule (9) used in our experiments, and is a special case of Theorem G.1, which allows any sampling scheme under which every layer is updated with positive probability. As is usual, the experiments depart from it in the momentum and learning-rate schedules.

  • Compute optimality (Theorem 4.3, and Theorem F.1 for the generalized-smooth case) is a deterministic result, derived for Algorithm 2 with subset-dependent stepsizes γik\gamma_i^k. We focus on the deterministic setting because the stochastic complexity bound of Theorem 4.4 contains several interacting terms involving the sampling probabilities, the momentum parameter and the variance constants, which preclude a clean characterization of the optimal sampling distribution (see the discussion after Theorem 4.4). The deterministic setting isolates the trade-off between optimization progress and computational cost, and allows an exact characterization of when full-network updates are compute-optimal. As is common in the analysis of optimization complexity, practical components such as stochasticity, momentum and learning-rate schedules are left out of this part of the analysis.

To answer the questions, we computed the smoothness-informed optimal RPT distribution for NanoGPT-162M by solving (38) with the constants of Table 2. Since L0≈0L^0 \approx 0 there, the O(1/ε)O(1/\varepsilon) regime (37) is the relevant one. We calibrated covc_{ov}, cic_i and ci♯c_i^\sharp from the profiling in Section H.7 and tried several values for the unmeasured embedding cost. The solution is p1⋆≈0.7p_1^\star \approx 0.7, in line with the smooth case where the optimal probabilities depend only on the smoothness constants. The resulting bound on the total cost is about 8-9% below the corresponding bound for full-network updates. Rule (9) drops layers more aggressively (p1≈0.14p_1 \approx 0.14), so in the worst-layer criterion of (37) its bound is limited by the rarely updated embedding layer. The difference comes from the stepsizes. The gain of p⋆p^\star relies on the larger subset-dependent stepsizes of Algorithm 2, while our implementation applies the same learning rate to every active subset, so its main source of savings is a cheaper step. The late wake-up partly compensates, raising the overall fraction of full-network steps.

Some claims are broader than the evidence warrants... How should the empirical smoothness estimates be interpreted relative to the global constants required by the theory?

We agree that these are trajectory-dependent estimates rather than a verification of a global assumption. They are nevertheless the relevant quantity for our theory.

First, although Assumptions 4.2-4.4 are stated globally for simplicity, the proofs of Theorems 4.1 and 4.2 use them only at consecutive iterates (X=XkX = X^k and Γ=Xk+1−Xk\Gamma = X^{k+1} - X^k). Our empirical estimates probe the same kind of pairs. At each XkX^k, Section H.6.1 applies the update to every suffix SjS_j and fits the constants to the resulting gradient-difference ratios. Second, global constants would be ideal, but they cannot be estimated in practice. Trajectory estimates are therefore standard in the literature. For example, [1] motivated (L0,L1)(L^0, L^1)-smoothness by smoothness estimated along training trajectories, [2] tracks sharpness during training, and [3] estimates layer-wise (L0,L1)(L^0, L^1) constants from consecutive NanoGPT iterates. The approach is imperfect but is a good proxy. Lastly, the cost conclusions depend only on an ordering. Theorems 4.3 and F.1 ask whether L1,[b]=max⁡iLi,[b]L_{1,[b]} = \max_i L_{i,[b]}. The estimates show a clear margin (35.235.2 versus 52.352.3).

[1] Zhang, J., He, T., Sra, S. and Jadbabaie, A., 2019. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv:1905.11881.

[2] Cohen, J.M., Kaur, S., Li, Y., Kolter, J.Z. and Talwalkar, A., 2021. Gradient descent on neural networks typically occurs at the edge of stability. arXiv:2103.00065.

[3] Riabinin, A., Shulgin, E., Gruntkowska, K. and Richtárik, P., 2025. Gluon: Making muon & scion great again!(bridging theory and practice of LMO-based optimizers for LLMs). arXiv:2505.13416.

The largest-scale experiments have limited statistical repetition

This is a fair point. However, the CNN results do average several runs. Moreover, our component profiling repeated over three paired seeds shows a consistent 12.9% reduction in per-step computation time. These results confirm that Drop-Muon reduces the cost of backward computation, orthogonalization, and gradient communication.


We hope this clarifies how the theory and the experiments relate. We thank the reviewer again for the positive assessment of our work.

Please engage with the authors during the discussion period.

Reviewer_xc11OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses a fundamental question: "*When training neural networks using Muon, should we update all the network layers together?" The authors propose Drop-Muon, an optimizer which selectively updates consecutive output-side layers. The starting layer is randomly selected at every iteration. The paper primarily focuses on the theoretical aspects of the layer-wise update scheme for Muon, with respect to the iteration complexity under deterministic gradient (section 4.1) and stochastic gradient (section 4.2) settings. As a result, the authors show that the stochastic setting achieves O(K−1/4)\mathcal{O}(K^{-1/4}), which is the optimal rate under smoothness and bounded variance assumptions. Thus, the conclusion is that the sub-network updates allow the same convergence rate as the full-network updates, while it enjoys the reduced iteration cost. As numerical results, the authors present a few experimental results obtained from relatively small networks.

Critical strengths
  1. Assumptions are wisely defined to consider the proposed layer-wise update scheme.

  2. The reported theoretical result matches the state-of-the-art convergence rate, O(K−1/4)\mathcal{O}(K^{-1/4}). This result will allow researchers to more flexibly design advanced layer-wise optimizers.

  3. Extensive and thorough proofs are provided in the appendix. These proofs will allow researchers to take advantages of the proof framework and several proof techniques in their own studies. I always appreciate these rich supplementary materials.

Critical weaknesses

Concerns regarding presentation quality

  1. [Readability of Algorithm 1 and 2] In Algorithm 1 and 2, the data samples are sampled from P\mathcal{P} and active layers from D\mathcal{D}. However, these distributions are not defined in the algorithm. To make things self-contained, please define these notations in the algorithm or the caption. I believe both are uniform random distributions.

  2. [Readability of Figures] The texts in Figure 1 and 2 are too small to read. Make sure the fonts are as large as the main text. In addition, the performance gap is nearly not visible. For example, MNIST bars in Figure 1 is almost not distinguished. All the learning curves in Figure 2 have the same issue.

  3. [Small models only] All numerical experiments use relatively small models like million-sized language models. Will the large models like billion-sized models enjoy the same performance improvement? NVIDIA V100 has a sufficiently large memory space, and the authors could have used popular models like GPT-2, LLaMA, Qwen, which contain 1B~3B. In order to provide more practical results, I strongly recommend collecting empirical results using larger models and verify whether the proposed ideas generally apply to neural networks.

  4. [Inappropriate exclamation mark] At line 348, a sentence ends with an exclamation mark. I believe it is not appropriate to use such marks in academic writing. It should be replaced with just a period.

Concerns regarding main ideas

  1. [Sampling condition] The umbrella paragraph of Section 4 explains that sks^k is sampled at each iteration kk. I assume the samplings are independent across kk. The authors should clarify this in the main text.

  2. [Connection to intuitive interpretation] The theorems are sound and match the best known convergence rates for SGD- and Muon-type methods. However, I recommend adding a short remark after each theorem that explains how the key parameters affect the bound. For example, it would be helpful to discuss how the sampling probabilities psp_s (in particular p1p_1) influence the layer weights CiC_i and the step sizes.

Questions for authors

I summarized a few questions and concerns in the above weakness section. Please check and carefully address them.

4 comments

We thank the reviewer for the careful reading and for the positive assessment of our work! Below we answer each point.


Readability of Algorithm 1 and 2 (notation P\mathcal P and D\mathcal D)

P\mathcal P is the underlying data distribution (see lines 80 and 88), and D\mathcal D is a user-defined layer sampling distribution introduced at the start of Section 2 (line 101).

Neither of them needs to be uniform. P\mathcal P is determined by the data, and D\mathcal D can be any distribution over subsets of the layers 1,…,b\\{1, \dots, b\\} (including the uniform distribution as a special case). In the main text, D\mathcal D is Randomized Progressive Training: draw a cutoff sks^k with P(sk=i)=pi\mathbb P(s^k = i) = p_i and set Sk=sk,…,bS^k = \\{s^k, \dots, b\\}, and our experiments use the non-uniform rule (9). Other choices of D\mathcal D are analyzed in Appendices D and F.

We agree that the definition of D\mathcal D belongs in the pseudocode and have added it to the captions of Algorithms 1 and 2 in the updated PDF.

Readability of Figures

We agree, and we increased the size of figure text in the revised PDF.

On the visibility of the gaps: in Figure 1, the relevant comparison is between each bar and the red dashed "no speedup" reference line at 1. Each bar is already the paired ratio TMuon/TDrop−MuonT_{Muon}/T_{Drop-Muon} at one accuracy target (shown on the x-axis). On MNIST the bars have similar heights (about 1.30×1.30\times to 1.35×1.35\times), which means that the speedup is consistent across accuracy targets. In Figure 2, the curves separate mostly in the second half of training, and we can add zoomed insets on that phase if the reviewer finds this helpful.

Small models only

The per-step savings can come from three sources: a truncated backward pass, fewer Newton–Schulz orthogonalizations, and less gradient communication. All three are proportional to the work in the frozen prefix. Under rule (9) the expected number of active groups is ∑i=1bpi (b−i+1)=2(b+1)3,\sum_{i=1}^b p_i\,(b - i + 1) = \tfrac{2(b+1)}{3}, so roughly one third of the network is frozen on average, independently of depth. Nothing in this mechanism shrinks as models get deeper or wider. Gradient communication, which frozen layers skip, can even take a larger share of step time in multi-node training of large models. Our current runs already span GPT models with 162M to 405M parameters, and we expect the qualitative behavior to be similar at larger model sizes.

Exclamation mark

We will replace it by a period.

Sampling condition

Yes. The cutoffs skk≥0\\{s^k\\}_{k\geq 0} are sampled independently across iterations. The opening paragraph of Section 4 now also states it as a standing assumption.

Connection to intuitive interpretation

Let Ci:=∑s=1ips,Ci1:=∑s=1ips Li,s,…,b1.C_i := \sum_{s=1}^i p_s, \qquad C^1_i := \sum_{s=1}^i p_s\, L^1_{i,\\{s,\dots,b\\}}. Layer ii is active exactly when the cutoff satisfies sk≤is^k\leq i, so CiC_i is the probability that layer ii is updated. Moreover, the constants Li,s,…,bαL^\alpha_{i,\\{s,\dots,b\\}} can be chosen nonincreasing in ss (Lemma C.1).

In Theorems 4.1 and 4.2, the weights are wi=∑s=1ips2Li,s,…,b0andwi=Ci2Ci1=CiCi1/Ci,w_i = \sum_{s=1}^i \frac{p_s}{2L^0_{i,\\{s,\dots,b\\}}} \qquad\text{and}\qquad w_i = \frac{C_i^2}{C^1_i} = \frac{C_i}{C^1_i/C_i}, where Ci1/CiC_i^1/C_i is the average of LI,s,…,b1L^1_{I,\\{s,\dots,b\\}} over the cutoffs at which layer ii is active. Moving probability from p1p_1 to a later cutoff jj has two effects. First, it lowers CiC_i for all layers i<ji<j, in particular C1=p1C_1=p_1 for the input layer, which is why p1>0p_1>0 is required. Second, it shifts mass to shorter active suffixes, where the constants are smaller, so wiw_i can only increase for layers i≥ji\geq j. A similar mechanism enlarges the stepsizes γik=1/Li,Sk0\gamma_i^k=1/L_{i,S^k}^0 (Theorem 4.1) and γik=(Li,Sk0+Li,Sk1∣∇if(Xk)∣(i)⋆)−1\gamma_i^k = (L^0_{i,S^k} + L^1_{i,S^k}\\|\nabla_i f(X^k)\\|_{(i)\star})^{-1} (Theorem 4.2) in iterations with a short active suffix.

In Theorem 4.4, layer ii is weighted by Ciηi/CC_i\eta_i/C. An inactive layer keeps its momentum buffer, so the momentum of layer ii is refreshed with probability CiC_i and its error decays at rate 1−Ci1-C_i in expectation. The slowest rate, 1−p11-p_1, belongs to the input layer, and this is where p1p_1 appears in the second bound on ηi2\eta_i^2. The smoothness terms in the bounds on ηi2\eta_i^2, namely Ci1C_i^1 and ∑s=1bpsmax⁡i∈[b]Li,{s,…,b}1\sum_{s=1}^b p_s \max_{i\in[b]} L^1_{i,\{s,\dots,b\}}, decrease as probability moves away from p1p_1. This relaxes the first bound, while in the second bound it competes with the smaller factor p1p_1.


We thank the reviewer again for the positive assessment and feedback, and hope our responses have addressed all the questions.

Thank you for the rebuttal. However, some of my concerns have not be adequately addressed.

  1. Model size issue. The authors did not explain reasons why million-scale models were used. The reported system configurations would allow them to easily train larger models like 1B ~ 3B LLMs. In addition, there is no evident reasons for the proposed method to work well with larger models.

  2. Connection to the intuitive interpretation. The provided explanation does not deliver practical messages. It just explains the interactions between terms and still does not connect to the high-level performance gains or guarantees.

My final score will depend on how well these concerns are addressed in the rebuttal.

Thank you for the follow-up.

  1. Model size. We agree that billion-scale experiments would strengthen the paper. However, every comparison requires paired Muon/Drop-Muon runs, with an independent learning-rate sweep for each method (e.g., 11 learning rates for NanoGPT-162M). Training cost scales with parameters × tokens: a 162M model trained on 2B tokens takes about 4 hours on one H200 GPU, whereas a 1B model at a comparable tokens-per-parameter ratio requires roughly 60× more compute per run, before multiplying by the number of methods and learning rates. Consequently, a properly tuned 1B experiment with sufficient training tokens is not feasible within the discussion period.

    Although we cannot complete these experiments within the discussion period, we commit to including experiments with a model exceeding 1B parameters, following the same evaluation protocol, in the final version. We expect the gains to persist at scale, and see concrete reasons to expect the gains to carry over to larger models. (i) The savings come from skipping work in the frozen prefix, and the expected frozen fraction does not depend on depth: about one third of the layers is frozen on average, for any number of layers. (ii) At all three model sizes (162M, 247M, 405M), Drop-Muon reaches a lower test loss than Muon at equal training time (Figure 2). (iii) Two of the savings become more important at scale: gradient communication, which frozen layers skip, takes a larger share of the step time in multi-node training, and frozen layers store no activations for backpropagation.

  2. Practical messages. Following the suggestion, we added a remark after each of Theorems 4.1, 4.2 and 4.4 explaining how the sampling probabilities, and p1p_1 in particular, affect the weights and stepsizes. The theory also gives three concrete prescriptions:

    • Whether to drop layers is decided by one checkable condition. Full-network updates minimize the cost bound (8) if and only if the first layer has the largest smoothness constant (Theorems 4.3 and F.1). For NanoGPT-162M this fails (L1,[b]1=35.2L^1_{1,[b]}=35.2 vs. max⁡iLi,[b]1=52.3\max_i L^1_{i,[b]}=52.3), so the theory predicts that full-network training is not compute-optimal, and our experiments agree.
    • Dropping keeps the rate. Theorem 4.4 gives the same O(K−1/4)\mathcal O(K^{-1/4}) rate as full-network Muon and reduces to it for p1=1p_1=1. Dropping changes only the constants, in exchange for cheaper iterations.
    • The optimal distribution is explicit. It follows from a recursion that uses only the smoothness constants (Section F.1.1), so per-layer costs need not be measured.

We thank the reviewer again for the constructive feedback and the positive assessment, and we will include the larger-scale experiment in the final version.

Thank you for the clarification. Although the response to the first concern still has not been adequately resolved, I am satisfied with the rebuttal. Will keep my current score.

Reviewer_qZ1NOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This paper proposes Drop-Muon, an optimizer that randomly selects a subset of continuous layers to update with orthogonalized momentum. The authors provide convergence and convergence rate analyses leveraging L0L^0-smoothness and (L0,L1)(L^0,L^1)-smoothness. Empirically, Drop-Muon is evaluated with small CNNs with MNIST and Fashion-MNIST, ResNet-50 on Tiny ImageNet, and GPT-style models (162M, 247M, and 405M parameters) on Fine-Web-10B dataset. The experiments use time-to-target, test accuracy, training accuracy, and test loss as metrics to evaluate convergence.

Critical strengths
  1. This paper investigates a very crucial problem: the optimization of modern neural networks.

  2. The evaluation with multiple types of neural networks.

Critical weaknesses
  1. The empirical experiment design. The experiments mix the metrics of training accuracy, testing accuracy, training loss across models and datasets. In general, one should compare Drop-Muon with baselines using validation/testing metrics. For LLMs, it should include downstream tasks in addition to validation loss/perplexity.

  2. The assumption of layer independence in Assumption 4.4 is not valid, as perturbing one layer can indeed change the gradients of another layer. Thus, Theorem 4.4 does not hold.

  3. Drop-Muon should be compared with SOTA methods that exploit sparsity (https://arxiv.org/abs/2512.16928) or low-rank approximation (https://arxiv.org/abs/2509.11983) for its runtime reduction.

Questions for authors

Please address my concerns in Critical Weaknesses.

7 comments

We thank the reviewer for their constructive feedback and for helpful suggestions. We address each point below.


The empirical experiment design

Optimization vs. generalization. Our theory concerns minimizing the training objective. Training metrics therefore test that claim directly. Held-out metrics test whether faster optimization also yields a model that performs well on unseen data. Reporting both training and held-out metrics is common practice in optimization papers, since they measure different aspects of performance.

Existing evaluation: held-out results are already in the paper. As the reviewer's summary notes, both main-text experiments use held-out metrics:

  • Figure 1 (MNIST, Fashion-MNIST): the time Muon needs to reach a fixed test-accuracy target, divided by the time Drop-Muon needs (paired runs, averaged). The ratio is about 1.30−1.34×1.30-1.34\times on MNIST and 1.15−1.39×1.15-1.39\times on Fashion-MNIST.
  • Figure 2 (GPT-162M, 247M, 405M on FineWeb): test loss against cumulative training time. At every model size, Drop-Muon reaches a lower test loss than Muon at the same training time. Validation time is excluded from the time axis.

Downstream tasks for the language models. We agree that downstream evaluation is a good complement to validation loss, and we added it to the revised paper (new Appendix H.6.1). We train NanoGPT-162M with Muon and Drop-Muon using the recipe and token budget of Figure 2 (2.0B tokens, two seeds per method), and evaluate zero-shot on PIQA, HellaSwag and ARC-Easy (full evaluation splits, lm-evaluation-harness) after 25, 50, 75 and 100% of training. At the end of training (mean over two seeds, normalized accuracy in %):

PIQAHellaSwagARC-EasyTraining time
Muon62.9529.9237.654.17 h
Drop-Muon63.1929.9937.524.01 h

All differences are within evaluation noise (each is below a quarter of its standard error), providing evidence that Drop-Muon does not degrade model quality while being faster. The same holds at every intermediate checkpoint, which Drop-Muon reaches earlier. A second setting (a quarter of the token budget, four V100 GPUs per run, two seeds) leads to the same conclusion, with 8% shorter training. Drop-Muon's wall-clock gains therefore come at no measurable cost in downstream accuracy.

Assumption 4.4

We thank the reviewer for this careful observation. The gradient form of Assumption 4.4 is needed in the proofs only to compare consecutive iterates XkX^k and Xk+1X^{k+1} (Lemma G.2), where by construction exactly the layers in SkS^k change. Elsewhere, only the descent inequality of Assumption 4.3 is used. Along the trajectory, every active layer moves by exactly its radius ti>0t_i>0. The situation behind the reviewer's argument, where one active layer stays fixed while another moves, therefore does not arise for the active layers.

We nevertheless agree that the global formulation is stronger than needed. This is resolved by replacing ∣Γi∣(i)\\|\Gamma_i\\|_{(i)} with the norm of the largest block of the perturbation.

Assumption 4.4 (revised). For every S∈supp⁡(D)S\in\operatorname{supp}(\mathcal{D}), every i∈[b]i\in[b], and all X,Γ∈XX,\Gamma\in\mathcal{X} with Γj=0\Gamma_j=0 for j∉Sj\notin S,

\leq \left(L_{i,S}^0+L_{i,S}^1\lVert\nabla_i f(X)\rVert_{(i)\star}\right) \max_{j\in S}\lVert\Gamma_j\rVert_{(j)}.$$ This form has several advantages: - It allows arbitrary interactions between layers. Any $f$ with $L$-Lipschitz gradient satisfies it. - It is weaker than the submitted assumption. The old Assumption 4.4 implies the new one and also implies Assumption 4.3. - For active layers along the trajectory, it agrees with the old form up to a factor of at most $\max_j t_j/t_i$.

Theorem 4.4 (revised). Let Assumptions 4.1, 4.3, 4.4 and 4.5 hold. Let Xk\\{X^k\\}, K≥0K\ge0, be the iterates of Algorithm 1 initialized with Mi−1=∇if(X0;ξ0)M_i^{-1}=\nabla_i f(X^0;\xi^0) and run with tik≡ti=ηi/(K+1)3/4t_i^k\equiv t_i=\eta_i/(K+1)^{3/4} and βi≡β=(K+1)−1/2\beta_i\equiv\beta=(K+1)^{-1/2}, where

\eta_i\eta_{\max}\le\frac{\rho_i p_1}{16\bar\rho_i C_i\bar C_i^1L^1_{i,[b]}}.$$ Then $$\min_{k=0,\dots,K}\sum_{i=1}^b\frac{C_i\eta_i}{C}\,\mathbb{E}\left[\\|\nabla_i f(X^k)\\|_{(i)\star}\right]\le$$ $$\frac{3\delta^0}{(K+1)^{1/4}C} + \sum_{i=1}^b\frac{6\eta_i\bar\rho_i\sigma_i}{(K+1)^{1/2}C} + \sum_{i=1}^b\frac{4\bar\rho_i\sigma_iC_i\eta_i}{(K+1)^{1/4}C}$$ $$+ \sum_{i=1}^b\frac{2\bar\rho_i\eta_i\eta_{\max}\left(\bar C_i^0+\bar C_i^1r_i\right)}{\underline{\rho}_i(K+1)^{1/4}C}$$ $$+\sum_{i=1}^b\frac{\eta_i^2\left(C_i^0+C_i^1r_i\right)}{2(K+1)^{3/4}C},$$ where $C$, $C_i$, $C_i^0$, $C_i^1$ are as before, $\bar C_i^\alpha:=\sum_{s=1}^{b}p_sL^\alpha_{i,\\{s,\dots,b\\}}$, $r_i:=L^0_{i,\\{1,\dots,b\\}}/L^1_{i,\\{1,\dots,b\\}}$ and $\eta_{\max}:=\max_j\eta_j$. **What changes.** The $\mathcal{O}(K^{-1/4})$ rate and the structure of the bound are unchanged. - Assumption 4.3 is now stated explicitly, because the new Assumption 4.4 no longer implies it. As noted above, this is not a stronger requirement than in the submission. - Frozen layers enter through $\bar C^\alpha_i$, which averages over all cutoffs, including those at which layer $i$ is frozen. - The drift term and the second step-size condition involve $\eta_i\eta_{\max}$ instead of $\eta_i^2$, since under the revised assumption the change in a layer's gradient is controlled by the largest step among the active layers. The change affects only Theorem 4.4 (and its general form, Theorem G.1), whose rate is unchanged. The deterministic convergence results (Theorems 4.1 and 4.2) and the cost-optimality results rely only on Assumptions 4.2 and 4.3 and do not depend on Assumption 4.4. > Drop-Muon should be compared with SOTA methods **Different mechanisms.** Dion2 reduces the cost of orthogonalization by orthonormalizing only a selected fraction of the rows or columns of each momentum matrix, while Low-Rank Muon orthogonalizes a low-rank approximation of it. These methods target a different source of computational cost than Drop-Muon: they make the orthogonalization of every layer cheaper, but still update every layer at every step. Drop-Muon instead skips the updates of a prefix of layers, which reduces not only orthogonalization cost but also backward computation, gradient communication, and optimizer overhead. The two ideas are not mutually exclusive: layer dropping can in principle be applied on top of either optimizer, and evaluating such combinations is an interesting direction for future work. **Where Drop-Muon's savings come from.** Our profiling (Appendix H.7) shows an 836 ms reduction per step (12.9%), including 716 ms from backward computation, compared with only 54 ms from Newton–Schulz orthogonalization. Most of Drop-Muon's savings therefore come from computation that Dion2 and Low-Rank Muon do not target. **Direct comparison.** We trained NanoGPT-162M (0.5B tokens, four V100 GPUs) with Drop-Muon, Dion2, and Low-Rank Muon, using the authors' official implementations. The learning rate of every method was tuned, and the baselines additionally received a tuned AdamW learning rate for their embedding and output-head parameters, i.e., a larger tuning budget than Drop-Muon (five runs vs. two). The raw learning rates are parametrized differently: our implementation applies a fixed update scale of 50, so Drop-Muon's 3.6e-4 corresponds to an effective step of 0.018, comparable to the baselines' selected 0.02. Results: | Method | Final validation loss | Mean step time | |---|---:|---:| | Drop-Muon | 3.645 | 6.24 s | | Low-Rank Muon | 3.675 | 6.41 s | | Dion2 | 3.683 | 6.40 s | Drop-Muon attains the lowest validation loss from about one hour of training onward, reaching a lower final loss after the same number of steps and tokens, without taking longer to train. Full details, including the tuning grids and implementation settings, are in Appendix H.8. --- We thank the reviewer again for their constructive feedback and suggestions, which have helped us strengthen the paper. We believe that the additional experiments and theoretical clarifications address all the concerns raised in the review. In particular, we have clarified the theoretical assumptions, revised the convergence result, and provided additional empirical comparisons with existing methods. Given these substantial revisions and our efforts to address all the points raised, we would be extremely grateful if the reviewer could reconsider their overall assessment of the paper.

Please engage with the authors during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.