acceptodds
Under review as a conference paper at ICLR 2027

StructMuon: Compact Core Communication by Reusing Muon History Subspaces

Abstract

Muon improves large model training through momentum orthogonalization, but scaling requires efficient communication across devices. Low-rank compression can reduce communication costs, but existing methods compute their projection bases separately from the optimizer and must transmit these bases or recompute them with SVDs. Can the optimizer itself provide these bases? We find that Muon's momentum accumulates gradient history, while its orthogonalized update retains paired left and right singular directions. We extract history subspaces from these shared optimizer states, and each worker can reproduce them locally. We propose StructMuon, which uses these subspaces as communication bases and transmits a compact core. A residual channel exploits concentrated residual energy to send information outside these subspaces, while local error feedback retains unsent information. With contractive residual compression, we prove communication error contraction without SVD safeguards and derive a nonconvex stationarity bound that tightens with better subspace capture. Experiments on GPT-2 Large and NanoGPT-124M show 92.0-97.5% less communication than Full Muon synchronized once per update at matched validation perplexity. These results suggest that communication can reuse the optimizer's own structure.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Muon, communication compression, LLM pretraining, distributed optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xZQ4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed StructMuon, which reuses Muon's internal states as communication bases. Every worker can rebuild left and right bases locally, so no SVD or basis exchange is needed. On the theory side, the paper proves that communication error contracts whenever the residual compressor is contractive. It also gives a nonconvex stationarity bound whose error term shrinks as the subspaces capture more of the gradient. Experiments on GPT-2 Large and NanoGPT-124M demonstrated that at matched validation perplexity, StructMuon needs 92 - 97.5% less communication than Full Muon.

Critical strengths
  • The idea is well-motivated: The bases come from optimizer states every worker already shares, so only r*r values are sent and the basis-maintenance cost of PowerSGD is avoided.
  • Careful and honest evaluation: The paper uses five seeds, and reports wall-clock and bidirectional traffic. This comprehensive result makes the claims concrete.
Critical weaknesses
  • The practical gain over baseline is small: To reach the same targets, StructMuon is only 2–18% faster than Full Muon (see Table 12). Counting the dense broadcast, total traffic savings fall to about 47–49% (Table 13).
  • Key baselines are missing: Dion and PowerSGD+, which were designed for distributed Muon, are cited but not compared, making the experimental results less convincing.
Questions for authors
  • How does StructMuon compare against EF-PowerMuon with the same Top-k residual budget, and against Dion? These results would show whether the history bases themselves are responsible for the gains.
  • The bases require every worker to compute bitwise-identical momentum and Newton–Schulz outputs. How do you guarantee this under nondeterministic GPU kernels or mixed precision, and what happens to the training performances if bases vary across workers?
3 comments

Thank you for your careful reading. Time constraints limited the original experiments; we have now expanded the evaluation to address your concerns.

1. Practical runtime benefits.

The runtime benefits of compression depend on the setting. Agarwal et al. [1] found speedups in only 6 of over 200 configurations in typical data centers. The Dion paper also reports 14.5% longer training time than Muon in its 350M NanoGPT speedrun experiment [2].

Appendix Table 12 reports NanoGPT-124M times calculated from recorded local costs plus upload/download times. Targets use five-seed mean validation-loss curves. Both methods accumulate gradients before one upload per update, with the same effective batch size and one return per update.

Target PPLUplink / downlink (Gbps)Full Muon (min)StructMuon (min)Speedup
≤ 1805.80 / 18.6037.1717.242.16×
≤ 1804.35 / 13.9546.4717.362.68×
≤ 1802.90 / 9.3065.0717.603.70×
≤ 1305.80 / 18.6054.7823.412.34×
≤ 1304.35 / 13.9568.4923.572.91×
≤ 1302.90 / 9.3095.9023.894.01×
≤ 855.80 / 18.6078.1142.331.85×

With one upload, speedup at PPL ≤ 130 rises from 2.34× to 4.01× as bandwidth decreases. At reference bandwidth, the stricter PPL ≤ 85 target reduces the gain to 1.85×.

2. Communication savings including downlink traffic.

The original manuscript counted full-model broadcasts after each update. The revised protocol returns combined compressed messages once per update; workers reconstruct the aggregated update locally. Appendix Table 11 reports communication over 25K updates:

Uploads per updateMethodUplink (TB/worker)Downlink (TB/worker)Total (TB/worker)
FourFull Muon49.6312.4162.04
FourStructMuon0.380.270.65
OneFull Muon12.4112.4124.82
OneStructMuon0.100.130.23

Total communication falls by approximately 98.9% with four uploads and 99.1% with one upload. The runtime results above use this revised protocol.

3. Additional baseline comparisons.

We added EF-PowerMuon(Top-k) and Dion. EF-PowerMuon(Top-k) uses the same residual budget as StructMuon: k=150 on GPT-2 Large and k=100 on NanoGPT-124M. Table 1 reports the final PPL at 25K updates:

MethodGPT-2 Large PPL ↓NanoGPT-124M PPL ↓
Full Muon19.143.7
EF-PowerMuon(Top-k)40.1268.94
Dion46.279.4
StructMuon24.3654.84

StructMuon outperforms both compressed baselines, including EF-PowerMuon with the same Top-k budget.

PowerSGD+ [3] uses periodic SVD-based subspace refreshes. In our tested implementation, these caused out-of-memory (OOM) errors during large-model training, preventing completion.

4. Basis consistency across workers.

One worker can construct the bases and broadcast them to all other workers at each refresh, ensuring a shared coordinate system. Related work such as TSR-Adam [4] also uses explicit basis communication.

StructMuon constructs bases locally from shared optimizer states with consistent row selections, QR sign conventions, and refresh schedules. We measure cross-worker numerical differences and their effect on training. Appendix A.8, Figure 14(d) reports the deviation across the tested workers and matrix instances, with (r,k)=(123,100).

QuantityMaximum relative deviation
Momentum≈4.10×10⁻⁸
Newton–Schulz output≈1.20×10⁻⁶
Left basis≈1.20×10⁻⁶
Right basis≈1.10×10⁻⁷
Probe core≤1.42×10⁻⁶

On NanoGPT-124M/OpenWebText at (r,k)=(94,100), we compare local construction on each worker against construction on one worker followed by broadcast to the others. Local results at 10K and 15K correspond to Table 14:

UpdatesLocal TrainBroadcast TrainLocal ValBroadcast ValLocal PPLBroadcast PPL
5K5.1045.1024.5974.59599.2298.94
10K4.764.7474.224.20668.1567.11
15K4.564.5094.074.08058.4359.15

In practice, one can choose between local basis construction and construction on one worker followed by broadcasting, depending on the system setup.

References

[1] Agarwal et al. On the Utility of Gradient Compression in Distributed Training Systems. MLSys 2022.

[2] Ahn et al. Dion: Distributed Orthonormalized Updates. 2025.

[3] Xie, Chen, and Yuan. From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guarantees. 2025.

[4] Dang et al. From O(mn) to O(r²): Two-Sided Low-Rank Communication for Adam in Distributed Training with Memory Efficiency. 2026.

Dear Reviewer,

We have added the requested baseline comparisons and basis-consistency checks.

We would be grateful if you could review our response and share your feedback. Thank you for your time!

Dear Reviewer xZQ4,

As the discussion period is nearing its end, we would be grateful if you could revisit your assessment in light of our responses and the added experiments. Please let us know if any concerns remain; we would be happy to clarify.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_6bHBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

StructMuon compresses communication for Muon by reusing optimizer states as the projection bases. Each worker forms a right basis from rr fixed sampled rows of the shared effective momentum, Vt+1=qf((M~I,:t+1)⊤)V^{t+1} = \mathrm{qf}\big((\tilde M^{t+1}_{\mathcal I,:})^\top\big), and a left basis Ut+1=qf(Qt+1Vt+1)U^{t+1} = \mathrm{qf}(Q^{t+1} V^{t+1}) from the Muon direction Qt+1Q^{t+1}. It then transmits an r×rr \times r core Ait=(Ut)⊤ΔitVtA_i^t = (U^t)^\top \Delta_i^t V^t, where Δit=Git+Hit\Delta_i^t = G_i^t + H_i^t is the gradient plus local error-feedback memory, together with a Top-kk residual of (I−Πt)Δit(I - \Pi_t)\Delta_i^t; the reconstruction is Δ^it=Πt(Δit)+DtCt(Δit−Πt(Δit))\hat\Delta_i^t = \Pi_t(\Delta_i^t) + \mathcal D_t \mathcal C_t\big(\Delta_i^t - \Pi_t(\Delta_i^t)\big). Because the optimizer states are shared, bases are never transmitted and no SVD is needed.

Experiments are on GPT-2 Large and NanoGPT124M down to losses in the range of 3.4 on OpenWebText, with analysis showing StructMuon matches perplexity of full muon with 46% less communication.

Critical strengths

The core idea, that communication bases can be obtained from the optimizer for free, is strong, the presentation is clear, appendices are extensive, and the derivations appear sound.

Critical weaknesses

My main concerns are that the experiments are very limited; results at this model size, for this duration of training, simply do not translate to any practical training regime and hence applicability is very limited. While training large models may be out of reach for many groups, at minimum, a long duration training run should be carried out to correctly understand performance. It is also very important to understand if the method ever eventually reaches performance similar to full muon, or if it never does. There are many situations where one would be happy to trade slower convergence for communication efficiency, however if training dynamics are significantly damaged, the value prop becomes much reduced.

Related to the above, several baselines in the main experimental results appear weak - power-SGD, for example, effectively totally failing to learn, does not match my expectation of that algorithm and performance I have seen demonstrated elsewhere. The very large gaps between EF21-muon and full muon also seem larger than they should be this early in training. It leads me to believe the setup is not representative of the various algorithms true performance. This appears substatiated by the baseline in Table 12, where StructMuon’s time-to-same-loss is only 3.6–5.0% faster.

Questions for authors

Are you sure the baselines in figure 5 are tuned correctly? If so - how did you obtain the hyperparameters?

3 comments

Thank you very much for your constructive feedbacks. Time constraints limited the original experiments; we have now added complete training schedules and expanded the evaluation.

1. Learning-rate settings and tuning.

We separately retuned PowerSGD’s momentum SGD peak learning rate and selected 0.005.

Parameter group / methodCoarse searchFine searchSelected
Full Muon, EF-PowerMuon, EF-PowerMuon(Top-k), EF21-Muon(Top-k): main LR0.0002, 0.002, 0.01, 0.10.02, 0.03, 0.05, 0.080.02
Auxiliary AdamW groups0.00001, 0.00005, 0.0001, 0.0002, 0.00040.00016, 0.00018, 0.0002, 0.00022, 0.000240.0002
Dion: main LR0.003, 0.01, 0.02, 0.03—0.02
PowerSGD: momentum SGD main LR0.0001, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02—0.005

Comparisons use the same data splits, effective batch size, update count, and evaluation intervals, with five seeds.

2. Communication budgets and quality gaps.

EF21-Muon reports substantially impaired convergence with Top-5% compression, which removes approximately 91% of per-round uplink communication [1], while our main experiments reduce uplink by more than 95%. PowerSGD reports PPL 102 at rank 1 versus 91 for its uncompressed SGD baseline on WikiText-2 [2]. Our comparison uses Full Muon as the uncompressed baseline.

Increasing the budget improves every compressed method. Appendix Table 13 reports NanoGPT-124M results at 25K updates; q_ref=5 is the main budget:

Methodq_ref = 3: PPL ↓q_ref = 5: PPL ↓q_ref = 8: PPL ↓
Full Muon (uncompressed)43.7143.7143.71
PowerSGD149.5486.4064.18
EF-PowerMuon110.9172.9055.27
EF-PowerMuon(Top-k)98.5068.9451.84
EF21-Muon(Top-k)107.1074.8053.62
Dion113.8879.4057.43
StructMuon64.5254.8445.31

StructMuon leads the compressed methods at all three budgets, approaching Full Muon at the highest budget (45.31 vs. 43.71 PPL).

3. Complete training schedules and final performance.

Both models complete 25K optimizer updates, processing approximately 1.64B tokens each. Figures 5–6 and Table 1 report complete curves and final PPL at the main budgets:

MethodGPT-2 LargeNanoGPT-124M (q_ref=5)
Full Muon19.143.71
EF-PowerMuon(Top-k)40.1268.94
StructMuon24.3654.84

4. Communication savings at stricter quality targets.

At the main budget, StructMuon reaches PPL ≤ 85 on NanoGPT-124M and PPL ≤ 30 on GPT-2 Large with lower uplink communication than Full Muon. Targets are evaluated using the mean validation-loss curve over five seeds (Appendix A.5).

Model / targetFull Muon (TB/worker)StructMuon (TB/worker)
NanoGPT-124M, PPL ≤ 857.77570.0991
GPT-2 Large, PPL ≤ 3069.78364.1589

StructMuon is the only compressed method reaching GPT-2 Large PPL ≤ 30 within 25K updates. On NanoGPT-124M, all compressed methods except PowerSGD reach PPL ≤ 85.

5. Practical runtime benefits and bandwidth sensitivity.

Communication savings do not automatically imply faster training. Agarwal et al. found speedups in only 6 of over 200 configurations against optimized synchronous training in typical data centers [4]. Dion reports 14.5% longer training time than Muon in its 350M NanoGPT speedrun experiment [3].

Table 4 and Appendix Table 12 calculate NanoGPT-124M times from recorded local costs and upload/download times. Both methods use the same effective batch size and upload schedule, with one return per update; one upload means synchronization after gradient accumulation.

Target PPLUploads/updateUp/down (Gbps)Full Muon (min)StructMuon (min)Speedup
≤13015.80/18.6054.7823.412.34×
≤13045.80/18.60149.2131.854.68×
≤13012.90/9.3095.9023.894.01×
≤13042.90/9.30284.3433.388.52×
≤8515.80/18.6078.1142.331.85×
≤8545.80/18.60212.7557.953.67×

The gain is smaller with one upload and stricter targets: at reference bandwidth, speedup falls from 2.34× at PPL ≤130 to 1.85× at PPL ≤85. Thus, the benefit persists after accumulation, while its magnitude depends on bandwidth, synchronization frequency, and target quality.

References

[1] Gruntkowska et al. Error Feedback for Muon and Friends. 2025. https://arxiv.org/abs/2510.00643v1

[2] Vogels et al. PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization. NeurIPS 2019. https://arxiv.org/abs/1905.13727

[3] Ahn et al. Dion: Distributed Orthonormalized Updates. 2025. https://arxiv.org/abs/2504.05295v3

[4] Agarwal et al. On the Utility of Gradient Compression in Distributed Training Systems. MLSys 2022. https://arxiv.org/abs/2103.00543

Dear Reviewer,

We have added longer training results and clarified baseline tuning in response to your concerns.

We would appreciate your feedback on our response when you have a chance. Thank you so much !

Dear Reviewer 6bHB,

As the discussion period is nearing its end, we would be grateful if you could revisit your assessment in light of our responses and the added experiments. Please let us know if any concerns remain; we would be happy to clarify.

Thank you again for your time and consideration.

Best regards, The Authors

Reviewer_3cuVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a method to reduce communication in data-parallel (multi-gpu) Muon training of LLMs, and call it StructMuon. Fundamentally, they highlight that each worker builds the same communication bases from Muon’s own optimizer state, so workers only need to send a small core that describes the gradient in those bases. Since the bases miss part of the gradient, workers also send the largest few entries of the missing part and keep the rest locally for later steps, They then empirircally compare their StrucMuon with other compressed Muon methods, StructMuon reaches the same validation perplexity with less communication

Critical strengths
  • The problem overall seems well motivated and relevant, and it matters both in practice and in theory.
  • The core idea is simple and, as far as I can tell, novel
  • The choice of Muon is tested as well as argued and this seems consistent with assumption 1. I specifically refer to figure 2 where bases are built from Muon, AdamW, and SGD states on the same inputs and shows that Muon’s capture more of the gradient at every rank (using cosine similarity an relative error, this choice tho, isn’t fully clear to me, see weaknesses).
  • The theory rests mostly on standard assumptions, and in figure 4 they show empirically that, in practice, the fraction is much larger than the worst-case guarantee
  • The empirical results are promising as StructMuon sends much less data than uncompressed Muon. Among the compressed Muon methods it reaches each target perplexity with the least communication, and at the same message size, it gives a more accurate copy of the gradient and needs fewer steps (measured by cosine similarity and relative error). Also, with their ablations they show that the residual channel and error feedback each help.
  • The claims also feel generally well scoped.
Critical weaknesses
  • the authors report a speed up of StructMuon against full Muon synchronized at each microstep. But, in the appendix table12 StructMuon is only a few percent faster on GPT2 Large, which seems deliberately left out of the main text

  • Worth noting that prior work (https://arxiv.org/abs/2103.00543) has shown that gradient compression rarely speeds up training once it is compared against a well-optimized synchronous baseline at realistic bandwidth. Since all experiments run on a single node with four GPUs, where communication is rarely the bottleneck, I think the paper should either add multi-node or bandwidth-limited experiments, or state its claim in terms of data sent and move the once-per-update time comparison into the main text

  • Similar to my previous point, all experiments also use 4 workers and models up to 774M parameters, so it remains unclear how the method behaves as the number of workers grows.

  • On the empirical findings, it seems that the perplexity targets (table 2) are reached early, the main results are reported partway through the training schedule. So, by early, I mean that the models appear to still be far from the loss a fully trained model of the same size. So, matched perplexity holds only at these early targets, and ablations also stop at very high perplexity. I would like the paper to state how far the runs are from convergence clearly, say whether there is a reason beyond the trend to expect the gains to hold, and, if possible, report some smaller scale model to the end of the schedule

  • Regarding the theory, convergence appears to be only shown when momentum approaches 1 as training gets longer, but with fixed momentum, as in all experiments ran, the bound keeps a term for the part of the true gradient outside the history subspaces, and the authors note that whether this term decays remains open. I believe this matters because the paper presents the absence of SVD safeguards as an advantage, while PowerSGD+, which is cited, adds those safeguards to guarantee convergence. It'd help to say this trade-off and to measure the term during training, for example, by projecting a large-batch gradient onto the history subspaces at a few checkpoints

  • Cosine similarity and relative error are used as evidence of reconstruction quality (in figures 2 and 3), but the paper doesn’t really motivate them

Questions for authors
  • Dion is cited but not compared or discussed, how do the aims and design of StructMuon differ from Dion’s, and why was Dion not included as a baseline? Generally what was the criteria to choose some of the compression Muon baselines?
  • StructMuon communicates at every microstep, while full Muon can accumulate gradients locally and synchronize once per update, and since the core is linear in the input, compressing once per update could reduce StructMuon’s communication further. The paper notes that this would change the compressed inputs but does not test it. Have the authors tried this, and how do top-k and error feedback behave in that setting?
  • In the ablations, the residual channel improves validation loss only slightly on WikiText103 , but the improvement on OpenWebText is much larger (appendix). What do the authors think explains this difference, and which setting do they consider more representative?
  • top-k selects residual entries by magnitude, which keeps the most energy, but muon’s update gives every singular direction the same weight, so the entries that hold the most energy may not be the ones that matter most for the Muon step. Did the authors consider a selection rule that matches Muon’s geometry and would they expect it to help?
  • powerSGD baseline uses momentum sgd with the same learning rate as the Muon runs (table 7). Was this learning rate tuned for sgd? If not, this baseline may be weaker than it should be?
  • The ablations do not report the number of seeds or any measure of variation, do you have multipe seed runs?
7 comments
AuthorsOpenReview ↗

Part I: Runtime and training quality

Thank you for your careful reading and constructive comments. Time constraints prevented us from completing all planned experiments before submission; we have since expanded the evaluation. We respond in three parts: runtime and training quality; baseline comparisons and convergence; reconstruction metrics and ablations.

1. Practical runtime benefits and bandwidth sensitivity.

Runtime gains depend on computation and communication costs. Dion, for example, reports 14.5% longer training time than Muon in its 350M NanoGPT speedrun experiment [1].

Using a system communication setup similar to [2], Table 4 and Appendix Table 12 cover four uploads per optimizer update and one upload after gradient accumulation. NanoGPT-124M times are calculated from recorded local computation costs plus upload/download times. Target crossings use the mean validation-loss curve over five seeds. Within each comparison, both methods use the same upload schedule and effective batch size, with one return per update.

Target PPLUploads/updateUp/down (Gbps)Full Muon (min)StructMuon (min)Speedup
≤18015.80/18.6037.1717.242.16×
≤18012.90/9.3065.0717.603.70×
≤18045.80/18.60101.2422.254.55×
≤18042.90/9.30192.9323.318.28×
≤13012.90/9.3095.9023.894.01×
≤13042.90/9.30284.3433.388.52×
≤8515.80/18.6078.1142.331.85×
≤8545.80/18.60212.7557.953.67×

At the stricter PPL ≤85 target, the calculated speedup is smaller but remains 1.85× with one upload and 3.67× with four at reference bandwidth. Thus, the benefit persists when both methods accumulate gradients and synchronize once per update. Scaling to more workers and multiple nodes remains future work.

2. Longer training and the quality–communication trade-off.

Aggressive compression also affects prior methods: EF21-Muon reports impaired convergence when Top-k retains 5% of entries [3]; PowerSGD reports test PPL 102 at rank 1 versus 91 for uncompressed SGD on WikiText-2 [4]. Our main comparisons reduce uplink by more than 95% relative to Full Muon.

Figures 5–6 now cover the full 25K-update schedules, processing approximately 1.64B tokens per model. Table 1 reports five-seed results:

MethodGPT-2 Large ValPPLNanoGPT-124M ValPPL
Full Muon2.9519.13.7843.7
EF-PowerMuon(Top-k)3.6940.124.2368.94
EF21-Muon(Top-k)3.8044.74.3174.8
Dion3.8346.24.3779.4
StructMuon3.1924.364.0054.84

StructMuon retains its advantage over compressed baselines at the end of training, although a gap to Full Muon remains. Appendix Table 13 shows that this gap narrows as the budget increases:

NanoGPT-124M budgetStructMuon uplink (GB/worker)Final ValFinal PPL
q_ref=3263.914.1764.52
q_ref=5384.744.0054.84
q_ref=8561.513.8145.31

Here q_ref is the reference factor-rank budget; the main setting is q_ref=5. Full Muon reaches PPL 43.71.

The savings also persist at stricter targets (Appendix A.5), determined from mean validation-loss curves:

ModelTarget PPLFull Muon (TB/worker)StructMuon (TB/worker)Uplink reduction
GPT-2 Large≤3069.78364.158994.0%
NanoGPT-124M≤857.77570.099198.7%

The ablations also extend to 15K updates: StructMuon reaches PPL 138.38 on WikiText-103 and 58.43 on OpenWebText (Table 3 and Appendix Table 14).

These results address performance beyond early training. Part II explains the baseline comparisons and tuning, then addresses whether the convergence guarantee requires gradient omission to vanish.

References

[1] Ahn et al. Dion: Distributed Orthonormalized Updates. 2025. https://arxiv.org/abs/2504.05295v3

[2] Agarwal et al. On the Utility of Gradient Compression in Distributed Training Systems. MLSys 2022. https://arxiv.org/abs/2103.00543

[3] Gruntkowska et al. Error Feedback for Muon and Friends. 2025. https://arxiv.org/abs/2510.00643v1

[4] Vogels et al. PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization. NeurIPS 2019. https://arxiv.org/abs/1905.13727

AuthorsOpenReview ↗

Part II: Baseline comparisons and convergence

Part I reported runtime, final quality, and communication savings. We now explain the baseline design and learning-rate tuning, then address the theoretical effect of information omitted by the history bases.

3. Dion and baseline selection.

Dion constructs low-rank orthonormalized optimizer updates. StructMuon instead uses optimizer history to compress the input to Muon, with a residual channel and error feedback preserving omitted information. Our baselines cover factor compression with SGD and Muon, sparse compression with Muon, and Dion's low-rank updates; Appendix A.3 details their implementations.

Table 2 reports cumulative uplink per worker at matched validation PPL:

ModelTarget PPLDion (TB)StructMuon (TB)
GPT-2 Large1301.00440.7699
GPT-2 Large1101.18260.8699
NanoGPT-124M1600.06220.0440
NanoGPT-124M1400.07410.0507

At GPT-2 Large PPL 110 and NanoGPT-124M PPL 140, StructMuon uses 26.4% and 31.6% less uplink than Dion, respectively. Its final PPL is also lower on both models, as reported in Part I and Table 1.

4. Separate learning-rate tuning for PowerSGD.

We retuned PowerSGD's momentum SGD peak learning rate through the grid {0.0001, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02}. We selected 0.005, compared with 0.02 for the Muon methods (Appendix A.3).

5. Convergence guarantee and residual terms.

Non-vanishing residual terms in fixed-momentum stochastic bounds are not unique to StructMuon. They also appear in EF21-Muon [1], EF21-SGDM and its double-momentum variant [2], and uncompressed normalized SGD with momentum [3]. A residual term in an upper bound does not itself establish an unavoidable error floor.

Following the same distinction between fixed-parameter guarantees and vanishing bounds obtained through parameter selection, we refined our analysis. For a horizon of TT optimizer updates, choose

β=1−T−1/2,ηt=T−3/4,0≤t<T.\beta = 1-T^{-1/2}, \qquad \eta_t = T^{-3/4}, \qquad 0 \leq t < T.

Here β\beta is selected before training and held constant within each run. This choice establishes the asymptotic guarantee; the experiments use β=0.95\beta=0.95, as specified in Appendix A.3.

Under the stated assumptions, for fixed problem parameters and initial quantities and sufficiently large TT,

1T∑t=0T−1E[∑b=1B∥∇(b)f(Xt)∥∗]≤O(T−1/4)(1+σ)+O(T−1/2)ζ.\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\left[\sum_{b=1}^{B}\left\lVert\nabla^{(b)}f(X^t)\right\rVert_{*}\right] \leq \mathcal{O}(T^{-1/4})(1+\sigma)+\mathcal{O}(T^{-1/2})\zeta.

Here σ:=(∑b=1Bσb2)1/2\sigma := (\sum_{b=1}^{B}\sigma_b^2)^{1/2} measures stochastic gradient noise, and ζ\zeta measures worker heterogeneity. For fixed σ\sigma and ζ\zeta, the bound vanishes at an overall rate of O(T−1/4)\mathcal{O}(T^{-1/4}).

Error-feedback cancellation gives the omission term a factor of 1−β1-\beta. Projection nonexpansiveness then allows it to be absorbed without assuming that the omitted gradient vanishes. Remark 2 and Appendix A.2.5 give the statement and proof, including the case of momentum fixed independently of TT.

We also measure omission during training. Appendix A.8, Figure 14(a–c) reports the following at (r,k)=(123,100):

Quantity2K10K25K
Relative omitted gradient norm0.810.740.75
Omitted norm / early-stage mean1.000.470.24
Final Muon update cosine0.930.950.95
Final Muon update relative error0.380.300.31

Each stage uses 15 matrix instances across three shapes. A substantial fraction remains outside the history subspaces, while its absolute magnitude decreases; final updates maintain high directional alignment.

StructMuon uses residual compression and error feedback to control communication error when the history bases miss information, avoiding separate SVD refreshes. Its contraction guarantee does not require the bases alone to capture the entire gradient.

References

[1] Gruntkowska et al. Error Feedback for Muon and Friends. 2025. https://arxiv.org/abs/2510.00643v1

[2] Fatkhullin et al. Momentum Provably Improves Error Feedback!. NeurIPS 2023. https://arxiv.org/abs/2305.15155

[3] Cutkosky and Mehta. Momentum Improves Normalized SGD. ICML 2020. https://proceedings.mlr.press/v119/cutkosky20b.html

AuthorsOpenReview ↗

Part III: Reconstruction metrics and ablations

Part III of III: Reconstruction metrics and ablations

Part II explained how residual compression and error feedback control communication error despite incomplete history-subspace capture. We now clarify what the reconstruction metrics measure and test the contributions of these components.

6. Motivation for the reconstruction metrics.

Figures 2–3 compare representations of the same corrected input under limited communication budgets. For input Δ\Delta and reconstruction Δ^\widehat{\Delta},

Cos=⟨Δ^,Δ⟩F∥Δ^∥F∥Δ∥F,RelErr=∥Δ−Δ^∥F∥Δ∥F.\mathrm{Cos}=\frac{\langle\widehat{\Delta},\Delta\rangle_F}{\lVert\widehat{\Delta}\rVert_F\lVert\Delta\rVert_F},\qquad \mathrm{RelErr}=\frac{\lVert\Delta-\widehat{\Delta}\rVert_F}{\lVert\Delta\rVert_F}.

Cosine measures directional alignment; relative error normalizes the Frobenius reconstruction error in our communication analysis. For the orthogonal core projection Δ^=Πt(Δ)\widehat{\Delta}=\Pi_t(\Delta), when input and projection are nonzero,

Cos2=∥Πt(Δ)∥F2∥Δ∥F2,RelErr2=1−∥Πt(Δ)∥F2∥Δ∥F2.\mathrm{Cos}^2=\frac{\lVert\Pi_t(\Delta)\rVert_F^2}{\lVert\Delta\rVert_F^2},\qquad \mathrm{RelErr}^2=1-\frac{\lVert\Pi_t(\Delta)\rVert_F^2}{\lVert\Delta\rVert_F^2}.

Thus, the metrics express retained and omitted projection energy. Figure 2 compares optimizer-derived bases; Figure 3 compares representations under a shared budget. At r=94, the StructMuon core versus PowerSGD gives cosine 0.82 versus 0.75 and relative error 0.57 versus 0.66.

These metrics concern reconstruction before the Muon update. Figure 14(c) separately evaluates final update fidelity; validation PPL and communication to matched quality assess training performance.

7. Gradient accumulation and once-per-update compression.

We tested local gradient accumulation followed by one compression and synchronization per optimizer update. The accumulated gradient is added to the local EF buffer, then compressed into the history core and Top-k residual. The unsent remainder is retained for the next update. Both Top-k selection and EF updates occur once per optimizer update.

Figure 17 and Table 15 report five-seed means on NanoGPT-124M/OpenWebText at 10K updates, r=94 and k=100; Core-only sets k=0 and retains EF:

VariantVal lossPPLUplink (GB/worker)
Core-only + EF4.79119.7026.01
Without EF4.61100.3826.39
StructMuon4.4081.7026.39

Residual correction lowers loss from 4.79 to 4.40 for approximately 1.5% more uplink. EF lowers it from 4.61 to 4.40 at unchanged communication cost. Both components remain effective with once-per-update compression.

The residual also helps under four uploads per update (Figures 7 and 16). We hypothesize that OpenWebText's broader content diversity produces more variable gradient directions, making immediate residual correction more valuable. Its benefit can be smaller when the history core already captures useful directions well.

8. Why use magnitude-based Top-k for the residual?

Retaining the most gradient energy need not minimize distortion of the final Muon update. The history core uses momentum and Muon's orthogonalized updates; Top-k provides a low-cost correction to the remaining reconstruction error. It operates before the Muon update, rather than selecting singular directions of the final update.

For residual RR, magnitude-based Top-k is the best kk-entry approximation in Frobenius norm:

Top⁡k(R)∈arg⁡min⁡∥S∥0≤k∥R−S∥F2.\operatorname{Top}_k(R)\in\arg\min_{\lVert S\rVert_0\leq k}\lVert R-S\rVert_F^2.

It also satisfies the contraction property required by our communication-error analysis. Empirically, adding the residual lowers PPL from 206.44 to 190.38 on WikiText-103 and 80.43 to 68.15 on OpenWebText, for approximately 2.2% and 1.5% additional uplink, respectively (Figures 7 and 16).

Top-k is one effective choice within our framework, which supports residual compressors satisfying the contraction condition. Future work will explore compressors better suited to Muon's update geometry.

9. Multiple seeds and ablation variability.

The revised basis, residual, and EF ablations use five seeds. Figures 7, 16, and 17 show mean validation loss with ±1 sample SD bands. Figure 15 reports seed variability, with PPL bands obtained by exponentiating mean validation loss ±1 sample SD.

For example, Figure 7 reports WikiText-103 validation loss at r=94 and 10K updates:

VariantVal loss (mean ± SD)
Core-only + EF5.33 ± 0.07
Without EF5.36 ± 0.08
StructMuon5.25 ± 0.06

These results report both average improvements and variation across runs, complementing the training comparisons in Part I and the convergence analysis in Part II. Thank you for your reading.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.