- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims at extending Muon to federated settings. The authors propose FedCALM-D (deterministic) and FedCALM (stochastic with momentum). Both of them build on SCAFFOLD-style control variates to correct client drift during LMO local steps. Moreover, under L-smoothness and non-convexity assumption, the paper establishes convergence rates to stationary points. Experiments on CIFAR-10 (ResNet-18) and SST-2 (RoBERTa with LoRA) are used to validate their design.
Critical strengths
1.The problem is meaningful and well motivated. The compression and local steps analysis is a critical topic in extending Muon to FL settings.
2.The observations in Section 4.2 are valuable, which indicate that the communication parameters cannot be encoded into on message due to the processing of LMO.
3.The convergence rate is provided for both FedCALM-D (deterministic) and FedCALM (stochastic with momentum).
4.Limitations are discussed in the paper.
Critical weaknesses
1.The novelty of this work is limited. The ideal of control variates originates from SCAFFOLD, which is used in FedMuon. The compression strategy originates from EF/EF21 that used in SCAFCOM.
2.The convergence rates in Theorem 4.7 and Theorem 4.9 are independent of K, which is the local steps that claimed in the contributions.
3.The authors defined a spectral based compressor in Definition 4.1, but uses element-wise top-K compression strategy. These two strategies are inconsistent.
4.The experiments have several problems. For instance, the details of data partitions (e.g., Heterogeneous or Pathological) are not mentioned in the paper. lacking strongly heterogeneous experiment. Anomalies in experiment results, FedCALM with 1% compression achieves the accuracy of 84.6%, high than training with 100% parameters (82.6%).
5.The structure of the paper makes it harder to follow than necessary.
Questions for authors
1.In Muon, the singular values of the matrix after processing is equal to each other, hence of the compressor becomes .
2.The local steps is neither theoretical analyzed nor ablation studied in this work.
3.Does the element-wise TopK used in the experiments satisfy Definition 4.1?
4.Please provide the details of the baselines, for instance the number of local steps K. Moreover, an ablation study on K is critical for demonstrating the effectiveness of the proposed algorithm.
5.Explain Pathological / Heterogeneous in the experiments. Moreover, using more heterogeneous partitions (e.g., partition with Dirichlet distribution) and more participants to validate these algorithms.
6.Experiment results of LLM fin-tuning is missing.
7.Please carefully revise this manuscript. For instance, there are many typos in the paper and the references Figure 5.1 and Table 5.1 are not exist in the paper.
2 comments
Response to reviewer 77ss (continuation)
- Compression in spectral norm (W3, Q1, Q3)
We agree with the reviewer that the Top- compressor does not strictly satisfy the theoretical definition of contractiveness, as demonstrated by the counterexample of an identity matrix. However, in practical scenarios, the singular value spectrum of the matrices deviates substantially from a single uniform value. To investigate this behavior, we include an ablation study on the compression constant in Figure 1.
Specifically, the contractiveness condition requires $ ||x-C(x)|| \leq (1-\frac1\beta) ||x|| $ (Eq (2)) to hold across all compressed matrices x. Rearranging this inequality yields an equivalent condition:
In our experiments, we recorded the spectral norms of the matrices before and after compression to empirically compute (x). Our findings show that remains bounded and is approximately equal to or smaller than the compression rate. Consequently, the applied compression operator behaves as practically contractive in real-world settings. We have incorporated these experimental results and an expanded discussion into Appendix A.3 of the revised manuscript.
- Experimental details (W4, Q2, Q4, Q5, Q6)
We provide the details of our experimental validation. The explicit data partitionings are described in Tables 10-13 in Appendix A.4. We found the optimal number of local steps to be for all methods. LLM experiments are presented and discussed in the section Additional Experiments in Appendix A.1. The missing specifics of the setup are added to Appendix A.4 of the revised paper.
- More heterogeneous partition (Q5)
We agree with the reviewer that the experimental validation with a more heterogeneous environment would strengthen the paper. Below, we provide our results for Dirichlet distribution with 50 clients for CIFAR-10 setup.
Dirichlet (=0.1)
| Method | Compression | Best Accuracy | Uplink | Time |
|---|---|---|---|---|
| FedCALM | Top1% | 0.635 | 1.83 GB | 7.20 s |
| FedCALM | Top10% | 0.643 | 11.7 GB | 22.0 s |
| SCAFCOM | Top1% | 0.443 | -- | -- |
| SCAFCOM | Top10% | 0.399 | -- | -- |
| EF21-Muon | Top1% | 0.409 | -- | -- |
| EF21-Muon | Top10% | 0.426 | -- | -- |
| FedMuon | Top100% | 0.664 | 104 GB | 174 s |
| LocalSGD | Top100% | 0.621 | 96.0 GB | 159 s |
Dirichlet (=0.5)
| Method | Compression | Best Accuracy | Uplink | Time |
|---|---|---|---|---|
| FedCALM | Top1% | 0.813 | 0.43 GB | 1.71 s |
| FedCALM | Top10% | 0.818 | 3.05 GB | 5.70 s |
| SCAFCOM | Top1% | 0.482 | -- | -- |
| SCAFCOM | Top10% | 0.568 | 25.9 GB | 48.5 s |
| EF21-Muon | Top1% | 0.309 | -- | -- |
| EF21-Muon | Top10% | 0.427 | -- | -- |
| FedMuon | Top100% | 0.812 | 21.8 GB | 36.2 s |
| LocalSGD | Top100% | 0.628 | 218 GB | 362 s |
Evidently, FedCALM demonstrates high final accuracy with FedMuon and LocalSGD as the only competitors. Moreover, the proposed method provides roughly improvement in terms of uplink and aggregation time to 50% accuracy. This serves as a strong validation of FedCALM’s performance in strongly heterogeneous settings. Those tables, as well as the convergence plots are added to the Appendix A.3 of the updated paper.
6. Structure of the paper (W5)
The lay out of our paper follows classic optimization structure. We provide the background in Introduction and Related Works, then describe our algorithmic contribution in Motivation and Algorithm sections, then transition to experimental validation. Many seminal works in optimization follow this template [1,2,3,4]. If the reviewer has any particular suggestions about improving the lay out, we will gladly take those into consideration.
We hope that our responses have thoroughly addressed all of the reviewer's concerns, and we remain open to any further discussion. Should the reviewer have no remaining questions, we kindly ask that they reconsider their evaluation of our work.
References
[1] Huang, Xinmeng, Ping Li, and Xiaoyun Li. "Stochastic controlled averaging for federated learning with communication compression." International Conference on Learning Representations. Vol. 2024. 2024.
[2] Mishchenko, Konstantin, et al. "Distributed learning with compressed gradient differences." Optimization Methods and Software 40.5 (2025): 1181-1196.
[3] Stich, Sebastian U. "Local SGD converges fast and communicates little." arXiv preprint arXiv:1805.09767 (2018).
[4] Bernstein, Jeremy, et al. "signSGD: Compressed optimisation for non-convex problems." International conference on machine learning. PMLR, 2018.
Response to reviewer 77ss
Dear reviewer 77ss, we are thankful for the time and effort put into evaluating our work. Below, we gladly respond to the raised concerns.
We respectfully disagree with the reviewer's characterization of our proposed method as merely a combination of existing techniques. Spectral methods represent a relatively new research direction, where much of the current literature focuses on incorporating Muon into established optimization frameworks. This trend extends to Federated Learning (FL), where recent works have begun adopting spectral schemes [1,2]. Both of the approaches apply only a single Muon update at the server to the averaged compressed messages, omitting local steps altogether. The papers introduce a method with a few variations and respective theoretical guarantees, without deeply investigating the underlying problem structure.
In contrast, our work thoroughly examines the core challenges of this setting and motivates every aspect of our algorithmic design in the Motivation section. Specifically, we introduce SCAFFOLD control variates to mitigate data drift across clients. Then, we demonstrate why two distinct compression mechanisms are required rather than a single one, and analyze which specific compression operators are best suited for each respective task. This systematic investigation shows that naively combining techniques fails to yield a viable algorithm.
Beyond algorithmic design, our convergence analysis introduces novel technical insights not found in prior work. Because the algorithm relies on two separate compression mechanisms, the proof requires tracking two coupled recursion equations. A central difficulty in the proof is preserving the exact dependence on the compression parameter β. Initially, the first Error Feedback scheme introduces a factor of into the bounds established in Lemmas B.5 and B.6: ∣∣eir∣∣≤ηKβ(1+εq), ∣∣xr−xr−1∣∣≤6ηKβ. When these bounds are subsequently applied (Lines 1647–1652, 1704–1717), EF21 introduces an additional factor of β. Preserving the exact power of β is essential to match the rates in [3]. We achieve this through a delicate recursion tailored to EF21 (Lines 1647–1652): E∣∣cir−mir,0∣∣≤(1−β1)E∣∣cir−1−mir−1,0∣∣+E∣∣mir−1,K−mir−1,0∣∣. In turn, bounding the momentum difference yields a factor of (1−(1−α)K): mir,K−mir,0=(1−(1−α)K)(∇fir,0−mir,0)+… This factor elevates the exponent of α when we invoke Lemma B.9 to bound ∇fir,0−mir,0. Consequently, the resulting variance term becomes non-leading and drops out of the dominant rate.
We agree that since one of the key contributions of our paper is local steps implementation, it is crucial to demonstrate how it affects convergence. We examined the proof and found a way to improve the derived bound. To be precise, the original version of the derivation used a trivial bound $1-(1-\alpha)^K \geq \alpha $, while a tighter one holds: 1−(1−α)K≥21min{1,αK}. The inclusion of the new bound allows to introduce the number of local steps K. The new version of the Theorem states R=O(ε2(1−εq)2β2ΔL∗+ε4(1−εq)4MKρ2βΔL∗σ2). Now, the bound properly matches ones of SCAFCOM and FedMuon. The changes in the proof can be found in Lines 1894-1918.
We kindly ask the reviewer to follow our next Official Comment, in which we provide a detailed response for the remaining questions.