acceptodds
Under review as a conference paper at ICLR 2027

Bridging Compressed Communication and Local Steps via Spectral Method in Federated Learning

Abstract

Federated Learning (FL) enables collaborative training of complex models across large-scale networks of edge devices while keeping data localized. However, data heterogeneity and high communication overhead remain fundamental bottlenecks. Recently, the spectral optimizer Muon has demonstrated remarkable advantages over Adam-like methods in centralized training. While recent efforts have attempted to transition Muon to FL, naive integrations into traditional architectures fail short. Crucially, existing approaches do not support the the communication compression and local steps — the two foundational techniques of efficient FL. To address this limitation, we introduce a novel spectral FL method that utilizes both local updates and compression. We establish rigorous convergence guarantees under mild assumptions, notably eliminating the restrictive bounded heterogeneity condition. Empirical evaluation on standard image classification benchmarks validates the efficiency and robustness of our approach.

Primary area:
optimization
Keywords:
non-convex optimization, stochastic optimization, federated learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_77ssOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims at extending Muon to federated settings. The authors propose FedCALM-D (deterministic) and FedCALM (stochastic with momentum). Both of them build on SCAFFOLD-style control variates to correct client drift during LMO local steps. Moreover, under L-smoothness and non-convexity assumption, the paper establishes convergence rates to stationary points. Experiments on CIFAR-10 (ResNet-18) and SST-2 (RoBERTa with LoRA) are used to validate their design.

Critical strengths

1.The problem is meaningful and well motivated. The compression and local steps analysis is a critical topic in extending Muon to FL settings.

2.The observations in Section 4.2 are valuable, which indicate that the communication parameters cannot be encoded into on message due to the processing of LMO.

3.The convergence rate is provided for both FedCALM-D (deterministic) and FedCALM (stochastic with momentum).

4.Limitations are discussed in the paper.

Critical weaknesses

1.The novelty of this work is limited. The ideal of control variates originates from SCAFFOLD, which is used in FedMuon. The compression strategy originates from EF/EF21 that used in SCAFCOM.

2.The convergence rates in Theorem 4.7 and Theorem 4.9 are independent of K, which is the local steps that claimed in the contributions.

3.The authors defined a spectral based compressor in Definition 4.1, but uses element-wise top-K compression strategy. These two strategies are inconsistent.

4.The experiments have several problems. For instance, the details of data partitions (e.g., Heterogeneous or Pathological) are not mentioned in the paper. lacking strongly heterogeneous experiment. Anomalies in experiment results, FedCALM with 1% compression achieves the accuracy of 84.6%, high than training with 100% parameters (82.6%).

5.The structure of the paper makes it harder to follow than necessary.

Questions for authors

1.In Muon, the singular values of the matrix after processing is equal to each other, hence β1\beta_1 of the compressor becomes ∞\infty.

2.The local steps KK is neither theoretical analyzed nor ablation studied in this work.

3.Does the element-wise TopK used in the experiments satisfy Definition 4.1?

4.Please provide the details of the baselines, for instance the number of local steps K. Moreover, an ablation study on K is critical for demonstrating the effectiveness of the proposed algorithm.

5.Explain Pathological / Heterogeneous in the experiments. Moreover, using more heterogeneous partitions (e.g., partition with Dirichlet distribution) and more participants to validate these algorithms.

6.Experiment results of LLM fin-tuning is missing.

7.Please carefully revise this manuscript. For instance, there are many typos in the paper and the references Figure 5.1 and Table 5.1 are not exist in the paper.

2 comments
AuthorsOpenReview ↗

Response to reviewer 77ss

Dear reviewer 77ss, we are thankful for the time and effort put into evaluating our work. Below, we gladly respond to the raised concerns.

  1. The novelty of the work (W1)

We respectfully disagree with the reviewer's characterization of our proposed method as merely a combination of existing techniques. Spectral methods represent a relatively new research direction, where much of the current literature focuses on incorporating Muon into established optimization frameworks. This trend extends to Federated Learning (FL), where recent works have begun adopting spectral schemes [1,2]. Both of the approaches apply only a single Muon update at the server to the averaged compressed messages, omitting local steps altogether. The papers introduce a method with a few variations and respective theoretical guarantees, without deeply investigating the underlying problem structure.

In contrast, our work thoroughly examines the core challenges of this setting and motivates every aspect of our algorithmic design in the Motivation section. Specifically, we introduce SCAFFOLD control variates to mitigate data drift across clients. Then, we demonstrate why two distinct compression mechanisms are required rather than a single one, and analyze which specific compression operators are best suited for each respective task. This systematic investigation shows that naively combining techniques fails to yield a viable algorithm.

Beyond algorithmic design, our convergence analysis introduces novel technical insights not found in prior work. Because the algorithm relies on two separate compression mechanisms, the proof requires tracking two coupled recursion equations. A central difficulty in the proof is preserving the exact dependence on the compression parameter β\beta. Initially, the first Error Feedback scheme introduces a factor of into the bounds established in Lemmas B.5 and B.6: ∣∣eir∣∣≤ηKβ(1+εq), ∣∣xr−xr−1∣∣≤6ηKβ.||e_i^{r} ||\leq \eta K \beta (1+\varepsilon_q), \ ||x^r - x^{r-1}|| \leq 6\eta K \beta. When these bounds are subsequently applied (Lines 1647–1652, 1704–1717), EF21 introduces an additional factor of β\beta. Preserving the exact power of β\beta is essential to match the rates in [3]. We achieve this through a delicate recursion tailored to EF21 (Lines 1647–1652): E∣∣cir−mir,0∣∣≤(1−1β)E∣∣cir−1−mir−1,0∣∣+E∣∣mir−1,K−mir−1,0∣∣.\mathbb{E}||c_i^r - m_i^{r,0}|| \leq (1-\frac1\beta)\mathbb{E}||c_i^{r-1} - m_i^{r-1,0}|| + \mathbb{E}||m_i^{r-1,K} - m_i^{r-1,0}||. In turn, bounding the momentum difference yields a factor of (1−(1−α)K)(1-(1-\alpha)^K): mir,K−mir,0=(1−(1−α)K)(∇fir,0−mir,0)+…m_i^{r,K} - m_i^{r,0} = (1-(1-\alpha)^K)(\nabla f_i^{r,0} - m_i^{r,0}) + \dots This factor elevates the exponent of α\alpha when we invoke Lemma B.9 to bound ∇fir,0−mir,0\nabla f_i^{r,0} - m_i^{r,0}. Consequently, the resulting variance term becomes non-leading and drops out of the dominant rate.

  1. Convergence over the local steps (W2)

We agree that since one of the key contributions of our paper is local steps implementation, it is crucial to demonstrate how it affects convergence. We examined the proof and found a way to improve the derived bound. To be precise, the original version of the derivation used a trivial bound $1-(1-\alpha)^K \geq \alpha $, while a tighter one holds: 1−(1−α)K≥12min⁡1-(1-\alpha)^K \geq \frac12 \min{1,αK1,\alpha K}. The inclusion of the new bound allows to introduce the number of local steps KK. The new version of the Theorem states R=O(β2ΔL∗ε2(1−εq)2+ρ2βΔL∗σ2ε4(1−εq)4MK).R = \mathcal{O}\left(\frac{\beta^2\Delta L_*}{\varepsilon^2(1-\varepsilon_q)^2} + \frac{{\rho^2\beta}\Delta L_*\sigma^2}{\varepsilon^4(1-\varepsilon_q)^4M{K}}\right). Now, the bound properly matches ones of SCAFCOM and FedMuon. The changes in the proof can be found in Lines 1894-1918.

We kindly ask the reviewer to follow our next Official Comment, in which we provide a detailed response for the remaining questions.

AuthorsOpenReview ↗

Response to reviewer 77ss (continuation)

  1. Compression in spectral norm (W3, Q1, Q3)

We agree with the reviewer that the Top-KK compressor does not strictly satisfy the theoretical definition of contractiveness, as demonstrated by the counterexample of an identity matrix. However, in practical scenarios, the singular value spectrum of the matrices deviates substantially from a single uniform value. To investigate this behavior, we include an ablation study on the compression constant in Figure 1.

Specifically, the contractiveness condition requires $ ||x-C(x)|| \leq (1-\frac1\beta) ||x|| $ (Eq (2)) to hold across all compressed matrices x. Rearranging this inequality yields an equivalent condition: 1β≤1−∣∣x−C(x)∣∣∣∣x∣∣⇔β≥(1−∣∣x−C(x)∣∣∣∣x∣∣)−1=defβ^(x).\frac1\beta \leq 1 - \frac{||x-C(x)||}{||x||} \Leftrightarrow \beta \geq (1 - \frac{||x-C(x)||}{||x||} )^{-1} \overset{\text{def}}{=} \hat{\beta}(x).

In our experiments, we recorded the spectral norms of the matrices before and after compression to empirically compute (x). Our findings show that remains bounded and is approximately equal to or smaller than the compression rate. Consequently, the applied compression operator behaves as practically contractive in real-world settings. We have incorporated these experimental results and an expanded discussion into Appendix A.3 of the revised manuscript.

  1. Experimental details (W4, Q2, Q4, Q5, Q6)

We provide the details of our experimental validation. The explicit data partitionings are described in Tables 10-13 in Appendix A.4. We found the optimal number of local steps to be 4−54-5 for all methods. LLM experiments are presented and discussed in the section Additional Experiments in Appendix A.1. The missing specifics of the setup are added to Appendix A.4 of the revised paper.

  1. More heterogeneous partition (Q5)

We agree with the reviewer that the experimental validation with a more heterogeneous environment would strengthen the paper. Below, we provide our results for Dirichlet distribution with 50 clients for CIFAR-10 setup.

Dirichlet (α\alpha=0.1)

MethodCompressionBest AccuracyUplinkTime
FedCALMTop1%0.6351.83 GB7.20 s
FedCALMTop10%0.64311.7 GB22.0 s
SCAFCOMTop1%0.443----
SCAFCOMTop10%0.399----
EF21-MuonTop1%0.409----
EF21-MuonTop10%0.426----
FedMuonTop100%0.664104 GB174 s
LocalSGDTop100%0.62196.0 GB159 s

Dirichlet (α\alpha=0.5)

MethodCompressionBest AccuracyUplinkTime
FedCALMTop1%0.8130.43 GB1.71 s
FedCALMTop10%0.8183.05 GB5.70 s
SCAFCOMTop1%0.482----
SCAFCOMTop10%0.56825.9 GB48.5 s
EF21-MuonTop1%0.309----
EF21-MuonTop10%0.427----
FedMuonTop100%0.81221.8 GB36.2 s
LocalSGDTop100%0.628218 GB362 s

Evidently, FedCALM demonstrates high final accuracy with FedMuon and LocalSGD as the only competitors. Moreover, the proposed method provides roughly ×10−×20\times10-\times20 improvement in terms of uplink and aggregation time to 50% accuracy. This serves as a strong validation of FedCALM’s performance in strongly heterogeneous settings. Those tables, as well as the convergence plots are added to the Appendix A.3 of the updated paper.

6. Structure of the paper (W5)

The lay out of our paper follows classic optimization structure. We provide the background in Introduction and Related Works, then describe our algorithmic contribution in Motivation and Algorithm sections, then transition to experimental validation. Many seminal works in optimization follow this template [1,2,3,4]. If the reviewer has any particular suggestions about improving the lay out, we will gladly take those into consideration.

We hope that our responses have thoroughly addressed all of the reviewer's concerns, and we remain open to any further discussion. Should the reviewer have no remaining questions, we kindly ask that they reconsider their evaluation of our work.


References

[1] Huang, Xinmeng, Ping Li, and Xiaoyun Li. "Stochastic controlled averaging for federated learning with communication compression." International Conference on Learning Representations. Vol. 2024. 2024.

[2] Mishchenko, Konstantin, et al. "Distributed learning with compressed gradient differences." Optimization Methods and Software 40.5 (2025): 1181-1196.

[3] Stich, Sebastian U. "Local SGD converges fast and communicates little." arXiv preprint arXiv:1805.09767 (2018).

[4] Bernstein, Jeremy, et al. "signSGD: Compressed optimisation for non-convex problems." International conference on machine learning. PMLR, 2018.

Reviewer_zuibOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FedCALM combines Muon, local steps, SCAFFOLD, and EF/EF21 compression for communication-efficient federated learning. The paper includes convergence analysis and experiments on CIFAR-10 and SST-2.

Critical strengths
  1. Reducing communication costs while allowing more local steps is useful for federated learning, especially when communication is expensive.
  2. Both convergence analysis and experiments are provided, with comparisons based on communication costs as well as accuracy.
Critical weaknesses
  1. FedCALM combines existing methods like Muon, local steps, SCAFFOLD and EF/EF21, but it's unclear what challenges this combination brings or what new benefits it offers. If it's mostly putting existing methods together, the novelty is limited.
  2. Issues with the convergence analysis. Eq. (2) and Remark 4.2 in Sec. 4.1 require contraction under both the spectral and nuclear norms, but this is not guaranteed by TopK. For example, after Top1% compression of a (10×10)(10\times10) identity matrix, both the original matrix and the compression error still have spectral norm 1, violating Eq. (2).
  3. FedCALM combines several existing components, but the experiments mainly test different numbers of Newton-Schulz iterations, with no ablations for SCAFFOLD, EF or EF21. It's unclear what each component adds, whether they're all needed, or if combining them brings any extra gains.
  4. The appendix proof doesn't match the main theorem. The second term in Theorem 4.9 (Sec. 4.4) contains (ρ2β)(\rho^2\beta), while Theorem B.13 has (ρ4)(\rho^4).
Questions for authors
  1. What does this combination actually add over existing methods? Is there anything beyond putting them together?
  2. Do the convergence guarantees hold for the TopK used in the experiments? And why do Theorem 4.9 and Theorem B.13 give different bounds?
  3. What does each of SCAFFOLD, EF and EF21 add? Are they all needed, and do they actually bring extra gains?
2 comments
AuthorsOpenReview ↗

Response to reviewer zuib

Dear reviewer zuib, thank their for the careful analysis of our paper.

To start with, we want to highlight one of the results of the rebuttal process: we clarified the convergence bound in Theorem 4.9, it now matches the baselines in terms of local steps. See details in the answer to Weakness 2 for reviewer 77ss and Lines (1894-1918) in the updated paper. Below, we are pleased to address the raised concerns.

  1. Combination of existing methods (W1, Q1, Q3)

We disagree with the reviewer saying that design of the proposed method comes down to putting together existing techniques. The spectral methods is a relatively new field of research, and a wide stream of works revolves around integration of Muon into well-known optimization paradigms. This holds for Federated Learning, with a number of papers adopting spectral schemes in FL settings [1,2]. Both papers only employ a single Muon/Gluon update at the server for the average of compressed messages, foregoing the local steps. Both works simply propose a method with a few variations and respective theoretical guarantees, without the deeper exploration of the setting. On the contrary, our paper studies the challenges of the problem and provides insight over the algorithmic design in the Motivation section (Section 4.2). First, we propose to use control variates of SCAFFOLD to overcome the client heterogeneity. Then, we explain why two separate compression mechanisms are necessary. Finally, we explore which compression schemes fit corresponding tasks. This comprehensive analysis suggests that naive combination of the techniques does not result in a feasible method. Beside the algorithmic contribution, the convergence proof contains insightful analysis, not present in the existing literature, and the necessity to maintain two recursion equations over two respective compression mechanisms facilitates a novel approach. One of the core challenges in the proof is to preserve dependence over compression rate β\beta. First Error Feedback scheme adds a factor of β\beta to the bounds in Lemmas B.5, B.6: ∣∣eir∣∣≤ηKβ(1+εq), ∣∣xr−xr−1∣∣≤6ηKβ.||e_i^{r} || \leq \eta K \beta (1+\varepsilon_q), \ ||x^r - x^{r-1}|| \leq 6\eta K \beta.
Later in the proof, when those bounds are employed, an additional factor of β\beta stems from EF21(Lines 1647-1652, 1704-1717). It is crucial to maintain the proper power of β\beta, which would match [3]. In the derivation, this is achieved via tricky recursion for EF21 (Lines 1647-1652):

E∣∣cir−mir,0∣∣≤(1−1β)E∣∣cir−1−mir−1,0∣∣+E∣∣mir−1,K−mir−1,0∣∣\mathbb{E}||c_i^r - m_i^{r,0}|| \leq (1-\frac1\beta) \mathbb{E}||c_i^{r-1} - m_i^{r-1,0}||+ \mathbb{E}||m_i^{r-1,K} - m_i^{r-1,0}||

The bound for the momentum difference, in turn, carries a factor (1−(1−α)K)(1-(1-\alpha)^K):

mir,K−mir,0=(1−(1−α)K)(∇fir,0−mir,0)+…m_i^{r,K} - m_i^{r,0} = (1-(1-\alpha)^K)(\nabla f_i^{r,0} - m_i^{r,0}) + \dots The factor allows to increase the power of α\alpha, while bringing the results from Lemma B.9 to bound ∇fir,0−mir,0\nabla f_i^{r,0} - m_i^{r,0}. This makes the variance term here non-leading, effectively removing it from the equation.

  1. Contractiveness under spectral norm (W2)

We agree that TopK compressor does not satisfy the contractiveness definition with a counterexample of an identity matrix. However, in practice the spectrum of matrices strongly differs from a single value. We provide an ablation study of the compression constant β\beta in our experiments in Figure 1. The contractiveness requires $ ||x-C(x)|| \leq (1-\frac1\beta) ||x|| $ (Eq (2)) to hold for all compressed matrices xx, which is equivalent to 1β≤1−∣∣x−C(x)∣∣∣∣x∣∣⇔β≥(1−∣∣x−C(x)∣∣∣∣x∣∣)−1=defβ^(x).\frac1\beta \leq 1 - \frac{||x-C(x)||}{||x||} \Leftrightarrow \beta \geq (1 - \frac{||x-C(x)||}{||x||} )^{-1} \overset{\text{def}}{=} \hat{\beta}(x). We gathered the spectral norm for the matrices before and after applying compression and used it to obtain β^(x)\hat{\beta}(x). Our study demonstrates that β^\hat{\beta} is bounded and approximately equal to or less than the compression rate. Thus, the utilized compression can be considered practically contractive. We added the corresponding figures and relevant discussion in Appendix A.3 of the revised paper.

We kindly ask the reviewer to follow our next Official Comment, where we address the remaining concerns.

AuthorsOpenReview ↗

Response to reviewer zuib (continuation)

  1. The impact of separate techniques (W3)

As discussed in the answer to Weakness 1, our paper provides the analysis of different compression schemes and explores which of them are suitable from the theoretical standpoint. However, we agree that an experimental validation for this claim would strengthen the paper. Below, we present the results for different versions of Error Feedback mechanisms for FedCALM. We compare EF21+EF21 run, where the model update is performed via EF21 (modified Lines 12-13 of Algorithm 2), and EF+EF, where the local control is updated with EF (modified Lines 8-10 of Algorithm 2). In the table we report best accuracy over 200 rounds in the setup of CIFAR-10 experiments with Dirichlet distribution (=0.1). The chosen mechanisms of Error Feedback provide best performance, which supports the motivation for the algorithmic design.

CombinationBest Accuracy
FedCALM EF+EF21 (Base)0.645
FedCALM EF21+EF210.600
FedCALM EF+EF0.606
  1. Inconsistent Theorem statement (W4, Q2)

We thank the reviewer for pointing out the discrepancy. This is a typo, and the correct value is ρ2β\rho^2\beta. This mistake was only in the theorem statement in the appendix, the proper bound with ρ2β\rho^2\beta is derived in the proof. The updated version of the paper now has a proper theorem statement, which includes improved rates in terms of local steps.

We hope these clarifications address all of the reviewer's concerns and remain available for further discussion. If no questions remain, we kindly request a reconsideration of the score.


References

[1] Gruntkowska, Kaja, et al. "Error feedback for muon and friends." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Qian, Xun, et al. "Communication-efficient gluon in federated learning." arXiv preprint arXiv:2604.10689 (2026).

[3] Gao, Yuan, et al. "Accelerated distributed optimization with compression and error feedback." arXiv preprint arXiv:2503.08427 (2025).

Reviewer_3YoNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedCALM-D and its stochastic extension, FedCALM, to combine Muon-style spectral updates with local training and communication compression in federated learning. The method uses control variates to address client drift, error feedback for compressed model updates. The experiments evaluate ResNet-18 on CIFAR-10 and LoRA fine-tuning of RoBERTa-base on SST-2, using 10 clients. The paper reports accuracy against communicated bits and provides additional communication and resource statistics.

Critical strengths
  1. The paper addresses a useful problem: combining local spectral updates with communication compression under heterogeneous client data.
Critical weaknesses
  1. Table 3 reports 44.7 MB and 407 ms per round for SCAFCOM Top10%, while Table 4 reports cumulative costs of 11,803 MB and 11.8 s to reach the target accuracy. The byte count implies approximately 264 rounds, whereas the time implies approximately 29 rounds. The former also exceeds the stated 200-round training budget.

  2. Algorithm 2 broadcasts both the global model and global control variate every round, while Appendix A.2 mainly reports uplink costs. Table 1’s compression ratios also do not explain whether sparse indices or the initial full-gradient communication are included.

  3. Both tasks use only 10 clients with full participation. Please include experiments with larger client populations, such as 50 and 100 clients, to show whether the accuracy and communication benefits hold as the number of devices increases. Experiments with partial participation would also help assess its practical applicability. The architecture coverage is limited to ResNet-18 and RoBERTa-base with LoRA adapters. Although this includes both CNNs and Transformers, it does not establish whether the gains extend to other architectures or full-model Transformer training.

Questions for authors
  1. Could you clarify the SCAFCOM Top10% numbers in Tables 3–4? Table 3 reports 44.7 MB and 407 ms per round, but Table 4 gives totals of 11,803 MB and 11.8 s. These seem to imply different numbers of rounds. Which entries need correcting?

  2. Could you confirm which CIFAR-10 split was used for each column of Table 1 and each panel of Figure 1? The descriptions in Appendix A.4 do not seem to match the labels in Tables 10–13.

  3. What exactly do the communication costs include? Do you count sparse indices, server-to-client messages, and the initial full-gradient transmission, or just the uploaded values? Also, were the reported times measured during training or estimated from the bandwidth and latency?

  4. For each model, how many parameters are trained, and how many are actually sent each round? Please give these numbers for both ResNet-18 and RoBERTa-LoRA. How many independent runs were used to obtain the reported accuracies?

2 comments
AuthorsOpenReview ↗

Response to reviewer 3YoN

Dear reviewer 3YoN, we would like to thank them for their efforts in evaluating our work. To start with, we want to highlight one of the results of the rebuttal process: we clarified the convergence bound in Theorem 4.9, it now matches the baselines in terms of local steps. See details in the answer to Weakness 2 for reviewer 77ss and Lines (1894-1918) in the updated paper.

Below, we address the concerns raised by reviewer 3YoN.

  1. Inconsistencies in Tables 3, 4 (W1,Q1).

This is an oversight on our part, thank their for bringing this to our attention. In this setup, SCAFCOM required 29 rounds to achieve 90% test accuracy with 44.7 MB per round on average, leading to 1296MB in total. While investigating the inconsistency, another miscalculation was found in the total FedMuon uplink. We replaced the previous values with the correct ones in Table 4 (Line 764) in the revised manuscript. We want to highlight that despite the inaccuracy the results still demonstrate the improved performance of FedCALM compared to baselines in terms of uplink memory, as well as training time.

2. Server broadcast (W2)

The foundational works in Federated learning establish the asymmetry of client and server communications [1]. In practice, downlink is much faster than uplink and is disregarded. Seminal works that study compression in Federated Learning do not consider the downlink, and only apply one-sided compression [2,3,4]. We adopt the same paradigm in our work. Here, we present the downlink memory values during training until 90% accuracy (same setting as Table 3). FedCALM requires less downlink than the direct competitor SCAFCOM, and other methods show comparable results. Overall, FedCALM outperforms all baselines in terms of uplink (expensive communication), while slightly overloading downlink (cheap communication), which leads to superior practical performance.

MethodCompressionDownlink
FedCALMTop1%14272 MB
FedCALMTop10%10728 MB
SCAFCOMTop1%52272 MB
SCAFCOMTop10%12959 MB
FedMuonTop100%9680 MB
LocalSGDTop100%13582 MB
Table 4 in the paper is now updated and also features downlink information.
  1. Communication costs clarifications (W2, Q3)

We agree that the details of the network load require more explanation. As discussed in the previous point, only client-to-server (uplink) costs are considered in our analysis. We clarified this in the revised version of the paper (Definition 4.6). Full gradient broadcasts at the starting point are not included in the costs. This careful initialization is mostly theoretical and allows to skip the derivation of some initialization terms which do not affect the leading terms of convergence, and makes the proof overall clearer. Both approaches are present in the literature. For comparison, we present the derived convergence bounds of SCAFCOM and FedMuon respectively: R=O(βΔLFε2+ΔLFσ2MKε4),R = \mathcal{O}\left(\frac{\beta\Delta L_F}{\varepsilon^2} + \frac{\Delta L_F\sigma^2}{MK\varepsilon^4} \right), R=O(βΔL∗ε2+ΔL∗ρ2σ2MKε4+σ0(1ε+ρ2σ2KL∗ΔMε2)).R = \mathcal{O}\left(\frac{\beta\Delta L_*}{\varepsilon^2} + \frac{\Delta L_*\rho^2\sigma^2}{MK\varepsilon^4} + \sigma_0 \left( \frac{1}{\varepsilon} + \frac{\rho^2\sigma^2K}{L_*\Delta M\varepsilon^2} \right) \right). In the bound for FedMuon, σ0\sigma_0 describes the error of initial approximation. If one sets it to zero, all of the bounds come to a uniform appearance. This signifies that exact initialisation does not inherently differ from stochastic one. In practice, we perform stochastic initialisation without full-gradient communications. Lastly, the compression rates describe the fraction of the weights transmitted. For example, Top1% compression for the ResNet-18 model of size 44.7 MB means that each client sends the compressed model of size 0.447 MB.

Regarding Weakness 3 and Questions 2-4, we gently ask the reviewer to follow our next Official Comment, in which we provide the detailed response.

AuthorsOpenReview ↗

Response to reviewer 3YoN (continuation)

  1. Additional experiments (W3)

We agree with the reviewer that additional experiments would strengthen the results. Below, we provide the experiment for a new setup: Dirichlet distribution across 50 clients for CIFAR-10. The uplink and aggregation time are calculated during the run to 50% accuracy. The absent values in the table correspond to the runs which did not meet target accuracy. Dirichlet (α\alpha=0.1)

MethodCompressionBest AccuracyUplinkTime
FedCALMTop1%0.6351.83 GB7.20 s
FedCALMTop10%0.64311.7 GB22.0 s
SCAFCOMTop1%0.443----
SCAFCOMTop10%0.399----
EF21-MuonTop1%0.409----
EF21-MuonTop10%0.426----
FedMuonTop100%0.664104 GB174 s
LocalSGDTop100%0.62196.0 GB159 s
Dirichlet (α\alpha=0.5)
MethodCompressionBest AccuracyUplinkTime
FedCALMTop1%0.8130.43 GB1.71 s
FedCALMTop10%0.8183.05 GB5.70 s
SCAFCOMTop1%0.482----
SCAFCOMTop10%0.56825.9 GB48.5 s
EF21-MuonTop1%0.309----
EF21-MuonTop10%0.427----
FedMuonTop100%0.81221.8 GB36.2 s
LocalSGDTop100%0.628218 GB362 s

Evidently, FedCALM demonstrates high final accuracy with FedMuon and LocalSGD as the only competitors. Moreover, the proposed method provides roughly (10--20) improvement in terms of uplink and aggregation time to 50% accuracy. This serves as a strong validation of FedCALM’s performance in strongly heterogeneous settings. Those tables, as well as the convergence plots are added to the updated paper. Regarding the full model fine-tuning, we believe that while this is a good suggestion from the reviewer, the compute workload for those experiments goes beyond the scope of the rebuttal time frame, considering the resources of our research center. We would not be able to carefully tune the results for a full model setup and run proper ablation studies.

5. Split clarification (Q2)

Thank their for pointing out the typos in the Tables 10-13 captions. We corrected those and the corresponding links in the text.

6. Time uplink (Q3)

We present the time derived from standard network statistics described in Appendix A.2 (Hardware). We consider the channel speed to be 100 Mb/s and latency to be 50 ms.

7.Experimental details (Q4)

For the ResNet-18 model, we used a full set of 11 689 512 parameters. For RoBERTa model we applied LoRA with rank 4 with a total of 1261826 trainable parameters. The accuracies are reported for a single run. We are currently working on repeated runs to provide trust intervals for the results. We are going to add those if the calculations are complete in the timespan of the rebuttal.

We hope that we have addressed all of the reviewer's concerns and remain open to further discussion. If the reviewer has no further questions, we kindly ask to reconsider the score.


References

[1] Konečný, Jakub, et al. "Federated learning: Strategies for improving communication efficiency." arXiv preprint arXiv:1610.05492 (2016).

[2] Huang, Xinmeng, Ping Li, and Xiaoyun Li. "Stochastic controlled averaging for federated learning with communication compression." International Conference on Learning Representations. Vol. 2024. 2024.

[3] Richtárik, Peter, Igor Sokolov, and Ilyas Fatkhullin. "EF21: A new, simpler, theoretically better, and practically faster error feedback." Advances in Neural Information Processing Systems 34 (2021): 4384-4396.

[4] Karimireddy, Sai Praneeth, et al. "Error feedback fixes signsgd and other gradient compression schemes." International conference on machine learning. PMLR, 2019.

Comments on the paper

1 comment

Dear Area Chair,

We would like to provide a brief summary of the review process, in case it may be helpful to them.

The reviewers highlighted some weaknesses of our paper, which allowed us to address those and strengthen our result. Most notably, reviewer 77ss identified a missing factor of local steps number KK in our convergence analysis of Theorem 4.7. We conducted a tighter analysis and obtained a proper factor. Now, the bound clearly matches ones of SCAFCOM and FedMuon, main competitors of our method.

Other issues highlighted by the reviewers overlap, mainly revolving around the following topics.

Reviewers zuib and 77ss raised concerns about the novelty of FedCALM. We argue that the method is more than a combination of existing methods. Our paper studies the intricacies of the setting and explains why certain design choices are unavoidable, such as using two separate compression mechanisms.

Reviewers zuib and 77ss pointed out that our definition of contractive compressor clashes with implemented TopK compression from the theoretical standpoint: the spectral norm of the identity matrix does not decrease after the compression. However, as we studied in our ablation experiments, this is not the case in practice. Both spectral and nuclear norm decrease due to TopK compression, allowing for a bounded compression rate constant. We added the corresponding experiments to the revised paper.

We acknowledged the limitations of our experimental setup, brought up by reviewers 3YoN and 77ss, and provided a setting with a larger number of clients and stronger heterogeneity (Dirichlet distribution). The new experiments serve as a strong validation of FedCALM’s performance in strongly heterogeneous settings.

Unfortunately, the reviewers did not engage in the discussion and did not answer to our responses. We hope the provided summary will reduce the workload put on the Area Chair.

Kind Regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.