acceptodds
Under review as a conference paper at ICLR 2027

FMuon: Coordinated Momentum Orthogonalization for Federated Training of Large Models

Abstract

Existing Federated Learning (FL) methods still primarily use element-wise local optimizers (e.g. AdamW, SGD) in each client, neglecting the geometric structure of the weight matrices. This often leads to the amplification of pathological directions in the weights during local updates, leading to deterioration in the condition number and slow convergence. Therefore, we introduce the Muon optimizer in each client (named Local Muon), which has matrix orthogonalization to optimize matrix-structured parameters. Experimental results show that, in the IID setting, Local Muon significantly accelerates the convergence of FL and reduces communication rounds compared to Local SGD and Local AdamW. However, in the non-IID setting, independent matrix orthogonalization based on the local distributions of each client induces strong client drift. Applying Muon in non-IID FL poses significant challenges: (1) Local–global lmo mismatch leading to client drift; (2) Momentum reinitialization. To address these challenges, we propose a novel Federated Muon optimizer (FMuon), which incorporates two key techniques: (1) momentum aggregation, where clients use the aggregated momentum for local initialization; (2) local-global alignment, where the local gradients are aligned with the global update direction to significantly reduce client drift. Theoretically, we establish a convergence guarantee for FMuon. Empirically, we validate the effectiveness of FMuon on language and vision models. Compared to several baselines including Local Muon, FMuon significantly reduces communication rounds and improves test accuracy. The code is available in https://anonymous.4open.science/r/FedMuon-935D.

Primary area:
optimization
Keywords:
Federated Learning, Muon Optimizer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jXiXOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

They propose FMuon, a federated adaptive optimization method that combines Muon optimizer on local devices, with shared momentum initialization, and gradient drift correction to align edge devices with global updates. They provided robust convergence gaurantees with evaluations across vision and language model tasks under data heterogeneity.

Critical strengths

The paper reports consistent improvements over the closely related FedMuon method, including gains of around 3 percent on ResNet-18 and ViT-Tiny, under the Dirichlet-0.1 setting in Table 1. The evaluation across larger ViT and GPT-2 models further supports the practical value of the proposed coordination mechanism.

Critical weaknesses
  1. The authors should modify the abstract and introduction to position the contribution more clearly relative to prior work. FedMuon [1] already studies Muon in heterogeneous federated learning, identifies the bias arising from local LMO updates, and proposes a correction with convergence guarantees. Although the submission acknowledges this relationship in Section 2, its opening presentation can give the impression that introducing Muon into federated learning and addressing the resulting drift are themselves new contributions. This creates tension with the more precise positioning later in the paper. The abstract and introduction should foreground the actual distinctions on momentum and post-LMO operations.

[1] Takezawa, Yuki, et al. "FedMuon: Federated learning with bias-corrected LMO-based optimization." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

Question on the source of communication gains: Tables 12/13 and the compression ablation provide useful evidence of communication efficiency. Could the authors clarify how much of the advantage over FedMuon comes from the proposed coordination mechanism versus compressing the communicated momentum? Since FedMuon’s communicated control variates are also derived from local momentum, could comparable low-rank compression benefit that method? A comparison under matched communication budgets, or a discussion of why compression is particularly suitable for FMuon, would help establish the source of the gains.

3 comments

W1. Positioning relative to FedMuon

Thank you for pointing out that the abstract and introduction could state our contribution more precisely. FedMuon identified the bias arising from local linear minimization oracle (LMO) updates in heterogeneous federated learning and proposed a control-variate correction with convergence guarantees. We will acknowledge this prior contribution in both sections and clarify the algorithmic differences. FedMuon retains client-specific momentum across rounds and uses control variates to correct the LMO input. FMuon initializes participating clients with the previous round’s aggregated momentum and mixes each local LMO output with the previous global update direction.

Algorithmic aspectFedMuon (Takezawa et al., ICLR 2026)FMuon (ours)
Local update directionPre-LMO correction. LMO(Mir,k+1−Cir+Cr)\mathrm{LMO}(M_i^{r,k+1}-C_i^r+C^r)Post-LMO alignment. (1−α)LMO(Mir,k+1)−αΔGr(1-\alpha)\mathrm{LMO}(M_i^{r,k+1})-\alpha\Delta_G^r
Momentum initializationEach selected client resumes its own stored momentum, Mir,0←Mir−1,KM_i^{r,0}\leftarrow M_i^{r-1,K}.All selected clients start from the same aggregated momentum, Mir,0←MˉrM_i^{r,0}\leftarrow\bar M^r.
Server auxiliary stateCr+1=Cr+1N∑i∈Sr(Cir+1−Cir)C^{r+1}=C^r+\frac{1}{N}\sum_{i\in S_r}(C_i^{r+1}-C_i^r), where Cir+1=Mir,KC_i^{r+1}=M_i^{r,K} for selected clients.Mˉr+1=1S∑i∈SrMir,K\bar M^{r+1}=\frac{1}{S}\sum_{i\in S_r}M_i^{r,K} and ΔGr+1=−1SKη∑i∈Srδir\Delta_G^{r+1}=-\frac{1}{SK\eta}\sum_{i\in S_r}\delta_i^r.
Model aggregationxr+1=xr+1N∑i∈Srδirx^{r+1}=x^r+\frac{1}{N}\sum_{i\in S_r}\delta_i^rxr+1=xr+1S∑i∈Srδirx^{r+1}=x^r+\frac{1}{S}\sum_{i\in S_r}\delta_i^r

Here, δir=xir,K−xir,0\delta_i^r=x_i^{r,K}-x_i^{r,0}, NN is the total number of clients, and SS is the number selected in round rr.

Q1. Where do the communication gains come from?

Thank you for suggesting a comparison with compressed FedMuon. We applied truncated SVD to FedMuon’s communicated control variates and matched the per-round payload of FMuon_SVD at 23.9 MB. The following results are for CIFAR-100 with ViT-Tiny under Dir-0.1 after 300 communication rounds.

MethodCommunicated auxiliary statePayload (MB/round) ↓Final test accuracy (%) ↑
FedMuonDense control variates45.645.17
FMuonDense shared momentum45.648.18
FedMuon + SVDLow-rank control variates23.944.28
FMuon_SVDLow-rank shared momentum23.947.68

With dense auxiliary states and equal per-round payloads, FMuon achieves 3.01 percentage points higher final accuracy than FedMuon. Compression reduces the per-round payload by 47.6% for both methods, with accuracy drops of 0.89 points for FedMuon and 0.50 points for FMuon. At the matched compressed payload of 23.9 MB per round, FMuon_SVD achieves 3.40 points higher final accuracy than FedMuon + SVD.

MethodPayload (MB/round) ↓Cumulative payload to reach 23% test accuracy (MB) ↓
FedMuon + SVD23.91,075
FMuon_SVD23.9550

At the same per-round payload, FMuon_SVD reaches 23% test accuracy using approximately 49% less cumulative communication because it reaches the target in fewer rounds.

One possible explanation for the smaller accuracy drop is how the two methods use the reconstructed state. FMuon uses reconstructed shared momentum to initialize the local optimizer at the start of each round and then updates it with fresh local gradients. FedMuon’s reconstructed global control variate CrC^r remains fixed throughout the round and enters the LMO input directly at every local step.

Thank you for the clear acknowledgment of FedMuon and the additional matched-payload experiments. Please incorporate these results and the clarified positioning into the revised manuscript, and scope the 49% communication reduction to the reported accuracy target. My concerns are largely resolved, and I maintain my acceptance recommendation.

Thank you for reading our follow-up and for suggesting the matched-payload comparison. We’ll include the new results in the revision, clarify how FMuon differs from FedMuon, and specify that the 49% reduction refers to the cumulative payload needed to reach 23% test accuracy in this experiment. Thank you again!

Reviewer_x5DYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to improve model training in federated learning (FL) by extending Muon to the federated setting, proposing a new algorithm, namely FMuon. Specifically, FMuon uses the direction of the global update, i.e., the update averaged across participating clients, to reduce the client drift and address non-IID datain FL. In addition, it initializes each local Muon optimizer with the averaged momentum, which accelerates local training. To reduce communication overhead, FMuon applies SVD to the resulting momentum before communication to reduce the transmitted dimension.

The paper provides a convergence analysis of FMuon and evaluates its performance under Dirichlet data partition.

Critical strengths

1.Clear motivation. This paper aims at solving the challenges of Local-global LMO mismatch and momentum reinitialization in federated Muon, which is clearly stated in the introduction.

2.Providing convergent analysis of Local Muon and FMuon, revealing that the gradient converges to 0 in proposed FMuon.

3.The SVD-based solution does not introduce much communication overhead.

4.Evaluating FMuon with multiple tasks, including vision and language.

Critical weaknesses

1.The SVD-based mechanism is difficult to extend to high-dimensional models. In that case, the proposed FMuon still incurs 2$\times$ the communication overhead when training these models.

2.After orthogonalization, the spectrum of each matrix becomes more balanced. As a result, using SVD to reduce communication overhead discards much more information, which should cause a larger performance loss. However, the authors only provide an empirical analysis of this phenomenon.

3.The experimental results contain anomalies. For instance, in Table 1, the loss of Local Muon is 1.504, nearly three times that of FedLADA, yet its accuracy is 7% higher. This discrepancy should be analyzed.

4.The experiments only use two non-iid cases: dirichlet-0.1 and dirichlet 0.6, which are not enough to comprehensively evaluate the performance of FMuon.

Questions for authors

1.FedMuon, which is closely related to FMuon, should be included in Figure 3 and Figure 4.

2.The comparison betweem Theorem 5.4 and Theorem 5.5 is unfair because Theorem 5.3 uses 1−β=R−1/21-\beta = R^{-1/2}, which is a diminishing term. This difference should be discussed in the paper.

3.Because computation time changes with the number of local steps, it gives a biased measure of the SVD overhead.

4.The notation in this paper is difficult to follow. For instance, GtG_t denotes the gradient estimation in Section 4 but represents the dual norm of the gradients in Section 5. Moreover, the gradient is denoted by vv in Section 3.

5.The text in the figures are too small to read. They should be at least as large as the font size of the main body text.

6.The abbreviation LMO is used in the abstract and introduction but is not interpreted until Section 3.1.

7.Figure 2.(b) is misleading, dimension dd is not changed during SVD-based dimension reduction.

2 comments

Responses to the Main Weaknesses

W1. Scalability of SVD-based momentum compression

Thank you for raising this concern. Transmitting dense momentum approximately doubles the communication payload relative to sending the model update alone. Randomized truncated SVD offers a way to compute the leading singular components more efficiently while retaining the communication savings of low-rank transmission. Q3 reports a direct timing comparison at the same retained rank.

Scalability on LLaMA-350M with C4

To evaluate momentum compression at a larger model scale, we compare Local Muon, FMuon with dense shared momentum, and two compressed variants on C4 using the same training protocol. FMuon_SVD uses standard SVD followed by truncation, and FMuon_RSVD uses randomized truncated SVD. Both variants retain 2.5% of the singular components. We report the time spent on SVD-based compression per round, communication cost relative to Local Muon, and validation perplexity at the same communication round.

MethodSVD computation time (s/round) ↓Relative communication cost ↓C4 validation perplexity ↓
Local Muon01.00×3.596
FMuon02.00×3.223
FMuon_SVD25.361.05×3.240
FMuon_RSVD1.92≈1.05×3.246

Randomized truncated SVD reduces compression time from 25.36 s to 1.92 s per round (92.4% less time) at approximately the same communication cost as standard SVD. Validation perplexity increases by 0.006 relative to standard SVD and by 0.023 relative to dense FMuon. FMuon_RSVD thus reduces the per-round communication cost from 2.00× to approximately 1.05× the Local Muon baseline, with a small increase in validation perplexity relative to dense FMuon.

For local Muon updates, we use Newton–Schulz iterations to approximate orthogonalization. SVD is used for compressing the momentum to be transmitted. In our timing measurements, Local Muon took approximately 5% more computation time than Local AdamW.

W2. Spectral flattening and momentum compression

Thank you for raising this concern. In FMuon, Newton–Schulz iterations transform the momentum matrix into a local update direction. FMuon_SVD compresses the momentum matrix before orthogonalization. The spectral flattening of the local update direction therefore does not directly determine the compressibility of the transmitted momentum.

We evaluated sensitivity to the retained rank on CIFAR-100 with ViT-Tiny under Dir-0.1 for 300 communication rounds.

MethodRetained singular componentsFinal test accuracy (%) ↑
FMuon_SVD1%45.26
FMuon_SVD2.5%47.68
FMuon_SVD5%47.82
FMuon_SVD10%47.94
FMuon_SVD15%48.01
FMuon_SVD20%48.12
FMuon (dense momentum)100%48.18

Retaining 2.5% of the singular components yields 47.68% test accuracy, 0.50 percentage points below dense momentum. At 1% retention, the gap increases to 2.92 points.

W3. Why does Local Muon have higher training loss but higher test accuracy?

Thank you for pointing out this discrepancy. Table 1 reports training loss and held-out test accuracy, which are evaluated on different data. To examine the discrepancy, we evaluated test cross-entropy for both methods on the same held-out CIFAR-100 test set after 300 communication rounds with ResNet-18 under Dir-0.1.

MethodTraining loss ↓Test cross-entropy ↓Test Top-1 accuracy (%) ↑
FedLADA0.4982.58157.78 ± 0.19
Local Muon1.5041.67266.71 ± 0.15

FedLADA achieves lower training loss. On the same held-out test set, Local Muon achieves both lower cross-entropy (1.672 vs. 2.581) and higher Top-1 accuracy (66.71% vs. 57.78%). The two test metrics therefore rank the methods consistently. We will clarify the evaluation split for each metric in Table 1 and include this test cross-entropy comparison.

W4. Evaluation under stronger client heterogeneity

Thank you for suggesting a broader evaluation of non-IID settings. We added a CIFAR-100 experiment with a Dirichlet concentration parameter of 0.05 to evaluate stronger label heterogeneity. We followed the protocol in Table 1, using 300 communication rounds, 100 clients, 10% participation per round, a batch size of 50, and 50 local steps. Final test accuracy is reported as mean ± standard deviation across seeds.

MethodResNet-18 Acc. (%) ↑ViT-Tiny Acc. (%) ↑
FedAvg55.19 ± 0.4723.42 ± 0.61
FedCM59.36 ± 0.5125.74 ± 0.23
Local AdamW54.91 ± 0.4035.33 ± 0.20
Local Muon35.27 ± 2.5735.51 ± 1.25
FedMuon65.30 ± 0.8644.65 ± 0.56
FMuon (ours)66.75 ± 0.4646.25 ± 0.73

Under Dir-0.05, FMuon exceeds FedMuon by 1.45 percentage points on ResNet-18 and 1.60 points on ViT-Tiny.

Responses to the Questions

Q1. Including FedMuon in Figures 3 and 4

Thank you for this suggestion. We will add FedMuon to Figures 3 and 4 using the same experimental settings and evaluation protocol as FMuon. The curves will complement the final-round results in Table 1 by showing how the two methods perform over the course of training.

Q2. Comparing Local Muon and FMuon under the same momentum schedule

Thank you for pointing out that Theorems 5.4 and 5.5 use different momentum schedules. Even under the same momentum schedule as FMuon, the heterogeneity term 2ρζ2\rho\zeta in the Local Muon bound remains unchanged because it is independent of β\beta. The Local Muon bound in Theorem 5.4 contains the heterogeneity term 2ρζ2\rho\zeta, which is independent of β\beta. Taking β\beta toward 1 therefore leaves this term in the stated bound. The bound alone leaves open whether Local Muon converges under a different schedule.

Theorem D.1 compares both methods on a constructed family of heterogeneous matrix problems using the same learning rate ηR=η0/(KR2/3)\eta_R=\eta_0/(KR^{2/3}), the same momentum schedule 1−βR=R−1/31-\beta_R=R^{-1/3}, and the same budget of 2KR2KR local gradient queries. Under these matched conditions, Local Muon satisfies GR(Local Muon)=dc>0\mathcal{G}_R(\mathrm{Local\,Muon})=dc>0, while FMuon satisfies GR(FMuon)≤C∗R−1/3\mathcal{G}_R(\mathrm{FMuon})\le C_*R^{-1/3}, which tends to zero as RR increases. We will clarify the parameter schedules in Section 5 and refer readers to this matched comparison.

Q3. Measuring the computational overhead of SVD

Thank you for pointing out that per-round computation time depends on the number of local steps, KK. Under the same experimental settings, Table 4 reports 6.44 seconds per round with dense momentum and 6.48 seconds per round with compressed momentum. These figures measure total computation time per round at the reported KK.

To isolate the compression cost, we timed the computation of the low-rank factors using the same momentum matrices, retained rank, and hardware. Vit-tiny,CIFAR-100,Dir-0.1.

Momentum compression methodRetained singular componentsMomentum compression time (s/round) ↓
Standard SVD followed by truncation2.5%0.156
Randomized truncated SVD2.5%0.0053

Randomized truncated SVD reduced the measured compression time by approximately 96.6% in this benchmark.

Q4. Consistency of notation

Thank you for pointing out the ambiguity. We will revise Sections 3–5 to distinguish the client gradient ∇fi(x)\nabla f_i(x), the stochastic gradient estimate Gir,kG_i^{r,k} at local step kk of round rr, and the dual norm of the global gradient ∥∇f(xr)∥⋆\|\nabla f(x^r)\|_\star. We will reserve GR\mathcal{G}_R for the expected average of this dual norm over RR rounds. We will also clarify the use of vv in Section 3 and define each symbol at its first occurrence.

Q5. Font sizes in figures

Thank you for pointing this out. We will enlarge the axis labels, tick labels, legends, and annotations so that they are at least as large as the main text at the final figure size. We will adjust the layouts where needed to prevent labels from overlapping.

Q6. Defining LMO at first use

Thank you for pointing this out. We will spell out “linear minimization oracle (LMO)” at its first occurrence in both the abstract and the introduction and retain the formal definition in Section 3.1.

Q7. Clarifying the dimensions in Figure 2(b)

Thank you for pointing this out. In the revised manuscript, we updated Figure 2(b) to label the dimensions of the truncated SVD factors and clarify that the reconstructed momentum matrix retains its original shape, d×dd\times d. Retaining only rr singular components reduces the number of scalars transmitted. We also enlarged the text in the figure to make the dimension labels easier to read.

Reviewer_iRqROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies Muon as the local optimizer in FL. Local Muon helps under IID data but degrades under non-IID data. To address this problem, the authors propose FMuon, a federated optimization framework that combines Muon-based local updates with local-global direction alignment and cross-round momentum initialization. The method also adopts truncated SVD to reduce the communication overhead of sharing momentum states. The authors provide convergence analysis and evaluate the approach on vision and language models under heterogeneous client data distributions.

Critical strengths
  • The paper proposes a simple and practical approach to improving Muon-based optimization in heterogeneous FL.
  • The experimental evaluation is comprehensive, covering strong baselines and vision and language models of different scales.
  • The paper provides convergence analysis and evaluates the communication-accuracy trade-off of momentum compression.
Critical weaknesses
  • The methodological novelty needs to be further justified. FedMuon has already investigated LMO bias in heterogeneous FL using control-variate correction, while FedCM and other studies have explored local–global alignment and momentum coordination. The technical advantages of FMuon over these approaches remain unclear.
  • The communication-efficiency evaluation could be more comprehensive. Although SVD compression reduces momentum transmission overhead, the paper lacks sensitivity analysis of the retained rank and evaluation of end-to-end training time under different network conditions.
Questions for authors
  1. Could the authors clarify the key methodological differences from FedMuon and explain the specific advantages?
  2. How does FMuon compare with FedMuon when both use the same cross-round momentum initialization strategy?
  3. How sensitive is FMuon_SVD to the retained rank, and how does compression affect optimization performance and convergence stability?
  4. Since the convergence guarantee also holds when α = 0, what additional theoretical benefits does local-global alignment provide?
  5. Could the authors explain why FedCM achieves a lower accuracy on ViT-Tiny under IID (27.88%) than under Dir-0.6 (33.33%), as reported in Table 15?
3 comments

W1. Methodological novelty and relation to prior work

Thank you for pointing out the connections between FMuon, FedMuon, and FedCM. FedMuon uses client-specific and global control variates to correct the momentum input before applying the linear minimization oracle (LMO). FedCM uses historical global information to adjust local SGD updates. FMuon initializes participating clients with the previous round's aggregated momentum and mixes each local LMO output with the previous global update direction. Under strong client heterogeneity, local–global alignment, which draws on the same use of historical global information as FedCM, leaves substantial room for improvement. In FMuon, initializing all participating clients with shared momentum at the start of each round provides an additional gain of 3.93 percentage points on ResNet-18 and 4.51 points on ViT-Tiny under Dir-0.1 (Table 5).

Our ablations show that both components contribute under strong client heterogeneity. On CIFAR-100 under Dir-0.1, with post-LMO alignment enabled, adding shared momentum initialization improves accuracy from 69.12% to 73.05% on ResNet-18 and from 43.67% to 48.18% on ViT-Tiny (Table 5). Table 4 isolates the effect of post-LMO alignment with shared initialization held fixed (Q4). We also compare FMuon and FedMuon under the same shared initialization (Q2). We will revise the abstract and introduction to acknowledge the prior work and foreground these two design choices.

W2. Retained-rank sensitivity and communication efficiency

Thank you for identifying these gaps. We added a retained-rank sweep on CIFAR-100 with ViT-Tiny under Dir-0.1 (Q3). At 2.5% retention, FMuon_SVD has a reported payload of 23.9 MB per round and achieves 47.68% final test accuracy, 0.50 percentage points below dense momentum.

We also timed the momentum compression step separately. At the same 2.5% retention, randomized truncated SVD reduced the measured compression time from 0.156 s to 0.0053 s per round. This benchmark measures the cost of compression itself. End-to-end training time under specified bandwidth and latency conditions remains to be evaluated.

Q1. What are the key methodological differences from FedMuon, and what advantages do they offer?

FedMuon corrects the input to the LMO with client-specific and global control variates. FMuon starts each round from a shared momentum and mixes the output of the LMO with the previous global update direction at each local step. Because the LMO is nonlinear, these choices yield different update rules.

ComponentFedMuonFMuon (ours)
Local updateCorrects the momentum input before the LMO.Mixes the LMO output with the previous global update direction.
Round-start momentumResumes each client's own stored momentum.Initializes all selected clients with the previous round's aggregated momentum.
Server stateUpdates the global control variate using changes in the selected clients' control variates.Averages the selected clients' final momenta and computes a global direction from their model updates.
Model aggregationDivides the sum of selected client updates by N.Divides the sum of selected client updates by S.

Here, N is the total number of clients and S is the number selected in the current round.

Local update directions. FedMuon uses LMO(Mir,k+1−Cir+Cr)\mathrm{LMO}(M_i^{r,k+1}-C_i^r+C^r). FMuon uses (1−α)LMO(Mir,k+1)−αΔGr(1-\alpha)\mathrm{LMO}(M_i^{r,k+1})-\alpha\Delta_G^r.

Round-start momentum. FedMuon sets Mir,0←Mir−1,KM_i^{r,0}\leftarrow M_i^{r-1,K}. FMuon sets Mir,0←MˉrM_i^{r,0}\leftarrow\bar M^r.

Server state. FedMuon updates Cr+1=Cr+1N∑i∈Sr(Cir+1−Cir)C^{r+1}=C^r+\frac{1}{N}\sum_{i\in S_r}(C_i^{r+1}-C_i^r), where Cir+1=Mir,KC_i^{r+1}=M_i^{r,K} for selected clients. FMuon computes Mˉr+1=1S∑i∈SrMir,K\bar M^{r+1}=\frac{1}{S}\sum_{i\in S_r}M_i^{r,K} and ΔGr+1=−1SKη∑i∈Srδir\Delta_G^{r+1}=-\frac{1}{SK\eta}\sum_{i\in S_r}\delta_i^r, where δir=xir,K−xir,0\delta_i^r=x_i^{r,K}-x_i^{r,0}.

Here, δir=xir,K−xir,0\delta_i^r=x_i^{r,K}-x_i^{r,0}, NN is the total number of clients, and SS is the number selected in round rr.

Shared initialization gives participating clients the same momentum at the start of each round. Post-LMO alignment then uses α\alpha to control how much the previous global update contributes to each local step. Table 5 quantifies the benefit of shared initialization with alignment enabled, and Table 4 quantifies the benefit of alignment with shared initialization held fixed.

Q2. How do FMuon and FedMuon compare under the same cross-round momentum initialization?

Thank you for suggesting this comparison. We modified FedMuon so that each selected client starts from the average of the previous round's final client momenta, as in FMuon. We retained the original FedMuon update and aggregation rules, including its pre-LMO control-variate correction. We evaluated the methods on CIFAR-100 under Dir-0.1 for 300 communication rounds.

MethodCross-round momentum initializationResNet-18 Acc. (%) ↑ViT-Tiny Acc. (%) ↑
FedMuon (original)Client-specific stored momentum69.71 ± 0.2345.17 ± 0.27
FedMuon + shared momentum initializationShared momentum from the previous round70.86 ± 0.3146.02 ± 0.38
FMuon (ours)Shared momentum from the previous round73.05 ± 0.1648.18 ± 0.12

Shared initialization improves FedMuon by 1.15 percentage points on ResNet-18 and 0.85 points on ViT-Tiny, narrowing the gap to FMuon. Under the same initialization strategy, FMuon remains 2.19 and 2.16 points ahead, respectively. This comparison measures the difference between the two algorithms under a common initialization strategy. Table 4 separately quantifies the effect of post-LMO alignment within FMuon.

Q3. How sensitive is FMuon_SVD to the retained rank?

We evaluated FMuon_SVD on CIFAR-100 with ViT-Tiny under Dir-0.1 for 300 communication rounds.

MethodRetained singular componentsFinal test accuracy (%) ↑
FMuon_SVD1%45.26
FMuon_SVD2.5%47.68
FMuon_SVD5%47.82
FMuon_SVD10%47.94
FMuon_SVD15%48.01
FMuon_SVD20%48.12
FMuon (dense momentum)100%48.18

Retaining 2.5% of the singular components gives 47.68% test accuracy, 0.50 percentage points below dense momentum. Increasing retention from 1% to 2.5% improves accuracy by 2.42 points, while increasing it from 2.5% to 20% adds 0.44 points. These results quantify sensitivity in final accuracy.

Q4. What theoretical benefit does local–global alignment provide if convergence already holds at α=0\alpha=0?

Thank you for raising this distinction. Under full participation with exact LMOs and uncompressed momentum, Theorem 5.5 gives an O(R−1/4)O(R^{-1/4}) upper bound on the average stationarity measure for any fixed α∈[0,1)\alpha\in[0,1), including α=0\alpha=0. Shared initialization removes the inherited differences in client momenta at the start of each round. The virtual-iterate analysis shows that mixing the local LMO output with the historical global direction preserves the asymptotic order of the bound. Whether positive α\alpha improves the convergence rate or finite-round stationarity remains open.

Table 4 quantifies the empirical benefit of alignment. With shared initialization held fixed on CIFAR-100 under Dir-0.1, setting α=0.5\alpha=0.5 instead of α=0\alpha=0 improves test accuracy from 68.05% to 73.05% on ResNet-18 and from 44.56% to 48.18% on ViT-Tiny.

Q5. Why was FedCM's IID accuracy on ViT-Tiny lower than its Dir-0.6 accuracy in Table 15?

Thank you for flagging this result. We rechecked the experimental configuration and reran FedCM with ViT-Tiny on the IID split of CIFAR-100. After 300 communication rounds, the test accuracy was 33.94%. We will correct the IID entry in Table 15 from 27.88% to 33.94%. The corrected IID accuracy is 0.61 percentage points higher than the reported Dir-0.6 accuracy of 33.33%.

Thank you again for your careful review and constructive feedback. We have provided point-by-point responses to your concerns, supported by additional analyses and experiments. We would be happy to provide further clarification or address any additional questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.