acceptodds
Withdrawn from ICLR 2027

Scalable and Decentralized Training of Mixture of Experts with Collaborative Gating

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Decentralized federated learning with Mixture-of-Experts (MoE) allows clients to directly share lightweight experts and combine them locally, avoiding the server dependency and knowledge degradation of centralized aggregation. However, existing decentralized FL-MoE methods assume fully-connected communication among all clients, which incurs quadratic per-round cost and does not scale. In addition, each client’s gate is trained only on local data, which is insufficient to learn reliable routing under non-IID distributions. We present DCGMoE (Decentralized Collaborative Gating MoE), a decentralized FL-MoE framework that addresses both issues. First, DCGMoE introduces a data-aware grouping strategy that partitions federations into groups of complementary clients using local statistics such as label distribution. Each client then communicates only with members of its own group, which reduces the per-round communication cost from quadratic to sub-quadratic growth as the federation size increases. Second, DCGMoE proposes a collaborative gating mechanism that jointly trains the routing module across clients so that each gate learns to exploit the expert pool within its group. The proposed DCGMoE achieves higher global accuracy than both centralized and decentralized FL baselines on Fashion-MNIST, CIFAR-10, and CIFAR-100 under non-IID partitioning with α=0.3. At a federation of 100 clients, DCGMoE further achieves substantially lower per-round communication cost and shorter per-round training time than baselines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Decentralized Mixture-of-Experts, Collaborative Gating

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nBxtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presented a federated learning for MoE model with stage wise learning strategies to learn gates and heads, and data-aware grouping for partition clients, in hope to reduce the communication load. The proposed model performance was tested on three datasets and compared with baseline models. The results suggested the proposed model has better generalizability for unseen data, and grouping provides greater speed-ups at larger scales, at the cost of the accuracy.

Critical strengths
  1. a multi-stage training to clearly separate out modules with local training , and modules trained with federated updates,
  2. Grouping of clients is proposed to reduce the communication load, and being tested for the effect on accuracy ,
Critical weaknesses
  1. The grouping effects, which is the main focus of this paper, are only tested or presented using one dataset.
  2. Different gate synchronization strategies affect communication load. How is it reflected in Equations 3 and 4?
  3. Missing ablution tests for show effects of each module.
Questions for authors

Can the authors provide more detailed explanation for the remarked improvement of DCGMoe model in classification accuracy of CIFAR-100 dataset ? A related question is what is importance of each module,each staged training strategies ?

Reviewer_tL5VOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper present DCGMoE a decentralized FL-MoE framework where its uses clients data distribution to group the clients for training the gate and transfer of experts. This allows the DCGMoE to reduce the communication cost and utilize other clients knowledge for training of the gate.

Critical strengths

This paper study an interesting problem of using MoE in decentralized FL settings and attempts to improve the communication cost.

The paper is written clearly and all the related works have been discussed.

Critical weaknesses

My main concern is around the novelty of this paper. The main idea of the paper is very close to dFLMoE and only the grouping mechanism were added for exchange of experts and training of the gate. To me this does not add any significant insight or value for the research community.

The second concern is around the experiments setups. DCGMoE has been evaluated only on small scale datasets (CIFAR and MNIST).

Based on my understanding, all the clients to communicate to their label histograms to be able to do the grouping. 1) doesn't that violate privacy of each client and 2) how exactly can we centralize all of those histograms in a decentralized settings?

Questions for authors

Please see above.

Reviewer_EoWmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DCGMoE, a decentralized federated learning framework with MoE. It clusters clients by label distributions to cut communication and adds collaborative gate training to improve routing. Experiments on three image datasets show global accuracy gains and lower per-round communication when scaling to 100 clients.

Critical strengths

Practical setting. Combining client grouping with MoE gating is intuitive for decentralized FL.

Good accuracy gains on CIFAR-10 and CIFAR-100.

Scalability experiments go up to 100 clients, which is nice to see.

Critical weaknesses

Unfair baseline comparisons. DCGMoE changes the gating architecture and adds extra components compared to dFLMoE. Plus, the baseline Avg-Sync aggregates the gate, adapter, and classifier all at once. Because of this, it is impossible to tell if the large accuracy bump comes from collaborative gating itself, or just from extra module capacity and confounding sync choices.

Questionable decentralization claims. The paper assumes group-wise weighted averaging is cheap, but how is this actually implemented peer-to-peer without a central coordinator or master node?

Hidden communication overhead. The cost calculations ignore the initial all-to-all label histogram exchange and client grouping step. In a realistic dynamic edge setting, this initialization overhead is non-trivial and should be factored into total communication.

Missing baseline. To fairly test the gating idea, why not just apply the same grouping strategy to the vanilla dFLMoE baseline? Without a grouped-dFLMoE point of comparison, we cannot separate the gain of grouping from the gain of DCGMoE.

Questions for authors

Can you run a clean ablation that isolates gate synchronization from adapter/classifier synchronization under the exact same backbone architecture and token budget?

How exactly is the group-wise averaging executed in a decentralized way without a coordinator? Does it require multi-hop gossip or a local cluster head?

What is the total end-to-end communication volume if you include the initial histogram sharing and cluster formation step?

What happens if you run grouped dFLMoE using your exact same client clusters?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.