- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presented a federated learning for MoE model with stage wise learning strategies to learn gates and heads, and data-aware grouping for partition clients, in hope to reduce the communication load. The proposed model performance was tested on three datasets and compared with baseline models. The results suggested the proposed model has better generalizability for unseen data, and grouping provides greater speed-ups at larger scales, at the cost of the accuracy.
Critical strengths
- a multi-stage training to clearly separate out modules with local training , and modules trained with federated updates,
- Grouping of clients is proposed to reduce the communication load, and being tested for the effect on accuracy ,
Critical weaknesses
- The grouping effects, which is the main focus of this paper, are only tested or presented using one dataset.
- Different gate synchronization strategies affect communication load. How is it reflected in Equations 3 and 4?
- Missing ablution tests for show effects of each module.
Questions for authors
Can the authors provide more detailed explanation for the remarked improvement of DCGMoe model in classification accuracy of CIFAR-100 dataset ? A related question is what is importance of each module,each staged training strategies ?