- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper targets the high memory and inter-GPU communication costs of training MoE LLMs. The authors propose MoE-DisCo, a staged framework that splits an MoE into several dense single-expert submodels and partitions the corpus into subsets via K-means on mean-pooled token embeddings. Each submodel is trained independently on a single GPU without communication, after which the shared backbones are averaged and the full MoE is briefly fine-tuned on multiple GPUs. Across 10B-parameter Qwen and Llama MoEs, MoE-DisCo matches or improves upon full-parameter training with reduced training costs.
Critical strengths
-
This paper targets a very important problem, as training a modern MoE LLM is resource-intensive. Designing meaningful systems for training such models is important.
-
The theoretical formulation in Section 3.5 and the appendices offers a clear conceptual decomposition of the error introduced by staged training.
Critical weaknesses
-
One of the main contributions claimed by the authors is that training is “shifted from expensive multi-GPU clusters to fully parallelized, low-cost single-GPU instances.” However, this claim is not supported by the experiments. As described in Sec. 4.4, the experiments are all conducted on A100/A800 GPUs, both of which have large GPU memory capacities. The whole training framework is not tested on low-cost, low-memory GPUs such as the RTX 4090 or RTX 5090. The hybrid GPU training settings shown in Figure 2 are also not validated in the experiments.
-
The main difference between the proposed training framework and BTX is that the authors use K-means clustering to divide the pretraining datasets into several subsets without using labels. However, this clustering process requires tokenization, embedding computation, and running a clustering algorithm on trillions of tokens, making the cost unacceptable for real-world pretraining. Although the authors provide a preprocessing time for the C4 dataset, this is not a meaningful analysis, as C4 is too small to be representative of a real-world corpus.
-
Some experimental settings are confusing, and the authors do not provide sufficient explanations. The official Qwen1.5-MoE-2.7B has 60 experts, but the authors set the number of experts to 4 and increase the number of layers. Why do the authors change the official architecture? How is the conversion performed?
-
The authors use strong open-source models to conduct experiments, but the training loss and PPL are very high. Why is that? The performance is also substantially lower than the results reproduced using the officially released checkpoints.
-
The loss comparison in Figure 4 is unfair. Full-parameter training lacks an additional submodel training stage. What would the loss curve look like if the full-parameter baseline were first trained on the same number of tokens used in MoE-DisCo's submodel training stage?
-
Many important baselines for communication-efficient training are neither discussed in the related work nor compared in the experiments, such as [1] and [2].
-
Modern MoE models use fine-grained experts. However, all experiments in the current paper are conducted using 4 experts. How does MoE-DisCo perform on models with more experts?
[1] DiLoCo: Distributed Low-Communication Training of Language Models, arXiv 2023.
[2] Photon: Federated LLM Pre-Training, MLSys 2025.
Questions for authors
- How were the 10B-parameter, 4-expert Qwen and Llama models initialized?
- What causes the staircase-like loss drops every 3,000 steps?
- How does MoE-DisCo compare against a random data partition baseline?