acceptodds
Under review as a conference paper at ICLR 2027

MoE-DisCo: Low Economic Cost Training Mixture-of-Experts Models

Abstract

Training large-scale Mixture-of-Experts (MoE) models faces prohibitive memory capacity constraints. While only a subset of parameters is activated during inference, the entire parameter space must be updated during training. This forces the use of multi-GPU parallelism (e.g., Tensor Parallelism), introducing massive inter-GPU communication overhead that significantly bottlenecks training efficiency. To address this, we propose MoE-DisCo (Mixture-of-Experts with Disentangled Clustering and Coordination)—a staged, communication-efficient training framework. MoE-DisCo decomposes the massive MoE model into multiple dense submodels, each consisting of a shared backbone and a single expert, ensuring each submodel fits within the memory of a single GPU. Concurrently, it partitions the training data into subsets using unsupervised clustering. Each submodel is trained independently and in parallel on its assigned data subset on isolated GPUs, completely eliminating inter-GPU communication overhead. Subsequently, all experts are integrated into a complete MoE model and fine-tuned globally for a short period. Experiments demonstrate that MoE-DisCo matches or exceeds the performance of full-parameter training across downstream tasks and perplexity (PPL), while drastically reducing training cost and communication overhead on architectures like Qwen-MoE and Llama-MoE.

Primary area:
optimization
Keywords:
MoE

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kRxHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets the high memory and inter-GPU communication costs of training MoE LLMs. The authors propose MoE-DisCo, a staged framework that splits an MoE into several dense single-expert submodels and partitions the corpus into subsets via K-means on mean-pooled token embeddings. Each submodel is trained independently on a single GPU without communication, after which the shared backbones are averaged and the full MoE is briefly fine-tuned on multiple GPUs. Across 10B-parameter Qwen and Llama MoEs, MoE-DisCo matches or improves upon full-parameter training with reduced training costs.

Critical strengths
  1. This paper targets a very important problem, as training a modern MoE LLM is resource-intensive. Designing meaningful systems for training such models is important.

  2. The theoretical formulation in Section 3.5 and the appendices offers a clear conceptual decomposition of the error introduced by staged training.

Critical weaknesses
  1. One of the main contributions claimed by the authors is that training is “shifted from expensive multi-GPU clusters to fully parallelized, low-cost single-GPU instances.” However, this claim is not supported by the experiments. As described in Sec. 4.4, the experiments are all conducted on A100/A800 GPUs, both of which have large GPU memory capacities. The whole training framework is not tested on low-cost, low-memory GPUs such as the RTX 4090 or RTX 5090. The hybrid GPU training settings shown in Figure 2 are also not validated in the experiments.

  2. The main difference between the proposed training framework and BTX is that the authors use K-means clustering to divide the pretraining datasets into several subsets without using labels. However, this clustering process requires tokenization, embedding computation, and running a clustering algorithm on trillions of tokens, making the cost unacceptable for real-world pretraining. Although the authors provide a preprocessing time for the C4 dataset, this is not a meaningful analysis, as C4 is too small to be representative of a real-world corpus.

  3. Some experimental settings are confusing, and the authors do not provide sufficient explanations. The official Qwen1.5-MoE-2.7B has 60 experts, but the authors set the number of experts to 4 and increase the number of layers. Why do the authors change the official architecture? How is the conversion performed?

  4. The authors use strong open-source models to conduct experiments, but the training loss and PPL are very high. Why is that? The performance is also substantially lower than the results reproduced using the officially released checkpoints.

  5. The loss comparison in Figure 4 is unfair. Full-parameter training lacks an additional submodel training stage. What would the loss curve look like if the full-parameter baseline were first trained on the same number of tokens used in MoE-DisCo's submodel training stage?

  6. Many important baselines for communication-efficient training are neither discussed in the related work nor compared in the experiments, such as [1] and [2].

  7. Modern MoE models use fine-grained experts. However, all experiments in the current paper are conducted using 4 experts. How does MoE-DisCo perform on models with more experts?

[1] DiLoCo: Distributed Low-Communication Training of Language Models, arXiv 2023.

[2] Photon: Federated LLM Pre-Training, MLSys 2025.

Questions for authors
  1. How were the 10B-parameter, 4-expert Qwen and Llama models initialized?
  2. What causes the staircase-like loss drops every 3,000 steps?
  3. How does MoE-DisCo compare against a random data partition baseline?
Reviewer_8bwAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MoE-DisCo, a staged framework for training Mixture-of-Experts models that removes inter-GPU communication from the bulk of training. An MoE with E experts is decomposed into E dense submodels, each a copy of the shared backbone plus one expert with the router removed, and the corpus is partitioned into E subsets by k-means over mean-pooled token embeddings. Each submodel trains independently on its own subset on a single GPU with no synchronisation. The experts are then reassembled into a full MoE, the backbone copies averaged with sample-count weights and the router reinitialised, and the model is fine-tuned jointly on the full corpus - the only stage requiring multi-GPU training. Experiments on Qwen- and Llama-based MoEs over C4, WikiText and OpenWebText report comparable or better performance than full-parameter training at 45.1% to 55.1% lower cost, supported by a convergence analysis and ablations on the partitioning strategy and expert count.

Critical strengths

The core idea is interesting and follows a sound intuition: if the expensive part of MoE training is the synchronisation, then decomposing the model into independently trainable dense submodels and reserving multi-GPU training for a short final stage could hold a real advantage over conventional parallel training. The framing of cost in monetary rather than purely computational terms is a reasonable way to make that case.

The architecture configuration in Section 4.1 is well suited to the hardware the authors had available. Fixing four experts and increasing depth to reach roughly 10B parameters keeps the model large enough that the baseline genuinely requires multi-GPU training, which is what the method needs to demonstrate against, while staying within reach of a four-GPU setup. It is a sensible way to showcase the method under a limited budget.

Critical weaknesses

The training loss curves show periodic sharp drops followed by plateaus. If these were epoch boundaries they should not appear on C4, which has far more data than the token budgets used here. This behaviour is unexpected in standard MoE training and was not addressed by the authors. The step counts for MoE-DisCo also appear to be measured from the start of merged fine-tuning, excluding the budget spent training the separate backbone-plus-expert submodels that produced the initialisation. Establishing optimal learning rate for both methods is an important factor that might distinguish reported results. Most likely each of those methods has a different optimal learning rate which needs to be fitted separately to compare them fairly together. Considering those factors, models can’t be deemed as fairly compared.

Assigning each expert a fixed cluster replaces a learned, token-level allocation under a load-balancing objective with a fixed, sequence-level one, which risks uneven expert utilisation and lower model quality. The second stage may undo this, since the router is reinitialised and trained on the full corpus, but the paper does not show that it does: no expert-utilisation or load-balance statistics are reported and no load-balancing loss is mentioned. As stage-one specialisation is the stated mechanism behind the gains, whether it survives matters.

Several existing approaches can combine separately trained dense models into an MoE, and those models can be trained much as they are here. The novel elements are using unsupervised clustering to define what each model trains on, and the efficiency framing when no parallelism is needed in the first stage. Sequence vectors are averages of token embeddings, so they carry no positional or contextual information. The partition is therefore over bag-of-words profiles. This is not semantic similarity in any strong sense, which weakens the method and makes the training comparison against a random data partitioning needed in reporting.

Questions for authors

Could you provide how learning rate was established or both methods?

Could you address "periodic sharp drops followed by plateaus" in training loss?

Reviewer_pCmkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoE-DisCo, a staged training framework for mixture-of-experts models. It clusters training examples using mean-pooled token embeddings, assigns one cluster to each expert, and trains independent dense submodels consisting of the shared backbone and one expert. The authors then merge the trained backbone copies using cluster-size weights, restore the experts and router, and perform short global fine-tuning. Experiments on Qwen- and LLaMA-based models report comparable or better language-modeling and downstream-task performance than full-parameter training, with estimated monetary cost reductions of 45.1%–55.1%. The paper also provides a convergence analysis under smoothness and strong-convexity assumptions.

Critical strengths
  1. The paper studies a practically relevant problem: reducing the cost of training MoE models by moving a substantial part of the work to independent, lower-cost single-GPU instances. The staged decomposition is intuitive and could be useful if its quality and cost claims are validated under controlled comparisons.

  2. The paper evaluates multiple datasets and two model families, and reports both model-quality metrics and an explicit monetary-cost estimate. This is a useful starting point for studying the trade-off between independent expert training and subsequent global coordination.

Critical weaknesses
  1. The experimental model construction and initialization are not described sufficiently to make the central results interpretable or reproducible. The paper says that it increases the number of layers in Qwen1.5-MoE-2.7B and LLaMA-MoE-3.5B to obtain models with approximately 10B total parameters, but does not clearly specify the resulting architectures, how additional layers are initialized, which exact checkpoints are used, or how the expert and router configurations are changed. These choices can materially affect both convergence and the reported benefits.

  2. The comparison does not establish that MoE-DisCo reaches the reported quality with a comparable training budget. The paper reports steps, but not the total number of tokens or total training FLOPs across the independent-submodel and global fine-tuning stages. The stages use different batch sizes and data partitions, and the rule for selecting the full-parameter baseline’s convergence point is unclear. Without token- or compute-matched comparisons, the results do not show whether the gains come from the proposed decomposition or from differences in the amount and distribution of training.

  3. The main results lack repeated runs and uncertainty estimates. This is particularly important for the downstream-task comparisons, where some reported differences are small, and for a method that depends on clustering, data order, and stochastic optimization. The paper does not establish that the reported improvements are robust across random seeds.

  4. The evidence for the claimed benefit of semantic clustering is limited. The BTX-style comparison is reported for only one model and dataset, and the data-assignment procedure, cluster-size matching, and training-budget matching are not sufficiently specified. The comparison therefore does not isolate the effect of K-means or establish that it is better than simpler alternatives such as random assignment.

  5. The theoretical analysis is only weakly connected to the empirical method. The convergence results assume strongly convex, smooth objectives and SGD, whereas the experiments use AdamW on nonconvex Transformer models. The partition-dependent quantities that determine the theoretical bound are not measured in the experiments. The analysis may motivate the method, but it does not currently explain or substantiate the empirical convergence and cost claims.

Questions for authors
  1. What are the exact architectures and initialization procedures used for the approximately 10B-parameter Qwen and LLaMA experiments? Please specify the source checkpoints, layer counts, expert counts, router configuration, and how any added layers or parameters are initialized.

  2. For each method and dataset, how many training tokens are processed during submodel training and global fine-tuning? Can the authors also report total training FLOPs and provide comparisons at matched token or compute budgets?

  3. How is the full-parameter baseline’s convergence point selected for Table 1? Is the same target-selection rule applied across datasets and architectures, and is it chosen using a held-out validation set?

  4. How many random seeds were used for the main results? Can the authors report variation across seeds for language-modeling and downstream-task metrics?

  5. What exact data-assignment procedure is used for the BTX-style comparison? Are cluster sizes, token counts, optimization steps, and total training compute matched to MoE-DisCo? Were random or other simple partitioning baselines evaluated?

  6. How are the GPU rental prices in the cost analysis sourced, and what assumptions are included in the estimate? Do the reported savings persist under alternative hardware prices and when accounting for utilization and data-processing overhead?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.