acceptodds
Under review as a conference paper at ICLR 2027

CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training

Abstract

Mixture-of-Experts (MoE) has been widely adopted in recent large language model (LLM) architectures. However, scaling up MoE in LLM training introduces system-level challenges on training, where non-uniform token routing can lead to highly imbalanced workloads across experts and devices, further destabilizing the training process. With trillion-scale LLMs, imbalanced expert workloads further amplify the resource cost of MoE training, resulting in degraded training efficiency and hardware utilization for underloaded experts, while hot experts require additional resources to accommodate excessive workloads. Recent studies address imbalanced MoE training through intricate parallelism strategies or resource reallocation. However, these system-level approaches often introduce additional resource requirements and considerable orchestration complexity, which become increasingly difficult to afford when training trillion-parameter LLMs under constrained computational resources. This work introduces CIPHER-MoE, which mitigates MoE workload imbalance while keeping the router's token-side Top-K selection unchanged. CIPHER-MoE applies affinity-aware Expert-to-Token filtering with explicit capacity control to reduce hotspot expert workloads without additional hardware resources or complex runtime design. The proposed method has been evaluated on large-scale MoE models, including DeepSeek-V4-Pro, showing up to 64.9 percentage points Top-1 expert workload reduction and 1.10-1.94 training acceleration, while preserving the training quality. The source code will be released soon.

Primary area:
generative models
Keywords:
Efficient LLM Training, Mixture of Expert

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rzB6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

At the scale of trillion parameter MoE models, nonuniform token routing can create hot experts, imbalanced computation and communication, and pressure on peak memory that may cause failures when memory is exhausted. CIPHER-MoE addresses this problem by applying token admission at the expert level after the router's Top-K selection. For each expert and micro-batch, it retains at most a capacity budget of candidates whose token hidden state has sufficient cosine affinity with the expert gate. Rejected assignments are either strictly dropped or rerouted to another expert. The method also adds the LocMoE locality loss.

The paper provides a theorem on capacity reduction using similarity scores, under assumptions about spherical geometry, independence, signal availability, and conditional ranking. It evaluates DeepSeek-V3, DeepSeek-V4-Flash, DeepSeek-V4-Pro, and GLM-5 on OR, MedMCQA, and SIQA, reporting training speedups from 1.10x to 1.94x and mostly comparable domain task scores. Across the two variants, strict dropping generally performs better than rerouting on downstream quality, while the appendix reports additional results on general benchmarks.

Critical strengths
  1. The paper targets a practically important systems bottleneck: capacity control at the expert level can reduce overloaded dispatches without requiring expert replication, migration, or a more elaborate runtime scheduler.

  2. The DeepSeek-V4-Pro result is a particularly compelling feasibility demonstration on a very large model: the vanilla MoE baseline runs out of memory before the first iteration, whereas CIPHER-MoE completes training under the reported configuration, reaches a global batch size of 8192, and outperforms the fixed router reference on the weighted OR average. Together with the evaluation across several large MoE models and the comparison of strict dropping, rerouting, and LocMoE, this supports the method's practical relevance. However, the different batch sizes mean the OOM result should be presented as a feasibility result rather than a directly matched speed comparison; the paper's attempt to separate signal retention from output fidelity in Appendix A.4.4 is a useful qualification of the theoretical claims.

Critical weaknesses
  1. The main empirical comparison is not sufficiently controlled, and the key ablation is missing. CIPHER-MoE combines admission using affinity scores with the LocMoE locality loss, but it does not include a CIPHER variant without the locality loss or a vanilla routing baseline with the same locality loss. Thus the reported gains cannot be attributed to token admission. In addition, Table 3 compares a baseline that OOMs at GBS 256 with CIPHER-MoE at GBS 8192, so it does not establish a speed or quality comparison at the same optimization budget. The paper needs matched data, training steps or tokens, global batch size, hardware, and reporting of assignment dropping and rerouting rates, as well as repeated runs or uncertainty estimates.

  2. The algorithmic and theoretical contracts are not sufficiently connected to the experiments. The exact rerouting procedure, candidate set, conflict resolution, gate renormalization, and whether additional router scores are computed are underspecified. The theorem only guarantees retention of an abstract discriminative witness under assumptions A1-A3; the manuscript does not estimate the certified margin, ranking parameters, or recall on learned representations, and explicitly distinguishes the theoretical margin from the implementation threshold. Consequently, the theorem does not support the claims about final model quality or general capability without substantially stronger empirical evidence.

Questions for authors
  1. Which exact values of τ, capacity factor, fill-target factor, training steps, total processed tokens, seeds, and rates of assignment dropping and rerouting were used for each model and dataset? Were all methods trained with the same data exposure and optimization budget? Please reconcile the GBS 256 and GBS 8192 setup and interpretation in the OOM comparison in Table 3.

  2. Please provide a 2x2 ablation separating token admission using affinity scores from the LocMoE locality loss: vanilla routing, locality loss only, admission only, and the full method. This ablation is required to determine which component causes the reported speed and quality changes.

  3. For rerouting, does the method search only the original Top-K experts or all experts? Please specify the complete deterministic algorithm, including how capacity conflicts at the destination expert are resolved, whether assignments between tokens and experts that have already been accepted can be evicted, and how routing weights are normalized after dropping or rerouting.

  4. How are the theoretical quantities δ, μ, η, and a₀ calibrated or measured on the evaluated checkpoints? Please report measurements of signal retention and output deviation, rather than using the illustrative Table 7 regime as evidence for the trained models.

  5. The related-work discussion could be strengthened by situating CIPHER-MoE among complementary approaches to MoE efficiency, including the following works:

  • Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts (He et al., ICLR 2026), which studies capacity-aware token dropping and rerouting during inference.
  • EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference (Wu et al., ICML 2026), which addresses workload imbalance through cross-layer scheduling.
  • Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression (Zhu et al., 2025), which explores expert reorganization, structured compression, and hierarchical routing.

These works address different settings and are not direct substitutes for CIPHER-MoE's training-time, post-Top-K admission mechanism. A concise comparison of their objectives, intervention points, and trade-offs would help clarify the novelty and scope of CIPHER-MoE. The discussion should focus on technically relevant distinctions rather than exhaustive citation coverage.

Reviewer_j9MuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

CIPHER-MoE tackles the challenge of imbalanced workloads across experts and devices during large-scale training of MoE models. The method is designed to couple with standard top-k routers, and through an additional module constrains over-loaded experts. This is done by each expert evaluating its candidate tokens using cosine similarity, and filtering the highest-affinity tokens up to a capacity budget. The two variants of the method CIPHER-strict and CIPHER-reroute then either drop the rejected token-expert assignments, or redirect them to other candidate experts that meet the similarity threshold. The authors extensively evaluate CIPHER-MoE on DeepSeek-V4-Flash, DeepSeek-V3, GLM-5, and 1.6T DeepSeek-V4-Pro focusing on large-model SFT. The method achieves training speedups of 1.10×1.10\times–1.94×1.94\times, while preserving domain-specific and general capabilities.

Critical strengths

The problem tackled in this paper is critical for efficient training of large-scale MoEs.

The major strength of the paper are solid evaluation of the method on large scale MoE models, measured meaningful speed/quality tradeoffs in domain SFT, and demonstrating trillion-scale feasibility results.

Critical weaknesses

The major weakness of the paper is over-claiming novelty on the core concept of CIPHER-MoE and neglecting highly similar related work from Li et al. [1]. There are very limited concrete methodological, theoretical, and practical contributions, when we isolate what CIPHER-MoE actually changes relative to [1].

  • In particular, the plug and play module in CIPHER-MoE, uses cosine similarity ci,ec_{i,e} on the expert side to filter and cap the candidate pool Pe\mathcal{P}_e. This is already proposed in [1] and the only difference is in how the threshold is applied.

  • The training objective remains the same as LocMoE locality KL objective.

Therefore there is minimal differences in methodology compared to LocMoE and [1] and the two variants of strict and reroute qualify more as ablations of the method rather than a new contribution (Previous works mainly consider dropping overflow tokens). In general, I am concerned about the significant overlap of CIPHER-MoE with this related work and its omission from the paper.

Additionally, important implementation details are missing. What exact value of τ\tau is used in CIPHER-strict and CIPHER-reroute across each model? Appendix A.5 only reports τ=0.01\tau = 0.01 for LocMoE. Could the authors, possibly on a small scale, report a sensitivity ablation of τ\tau (and possibly the capacity factor CeC_e) for both CIPHER variants showing how training step time, actual dropped/rerouted token ratios, and benchmark accuracy vary across τ\tau?

[1] Li, Jing, et al. "Expert-token resonance MoE: Bidirectional routing with efficiency affinity-driven active selection." arXiv preprint arXiv:2406.00023 (2024).

Questions for authors

Can the authors discuss why the speedup of CIPHER-MoE drops to 1.10×1.10\times–1.14×1.14\times on GLM-5 (matching LocMoE) compared to 1.68×1.68\times–1.94×1.94\times on DeepSeek-V3 despite both using the same granular 256 experts and Top-8 sigmoid routing?

Reviewer_w5KqOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CIPHER-MoE, which filters token–expert assignments after the original Top-K routing based on affinity and capacity constraints, dropping or rerouting excess assignments to reduce computation and memory pressure on overloaded experts. The authors report 1.10×–1.94× training speedups across domain-specific fine-tuning of several large MoE models and enable training of a trillion-parameter model that otherwise runs out of memory. However, the gains partly stem from reduced expert computation, and advantages over simpler pruning strategies remain insufficiently established. The theoretical signal-retention guarantee does not ensure unchanged training quality, and some general capabilities show substantial degradation.

Critical strengths

The paper targets a practical bottleneck in MoE fine-tuning and demonstrates that capacity-aware filtering can reduce training time and enable otherwise memory-infeasible training at trillion-parameter scale.

Critical weaknesses
  1. Causal consistency of cross-token filtering is unclear: If tokens are ranked across an entire sequence or micro-batch, whether a token receives expert computation may depend on future tokens, making training-time execution paths depend on information unavailable during inference. The paper should clarify the filtering scope and evaluate a causal alternative.

  2. Missing compute-matched baselines: Speedups partly result from executing fewer token–expert assignments, yet comparisons against random dropping, gate-score pruning, and reduced Top-K under matched compute budgets are insufficient. The benefit of affinity-based selection itself therefore remains unclear.

  3. Routing fidelity lacks direct evidence: Preserving the initial Top-K candidates does not preserve the executed paths, mixture weights, or gradients. The paper lacks direct measurements of retained routing probability mass, output deviation, and gradient deviation.

  4. Affinity filtering may reinforce existing preferences at the expense of learning coverage: High affinity does not necessarily imply high training value, and repeatedly retaining high-scoring tokens may exclude difficult examples and novel domain signals. The specialization metrics do not rule out narrower expert coverage as an alternative explanation.

  5. Expert load is insufficiently distinguished from device-level bottlenecks: Training latency depends on aggregate device load, communication hotspots, and pipeline stalls; limiting individual expert capacity does not automatically optimize overall execution. Device-level load and critical-path analyses are needed to separate load-balancing benefits from simply reducing computation.

  6. Large-scale feasibility does not establish a fair efficiency advantage: The vanilla trillion-parameter model running out of memory only demonstrates that the chosen configuration is infeasible. Comparisons against runnable baselines with adjusted micro-batch sizes, gradient accumulation, or activation recomputation are needed to establish an overall efficiency advantage.

  7. Filtering introduces systematic gradient selection bias: Experts receive samples selected by the current routing preferences, rather than an unbiased subset of the original training signal. The paper insufficiently analyzes how this bias affects the optimization objective and convergence, or distinguishes its effects from random expert dropout.

  8. The connection between capacity limits and peak resource usage is unclear: Filtering after dispatch may leave communication and receive buffers exposed to peak loads, while filtering before dispatch requires cross-rank capacity coordination. An expert-level capacity bound alone does not guarantee controlled peak resource usage during distributed training.

Questions for authors
  1. Does expert-side ranking allow future tokens to influence earlier tokens’ execution paths, and do the gains persist under strictly causal filtering?

  2. At matched expert FLOPs and memory budgets, does CIPHER outperform random dropping, gate-score pruning, and reduced Top-K?

  3. Where does filtering occur relative to dispatch, and how are cross-rank capacity limits enforced before communication and memory peaks arise?

  4. Do the authors plan to release the code and full training configurations, given that the current implementation details are insufficient for independent reproduction?

Clear answers and supporting evidence could persuade me to raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.