- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
At the scale of trillion parameter MoE models, nonuniform token routing can create hot experts, imbalanced computation and communication, and pressure on peak memory that may cause failures when memory is exhausted. CIPHER-MoE addresses this problem by applying token admission at the expert level after the router's Top-K selection. For each expert and micro-batch, it retains at most a capacity budget of candidates whose token hidden state has sufficient cosine affinity with the expert gate. Rejected assignments are either strictly dropped or rerouted to another expert. The method also adds the LocMoE locality loss.
The paper provides a theorem on capacity reduction using similarity scores, under assumptions about spherical geometry, independence, signal availability, and conditional ranking. It evaluates DeepSeek-V3, DeepSeek-V4-Flash, DeepSeek-V4-Pro, and GLM-5 on OR, MedMCQA, and SIQA, reporting training speedups from 1.10x to 1.94x and mostly comparable domain task scores. Across the two variants, strict dropping generally performs better than rerouting on downstream quality, while the appendix reports additional results on general benchmarks.
Critical strengths
-
The paper targets a practically important systems bottleneck: capacity control at the expert level can reduce overloaded dispatches without requiring expert replication, migration, or a more elaborate runtime scheduler.
-
The DeepSeek-V4-Pro result is a particularly compelling feasibility demonstration on a very large model: the vanilla MoE baseline runs out of memory before the first iteration, whereas CIPHER-MoE completes training under the reported configuration, reaches a global batch size of 8192, and outperforms the fixed router reference on the weighted OR average. Together with the evaluation across several large MoE models and the comparison of strict dropping, rerouting, and LocMoE, this supports the method's practical relevance. However, the different batch sizes mean the OOM result should be presented as a feasibility result rather than a directly matched speed comparison; the paper's attempt to separate signal retention from output fidelity in Appendix A.4.4 is a useful qualification of the theoretical claims.
Critical weaknesses
-
The main empirical comparison is not sufficiently controlled, and the key ablation is missing. CIPHER-MoE combines admission using affinity scores with the LocMoE locality loss, but it does not include a CIPHER variant without the locality loss or a vanilla routing baseline with the same locality loss. Thus the reported gains cannot be attributed to token admission. In addition, Table 3 compares a baseline that OOMs at GBS 256 with CIPHER-MoE at GBS 8192, so it does not establish a speed or quality comparison at the same optimization budget. The paper needs matched data, training steps or tokens, global batch size, hardware, and reporting of assignment dropping and rerouting rates, as well as repeated runs or uncertainty estimates.
-
The algorithmic and theoretical contracts are not sufficiently connected to the experiments. The exact rerouting procedure, candidate set, conflict resolution, gate renormalization, and whether additional router scores are computed are underspecified. The theorem only guarantees retention of an abstract discriminative witness under assumptions A1-A3; the manuscript does not estimate the certified margin, ranking parameters, or recall on learned representations, and explicitly distinguishes the theoretical margin from the implementation threshold. Consequently, the theorem does not support the claims about final model quality or general capability without substantially stronger empirical evidence.
Questions for authors
-
Which exact values of τ, capacity factor, fill-target factor, training steps, total processed tokens, seeds, and rates of assignment dropping and rerouting were used for each model and dataset? Were all methods trained with the same data exposure and optimization budget? Please reconcile the GBS 256 and GBS 8192 setup and interpretation in the OOM comparison in Table 3.
-
Please provide a 2x2 ablation separating token admission using affinity scores from the LocMoE locality loss: vanilla routing, locality loss only, admission only, and the full method. This ablation is required to determine which component causes the reported speed and quality changes.
-
For rerouting, does the method search only the original Top-K experts or all experts? Please specify the complete deterministic algorithm, including how capacity conflicts at the destination expert are resolved, whether assignments between tokens and experts that have already been accepted can be evicted, and how routing weights are normalized after dropping or rerouting.
-
How are the theoretical quantities δ, μ, η, and a₀ calibrated or measured on the evaluated checkpoints? Please report measurements of signal retention and output deviation, rather than using the illustrative Table 7 regime as evidence for the trained models.
-
The related-work discussion could be strengthened by situating CIPHER-MoE among complementary approaches to MoE efficiency, including the following works:
- Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts (He et al., ICLR 2026), which studies capacity-aware token dropping and rerouting during inference.
- EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference (Wu et al., ICML 2026), which addresses workload imbalance through cross-layer scheduling.
- Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression (Zhu et al., 2025), which explores expert reorganization, structured compression, and hierarchical routing.
These works address different settings and are not direct substitutes for CIPHER-MoE's training-time, post-Top-K admission mechanism. A concise comparison of their objectives, intervention points, and trade-offs would help clarify the novelty and scope of CIPHER-MoE. The discussion should focus on technically relevant distinctions rather than exhaustive citation coverage.