- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work replaces a global expert pool with (G) parallel subspace MoE branches. Each branch has its own expert bank and router and selects a fixed number of experts, (q=k/G), per token. Every token participates in every branch, ensuring exactly equal assignment counts across branches while allowing input-dependent expert selection within each branch.
In the evaluated setting, the method reduces per-call all-to-all communication time while maintaining training-loss trajectories close to those of the baselines. The authors acknowledge that fixed all-to-all latency makes the communication-time reduction smaller than the payload reduction. A corresponding end-to-end speedup has not been demonstrated.
Critical strengths
- The assumptions underlying the balance guarantee are clearly specified, and the architectural definition aligns well with the distributed implementation. Under the stated communication convention, the payload savings are well supported and clearly quantified.
Critical weaknesses
- Mandatory participation in every branch restricts the space of possible MoE routing assignments. Although two routing granularities are evaluated, the experiments remain limited in scale and scope. It remains unclear whether comparable performance can be maintained in larger models or across a broader range of expert granularities and activation ratios.
Questions for authors
- Why do you report only training loss and not validation loss?
- Have you explored routing assignment strategies based on linear programming?
3 comments
Q2: Have you explored LP-based routing assignment
Response: We considered it, and LP addresses a different problem. Routing as a constrained assignment problem can use LP either to achieve equal per-group load, or to optimise assignment quality within a fixed quota.
EquiMoE targets the first structurally: because every branch receives every token and selects exactly q = k/G, the equal-quota property is a counting property of the forward operator. It holds for any input and any router parameters, with zero runtime cost and no solver. An LP solution is a strictly weaker guarantee — it depends on the data distribution, on solver convergence and on feasibility, and it requires a solve on every forward pass. Using LP to obtain what EquiMoE obtains for free would be a regression.
EquiMoE's router is also intra-branch: the inter-group constraint is resolved before routing (one reduced representation per rank), so no group-level assignment problem remains. The only LP-shaped question left is expert-level balance inside a branch, which §4.2 of the paper explicitly does not claim, and where n = 16, q = 2 makes the feasible set too small to be worth solving.
Finally, LP solutions are discrete and non-differentiable, so training would require a relaxation (Sinkhorn / optimal transport). Such relaxations optimise assignment quality but do not yield a zero-variance quota guarantee, and they add an iterative solve per forward pass. The revision adds this discussion, together with the feasible-set sizes above, to §4.3.
Additional data for W2 / Q2 (quality evidence at a larger scale)
This comment supplements our reply to W2 / Q2, where you noted that the reported results do not establish a quality advantage and that the evidence is at a single scale. It adds one experiment that was not in that reply, and does not repeat the material already posted there.
A second, much larger scale. We trained a 30 B-class model for 100 B tokens: 48 layers, d = 2048, 128 experts, top-8, EP = 4, Standard MoE against EquiMoE.
The two are matched on activated compute, not on total parameters: 3.34 B parameters activated per token in each (experts + attention + embeddings, within 0.1 %), while the totals differ — 30.5 B baseline versus 24.4 B EquiMoE. That is a choice, not a constraint: with the expert FFN width F as the knob, F = 2384 preserves activated compute (ours), F = 3029 preserves total parameters, F = 2901 preserves per-rank memory. We took F = 2384 because equal FLOPs per token is what makes step time and inference cost comparable; F = 3029 would have handed EquiMoE about 21 % more expert compute per token.
| At 30 B-class, 100 B tokens | Standard MoE | EquiMoE |
|---|---|---|
| total parameters | 30.5 B | 24.4 B |
| activated parameters per token | 3.34 B | 3.34 B |
| expert FFN width F | 768 | 2384 |
| MMLU (0-shot, full-option-text CE) | 37.89 % | 37.05 % |
| nine-task commonsense mean | 56.55 % | 55.56 % |
Reading of the result. EquiMoE is 0.84 points below the baseline on MMLU and 0.99 points below on the commonsense suite — approximate parity, not an advantage. The held-out-perplexity gain we observe at ~1.9 B does not reproduce at this scale, so the large-scale claim we are prepared to defend is "no meaningful degradation".
What this scale does not cover. No step-time or throughput measurement exists at 30 B, so the efficiency claim remains a ~1.9 B result; we do not extrapolate it.
Dear Reviewer rPL7,
We thank you for the concise assessment. We address your weakness and the two questions as follows.
Response: We agree. In the paper's own notation (
r = d/G,n = N/G,q = k/G), at N = 64, k = 8, G = 4 (so n = 16, q = 2), the unrestricted per-token feasible set is C(64,8) ≈ 4.43 × 10⁹ while EquiMoE's is C(16,2)⁴ ≈ 2.07 × 10⁸:On scope — what we tested. Two activation granularities at ~1.9 B total / ~1.0 B active (16 layers, EP = 4): top-8-of-64, and top-32-of-256. On top of that we trained a second model scale on 100 B tokens, 48 layers, d = 2048, 128 experts, top-8, EP = 4:
Response: The reviewer is right, and we want to state it plainly: the submitted version reported only training loss. That was an omission on our side, not a deliberate choice — we had no held-out evaluation at submission time, and we should not have let the training curve stand in for model quality. The revision adds the missing held-out measurements; they are set out in full below.
Held-out validation cross-entropy (nats) / perplexity.
Downstream, nine-task commonsense suite at 7.6 B tokens (accuracy %, batch = 1):
MMLU, disclosed rather than relied upon (accuracy %), under the 0-shot options-hidden full-option-text cross-entropy protocol; source: the early-signal protocol of FineWeb's harness,
https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py, whose rationale is that letter targets "give generally random results on small/non instruction tuned models", so the full answer text is the target.