acceptodds
Under review as a conference paper at ICLR 2027

EquiMoE: A Mixture-of-Experts Architecture with Load Balance by Construction

Abstract

Sparse mixture-of-experts (MoE) routing can concentrate assignments on a few expert groups, creating uneven work across expert-parallel ranks. EquiMoE fixes each group's assignment budget in the forward operator: every token enters every learned subspace branch, where an independent router selects a fixed number of experts from a disjoint bank. This gives exactly equal group assignment counts for any input and router parameters, while expert loads within each group may remain unequal. With one branch per rank, EquiMoE sends one reduced representation per token to each rank before local routing and returns one aggregated result per branch. The nominal per-call all-to-all payload is 4.19 MB, versus 33.55 MB for an assignment-expanded Standard MoE baseline: an 87.5% reduction under this dispatch convention. In four C4 runs of 20,000 steps (2.62 billion processed token positions), EquiMoE has equal rank assignment counts, constant measured payload, and training-loss trajectories close to the baselines. A LatentMoE-style scale-up comparison shows that reduced expert width with assignment-expanded dispatch does not yield the same payload saving. The guarantee concerns group assignments, not equal per-expert load. While EquiMoE's per-call communication time is approximately 26% lower than the baseline, this does not translate to proportional end-to-end speedup due to fixed all-to-all latency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Load Balance, Subspace Projection

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rPL7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work replaces a global expert pool with (G) parallel subspace MoE branches. Each branch has its own expert bank and router and selects a fixed number of experts, (q=k/G), per token. Every token participates in every branch, ensuring exactly equal assignment counts across branches while allowing input-dependent expert selection within each branch.

In the evaluated setting, the method reduces per-call all-to-all communication time while maintaining training-loss trajectories close to those of the baselines. The authors acknowledge that fixed all-to-all latency makes the communication-time reduction smaller than the payload reduction. A corresponding end-to-end speedup has not been demonstrated.

Critical strengths
  1. The assumptions underlying the balance guarantee are clearly specified, and the architectural definition aligns well with the distributed implementation. Under the stated communication convention, the payload savings are well supported and clearly quantified.
Critical weaknesses
  1. Mandatory participation in every branch restricts the space of possible MoE routing assignments. Although two routing granularities are evaluated, the experiments remain limited in scale and scope. It remains unclear whether comparable performance can be maintained in larger models or across a broader range of expert granularities and activation ratios.
Questions for authors
  1. Why do you report only training loss and not validation loss?
  2. Have you explored routing assignment strategies based on linear programming?
3 comments

Dear Reviewer rPL7,

We thank you for the concise assessment. We address your weakness and the two questions as follows.


W1: Mandatory participation restricts the routing space; scale and scope are limited

Response: We agree. In the paper's own notation (r = d/G, n = N/G, q = k/G), at N = 64, k = 8, G = 4 (so n = 16, q = 2), the unrestricted per-token feasible set is C(64,8) ≈ 4.43 × 10⁹ while EquiMoE's is C(16,2)⁴ ≈ 2.07 × 10⁸:

ConfigurationN, k, G, n, qfeasible set per tokenvs unconstrained
Standard MoEN = 64, k = 8, no group structureC(64,8) ≈ 4.43 × 10⁹1×
EquiMoE64, 8, 4, 16, 2C(16,2)⁴ ≈ 2.07 × 10⁸21.4× smaller

On scope — what we tested. Two activation granularities at ~1.9 B total / ~1.0 B active (16 layers, EP = 4): top-8-of-64, and top-32-of-256. On top of that we trained a second model scale on 100 B tokens, 48 layers, d = 2048, 128 experts, top-8, EP = 4:

At the 30 B scaleStandard MoEEquiMoE
total parameters30.5 B24.4 B
activated parameters per token (total)3.34 B3.34 B (within 0.1 %)
activated expert-path parameters1.81 B1.81 B (0.17 % apart)
expert FFN width F7682384
MMLU (0-shot full-option-text CE)37.89 %37.05 %
nine-task commonsense mean56.55 %55.56 %

Q1: Why report only training loss and not validation loss

Response: The reviewer is right, and we want to state it plainly: the submitted version reported only training loss. That was an omission on our side, not a deliberate choice — we had no held-out evaluation at submission time, and we should not have let the training curve stand in for model quality. The revision adds the missing held-out measurements; they are set out in full below.

Held-out validation cross-entropy (nats) / perplexity.

Config2.6 B CE2.6 B PPL7.6 B CE7.6 B PPL
Standard MoE3.10173022.23642.83811817.0836
EquiMoE3.08369621.83902.81911716.7620
LatentMoE3.10880422.39422.82548916.8692
EquiMoE ScaleUp3.08682121.90732.80112916.4632

Downstream, nine-task commonsense suite at 7.6 B tokens (accuracy %, batch = 1):

TaskStandard MoEEquiMoELatentMoEScaleUp
hellaswag41.6242.5442.0343.09
winogrande50.3649.8850.5950.04
piqa69.8071.0671.0071.27
siqa39.9739.0540.4839.82
openbookqa21.8022.4023.2023.00
arc_challenge24.3224.5723.4625.00
arc_easy45.8347.5647.7349.33
commonsense_qa34.3234.1536.6135.71
boolq59.8557.1654.1057.19
mean43.1043.1543.2543.83

MMLU, disclosed rather than relied upon (accuracy %), under the 0-shot options-hidden full-option-text cross-entropy protocol; source: the early-signal protocol of FineWeb's harness, https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py, whose rationale is that letter targets "give generally random results on small/non instruction tuned models", so the full answer text is the target.

ObjectMMLU accuracy
Standard MoE, 2.6 B tokens26.68
Standard MoE, 7.6 B tokens27.81
EquiMoE, 2.6 B tokens26.65
EquiMoE, 7.6 B tokens28.08
LatentMoE, 2.6 B tokens26.84
LatentMoE, 7.6 B tokens28.15
EquiMoE ScaleUp, 2.6 B tokens27.06
EquiMoE ScaleUp, 7.6 B tokens27.80
4-way chance25.00

Q2: Have you explored LP-based routing assignment

Response: We considered it, and LP addresses a different problem. Routing as a constrained assignment problem can use LP either to achieve equal per-group load, or to optimise assignment quality within a fixed quota.

EquiMoE targets the first structurally: because every branch receives every token and selects exactly q = k/G, the equal-quota property is a counting property of the forward operator. It holds for any input and any router parameters, with zero runtime cost and no solver. An LP solution is a strictly weaker guarantee — it depends on the data distribution, on solver convergence and on feasibility, and it requires a solve on every forward pass. Using LP to obtain what EquiMoE obtains for free would be a regression.

EquiMoE's router is also intra-branch: the inter-group constraint is resolved before routing (one reduced representation per rank), so no group-level assignment problem remains. The only LP-shaped question left is expert-level balance inside a branch, which §4.2 of the paper explicitly does not claim, and where n = 16, q = 2 makes the feasible set too small to be worth solving.

Finally, LP solutions are discrete and non-differentiable, so training would require a relaxation (Sinkhorn / optimal transport). Such relaxations optimise assignment quality but do not yield a zero-variance quota guarantee, and they add an iterative solve per forward pass. The revision adds this discussion, together with the feasible-set sizes above, to §4.3.

Additional data for W2 / Q2 (quality evidence at a larger scale)

This comment supplements our reply to W2 / Q2, where you noted that the reported results do not establish a quality advantage and that the evidence is at a single scale. It adds one experiment that was not in that reply, and does not repeat the material already posted there.

A second, much larger scale. We trained a 30 B-class model for 100 B tokens: 48 layers, d = 2048, 128 experts, top-8, EP = 4, Standard MoE against EquiMoE.

The two are matched on activated compute, not on total parameters: 3.34 B parameters activated per token in each (experts + attention + embeddings, within 0.1 %), while the totals differ — 30.5 B baseline versus 24.4 B EquiMoE. That is a choice, not a constraint: with the expert FFN width F as the knob, F = 2384 preserves activated compute (ours), F = 3029 preserves total parameters, F = 2901 preserves per-rank memory. We took F = 2384 because equal FLOPs per token is what makes step time and inference cost comparable; F = 3029 would have handed EquiMoE about 21 % more expert compute per token.

At 30 B-class, 100 B tokensStandard MoEEquiMoE
total parameters30.5 B24.4 B
activated parameters per token3.34 B3.34 B
expert FFN width F7682384
MMLU (0-shot, full-option-text CE)37.89 %37.05 %
nine-task commonsense mean56.55 %55.56 %

Reading of the result. EquiMoE is 0.84 points below the baseline on MMLU and 0.99 points below on the commonsense suite — approximate parity, not an advantage. The held-out-perplexity gain we observe at ~1.9 B does not reproduce at this scale, so the large-scale claim we are prepared to defend is "no meaningful degradation".

What this scale does not cover. No step-time or throughput measurement exists at 30 B, so the efficiency claim remains a ~1.9 B result; we do not extrapolate it.

Reviewer_Z9Y4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EquiMoE divides an MoE layer into learned subspace branches. Every token enters every branch, where a local router selects a fixed number of experts. This gives each expert group the same number of assignments and allows reduced representations to be communicated before expert selection. In the tested configuration, the per-call payload falls significantly, as well as all-to-all latency. C4 training losses remain close. However, I am not convinced that the architectural additions offer enough value over existing designs.

Critical strengths
  1. The architecture directly supports the proposed execution scheme. Equal group assignment counts follow from the fixed quotas. The paper also clearly explains why this guarantee does not imply balanced individual experts or equal device execution times.

  2. The distributed implementation shows lower communication volume and per-call latency under the tested dispatch convention.

Critical weaknesses
  1. The contribution beyond existing designs is not established convincingly. Equal per-group activation quotas already appear in [1]. EquiMoE adds learned subspaces and moves local routing after communication. The paper acknowledges this difference, but does not show which benefits require these additions. A comparison with simpler designs that retain the same group quotas is required to assess the value of the proposed combination.

  2. The reported results do not establish an overall quality/efficiency advantage. The quality comparisons use training loss from 1 seed, with no reported validation results. There are no end-to-end throughput or step-time measurements. The 87.5% payload reduction depends on assignment-expanded dispatch, which sends duplicate token representations to experts on the same device. For the tested LatentMoE configuration, destination deduplication and local aggregation would bound the logical activation payload by the same 4.19 MB as EquiMoE. This makes an optimized dispatcher an important missing baseline. Besides, the benefit of exact group balance is also unclear when the baseline’s reported rank-level CV is already only 1.77%.

[1] Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity

Questions for authors
  1. With parameters and compute matched, how does EquiMoE compare with MoGE using group quotas alone, or with group quotas and a shared latent projection?
  2. How do the baselines perform when each token is sent only once per destination device and expert outputs are combined locally (in terms of communication latency and end-to-end throughput)?
  3. Can the authors report held-out C4 loss for the existing checkpoints and results from additional seeds? Do the conclusions hold at matched compute?
  4. What is the total training-step time, including projections, backward communication, and gradient synchronization? How much time is saved specifically by eliminating group imbalance?
5 comments

Dear Reviewer Z9Y4,

We thank the reviewer. The central objection — that the contribution over existing designs and the overall quality/efficiency advantage are not established — is fair for the submitted version. Four of the five sub-objections are now answered with new measurement; the fifth we state as future work. Where the new measurements narrow our claims, we say so rather than defend the original framing.


W1 / Q1: Contribution beyond MoGE is not established

Response: We agree the equal-quota property is not ours; §2 of the paper says so and Proposition 1 is explicitly framed as "a counting property of equal quotas, also exploited by MoGE". What EquiMoE attaches to the quota is a per-branch learned projection of width R = d/G and an independent router over a disjoint expert bank. The consequence is not better balance — the balance is identical — but a different dispatch cost, because the token is projected down before it is sent: a quota-only design dispatches at full hidden width d = 2048, EquiMoE dispatches R = 512.

So we wrote it and ran it. Quota-only MoGE: 64 experts, top-8, EP = 4 → G = 4 groups with q = 2 selections per group, expert FFN 1024 at full width d = 2048, with the same data, optimiser, auxiliary-loss coefficients and seed as Standard MoE. The implementation reuses the standard Megatron MoE path verbatim and changes only the selection rule, so the parameter count is bit-identical to Standard MoE — 2 252 933 120 per rank (read from the training log), which meets "matched parameters and compute" exactly rather than approximately. The quota holds for arbitrary inputs: we verified exactly T·q selections per group in standalone checks, including an all-equal-logits case and a case where one group's logits were shifted by +50. Run length 20 000 steps.

Step time, each row against its own baselinemean over ~19.5 k stepsΔ
Standard MoE — 2.6 B tokens, seed 423689.8 ± 141.3 msreference
MoGE, quota-only — 2.6 B tokens, seed 423347.8 ± 87.6 ms−9.3 %
Standard MoE — 7.6 B tokens, seed 423692.5 ± 114.4 msreference
EquiMoE — 7.6 B tokens, seed 42, fused permute3156.6 ± 112.8 ms−14.6 %
Quality at 2.6 B tokensStandard MoEMoGE (ours)EquiMoE
Held-out C4 CE3.0975563.092862 (−0.0047)3.069347 (−0.0282)
Nine-task CS mean40.9941.0441.10

W2 / Q2: Results do not establish an overall quality/efficiency advantage; dedup is the missing baseline; CV is only 1.77 %

Response: The dedup baseline. We implemented the accounting the reviewer describes and measured its exact payload and per-call latency: a forward-only hook on the router output, 200 batches per checkpoint, 8 NPU, EP = 4, 419.4 M routed (token, layer) pairs per checkpoint over six checkpoints per model; latency from an isolated all-to-all benchmark with 4 800 samples per configuration and two runs agreeing within 2.4 %.

For Standard MoE, E[distinct destination ranks] = 3.653 ± 0.509 and dedup gives 15.33 ± 2.13 MB against EquiMoE's 4.19 MB — a 3.66× ratio, with 67 % of tokens hitting all four ranks. For LatentMoE the reviewer's conjecture is exactly right: dedup lands on 4.19 MB, ratio 1.00, because top-32 over 64 experts per rank hits all four ranks for 99.93–99.98 % of tokens and dedup degenerates to broadcasting. We therefore state plainly: destination dedup matches EquiMoE only in the latent-width regime; on standard-width MoE it remains 3.7× heavier, and unlike EquiMoE it provides no bound, because its payload depends on the routing distribution.

Baselinepayload / callper-call p50
Standard MoE, assignment-expanded33.554 MB0.2791 ms
Standard MoE, destination-dedup15.33 MB (min 4.19 / max 16.78)0.3933 ms (+40.9 %)
LatentMoE, destination-dedup4.19 MB (degenerates to broadcast)not measured separately
EquiMoE4.19 MB, routing-independent0.2531 ms

MMLU, for completeness (accuracy %, under the 0-shot options-hidden full-option-text cross-entropy protocol described in the answer to W2; the protocol source is the early-signal protocol of FineWeb's harness, https://huggingface.co/datasets/HuggingFaceFW/fineweb/blob/main/lighteval_tasks.py):

ObjectMMLU accuracy
Standard MoE, 2.6 B tokens26.68
Standard MoE, 7.6 B tokens27.81
EquiMoE, 2.6 B tokens26.65
EquiMoE, 7.6 B tokens28.08
4-way chance25.00

Q3: Single seed, no error bars

Response: We ran a second seed. Standard MoE and EquiMoE were retrained for 20 000 steps at seed 2345 with an otherwise identical recipe, and every 1 000-step checkpoint was evaluated on the same fixed held-out C4 stream (5 242 880 tokens, token_checksum = 1338436).

Configseed 42seed 2345Δ across seeds
Standard MoE3.101730 / 22.23643.097556 / 22.1438−0.0042 nats / −0.093 PPL (0.42 %)
EquiMoE3.083696 / 21.83903.069347 / 21.5278−0.0143 nats / −0.311 PPL (1.42 %)

The EquiMoE advantage replicates, and is larger on the second seed.

seedStandard MoEEquiMoEgap
423.1017303.083696−0.0180
23453.0975563.069347−0.0282

Learning curves (20 points per configuration, identical eval set). EquiMoE is below Standard MoE at every checkpoint from 1 000 steps on and the curves never cross:

stepStandard MoEEquiMoEMoGE (quota-only)
1 0004.5321084.4537124.536941
5 0003.5718853.5571073.566281
10 0003.3139003.3013283.310048
15 0003.1655263.1449723.161360
20 0003.0975563.0693473.092862

Q4: Total training-step time including projections, backward communication and gradient sync

Response: Yes — the reported quantity is Megatron's elapsed time per iteration, which spans the entire iteration: forward, backward, both all-to-all directions including the backward pass, the projection pairs, and gradient synchronisation. It is not a communication-only figure.

The −14.6 % decomposes qualitatively into three effects — 4× lower permute traffic (k·T·R = 8·T·512 = 2·T·2048 versus 8·T·2048), 4× lower all-to-all payload (G·T·R versus k·T·d), and fixed-size rather than routing-dependent split sizes — against the cost of one extra projection pair per layer (2d² parameters and 4Td² FLOPs per layer, already accounted for in Appendix E of the paper). We cannot attribute the 14.6 % across the three components without a profiling run, so we leave the attribution open here; that run is planned.

Additional data for W1 (scale and scope)

This comment supplements our reply to W1, where you observed that "the experiments remain limited in scale and scope" and that it remains unclear whether comparable performance can be maintained in larger models. It adds one experiment that was not in that reply, and does not repeat the material already posted there.

A second, much larger scale. We trained a 30 B-class model for 100 B tokens: 48 layers, d = 2048, 128 experts, top-8, EP = 4, Standard MoE against EquiMoE. This is a different scale from the ~1.9 B runs behind the rest of our results.

The two are matched on activated compute, not on total parameters: 3.34 B parameters are activated per token in each (experts + attention + embeddings, within 0.1 %), while the totals differ — 30.5 B for the baseline, 24.4 B for EquiMoE. The matching is a choice: with the expert FFN width F as the knob, F = 2384 preserves activated compute (ours), F = 3029 would preserve total parameters and F = 2901 per-rank memory. We took F = 2384 because equal FLOPs per token is what makes step time and inference cost comparable; F = 3029 would have given EquiMoE about 21 % more expert compute per token and tilted the comparison in its favour.

At 30 B-class, 100 B tokensStandard MoEEquiMoE
total parameters30.5 B24.4 B
activated parameters per token3.34 B3.34 B
expert FFN width F7682384
MMLU (0-shot, full-option-text CE)37.89 %37.05 %
nine-task commonsense mean56.55 %55.56 %

On your question about larger models. The architecture does run at this scale and the balance guarantee holds by construction, but the quality advantage does not survive: EquiMoE is 0.84 points below on MMLU and 0.99 points below on the commonsense suite — parity, not an improvement. The held-out-perplexity gain seen at ~1.9 B does not reproduce here. So the defensible claim at larger scale is "no meaningful degradation", and we say this in the revision rather than leaving the small-scale result to imply more.

On scope. We still have only two activation granularities (top-8-of-64, top-32-of-256) plus this second scale; a wider (G, k/N) sweep remains open, and the step-time measurement at 30 B is not available — we do not extrapolate the efficiency claim to this scale.

Reviewer_L5oaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces EquiMoE, an MoE architecture that ensures strictly the same load for each rank. To address the load balancing problem across expert-parallel ranks, EquiMoE splits the whole architecture into GG disjoint branches, each corresponding to a specific rank. When entering an EquiMoE layer, the token hidden state is projected into the sub-spaces of each branch, and each branch independently selects a fixed number q=k/Gq=k/G of experts from each rank containing n=N/Gn=N/G experts. Therefore, every branch activates exactly the same number of experts for any input.

The authors prove such a strict rank-level balance property and provide an implementation in MindSpeed-LLM. The experiments report four 20,000-step C4 experiments using Standard MoE, EquiMoE, EquiMoE ScaleUp, and LatentMoE. An 87.5% reduction in all-to-all payload is claimed under assignment-expanded dispatch, and the training losses are close to baselines.

Critical strengths
  1. The rank-level load balance claimed in this paper is solid and theoretically ensured. As proved in Section 4.1, through the branch-style architecture, there is no doubt that EquiMoE reaches a strict load balance across branches. This is a technically sound property for those expert-parallel training or deployment infrastructures.

  2. The communication reduction property of EquiMoE is well supported from multiple perspectives. As proved in Section 6.4, the authors not only demonstrate the communication reduction theoretically, but also measure the all-to-all communication volume in a real engineering implementation. I respect such a concrete and informative effort.

  3. This paper presents good transparency. The authors provide a complete forward operator, backward pass, and distributed cost accounting. The experimental details are well documented in the Appendix. These efforts promote the reliability of this work.

Critical weaknesses
  1. The efficiency claim is based on a favorable baseline and not well validated end to end.

The 87.5% communication reduction is measured against assignment-expanded dispatch, which replicates a full-width token for every selected expert. However, a standard MoE dispatcher can deduplicate tokens by destination rank and send each token once to that rank. In the tested E=4E=4 setting, this can eliminate most of the apparent advantage. This is also a problem that the authors acknowledge briefly in Appendix E.

More importantly, only per-call all-to-all time is reported. There is no end-to-end training step time, wall-clock throughput, inference latency, or memory usage data in this article. Admittedly, this is really a bad practice for a paper that focuses on the efficiency problem and training infrastructure.

  1. There is a lack of evidence supporting that the model quality of EquiMoE is close to baselines.

Concretely, the experiments in this paper only consist of one run per setting at one single model scale. While the paper reports training loss, it provides no validation perplexity, downstream-task accuracy, or statistical uncertainty. Besides, only the training loss curve is presented, but no more quantitative results (e.g., the average training loss of the last 100 steps) are included.

  1. The research question proposed in the Introduction is not fully answered.

In the second paragraph of Introduction, the authors raise the main research question: Can an MoE layer learn input-dependent expert specialization while keeping the amount of computation assigned to each expert group fixed?

However, there is no direct evidence that demonstrates the existence of input-dependent expert specialization in EquiMoE. The authors only claim that "complementary specialization may emerge from training". No analysis concerning expert specialization is conducted throughout this work.

Questions for authors
  1. Can the authors report end-to-end training throughput and inference latency data for both assignment-expanded and destination-deduplicated dispatch?

  2. Can the authors report more data concerning model quality, such as the validation perplexity, downstream-task results, and experiments on multiple model scales?

  3. Can the authors provide more analyses to demonstrate that the "complementary specialization" really emerges in EquiMoE? You may analyze something like the branch similarity, expert similarity, or the expert activation pattern on different fields of texts.

4 comments

Dear Reviewer L5oa,

We thank the reviewer for the careful reading. All three weaknesses are addressable, two of them came from our own reporting omissions, and we have added the missing measurements and corrected one implementation defect in our code.


W1: The efficiency claim is based on a favourable baseline and not validated end to end

Response: The reviewer is right on both counts, and the end-to-end part turned out to matter much more than the baseline part.

Summary table (8 NPU, EP = 4, seq 1024, bf16; per-call figures from the isolated benchmark, step time from the training logs):

Baselinepayload / callper-call p50
Standard MoE, assignment-expanded33.554 MB0.2791 ms
Standard MoE, destination-dedup15.33 MB (min 4.19 / max 16.78)0.3933 ms (+40.9 %)
EquiMoE4.19 MB, routing-independent0.2531 ms (−9.3 %)

Payload decomposition — four quantities that must not be conflated. For T = 1024 tokens, d = 2048, R = d/4 = 512, bf16, EP = 4, k = 8, G = 4:

#QuantityCrossesStandard MoELatentMoEEquiMoEHow obtained
①all-to-all, assignment-expanded (one copy per selected expert)device -> device link33.55 MB33.55 MB4.19 MBmeasured
②all-to-all, destination-dedup (one copy per distinct destination rank)device -> device link15.33 MB (max 16.78)4.19 MB4.19 MBE[ranks] measured x analytic width
③permute traffic (reordering tokens by expert id)intra-device HBM33.55 MB33.55 MB8.39 MBderived from dispatch width
④wall-clock step time at 7.6 B tokensnot a data movement3692.5 +/- 114.4 ms3975.8 +/- 122.9 ms3156.6 +/- 112.8 msmeasured

Time decomposition — what the isolated primitive can and cannot explain. A training step performs 16 layers x 2 directions = 32 all-to-all calls:

ComponentStandard MoEEquiMoEBasis
all-to-all, 32 x measured per-call latency8.93 ms8.10 ms0.2791 / 0.2531 ms per call, measured
as a fraction of step time0.24 %0.26 %3692.5 / 3156.6 ms
permute traffic per step1.07 GB0.27 GB(3) x 32, i.e. 4x reduction
remainder (attention, expert GEMM, optimizer, gradient sync)3683.6 ms3148.5 msby subtraction

W2: Model quality is not shown to be close to baselines

Response: Protocol. Held-out C4 validation loss on a single fixed stream used for every checkpoint of every configuration: the last five shards of the 50-shard corpus (45–49, i.e. the trailing 10 % of the shard-granular 90/10 split), 1024 non-overlapping 1024-token windows from the start of each shard, so 5 242 880 tokens per measurement with token_checksum = 1338436 identical across all runs. Checkpoints use padded_vocab = 151936 while token ids stay ≤ 50276, so cross-configuration comparison is fair although absolute PPL is not comparable to published numbers with a 50304 vocabulary.

Cross-entropy (nats) / perplexity.

Config2.6 B tokens7.6 B tokens
Standard MoE3.101730 / 22.23642.838118 / 17.0836
EquiMoE3.083696 / 21.83902.819117 / 16.7620
LatentMoE3.108804 / 22.39422.825489 / 16.8692
EquiMoE ScaleUp3.086821 / 21.90732.801129 / 16.4632

MMLU under that protocol (accuracy %):

ObjectMMLU accuracy
Standard MoE, 2.6 B tokens26.68
Standard MoE, 7.6 B tokens27.81
EquiMoE, 2.6 B tokens26.65
EquiMoE, 7.6 B tokens28.08
LatentMoE, 2.6 B tokens26.84
LatentMoE, 7.6 B tokens28.15
EquiMoE ScaleUp, 2.6 B tokens27.06
EquiMoE ScaleUp, 7.6 B tokens27.80
4-way chance25.00

Nine-task commonsense suite at 7.6 B tokens (accuracy %, batch = 1):

TaskStandard MoEEquiMoELatentMoEScaleUp
hellaswag41.6242.5442.0343.09
winogrande50.3649.8850.5950.04
piqa69.8071.0671.0071.27
siqa39.9739.0540.4839.82
openbookqa21.8022.4023.2023.00
arc_challenge24.3224.5723.4625.00
arc_easy45.8347.5647.7349.33
commonsense_qa34.3234.1536.6135.71
boolq59.8557.1654.1057.19
mean43.1043.1543.2543.83

W3 / Q3: Is there evidence of input-dependent expert specialisation?

Response: Yes — on top of the assertion in the submission we now have four analyses. Setup. We instrumented the projection outputs and the routers of all four configurations over 10 batches × 4 domains (C4, commonsense, MMLU, CMMLU) × 2048 tokens = 81.9 k tokens per domain per configuration, across all 16 layers.

IndicatorStandard MoEEquiMoELatentMoEScaleUp
Branch-representation cosine, same tokennot applicable (single shared projection)0.000 ± 0.003not applicable (single shared projection)−0.003 ± 0.012
Same-group / cross-group co-activation ratio1.010.531.010.88
Load entropy H / log E0.94660.98260.91840.9325
Effective experts / E0.8030.9310.6450.696
KL_between / KL_within4.035.962.692.94
Route Jaccard under a context change0.1630.1660.1600.156

Q1: End-to-end training throughput and inference latency

Response: Training throughput (measured from the logs, same protocol as W1 above; throughput computed from p50 step time): EquiMoE 41,753 tok/s, EquiMoE ScaleUp 38,470, Standard MoE 35,649, LatentMoE 33,166 — i.e. +17.1 % for EquiMoE at 7.6 B tokens. Per-configuration distributions are reported as mean ± std with min / p50 / p90 / p99 / max / n and the CV (1.6–4.4 %). Percentiles are read from the ascending-sorted sample at index ⌊n·q⌋ (for even n the median is the upper of the two middle values); we quote p99 to expose tail latency rather than the typical step. The only outlier in each run is the first iteration after launch (kernel compilation); we keep it rather than trim silently.

Inference latency (2.6 B-token checkpoints, 8 NPU, EP = 4, PP = 1, bf16, KV cache enabled, 10 warm-up + 50 timed iterations, decode of 128 tokens from a 1024-token prompt; every decode run verified against a non-cached prefill with cosine ≈ 1.0 and max_abs_diff = 0.0):

Standard MoEEquiMoEScaleUpΔ (EquiMoE vs base)
prefill, seq 51265.80 ms59.19 ms60.78 ms−10.0 %
prefill, seq 102469.89 ms60.21 ms65.32 ms−13.9 %
prefill, seq 2048104.21 ms106.31 ms120.96 ms+2.0 %
decode63.27 ms/token55.14 ms/token54.00 ms/token−12.9 %
decode throughput15.81 tok/s18.13 tok/s18.52 tok/s+14.7 %
load-time peak HBM4301.6 MiB4555.9 MiB4281.3 MiB+5.9 %

Q2: Validation perplexity, downstream results, multiple scales

Response: All three are covered in W2 above; we repeat the essentials so this answer stands alone.

Validation perplexity on the fixed 5.24 M-token C4 stream (token_checksum = 1338436, trailing-10 % shards 45–49, 1024 × 1024-token windows): at 7.6 B tokens, ScaleUp 16.46 < EquiMoE 16.76 < LatentMoE 16.87 < Standard MoE 17.08 (PPL); at 2.6 B tokens the order is EquiMoE 21.839 < ScaleUp 21.907 < Standard MoE 22.236 < LatentMoE 22.394. Standard error ≈1×10⁻³ nats on 5.24 M tokens; the base-vs-ScaleUp gap of 0.037 nats is ≥10σ paired.

Downstream: nine commonsense tasks, all eight checkpoints. 2.6 B tokens → 7.6 B tokens means: base 41.42 → 43.10, EquiMoE 41.28 → 43.15, LatentMoE 41.30 → 43.25, ScaleUp 41.29 → 43.83. MMLU is disclosed as non-discriminative at this training budget (a 1.5-point band across all configurations, only 1.65 points above 4-way chance) and is not offered as evidence.

Multiple scales: two activation granularities (top-8-of-64 and top-32-of-256) at ~1.9 B total, plus a larger model (128 experts, top-8, 100 B tokens) where baseline and EquiMoE activate 3.34 B parameters per token each while totalling 30.5 B and 24.4 B respectively — matched on activated compute, with F = 2384 chosen over F = 3029 (total-parameter preserving) because FLOPs per token are what govern cost. On that matching EquiMoE is −0.84 points on MMLU and −0.99 on the CS suite — parity, reported as parity.

Additional data for W2 / Q2 (multiple scales)

This comment supplements our reply to W2 (model quality) and Q2 (validation perplexity, downstream results, multiple scales). It adds one experiment that was not in that reply, and does not repeat the material already posted there.

A second, much larger scale. We trained a 30 B-class model for 100 B tokens: 48 layers, d = 2048, 128 experts, top-8, EP = 4, Standard MoE against EquiMoE.

The two models are matched on activated compute, not on total parameters: 3.34 B parameters are activated per token in each (experts + attention + embeddings, within 0.1 %), while the totals differ — 30.5 B for the baseline and 24.4 B for EquiMoE. That difference is a deliberate choice, not a constraint: with the expert FFN width F as the knob, F = 2384 preserves activated compute (our choice), F = 3029 would preserve total parameters and F = 2901 per-rank memory. We chose F = 2384 because equal FLOPs per token is what makes step time and inference cost comparable; F = 3029 would have given EquiMoE about 21 % more expert compute per token and tilted the comparison the other way.

At 30 B-class, 100 B tokensStandard MoEEquiMoE
total parameters30.5 B24.4 B
activated parameters per token3.34 B3.34 B
expert FFN width F7682384
MMLU (0-shot, full-option-text CE)37.89 %37.05 %
nine-task commonsense mean56.55 %55.56 %

Reading of the result. EquiMoE is 0.84 points below the baseline on MMLU and 0.99 points below on the commonsense suite, i.e. approximate parity rather than an improvement. The positive held-out-perplexity delta we report at the small scale does not reproduce here, so the defensible large-scale claim is "no meaningful degradation", not "better".

What this scale does not cover. We have no step-time or throughput measurement at 30 B, so the efficiency result remains established at ~1.9 B only. We prefer to say so rather than extrapolate: the payload accounting is architecture-level and should carry over, but the step-time gain depends on the fixed-cost/per-payload balance, which we have measured only at the smaller scale.

why the dedup dispatcher is slower on this platform (W1)

This comment supplements our reply to W1 (the efficiency claim and the missing dedup baseline). It supplies the explanation behind the latency numbers already posted there, and does not repeat them.

That reply reports 0.3933 ms per call for dedup against 0.2791 ms for assignment-expanded (+40.9 %), and that a real dedup assignment is padded from 15.10 to 22.35 MB. The cause is a property of the transport, and it deserves to be stated plainly:

  • This stack has no one-sided communication. --moe-token-dispatcher-type accepts only allgather and alltoall, and token_dispatcher.py contains only MoEAllGatherTokenDispatcher and MoEAlltoAllTokenDispatcher.
  • The upstream implementation that does dedup natively is not available here. NVIDIA's Megatron-LM ships a flex dispatcher (MoEFlexTokenDispatcher), documented as removing "redundant tokens during cross-node communication" — precisely the baseline you asked for. It requires DeepEP/NVSHMEM, which has no Ascend backend.
  • So a dedup assignment must be expressed as a collective all-to-all, and HCCL requires equal per-rank send/recv totals (AICPU error 507018 otherwise). Padding a 4.94×-imbalanced assignment is what produces the +48 %, and with variable-length all-to-all being bandwidth-bound (9.2 µs/MB versus 1.05 µs/MB for fixed-length) it is also what makes the per-call time worse.

In short: the +48 % is a property of the transport available on Ascend, not of our packing, and this constraint does not exist on the NVIDIA path. Our dedup dispatcher and the all-to-all microbenchmark are research code, so their absolute timings carry prototype overhead; we argue from ratios rather than absolutes.

Comments on the paper

1 comment

Dear Area Chair,

We thank the reviewers for their comments and you for handling the submission. We ran the experiments they asked for and revised the manuscript accordingly. Below is a summary by reviewer; the detailed replies are posted under each review.

  • Reviewer L5oa (Rating 2, Confidence 3) asked for the destination-dedup baseline and for evidence that model quality is close to the baselines. We implemented the dedup accounting and the dispatcher: on Standard MoE its payload is 3.66× EquiMoE's, and on Ascend it is 40.9 % slower than assignment-expanded because the stack has no one-sided communication. We also report end-to-end step time for all configurations (−14.6 % for EquiMoE, after we found and fixed an unfused-permute defect in our own layer), plus held-out C4 perplexity, the nine-task commonsense suite, a second seed, and a 30 B-class scale.

  • Reviewer Z9Y4 (Rating 1, Confidence 3) asked what EquiMoE adds over MoGE, and noted the single seed and the 1.77 % rank-level CV. No public MoGE implementation exists, so we wrote one with bit-identical parameters and trained it: it recovers about two-thirds of the speed gain but only about one-sixth of the quality gain, a narrowing we state ourselves. A second seed replicates the EquiMoE advantage, and the CV argument is answered by the per-token distinct-rank count rather than the aggregate CV.

  • Reviewer rPL7 (Rating 1, Confidence 4) asked about the restricted routing space, about validation loss, and about LP-based assignment. We quantify the restriction (21.4× fewer feasible subsets at the tested configuration), report full held-out perplexity and commonsense results, and explain why an LP relaxation is a strictly weaker guarantee than a counting property.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.