acceptodds
Withdrawn from ICLR 2027

Learning Balanced Sparse Routing with Uniform Pairwise Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sparse Mixture‑of‑Experts (MoE) layers activate a small subset of experts per token, so practical runtime efficiency heavily depends on how evenly the router distributes token‑level demand. Skewed demand creates overloaded hot experts, underutilized experts, rank imbalance, and extra runtime repair overhead. Existing approaches either apply indirect global balancing losses, or only perform ephemeral runtime corrections. We introduce Uniform Pairwise Distillation (UPD), a lightweight local self‑distillation learning objective for training better‑balanced router proposals. For every route slot modified by the strict capacity‑aware execution policy during training, UPD constructs one sparse pairwise ranking target that favours the accepted replacement expert over the displaced expert. It assigns equal weight to each observed correction pair, reuses the router’s own logits as student signals, and introduces neither an external teacher network nor additional trainable parameters. Across matched from‑scratch MoE configurations ranging from 350M to 7B total parameters, UPD reduces raw‑demand imbalance and runtime reassignment work while maintaining perplexity comparable to the no‑distillation baseline. Improvements in routing behaviour transfer to held‑out in‑domain and out‑of‑domain test inputs. In a two‑rank expert‑parallel pipeline, these learned routing gains translate to higher prefill throughput and modest, repeatable decode improvements, while the original execution policy continues to enforce hard capacity feasibility guarantees.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Load Balancing, Router Distillation, Capacity-Constrained Routing, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bfqkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Uniform Pairwise Distillation (UPD), an auxiliary loss that uses capacity-induced rerouting decisions to improve MoE load balance. For each reassigned routing slot, UPD encourages the router to score the actual replacement expert above other experts. Experiments across several model sizes report reduced load imbalance and reassignment rates with comparable perplexity, alongside inference throughput gains in a two-GPU expert-parallel setting.

Critical strengths
  • UPD offers a simple way to turn rerouting decisions into router supervision, without adding trainable parameters or a separate forward pass.
  • The raw top-k evaluation in Table 1 shows improved load balance. The systems evaluation in Section 5.5 shows increased inference throughput in a two-GPU expert-parallel setting.
Critical weaknesses
  • The training budgets are too limited to establish effectiveness beyond early training. The 350M, 1B, and 7B models receive only 20M, 40M, and 20M tokens, respectively (Tables 5–6). It remains unclear whether the gains persist after substantial expert specialization develops.
  • Autoregressive causality is not clearly established. The repaired assignments are executed in the forward pass (§3.1), but the paper does not explain how competition for expert capacity excludes future positions within the same sequence. If future tokens can alter earlier tokens’ assignments, this introduces an information path that bypasses causal attention masking. The distillation loss further breaks it.
Questions for authors
  • Could the authors evaluate downstream task accuracies? Tables 1–2 report only perplexity, including for LAMBADA.
  • What is the end-to-end training overhead of UPD? The inference measurements in Section 5.5 do not quantify the training cost of applying the additional loss.
  • How does the full UPD pipeline compare with MoE with higher-capacity or dropless MoE [1] in model quality and end-to-end efficiency? Since all experiments use a capacity factor of 1.0 (Table 5), it is unclear whether the benefits extend beyond the chosen strict-capacity setting.

[1] MegaBlocks: Efficient Sparse Training with Mixture-of-Experts https://arxiv.org/abs/2211.15841

Reviewer_8ma4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Uniform Pairwise Distillation (UPD), a training-time auxiliary loss for a sparse Mixture-of-Experts router operating under a hard capacity-aware assignment policy. When a proposed token–expert route must be replaced to satisfy capacity, UPD uses the selected replacement and displaced expert as a detached pairwise target, increasing the former expert's router logit relative to the latter. It therefore teaches the router from execution corrections without adding a separate teacher model. In a three-seed 1B-scale scratch experiment on WikiText-103, the paper reports reducing the reassignment fraction from 11.177% to 4.173% and raw demand CV from 0.11875 to 0.01628, with similar perplexity. Under a route-equivalent optimized two-A800 implementation, prefill throughput rises by about 11.8%, while decode throughput rises by about 2.1%

Critical strengths
  1. Simple, interpretable, execution-aligned training signal. Equation (3) applies a smooth pairwise ranking loss only to slots altered by the feasibility policy. The mechanism is easy to implement when the runtime exposes before/after assignment indices, has no separate teacher forward pass, and does not add model parameters.

  2. Good decomposition of routing and systems effects. The paper evaluates raw router proposals separately from strict post-repair execution, and validates a route-equivalent optimized implementation before comparing speed. This is more informative than measuring the corrected, necessarily balanced expert load alone.

  3. Consistent directional improvements with appropriate controls. The 1B experiment uses matched initialization/data order across three seeds and includes Aux, Loss-Free Balancing, and MaxScore* baselines. Results are also reported on held-out C4 and LAMBADA, and at 350M and 7B-total model sizes. The observed router-demand effect is considerably larger and more consistent than the small perplexity changes.

Critical weaknesses
  1. The training signal represents batch-dependent feasibility rather than token-level semantic suitability. UPD supervises replacement r > displaced d solely because expert d was overloaded in that particular batch. The replacement can have a lower model score because it was chosen for availability, not because it is better for that token's prediction. The authors recognize the batch dependence by only supervising changed slots, but the objective still gives the same weight to every correction, including cases where the replacement is likely semantically inferior. This creates a possible conflict between balancing and expert specialization: the router is driven away from useful popular experts, and the expert identities available as replacements can change with batch composition, sequence packing, capacity factor, and expert parallelism. Can the authors quantify the quality/learned-specialization cost as training matures, e.g. expert-token affinity, routing entropy, or task loss conditional on changes? A simple variant weighted by the replaced-vs-original logit gap or score loss would also test whether uniform weighting is actually best.

  2. The capacity mechanism is a strong dependence of the method, not just an implementation detail. The experiments use capacity factor 1.0, exact top-k capacity, uniqueness constraints, and an augmenting-path rerouting policy. The objective receives labels only from this particular solver. Many production MoE stacks instead use dropless block-sparse kernels, different capacity factors, expert bias controllers, or alternative dispatch constraints; they may not present analogous replacements or may have very different bottlenecks. The paper acknowledges testing only one policy, but this sharply narrows the generality of the title and efficiency claims. The main revision priority is not a large new model: show at least a capacity-factor sweep (e.g., 1.0 versus moderate headroom) and a different batch/sequence configuration under the same model, identifying where the number of correction pairs becomes too small to help. Claims about general MoE serving should remain conditional on capacity-repair execution.

  3. The impressive routing metrics translate to modest optimized-system gains; the largest gains come from a slow original solver. The original rerouting implementation shows dramatic runtime improvement, but its repair path repeatedly reconstructs and sorts occupant lists. With the route-equivalent optimized solver, the headline benefit becomes +11.8% prefill throughput and +2.1% decode throughput, with about 5.9% improvement in optimized p95 latency (Appendix Table 17). This is still useful, but it indicates that much of the spectacular original-solver advantage depends on avoidable host-side overhead. Only two expert-parallel ranks on one host were tested, and optimized all-to-all breakdowns were not recorded (Appendix D). Please separate routing-kernel, host repair, all-to-all, and expert-GEMM time for the optimized path, and examine whether improvements persist at a second load/concurrency setting. A performance claim about MoE deployment needs to show which fraction of end-to-end latency is actually saved.

  4. The training regimes are too short to firmly establish quality preservation at deployment scale. The primary 973M-parameter model sees only 40M tokens; the randomly initialized 7B-total model is trained on just 20M tokens (Appendix Tables 5–6). These are useful controlled pilots for router dynamics, but they are very far from mature LM pretraining. The high frozen C4/LAMBADA perplexities in Table 2 (on the order of 10^3) underscore this limitation: lower reassignment on these inputs does not establish preservation of useful out-of-domain language-model quality. Longer training at one moderate scale, or router-only fine-tuning from a well-trained checkpoint, would substantiate the claim that UPD remains helpful after expert specialization forms. I would not count the 7B scratch pilot as evidence of production-scale language quality.

  5. The component ablation does not separate pairwise ranking from sparse targeting. Table 3 compares full-set cross-entropy, changed-token cross-entropy, and changed-slot pairwise UPD, simultaneously changing both which tokens receive labels and the loss form. The authors explicitly acknowledge this limitation. Also, broader CE alternatives have better PPL at that ablation point, even though UPD balances load best; the quality–balance trade-off matters. To support the specific proposed design, include pairwise loss on all changed tokens versus only changed slots, and a changed-slot cross-entropy/ranking control where only the surrogate differs. Sensitivity to the UPD coefficient, margin, temperature, and scheduling should be reported on one manageable-scale experiment.

Questions for authors
  1. How do the routing, perplexity, and systems gains change with capacity factor, batch size, and routing arity after pretraining has converged further?

  2. Under the optimized implementation, what are the detailed host repair, device compute, dispatch/all-to-all, prefill TTFT, and decode latency components? Which path explains +11.8% prefill but only +2.1% decode?

  3. In a factorized ablation, is the advantage due to changed-slot selection, pairwise loss form, or both? What is the accuracy trade-off relative to the broader CE targets?

  4. Does the learned router keep meaningful expert specialization, rather than simply moving toward uniformly spread but less useful assignments?

Reviewer_iMraOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a novel routing training method to improve the load balancing of sparse Mixture-of-Experts models. Using the routing corrections generated in the capacity-aware rerouting mechanism, the method constructs an additional supervision signal for self-distillation through pairwise ranking loss. Without introducing an external teacher or new parameters, it optimizes the load distribution of the router and boosts the throughput during model training. Experiments were conducted on MoE models with multiple parameter scales including 350M, 1B and 7B. The method shows better performance than baseline algorithms in multiple datasets, with sufficiently comprehensive ablation studies . In my opinion, this paper comes up with a simple yet effective routing training method that can be easily integrated into existing training systems to address the critical load balancing issue in MoE models,

Critical strengths

1.It proposes a straightforward approach that adds a ranking loss to enable the model to perform self-distillation during capacity-aware rerouting, so that the router can learn to achieve load balancing. 2.It is plug-and-play, and requires no additional parameters.

Critical weaknesses

1.The innovation is relatively limited. Both capacity-aware rerouting and pairwise ranking loss are existing technologies, and the main innovation of this paper lies in the combination. 2.According to Table 6 in the appendix, the 1B model was trained with only 40 million tokens, while the 7B model was trained with merely 20 million tokens. The limited amount of pre-training data only allows us to confirm that tge method works effectively in small training budget.

Questions for authors

The existing experiments cannot prove that the advantages of the proposed method can still be maintained on fully trained models. We suggest the authors provide new results after increasing the amount of training data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.