- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Uniform Pairwise Distillation (UPD), an auxiliary loss that uses capacity-induced rerouting decisions to improve MoE load balance. For each reassigned routing slot, UPD encourages the router to score the actual replacement expert above other experts. Experiments across several model sizes report reduced load imbalance and reassignment rates with comparable perplexity, alongside inference throughput gains in a two-GPU expert-parallel setting.
Critical strengths
- UPD offers a simple way to turn rerouting decisions into router supervision, without adding trainable parameters or a separate forward pass.
- The raw top-k evaluation in Table 1 shows improved load balance. The systems evaluation in Section 5.5 shows increased inference throughput in a two-GPU expert-parallel setting.
Critical weaknesses
- The training budgets are too limited to establish effectiveness beyond early training. The 350M, 1B, and 7B models receive only 20M, 40M, and 20M tokens, respectively (Tables 5–6). It remains unclear whether the gains persist after substantial expert specialization develops.
- Autoregressive causality is not clearly established. The repaired assignments are executed in the forward pass (§3.1), but the paper does not explain how competition for expert capacity excludes future positions within the same sequence. If future tokens can alter earlier tokens’ assignments, this introduces an information path that bypasses causal attention masking. The distillation loss further breaks it.
Questions for authors
- Could the authors evaluate downstream task accuracies? Tables 1–2 report only perplexity, including for LAMBADA.
- What is the end-to-end training overhead of UPD? The inference measurements in Section 5.5 do not quantify the training cost of applying the additional loss.
- How does the full UPD pipeline compare with MoE with higher-capacity or dropless MoE [1] in model quality and end-to-end efficiency? Since all experiments use a capacity factor of 1.0 (Table 5), it is unclear whether the benefits extend beyond the chosen strict-capacity setting.
[1] MegaBlocks: Efficient Sparse Training with Mixture-of-Experts https://arxiv.org/abs/2211.15841