- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes AIR-MoE, a hierarchical routing method for highly granular mixture-of-experts models, where routing costs can be substantial. Compared with straightforward hierarchical routing, AIR-MoE enables learnable, overlapping expert groups through a codebook updated via gradient-free online clustering. The paper provides a lower bound on retained routing probability mass and reports improved perplexity at comparable FLOP budgets relative to existing hierarchical routers like PEER.
Critical strengths
- The idea is straightforward: hierarchical routing with a dynamic hierarchy.
- The ablations in Table 4 help establish the importance of adaptive codebook updates.
Critical weaknesses
-
My main concern is the limited experimental scale. Based on the architecture described in Section 5.1 and Appendix E.3, I estimate that the largest model has fewer than 300M active parameters and is trained on fewer than 5B tokens. Table 3 reports training FLOPs for the largest configurations. This limited scale leaves the effectiveness of AIR-MoE in practical, larger-scale MoE pretraining uncertain.
-
The efficiency of the proposed method is insufficiently established. Figures 3 and 6 compare isolated router forward/backward latency for AIR-MoE, PEER, and hierarchical routing, but no end-to-end training time or throughput is reported. It therefore remains unclear whether the routing savings translate into overall training speedups after accounting for gradient-free codebook adaptation, shortlist refreshes, and expert execution.
Questions for authors
-
Section 5.1 states that "the middle FFN is replaced with a MoE layer". Does this mean exactly one MoE layer is used in every compared method? Could the authors list the detailed configurations like routing dimensions, exact total and active parameter counts, and actual training token counts?
-
Table 4 reports 68.1% dead experts for Expert Choice, while Appendix F.5 states that experts outside the shortlist receive scores of . How are expert capacity, insufficient candidates, and fully masked candidates handled? Over what window is the dead-expert fraction measured?
-
Could the authors clarify how load balancing is implemented and how effectively it balances actual token loads across experts, particularly given the overlapping shortlists and reuse of hot experts? Is AIR-MoE more imbalanced than existing structured hierarchical routing methods like PEER?
2 comments
I thank the authors for their response and additional experiments. The end-to-end step-time comparison and implementation clarifications address my corresponding concerns. I accept that the observed dead experts and imbalance does not hurt quality in the reported experiments, although it can complicate expert-parallel deployment.
However, I still consider the largest experiment scale of 247.5M active parameters trained on 2.5B tokens insufficient to validate an MoE architecture for LLM pretraining.
Overall, the paper largely follows the PEER (https://arxiv.org/abs/2407.04153) setup and introduces a more refined hierarchical routing method. The improvements over PEER are encouraging, but the limited experimental scope and scale lead me to recommend weak rejection.
We thank the reviewer for noting that "the ablations in Table 4 help establish the importance of adaptive codebook updates".
W1: Experimental scale
The reviewer's estimate is correct: our largest model has 247.5M active parameters and is trained on 2.50B tokens (App. E.4). We agree that frontier-scale evaluation would be valuable. With academic compute, we cannot claim that this persists at frontier scale. We decided therefore to add a paragraph discussing this limitation in Sec. 6.
W2: End-to-end training efficiency
We agree that router latency alone does not determine training cost and therefore added an end-to-end benchmark (App. F.3): full optimizer steps of the large model (E=10242, K=128), mean ±1 std over 20 steps (see App. F.3 for full information).
bold: granular methods wihtin one standard deviation of the fastest granular method.
AIR-MoE matches the most efficient granular routers in step time and memory, while achieving lower perplexity than PEER (Tab. 1). We stress that codebook updates, shortlist refreshes, and expert execution are included in the measured step times.
Q1: Architecture and configurations
Yes, all methods use exactly one MoE layer, following [1], which isolates the effect of the router. App. E.4 now lists the routing dimension, number of experts, active experts, total/active/router parameter counts, and training tokens for every run.
Q2: Dead experts and masked candidates
We define a dead expert to be an expert that receives zero top-K assignments over the entire validation set (summed over all GPUs). We clarified this in the metric description of the revised manuscript. Most dead experts are never shortlisted: for the large C5 model, 16.3% of experts appear in no shortlist, which accounts for 99% of its 16.5% dead experts (App. F.9).
In default (token-choice) AIR-MoE, masking cannot leave a token short of experts: each token selects its top-K (K=128) among the M=8192 experts of its shortlist, and candidate scoring is dropless, without expert capacity.
Masking matters for the expert-choice variant (Tab. 10). In the token-expert score matrix, every expert outside a token's shortlist has score −∞, so only a fraction M/E<1% of all scores is finite. Each expert keeps its top C=⌈TK/E⋅c⌉ tokens (capacity factor c=1.25). Due to the skewed shortlist coverage (App. F.9), this causes two problems:
Rarely shortlisted experts therefore end up with few or no tokens that contribute, which explains the 68.1% dead experts.
Q3: Load balancing with overlapping shortlists
We use a Switch-style auxiliary loss E∑efepe over each token's shortlist and jitter at both routing stages (App. E.6). AIR-MoE is indeed less balanced than PEER in expert utilization (e.g., 63.8% vs. 1.2% dead experts on OpenWebText2 small; Tab. 9). This is partly by design: overlapping shortlists let hot experts serve many tokens, whereas PEER's product keys spread assignments structurally (each sub-key is shared by E experts). Imbalance did not hurt quality: preliminary experiments indicated that stronger load balancing worsened perplexity for the small models.
Once again, we thank the reviewer for the critical feedback. We hope that our responses address the concerns and would be grateful if the reviewer considered updating their assessment.
References
[1] He (2024), Mixture of A Million Experts, arXiv