acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts

Abstract

Sparse Mixture-of-experts (MoE) models enable scalable transformer architectures by activating only a subset of experts per token. Recent evidence suggests that performance improves with increasingly granular experts, i.e., many small experts instead of a few large ones. However, this regime substantially increases routing cost, which can dominate computation and memory. We introduce adaptive inverted-index routing for MoE (AIR-MoE), an inverted-index-inspired routing architecture based on vector quantization (VQ). In a first stage, AIR-MoE performs coarse shortlisting by assigning tokens to VQ codewords to construct a candidate set of experts. In a second stage, fine scoring computes exact routing scores restricted to this shortlist. This two-stage procedure approximates true top-K routing while avoiding full expert scoring and, in contrast to prior work, imposing no structural constraints on expert parameters. AIR-MoE serves as a drop-in replacement for standard routers and requires no modifications to the model architecture or loss function. We further provide a lower bound on the mass recall achieved by AIR-MoE that yields insights into the inner workings. Empirically, AIR-MoE improves the perplexity–FLOPs trade-off over existing MoE routers in regimes with up to more than a million tiny experts, and achieves consistent perplexity improvements over the best baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts, transformer, large language model, inverted index

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_GQvuOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes AIR-MoE, a hierarchical routing method for highly granular mixture-of-experts models, where routing costs can be substantial. Compared with straightforward hierarchical routing, AIR-MoE enables learnable, overlapping expert groups through a codebook updated via gradient-free online clustering. The paper provides a lower bound on retained routing probability mass and reports improved perplexity at comparable FLOP budgets relative to existing hierarchical routers like PEER.

Critical strengths
  • The idea is straightforward: hierarchical routing with a dynamic hierarchy.
  • The ablations in Table 4 help establish the importance of adaptive codebook updates.
Critical weaknesses
  • My main concern is the limited experimental scale. Based on the architecture described in Section 5.1 and Appendix E.3, I estimate that the largest model has fewer than 300M active parameters and is trained on fewer than 5B tokens. Table 3 reports 7.8×10187.8\times10^{18} training FLOPs for the largest configurations. This limited scale leaves the effectiveness of AIR-MoE in practical, larger-scale MoE pretraining uncertain.

  • The efficiency of the proposed method is insufficiently established. Figures 3 and 6 compare isolated router forward/backward latency for AIR-MoE, PEER, and hierarchical routing, but no end-to-end training time or throughput is reported. It therefore remains unclear whether the routing savings translate into overall training speedups after accounting for gradient-free codebook adaptation, shortlist refreshes, and expert execution.

Questions for authors
  • Section 5.1 states that "the middle FFN is replaced with a MoE layer". Does this mean exactly one MoE layer is used in every compared method? Could the authors list the detailed configurations like routing dimensions, exact total and active parameter counts, and actual training token counts?

  • Table 4 reports 68.1% dead experts for Expert Choice, while Appendix F.5 states that experts outside the shortlist receive scores of −∞-\infty. How are expert capacity, insufficient candidates, and fully masked candidates handled? Over what window is the dead-expert fraction measured?

  • Could the authors clarify how load balancing is implemented and how effectively it balances actual token loads across experts, particularly given the overlapping shortlists and reuse of hot experts? Is AIR-MoE more imbalanced than existing structured hierarchical routing methods like PEER?

2 comments

We thank the reviewer for noting that "the ablations in Table 4 help establish the importance of adaptive codebook updates".

W1: Experimental scale

The reviewer's estimate is correct: our largest model has 247.5247.5M active parameters and is trained on 2.502.50B tokens (App. E.4). We agree that frontier-scale evaluation would be valuable. With academic compute, we cannot claim that this persists at frontier scale. We decided therefore to add a paragraph discussing this limitation in Sec. 6.

W2: End-to-end training efficiency

We agree that router latency alone does not determine training cost and therefore added an end-to-end benchmark (App. F.3): full optimizer steps of the large model (E=10242E=1024^2, K=128K=128), mean ±1\pm 1 std over 20 steps (see App. F.3 for full information).

MethodStep time (ms)Throughput (tokens/s)Rel. step timeAdditional peak memory per step (GiB)
AIR-MoE2801 ± 12011,7001.00×66.7
PEER2759 ± 12811,8800.99×66.9
Hierarchical2720 ± 13112,0500.97×66.4
Std. Granular (exact top-KK)4474 ± 1417,3201.60×65.6

bold: granular methods wihtin one standard deviation of the fastest granular method.

AIR-MoE matches the most efficient granular routers in step time and memory, while achieving lower perplexity than PEER (Tab. 1). We stress that codebook updates, shortlist refreshes, and expert execution are included in the measured step times.

Q1: Architecture and configurations

Yes, all methods use exactly one MoE layer, following [1], which isolates the effect of the router. App. E.4 now lists the routing dimension, number of experts, active experts, total/active/router parameter counts, and training tokens for every run.

Q2: Dead experts and masked candidates

We define a dead expert to be an expert that receives zero top-KK assignments over the entire validation set (summed over all GPUs). We clarified this in the metric description of the revised manuscript. Most dead experts are never shortlisted: for the large C5 model, 16.3%16.3\% of experts appear in no shortlist, which accounts for 99%99\% of its 16.5%16.5\% dead experts (App. F.9).

In default (token-choice) AIR-MoE, masking cannot leave a token short of experts: each token selects its top-KK (K=128K=128) among the M=8192M=8192 experts of its shortlist, and candidate scoring is dropless, without expert capacity.

Masking matters for the expert-choice variant (Tab. 10). In the token-expert score matrix, every expert outside a token's shortlist has score −∞-\infty, so only a fraction M/E<1%M/E < 1\% of all scores is finite. Each expert keeps its top C=⌈TK/E⋅c⌉C = \lceil TK/E \cdot c \rceil tokens (capacity factor c=1.25c = 1.25). Due to the skewed shortlist coverage (App. F.9), this causes two problems:

  • Rarely shortlisted experts have fewer than CC tokens with finite scores. The rest of their capacity goes to tokens with score −∞-\infty, which receive weight 00 and no gradient. An expert in no shortlist is never selected.
  • Frequently shortlisted experts have far more than CC candidate tokens and drop the rest. A token can therefore receive fewer than KK experts, and empty slots get weight 00. If all KK slots are empty, the MoE layer contributes nothing for that token (the residual connection is unaffected).

Rarely shortlisted experts therefore end up with few or no tokens that contribute, which explains the 68.1%68.1\% dead experts.

Q3: Load balancing with overlapping shortlists

We use a Switch-style auxiliary loss E∑efepeE\sum_e f_e p_e over each token's shortlist and jitter at both routing stages (App. E.6). AIR-MoE is indeed less balanced than PEER in expert utilization (e.g., 63.8%63.8\% vs. 1.2%1.2\% dead experts on OpenWebText2 small; Tab. 9). This is partly by design: overlapping shortlists let hot experts serve many tokens, whereas PEER's product keys spread assignments structurally (each sub-key is shared by E\sqrt{E} experts). Imbalance did not hurt quality: preliminary experiments indicated that stronger load balancing worsened perplexity for the small models.


Once again, we thank the reviewer for the critical feedback. We hope that our responses address the concerns and would be grateful if the reviewer considered updating their assessment.

References

[1] He (2024), Mixture of A Million Experts, arXiv

I thank the authors for their response and additional experiments. The end-to-end step-time comparison and implementation clarifications address my corresponding concerns. I accept that the observed dead experts and imbalance does not hurt quality in the reported experiments, although it can complicate expert-parallel deployment.

However, I still consider the largest experiment scale of 247.5M active parameters trained on 2.5B tokens insufficient to validate an MoE architecture for LLM pretraining.

Overall, the paper largely follows the PEER (https://arxiv.org/abs/2407.04153) setup and introduces a more refined hierarchical routing method. The improvements over PEER are encouraging, but the limited experimental scope and scale lead me to recommend weak rejection.

Reviewer_R6bhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Adaptive Inverted-Index Routing for Mixture-of-Experts (AIR-MoE), a two-stage routing architecture designed for granular MoE models that contain up to a million tiny experts.

Critical strengths
  1. Unconstrained and flexible coarse-to-fine routing for massive expert pools. Prior methods for scaling to a million experts rely on either rigid disjoint partitions (hierarchical MoE) or Cartesian-product structural constraints on expert weights (PEER), both of which create artificial routing patterns. By adapting inverted file indexing and vector quantization to MoE routing, AIR-MoE allows shortlists to overlap naturally so broadly useful experts can be shared across many regions of the input space while keeping runtime and memory comparable to PEER.

  2. Clean decoupled training design and thorough empirical analysis. Updating the quantization codebook via online exponential moving average k-means while letting task gradients flow directly through the fine-scoring stage avoids gradient estimation tricks and makes AIR-MoE a simple drop-in replacement. The paper supports this design with a clear theoretical mass-recall bound, multi-seed verification, hardware efficiency benchmarks using grouped matrix multiplication, and insightful visualizations of expert utilization and shortlist overlap.

Critical weaknesses
  1. Single-layer MoE evaluation and undertrained large-scale regime. Following the PEER setup, the experiments only replace a single middle feed-forward layer of the transformer with an MoE layer rather than placing MoE layers throughout the network.
  • The training token budget on C5 and OpenWebText2 is relatively small (ten times the non-MoE backbone parameter count), which appears to cause the Large model to achieve worse validation perplexity than the Medium model on C5 across all tested methods (46.84 versus 43.85 for AIR-MoE).
  1. High dead-expert rates and shortlist refresh overhead in smaller-batch regimes. Despite applying load-balancing losses and jitter noise at both routing stages, AIR-MoE still leaves a large fraction of experts completely unused in several configurations (for example, 34% to 64% dead experts across the OpenWebText2 models, which is often much higher than PEER).
Questions for authors
  1. In Table 1, why does the Large model have worse validation perplexity than the Medium model on the C5 dataset across all four routing methods?

  2. What causes AIR-MoE to have a noticeably higher percentage of dead experts than PEER in several settings? Does the Switch load-balancing loss struggle when applied only over the shortlisted candidates, or are certain experts never included in any codeword's top-M shortlist?

2 comments

We thank the reviewer for the positive assessment, in particular for highlighting that AIR-MoE "allows shortlists to overlap naturally" while "keeping runtime and memory comparable to PEER", and its "clean decoupled training design and thorough empirical analysis".

We will go through the remaining concerns, in order:

W1: Single MoE layer

We follow the PEER setup [1]: replacing one middle FFN isolates the effect of the router, since all other components are identical across methods. This setup allows to scale up the expert layer to the million-expert regime, while remaining feasible to train on our academic compute budget.

W2 / Q1: Token budget and Large vs. Medium on C5

Due to our compute budget, each model is trained on 10×10\times its dense parameter count in tokens (449449M / 1.381.38B / 2.502.50B; App. E.4). All models use one fixed recipe (token budget and hyperparameters) shared by all sizes and routers, which keeps the comparison controlled. Under this recipe, all models are still improving at the end of training (e.g., −4.3%-4.3\% validation PPL over the final 10%10\% of tokens for AIR-MoE Large; learning curves in App. F.2), and the Large backbone does not surpass Medium on C5. Since this holds for all four routers alike, it concerns the shared training setup and not the routing. AIR-MoE remains the best router at every size (Large: 46.8446.84 vs. 47.8647.86 for PEER). We discuss this in Sec. 5.2.

W3 / Q2: Dead experts and shortlist refresh overhead

The load-balancing loss acts on each token's shortlist. An expert that appears in no codeword's top-MM shortlist receives neither routing mass nor load-balancing signal, and its centroid receives no task gradient. It can only re-enter after a shortlist refresh once codewords or other centroids move. This is visible in the 0 bin of the shortlist-coverage histogram (App. F.9): for the large C5 model, 16.3%16.3\% of experts appear in no shortlist, which accounts for almost all (99%99\%) of its 16.5%16.5\% dead experts (Tab. 9). Experts that are contained in at least one shortlist are thus almost never dead, so the load-balancing loss is effective.

Effect on quality. Fewer active experts do not imply worse perplexity. For the small model, stronger load balancing improved expert utilization, but worsened perplexity. On C5 medium, cosine assignment reduces dead experts from 13.7%13.7\% to 8.4%8.4\% but raises PPL from 43.8543.85 to 44.1144.11 (Tab. 10). PEER's higher utilization is largely structural: each expert's score is the sum of two product-key scores, each shared with E−1\sqrt{E}-1 other experts, so every sub-key update shifts the scores of E\sqrt{E} experts at once. AIR-MoE scores each expert with its own centroid and can concentrate tokens on fewer experts, while reaching lower perplexity than PEER (Tab. 1).

Refresh overhead. It is true that AIR-MoE is particular efficient for large-batch training. However, we note that even for smaller physical batched, gradient accumulation can amortize the shortlist refresh over all micro-batches.


Once again, we thank the reviewer for the positive and constructive feedback. We hope that our responses further strengthen the reviewer's confidence in our work. We would be happy about a score increase, if this is the case.

I thank the authors for their response. I maintain my score of 3 as I believe the paper is a small but solid contribution to the literature.

Reviewer_S4M9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper targets the routing bottleneck in very granular MoE layers with up to about a million (~10⁶) experts. AIR-MoE borrows the inverted file indices idea, which performs an approximate nearest-neighbor search. It learns an exponential moving average k-means codebook with G codewords without gradients and maps every token to one codeword. Each codeword stores a cached shortlist of the top-M experts by inner product with the codeword and is refreshed once per optimizer step. Then, the token is scored exactly with those M experts. The paper also gives a lower bound on the routing mass the shortlist keeps. Across WikiText-103, C5 and OpenWebText2, AIR-MoE improves perplexity by 0.7%–5.4% over the best baseline at matched FLOPs, with router cost comparable to PEER and far below exact granular routing.

Critical strengths

The AIR-MoE has no structural constraints on expert parameters and it has a comparable routing cost to PEER. This core contribution sets it apart from prior granular routers.

The consistent perplexity gains over the best existing baseline. AIR-MoE beats PEER in every setting of Tab. 1 and in all three seeds of Tab. 2, at nearly identical training FLOPs Tab. 3.

Critical weaknesses

The claim that AIR-MoE approximates true top-K routing seems not clearly supported by the results. The paper presents AIR-MoE as an approximation of true top-K routing. However, on C5 medium, AIR-MoE overlaps with only 58% of the true top-K experts, and all variants with higher overlap (0.65–0.81) have worse perplexity (Tab. 4). Moreover, a static codebook with only 5% overlap still beats the hierarchical and coarse baselines (47.81 vs. 51.45 and 49.81 PPL; Tab. 1). The gains may therefore come from the shortlist structure itself rather than from approximating top-K. Since the paper does not include an exact top-K router in the quality experiments, it cannot tell these apart.

Questions for authors
  1. On C5 medium, AIR-MoE overlaps with only 58% of the true top-K experts, and all variants with higher overlap have worse perplexity (Tab. 4). The paper itself notes that “overlap alone is not a reliable proxy for effective routing” (Sec. F.5). Do you think AIR-MoE’s gains come from the quality of its top-K approximation, or the shortlist structure itself?
  2. Training uses multiple GPUs (Sec. E.6). Are codebook and EMA statistics synchronized across GPUs, and if so, how? If not, which GPU’s codebook is used for evaluation?
  3. AIR-MoE often has a large fraction of unused experts (e.g., 63.8% vs. 1.2% for PEER on OpenWebText2 small; Tab. 3), and it achieves the lowest PPL in every setting. How are dead experts measured, and can we reduce them to further improve AIR-MoE, for example, with stronger load balancing?
  4. On C5, perplexity drops from 547–918 for the small models to 44–51 for the medium models (Tab. 1), which is a much larger change than expected from scaling the model. Could you explain what causes this?
  5. I would recommend adding an exact top-K router to the quality experiments. This is the most direct way to tell whether AIR-MoE trades quality for efficiency or actually improves over exact top-K routing.
5 comments

We thank the reviewer for recognizing that AIR-MoE "has no structural constraints on expert parameters and it has a comparable routing cost to PEER", a core contribution that "sets it apart from prior granular routers", as well as its consistent gains: "AIR-MoE beats PEER in every setting of Tab. 1 and in all three seeds of Tab. 2".

W1/Q1: Top-KK approximation vs. shortlist structure

We agree: overlap with exact top-KK and perplexity are distinct and we have a simple explanation for the observed phenomenon. The key is to read overlap in the context of expert utilization (Tab. 10, C5 medium):

VariantOverlapDead expertsPPL
AIR-MoE58%13.7%43.85
Cosine assignment65%8.4%44.11
Expert choice81%68.1%48.77
No centroid normalization72%85.9%46.20
Static codebook5%89.1%47.81
Hierarchical––51.45
Std. Coarse––49.81

High overlap is easy to achieve when most experts are unused: the exact top-KK then concentrates on a few experts that appear in almost every shortlist. Expert choice and missing normalization reach 81%81\% and 72%72\% overlap but leave 68.1%68.1\% and 85.9%85.9\% of experts dead. This wastes model capacity and raises PPL to 48.7748.77 and 46.2046.20. Cosine assignment uses slightly more experts and overlaps more, at nearly the same PPL (44.1144.11 vs. 43.8543.85). The static codebook shows the opposite failure: its codewords do not follow the evolving representations, so overlap drops to 5%5\% and 89.1%89.1\% of experts are dead. It still outperforms both baselines, but is clearly worse than adaptive AIR-MoE. We therefore agree with the reviewer that the gains stem from both the overlapping shortlist structure and its adaptation, rather than from approximating exact top-KK alone.

Q2: Codebook synchronization across GPUs

The codebook is synchronized. Before every EMA update (see Alg. 2), per-codeword assignment counts and sums are all-reduced across GPUs, so all GPUs apply the same update. Initialization and dead-codeword re-initialization are performed on rank 0 and broadcast. Expert centroids are synchronized via the data-parallel gradient all-reduce. Shortlists are recomputed on each GPU from these synchronized parameters after every optimizer step. At evaluation, the codebook is frozen and no jitter is applied. We added these details to App. C.

Q3: Dead experts and load balancing

A dead expert receives zero top-KK assignments over the entire validation set (summed over all GPUs). Utilization and perplexity are not directly related: on C5 medium, cosine assignment lowers the dead-expert fraction from 13.7%13.7\% to 8.4%8.4\% but raises PPL from 43.8543.85 to 44.1144.11. Preliminary experiments also indicated that stronger load balancing or jitter worsened perplexity for the small models (even though fewer experts died). Overlapping shortlists let broadly useful experts appear in many shortlists (App. F.9), which hierarchical routing (disjoint clusters) and PEER (product-key structure) do not allow. We already use Switch-style load balancing and jitter at both routing stages (Sec. 3.1). We made the definition precise in the metrics paragraph of Sec. 5.1.

Q4: PPL gap between C5 small and medium

The two configurations differ in backbone size and in training tokens (449449M vs. 1.381.38B, 3.1×3.1\times; App. E.4, Tab. 4), since the token budget scales with the dense parameter count (Sec. 5.1). The size of the gap stems from the C5 data stream: as is common for streamed pretraining data, it is shuffled approximately, at the shard level and within a buffer of 100k documents, rather than globally. Its early part is not representative of the validation data. The small model is affected most: at the end of training, its training perplexity is about 2323, while its validation perplexity is 547547 (AIR-MoE) and 571571 (PEER), i.e., more than 20×20\times higher. On the same stream, the medium model has a similar training perplexity at 0.450.45B tokens but a validation perplexity of about 6767 (about 3×3\times). Since all routers see the identical stream, this affects absolute perplexities but not the comparison: AIR-MoE outperforms PEER at every size and seed, as well as on OpenWebText2, where this effect does not occur. We describe this in App. F.2 (learning curves), which we added in the revision.

Q5: Exact top-KK routing baseline

We agree that exact top-KK routing would be the most direct reference. However, it scores all EE experts per token, and its cost grows prohibitively with EE (Sec. 5.3, App. F.7): at E=10242E=1024^2, a single training step is 1.60×1.60\times slower than with AIR-MoE (App. F.3), and full training runs exceed our memory budget. Our quality experiments therefore compare routers that remain feasible in our academic hardware setup, and we state this scope explicitly in Sec. 6.


Once again, we thank the reviewer for the constructive feedback. We hope that our responses address the concerns and would be grateful if the reviewer considered updating their assessment.

Thank you for the authors' response and the revision. My main concern is still that there is no exact top-K baseline in the quality experiments. I understand exact routing is expensive at a million experts. A run with fewer experts would still show whether AIR-MoE keeps the quality of exact top-K, or only beats other approximate routers. Without that, it is hard to tie the perplexity gains to the approximation. I will keep my score.

Reviewer_FGCDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Adaptive Inverted-Index Routing for Mixture-of-Experts (AIR-MoE), a routing mechanism designed to reduce the computational overhead of highly granular MoE models containing very large numbers of small experts. The proposed method adopts a two-stage retrieval strategy inspired by inverted-file indexing. First, token representations are assigned to learned vector-quantization codewords, each associated with a precomputed shortlist of candidate experts. Second, exact token–expert similarity scores are computed only within the retrieved shortlist to select the final top-K experts. Unlike product-key routing and hierarchical MoE, AIR-MoE does not impose structural constraints on expert routing representations or require disjoint expert groups.

To accommodate evolving token representations during training, the authors employ a gradient-free adaptive k-means procedure with exponential moving averages and dead-code reinitialization. The paper further derives a lower bound on retained routing probability mass as a function of quantization error and shortlist mass. Experiments on WikiText-103, C5, and OpenWebText2, across model configurations containing up to approximately one million experts, show that AIR-MoE generally achieves lower perplexity than PEER, hierarchical routing, and standard coarse MoE. Additional experiments examine routing FLOPs, runtime, memory consumption, expert utilization, and component ablations.

Critical strengths

S1. Flexible and computationally efficient routing without structural constraints on expert representations.

A key strength is that AIR-MoE avoids the structural restrictions imposed by product-key retrieval and hierarchical expert grouping. The dynamically constructed, overlapping shortlists allow experts to be accessible from multiple regions of the representation space while retaining unrestricted learnable routing vectors.

This is consequential because it provides a plausible explanation for improved language modeling performance without requiring a full search over all experts. The approach also preserves the general form of conventional top-K MoE routing.

S2. Consistent empirical improvements over competitive granular MoE baselines.

The paper evaluates AIR-MoE across three language modeling datasets and multiple model scales, including configurations with approximately one million experts. Table 1 reports consistent perplexity improvements over PEER, and the additional multi-seed experiments in Table 2 show that the improvement persists across the evaluated seeds.

The efficiency experiments further demonstrate substantially lower routing overhead than standard full-scoring granular routing. Together, these results provide meaningful empirical support for the proposed architecture.

Critical weaknesses

W1. The practical end-to-end efficiency benefits remain insufficiently established.

The central motivation is to make highly granular MoE routing computationally practical. However, the proposed method introduces an additional global shortlist reconstruction operation with complexity O(EGd) after each optimizer step, alongside codebook assignment and candidate scoring.

Although the authors discuss amortizing this cost across gradient-accumulation windows, its practical impact depends on the effective batch size, codebook size, and training configuration.

The model-level performance–efficiency analysis in Figure 2 primarily considers estimated FLOPs, while the runtime and memory experiments in Figures 3 and 6 benchmark isolated routers rather than complete MoE training pipelines. In particular, the demonstrated router-level advantages over other efficient alternatives, including PEER, are less pronounced than those over full granular routing.

Consequently, it remains unclear whether AIR-MoE provides a substantial improvement in end-to-end training throughput, or in the practical perplexity-versus-wall-clock-time trade-off, compared with PEER.

This distinction is important because improved computational complexity does not necessarily translate into improved hardware utilization, especially when dynamic indexing, irregular candidate selection, and sparse expert execution are involved.

W2. The relationship between the theoretical guarantee, routing approximation quality, and downstream performance is not sufficiently characterized.

Proposition 1 establishes a lower bound on the probability mass retained by the expert shortlist. While mathematically reasonable, this guarantee does not directly establish recovery of the true top-K experts or bound the resulting deviation in MoE outputs or language modeling loss.

The empirical results suggest that these distinctions matter. For example, Table 4 reports a top-K overlap of only 0.58 for the default AIR-MoE configuration, while some alternatives achieve higher overlap but worse perplexity. This indicates that agreement with exact routing is not necessarily aligned with downstream performance.

Furthermore, the lower bound may be loose when the quantization error is large or the codeword shortlist captures limited probability mass. The paper does not report the empirical distribution of the bound's key quantities or evaluate its tightness.

As a result, the theoretical analysis offers useful intuition but provides limited explanatory support for the observed performance improvements. A stronger empirical characterization of shortlist recall, routing distortion, and downstream quality would substantially improve the paper.

Questions for authors

Q1. End-to-end training efficiency (related to W1).

Could the authors report end-to-end training throughput or wall-clock time for AIR-MoE and PEER under matched training configurations? Ideally, these measurements would include codebook updates, shortlist reconstruction, token routing, expert computation, and backpropagation.

A breakdown of these costs would also help establish whether the claimed efficiency advantage persists across realistic gradient-accumulation settings.

Q2. Approximation quality and theoretical interpretation (related to W2).

Could the authors report the empirical distributions of the quantization error ϵ(h)\epsilon(h), codeword shortlist mass ρM(c(h))\rho_M(c(h)), and actual MassRecall across training and validation tokens?

It would also be helpful to understand how top-K overlap, retained probability mass, and perplexity change with shortlist size M. In particular, can the authors explain why higher top-K overlap in some ablations does not lead to better perplexity?

Q3. Variability across random seeds.

Table 2 shows substantial differences in absolute C5 perplexity across seeds, despite consistent paired improvements over PEER. Could the authors clarify the source of this variability, including whether training budgets, checkpoint selection, and evaluation settings were identical across seeds?

Reporting aggregate statistics and, if available, learning curves would help determine the stability of the observed improvements.

3 comments

We thank the reviewer for recognizing the "flexible and computationally efficient routing without structural constraints on expert representations" and the "consistent empirical improvements over competitive granular MoE baselines". We address the remaining concerns below.

W1 / Q1: End-to-end training efficiency

We added an end-to-end benchmark under matched configurations (App. F.3): full optimizer steps of the large model (E=10242E=1024^2, K=128K=128, 32,76832{,}768 tokens per step) with identical backbone, experts, batch size, and precision. Each step includes codebook updates, shortlist update, token routing, expert computation, backpropagation, and the optimizer update.

MethodStep time (ms)Throughput (tokens/s)Rel. step time
AIR-MoE2801 ± 12011,7001.00×
PEER2759 ± 12811,8800.99×
Hierarchical2720 ± 13112,0500.97×
Std. Granular (exact top-KK)4474 ± 1417,3201.60×

bold: granular methods wihtin one standard deviation of the fastest granular method.

We agree that AIR-MoE's advantage over PEER is not a throughput advantage: both have the same step time within noise. Since all methods are trained on the same number of tokens, AIR-MoE reaches lower perplexity than PEER (Tab. 1) at practically the same wall-clock time. Regarding the breakdown, the shortlist reconstruction costs below 1010 ms per optimizer step in our router-level measurements (E=10242E=1024^2, G=512G=512), i.e., less than 1%1\% of the step.

W2 / Q2: Approximation quality and the bound

We measured the quantities of Prop. 1 on about 51,00051{,}000 validation tokens of each large model (E=10242E=1024^2, M=8192M=8192, K=128K=128; App. F.4):

Quantity (mean ± std)C5OpenWebText2
MassRecall(h)\mathrm{MassRecall}(\mathbf{h})0.030±0.0150.030 \pm 0.0150.072±0.0380.072 \pm 0.038
Codeword shortlist mass ρM(c(h))\rho_M(c(\mathbf{h}))0.041±0.0200.041 \pm 0.0200.066±0.0290.066 \pm 0.029
Quantization error ϵ(h)\epsilon(\mathbf{h})6.36±2.336.36 \pm 2.339.61±1.789.61 \pm 1.78
Lower bound of Prop. 19.1⋅10−5±2.2⋅10−39.1 \cdot 10^{-5} \pm 2.2 \cdot 10^{-3}1.3⋅10−5±3.9⋅10−41.3 \cdot 10^{-5} \pm 3.9 \cdot 10^{-4}
Bound violations0%0\%0%0\%
Top-KK overlap$^\dagger$0.600±0.1080.600 \pm 0.1080.584±0.0450.584 \pm 0.045

Std over tokens; †^\daggerstd over validation batches.

  • Retained mass. The shortlist (M/E<1%M/E < 1\% of all experts) holds 3.0%3.0\% (C5) and 7.2%7.2\% (OpenWebText2) of the total routing mass.
  • Recovery of the exact top-KK. Without jitter, every exact top-KK expert contained in the shortlist is selected, since at most K−1K-1 shortlisted experts score higher. The top-KK overlap therefore equals the fraction of exact top-KK experts contained in the shortlist: 60%60\% (C5) and 58%58\% (OpenWebText2) at M=8192M=8192.
  • Tightness. The bound holds for every token but it is somewhat loose. With ϵ≈6.4\epsilon \approx 6.4 (C5) and 9.69.6 (OpenWebText2), it allows the shortlist log-odds to change by up to 2ϵ2\epsilon between h\mathbf{h} and c(h)c(\mathbf{h}).
  • Dependence on MM. Without retraining, we rebuilt the shortlists of the trained model (M=8192M=8192) for smaller MM from its codebook and expert centroids:
DatasetMMTop-KK overlapMassRecall\mathrm{MassRecall}ρM(c(h))\rho_M(c(\mathbf{h}))BoundVal. PPL$^\ddagger$
C510240.1680.1680.00540.00540.00890.00892.2⋅10−52.2 \cdot 10^{-5}26.1626.16
20480.3090.3090.01040.01040.01720.01724.3⋅10−54.3 \cdot 10^{-5}24.5724.57
40960.5330.5330.01880.01880.03010.03017.5⋅10−57.5 \cdot 10^{-5}23.2123.21
8192 (trained)0.6000.6000.02960.02960.04140.04149.1⋅10−59.1 \cdot 10^{-5}22.5822.58
OpenWebText210240.2000.2000.01360.01360.01380.01381.8⋅10−61.8 \cdot 10^{-6}18.4618.46
20480.3140.3140.02400.02400.02410.02413.6⋅10−63.6 \cdot 10^{-6}17.0717.07
40960.4430.4430.04170.04170.04080.04087.0⋅10−67.0 \cdot 10^{-6}16.0416.04
8192 (trained)0.5840.5840.07010.07010.06590.06591.3⋅10−51.3 \cdot 10^{-5}15.5115.51

‡^\ddaggerOn a fixed validation subset; comparable across MM, but not with Tab. 1. On both datasets, overlap, retained mass, and perplexity improve monotonically with MM. Since the model was trained with M=8192M=8192, smaller MM truncates shortlists the experts were trained with, so these perplexities are pessimistic compared with models trained at that MM.

It is true that highler overlap does not necessarily lead to better (i.e., lower) perplexity. For the ablations, there are simple explanations: both the "expert choice" and the "missing normalization" ablation achieve high overlap at the cost of very low expert utilization (68.1%68.1\% and 85.9%85.9\% dead experts, respectively): in simple terms, it is easy to achieve high overlap if most experts are just never used. If too few experts are used, the full capacity of the model is not utilized, which in turn leads to poor (high) perplexity.

Q3: Variability across seeds

The seed determines the shuffling of the data, which sets both the training order and the held-out validation subset. Absolute perplexities of different seeds are therefore measured on different validation text and are not directly comparable. However, AIR-MoE and PEER with the same seed use identical training data, validation data, token budgets, checkpoint selection (best validation loss), and evaluation settings, so each per-seed comparison is paired. As suggested by the reviewer, we added validation learning curves for all three seeds on C5 and OpenWebText2 (App. F.2). Within each seed, both routers follow the same trajectory shape, set by the data order, and AIR-MoE attains the lower perplexity over the final part of training. The curves also show the source of the variability. As is common for streamed pretraining data, C5 is shuffled approximately, at the shard level and within a buffer of 100k documents, rather than with a global permutation. For some seeds, the early part of the stream is therefore not representative of the validation data, which shows as a sharp drop in validation perplexity once the stream reaches more diverse data (seeds 43 and 44). Since all routers of a seed see the identical stream, this affects absolute perplexities but not the comparison: AIR-MoE outperforms PEER on all seeds, with and without this effect, as well as on OpenWebText2, where it does not occur. Across seeds, AIR-MoE improves PPL over PEER by 2.9±1.6%2.9 \pm 1.6\% on C5 and 3.4±0.5%3.4 \pm 0.5\% on OpenWebText2 (App. F.1).


Once again, we thank the reviewer for the constructive feedback. We hope that our responses address the concerns and would be grateful if the reviewer considered updating their assessment.

Thanks for the author's efforts. My core concerns about the training efficiency, multi-seed experiement and approximation quality have been solved. And the author also revise those content into the manuscript.

Besides, I recommend the author discuss the combination possibility of below papers in related works:

  1. MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance. ICML 2025.

  2. KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Models. ICLR 2026.

I don't have other questions. I will raise my score.

Comments on the paper

2 comments

We thank the reviewers for their feedback. We were delighted that the reviewers recognized AIR-MoE's "flexible and computationally efficient routing without structural constraints on expert representations" (FGCD), which "sets it apart from prior granular routers" (S4M9), and its "consistent empirical improvements over competitive granular MoE baselines" (FGCD): "AIR-MoE beats PEER in every setting of Tab. 1 and in all three seeds of Tab. 2, at nearly identical training FLOPs" (S4M9). R6bh highlights that AIR-MoE "allows shortlists to overlap naturally so broadly useful experts can be shared across many regions of the input space while keeping runtime and memory comparable to PEER", and calls it a "clean decoupled training design" with a "thorough empirical analysis" that "avoids gradient estimation tricks and makes AIR-MoE a simple drop-in replacement". GQvu notes that "the ablations in Table 4 help establish the importance of adaptive codebook updates". The feedback help us improve the manuscript further. All changes to the manuscript are marked red in the revision. The main changes are the following:

  1. End-to-end efficiency (FGCD, GQvu; App. F.3, Tab. 6). Beyond the router-only benchmarks, we now measure the wall-clock time of full training steps of the large model. AIR-MoE, PEER, and the hierarchical router have step times within one standard deviation of each other and near-identical memory, while exact granular top-KK routing is 1.60×1.60\times slower than AIR-MoE. AIR-MoE's perplexity gains over PEER thus come at no measurable additional training cost.
  2. Approximation quality (FGCD, S4M9; Sec. 3.3, App. F.4). With the same argument, we tightened the MassRecall bound. Writing ρ=ρM(c(h))\rho = \rho_M(c(\mathbf{h})) and ϵ=ϵ(h)\epsilon = \epsilon(\mathbf{h}), it improves from ρ e−2ϵ\rho\, e^{-2\epsilon} to ρ/(ρ+(1−ρ) e2ϵ)\rho / (\rho + (1-\rho)\, e^{2\epsilon}). On C5 and OpenWebText2, the bound holds for every evaluated token, and a post-hoc sweep over the shortlist size MM shows that top-KK overlap, retained mass, and perplexity improve monotonically with MM.
  3. Learning curves and seed robustness (FGCD, R6bh; App. F.1, F.2). Validation curves for all seeds on both datasets show AIR-MoE ahead of PEER over the final part of training on every seed. Across seeds, AIR-MoE improves PPL over PEER by 2.9±1.6%2.9 \pm 1.6\% on C5 and 3.4±0.5%3.4 \pm 0.5\% on OpenWebText2.
  4. Dead experts (S4M9, R6bh, GQvu; App. F.9). Almost all dead experts are ones that are never shortlisted: for the large C5 model, the 16.3%16.3\% of experts in no shortlist account for 99%99\% of the 16.5%16.5\% dead experts. Experts that are in at least one shortlist are almost never dead.
  5. Scope of experiments (R6bh, GQvu; Sec. 6). Our models are far from frontier scale, since our academic compute limits model size and token budget. Within this budget, the results are consistent: AIR-MoE outperforms PEER for three model sizes, on two datasets, and for all seeds, with up to E=10242E = 1024^2 experts. Sec. 6 now states these limitations explicitly (single MoE layer, fixed training recipe), and we see validation at larger scale as the main next step.
  6. Clarifications. Codebook synchronization across GPUs (App. C).

We thank all reviewers again for their time and effort. We hope that the additional experiments and clarifications address the main concerns.

Area_Chair_LwUeOpenReview ↗

Please check author reponses

Dear reviewers,

We’re about halfway through the discussion period. This is a friendly reminder to check other's reviews and the author responses. If the authors have answered questions or responded to critiques in reviews, please leave a comment for the authors!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.