acceptodds
Under review as a conference paper at ICLR 2027

Batch-Aware Hierarchical Routing for Mixture-of-Experts with Reduced Memory Traffic and Theoretical Guarantees

Abstract

Mixture-of-Experts (MoE) architectures reduce computation by activating a small subset of experts per token. However, under batch-based execution, memory access is determined by the union of experts selected across the batch, so memory movement remains a key bottleneck. We propose Batch-Aware Hierarchical MoE Routing (BAR), which augments token-level routing with a batch-level router that first selects a small subset of experts per batch, followed by token-level refinement. This design explicitly reduces memory traffic while improving efficiency and expert specialization. We theoretically show that BAR reduces memory access, inference cost, and training computation, established through the training dynamics analysis of two-layer ReLU experts. Across evaluation on large-scale MoE language models, BAR substantially reduces HBM traffic and improves inference throughput while closely preserving base-model performance. For example, on DeepSeek-MoE-16B, BAR reduces HBM traffic by 60% and improves throughput by \(2.5\times\), while maintaining perplexity within 2% of the base model

Primary area:
learning theory
Keywords:
Mixture of experts, routing mechanism, training dynamics, generalization, batch-level routing, memory traffic

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q8X9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BAR (Batch-Aware Hierarchical Routing) to address a problem in MoE models: each token uses only a few experts, but a whole batch of tokens may still access most experts. This reduces the cost of moving expert weights. BAR adds a trainable batch router to a standard MoE model to select a shared subset of experts for the whole batch, then combines it with the original token-level routing to perform computation. This greatly reduces the cost of moving data in HBM. The paper devotes a large amount of space to carefully proving whether joint training of the two routing layers can converge. On the DeepSeek-MoE-16B, OLMoE-1B-7B, and Qwen1.5-MoE-A2.7B models evaluated by the authors, BAR achieves a better trade-off between throughput and model performance than direct pruning and FBR.

Critical strengths
  1. The two-level routing scheme proposed in this paper is reasonable, has important value for training and deploying MoE models in real-world settings, and is supported by theory. The paper introduces batch-level expert selection above token-level routing to directly control the set of experts accessed by the whole batch. This addresses the issue that sparse computation at the token level does not necessarily lead to sparse memory access at the batch level. The scheme can work with either token-choice or expert-choice routing. It reduces expert computation and weight movement while keeping expert selection flexible across batches, and the experiments show efficiency gains in both inference and fine-tuning.

  2. The paper puts substantial effort into providing a complete theoretical proof of training stability for the two-level routers. Beyond the design and experimental results, it describes the learning processes of the batch router, expert-choice router, and expert networks under assumptions such as a simplified data distribution and two-layer ReLU experts. It explains how they learn to specialize together and further provides guarantees for classification generalization and computational savings. This offers a theoretical explanation of how and why two-level routing works.

Critical weaknesses
  1. The paper seems to lack a detailed discussion of related work. This is important for assessing the contribution of the proposed algorithm and needs further discussion.

    • Token-level routing naturally leads to uneven loads, and batch routing is a natural way to improve this. Although this paper focuses more on jointly training routers at two levels, it still needs to discuss related work on batch routing. I know of at least two such papers, [1] and [2]. The authors should at least describe them in detail in the related work section or add baseline experiments to further explain their unique contribution among works on batch routing.
    • Also, neither of the two baselines in the experiments (pruning and FBR) directly trains the routers (or more precisely in this paper, fine-tunes them). Instead, they directly modify the fine-tuned models. Therefore, I do not think the current results fully separate the gains from the authors’ hierarchical algorithm from those of additional training. With the help of generative AI, I found concurrent work [3] related to this issue, and they also train the batch-aware router to achieve better performance. I also notice another concurrent work [4] but it seems not need to train the router. Since their release dates overlap with the ICLR submission deadline, I do not use these works to judge whether this paper should be accepted. I encourage the authors to discuss them in later versions.
  2. Although the paper provides extensive convergence proofs, its theoretical analysis assumes binary classification, exactly one token that determines the label in each group, mutually orthogonal features, two-layer ReLU experts, and expert-choice routing. The actual experiments mainly modify large models with token-choice routing, so there is a gap between the theory and the experiments. I recommend that the authors track some specific metrics in the experiments to explain exactly how training the two routers in this hierarchy affects expert routing, how to determine whether these effects are positive, how the model’s final performance is maintained, and so on.

[1] Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining https://arxiv.org/abs/2511.02237

[2] SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Models https://arxiv.org/abs/2602.07616

[3] BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding https://arxiv.org/abs/2609.36222

[4] ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration https://arxiv.org/abs/2609.36222

Questions for authors

(For questions 1–3 below, I fully understand that the authors may be limited by GPU resources and may not be able to run larger-scale experiments. However, since this paper mainly aims to optimize MoE efficiency, I believe these questions are still important.)

  1. Have the authors evaluated whether applying this approach during MoE pretraining would work better than applying it during fine-tuning? Have you analyzed in detail whether forcing group-level routing constraints on a pretrained MoE model would greatly affect its overall routing behavior?
  2. Why did the authors choose DeepSeek-MoE-16B, OLMoE-1B-7B, and Qwen1.5-MoE-A2.7B? Have you considered testing more advanced and larger MoE models?
  3. Following up on question 2, I noticed that the experiments use relatively few GPUs. Have the authors considered using this approach together with optimization techniques for large-model deployment, such as expert parallelism, to further improve throughput when deploying MoE models at a larger scale?
  4. Minor issue: “RELATED WORKS” should be a full section rather than a subsection of the introduction.
  5. Minor issue: In “4 EMPIRICAL EXPERIMENTS,” some space should be set aside for downstream task metrics, instead of putting them all in appendix. At least an overall score across downstream tasks should be included to better highlight that BAR achieves a better trade-off between efficiency and performance.

I'll carefully consider raising my rating score if the authors are willing to discuss (or resolve if possible) my concerns during the rebuttal phase.

Reviewer_tn2zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Batch-Aware Hierarchical Routing (BAR), which adds a batch-level router to select a shared subset of experts before token-level expert computation. In the token-choice implementation, token–expert assignments outside this subset are skipped, reducing the number of experts accessed and the associated computation and memory traffic. The paper provides theoretical analysis for a simplified expert-choice model and evaluates the method on several pretrained MoE language models, reporting efficiency improvements with varying degrees of quality degradation.

Critical strengths
  1. The paper addresses a relevant efficiency issue: sparse expert activation per token does not necessarily imply a small union of accessed experts across a batch. Explicitly controlling this union is a clear and well-motivated approach.

  2. The experiments include measured memory traffic, latency, and throughput, rather than relying exclusively on computational proxies. The evaluation also includes out-of-domain benchmarks and a router-only adaptation setting.

  3. The batch-composition experiments in Appendix A.5 are useful, since sensitivity to other requests in the batch is a natural concern for this method. The theoretical analysis also provides a mechanism-level explanation within its stated simplified setting.

Critical weaknesses

W1. The quality–cost advantage is not sufficiently established under fair comparisons.
BAR removes token–expert computations and incurs non-negligible quality degradation in some settings. The reported speedups therefore need comparisons at matched quality or controlled quality loss. Moreover, BAR is trained under the expert restriction, while pruning/FBR are applied after standard fine-tuning. This adaptation asymmetry makes it difficult to isolate the benefit of the proposed routing strategy.

W2. The model coverage provides limited evidence of broader applicability.
The evaluated backbones demonstrate feasibility, but do not establish whether substantial computation removal remains effective across different expert granularities, training regimes, and routing behaviors. The concern is architectural representativeness rather than model recency. Additional evidence is needed to characterize when the method preserves capability and when its benefits diminish.

W3. The theoretical guarantees have limited correspondence to the main experiments.
The theory considers a simplified setting in which each token group shares a single classification target, whereas practical batches contain independent requests with potentially conflicting expert requirements. It also primarily analyzes expert-choice routing, while the main experiments use token-choice routing. Consequently, the analysis does not directly establish quality preservation when restricting expert access in the evaluated pretrained language models.

Questions for authors
  1. How do BAR, FBR, and static pruning compare when each method is adapted under its own expert restriction using comparable training resources? Under this protocol, what inference savings remain at matched quality or fixed quality-loss thresholds?

  2. Which model and routing characteristics predict tolerance to the proposed computation removal? Can the authors support the claimed applicability with a distinct MoE configuration or an analysis relating quality degradation to actual dropped assignments and retained routing-weight mass?

  3. Which theoretical conclusions extend to a shared expert subset across heterogeneous, independently labeled requests and to token-choice routing? Please clarify which claims are formally guaranteed and which are supported only empirically.

Reviewer_82Y5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes batch-aware hierarchical routing for MoE, restrciting each batch to a learned subset of experts to reduce computation and memory traffic. It further conducts theoretical analysis in a simplified setting and reported the training and inference gains.

Critical strengths
  1. Under a restricted expert budget, such method retains the full expert pool while selecting different subsets for different batches, offering greater flexibility than static expert pruning.
Critical weaknesses

W1: The chice of expert subset selection at the batch level is unjustified.

  • BAR forces potentially unrelated sequences to share one expert pool, making the available experts for each sequence conditioned on other seuqneces in the batch. This is particularly concerning under continuous batching, a widely adopted LLM serving technique where requests enter and leave dynamically. Limited perplexity degradation in the reported setting does not establish the quality and throughput gains persist. I have serious doubts whether this approach can be pratically adopted.
  • On the other hand, document-level subset selection, explored in prior work like EMO (2026, arXiv:2605.06663), provides a more semantically motivated alternative, where documents act as the natural boundary for grouping subset of experts. The reshuffling experiments provide useful evidence of robustness to test the batch compositions, but not establish a trade-off over document-level selection. A direct quality-efficiency comparison should be made.

W2: The reported efficiency gains do not establish scalability to expert-parallel training.

  • A common technique to train MoE models at larger scale is to shard experts across GPUs. Selecting fewer experts may concentrate computation on a subset of devices while leaving others idle. Consequently, reductions in aggregate expert computation or weight accesses do not necessarily translate into higher training throuhgput.
  • An EP evaluation, even on a single node, would help address this concern by reporting per-GPU load balance and the end-to-end step time. At minimum, the paper should explicitly discuss this limitation and qualify its training-efficiecny claims accordingly.
Questions for authors

No additional questions beyond the requests raised in the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.