- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes BAR (Batch-Aware Hierarchical Routing) to address a problem in MoE models: each token uses only a few experts, but a whole batch of tokens may still access most experts. This reduces the cost of moving expert weights. BAR adds a trainable batch router to a standard MoE model to select a shared subset of experts for the whole batch, then combines it with the original token-level routing to perform computation. This greatly reduces the cost of moving data in HBM. The paper devotes a large amount of space to carefully proving whether joint training of the two routing layers can converge. On the DeepSeek-MoE-16B, OLMoE-1B-7B, and Qwen1.5-MoE-A2.7B models evaluated by the authors, BAR achieves a better trade-off between throughput and model performance than direct pruning and FBR.
Critical strengths
-
The two-level routing scheme proposed in this paper is reasonable, has important value for training and deploying MoE models in real-world settings, and is supported by theory. The paper introduces batch-level expert selection above token-level routing to directly control the set of experts accessed by the whole batch. This addresses the issue that sparse computation at the token level does not necessarily lead to sparse memory access at the batch level. The scheme can work with either token-choice or expert-choice routing. It reduces expert computation and weight movement while keeping expert selection flexible across batches, and the experiments show efficiency gains in both inference and fine-tuning.
-
The paper puts substantial effort into providing a complete theoretical proof of training stability for the two-level routers. Beyond the design and experimental results, it describes the learning processes of the batch router, expert-choice router, and expert networks under assumptions such as a simplified data distribution and two-layer ReLU experts. It explains how they learn to specialize together and further provides guarantees for classification generalization and computational savings. This offers a theoretical explanation of how and why two-level routing works.
Critical weaknesses
-
The paper seems to lack a detailed discussion of related work. This is important for assessing the contribution of the proposed algorithm and needs further discussion.
- Token-level routing naturally leads to uneven loads, and batch routing is a natural way to improve this. Although this paper focuses more on jointly training routers at two levels, it still needs to discuss related work on batch routing. I know of at least two such papers, [1] and [2]. The authors should at least describe them in detail in the related work section or add baseline experiments to further explain their unique contribution among works on batch routing.
- Also, neither of the two baselines in the experiments (pruning and FBR) directly trains the routers (or more precisely in this paper, fine-tunes them). Instead, they directly modify the fine-tuned models. Therefore, I do not think the current results fully separate the gains from the authors’ hierarchical algorithm from those of additional training. With the help of generative AI, I found concurrent work [3] related to this issue, and they also train the batch-aware router to achieve better performance. I also notice another concurrent work [4] but it seems not need to train the router. Since their release dates overlap with the ICLR submission deadline, I do not use these works to judge whether this paper should be accepted. I encourage the authors to discuss them in later versions.
-
Although the paper provides extensive convergence proofs, its theoretical analysis assumes binary classification, exactly one token that determines the label in each group, mutually orthogonal features, two-layer ReLU experts, and expert-choice routing. The actual experiments mainly modify large models with token-choice routing, so there is a gap between the theory and the experiments. I recommend that the authors track some specific metrics in the experiments to explain exactly how training the two routers in this hierarchy affects expert routing, how to determine whether these effects are positive, how the model’s final performance is maintained, and so on.
[1] Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining https://arxiv.org/abs/2511.02237
[2] SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Models https://arxiv.org/abs/2602.07616
[3] BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding https://arxiv.org/abs/2609.36222
[4] ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration https://arxiv.org/abs/2609.36222
Questions for authors
(For questions 1–3 below, I fully understand that the authors may be limited by GPU resources and may not be able to run larger-scale experiments. However, since this paper mainly aims to optimize MoE efficiency, I believe these questions are still important.)
- Have the authors evaluated whether applying this approach during MoE pretraining would work better than applying it during fine-tuning? Have you analyzed in detail whether forcing group-level routing constraints on a pretrained MoE model would greatly affect its overall routing behavior?
- Why did the authors choose DeepSeek-MoE-16B, OLMoE-1B-7B, and Qwen1.5-MoE-A2.7B? Have you considered testing more advanced and larger MoE models?
- Following up on question 2, I noticed that the experiments use relatively few GPUs. Have the authors considered using this approach together with optimization techniques for large-model deployment, such as expert parallelism, to further improve throughput when deploying MoE models at a larger scale?
- Minor issue: “RELATED WORKS” should be a full section rather than a subsection of the introduction.
- Minor issue: In “4 EMPIRICAL EXPERIMENTS,” some space should be set aside for downstream task metrics, instead of putting them all in appendix. At least an overall score across downstream tasks should be included to better highlight that BAR achieves a better trade-off between efficiency and performance.
I'll carefully consider raising my rating score if the authors are willing to discuss (or resolve if possible) my concerns during the rebuttal phase.