- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies adaptive expert-count routing for MoE models and proposes a dense credit assignment method to learn layer-wise expert counts. The paper replaces the terminal LM reward with a layer-wise MoE update alignment signal. However, the motivation for adaptive expert count and its practical benefits remain unclear, and the experimental evaluation is limited.
Critical strengths
The paper identifies a possible mismatch between the optimization signal and the expert-count decision in Ada-K. The proposed MoE-δ reward is a reasonable attempt to provide more localized feedback for this specific routing formulation.
Critical weaknesses
- Lack of motivation for adaptive expert count. The paper does not sufficiently justify why adaptive expert count is an important problem. To the best of my knowledge, current widely deployed large-scale MoE models rely on fixed expert counts, and the practical necessity of dynamic K remains unclear.
- Incomplete experimental evaluation. The experiments do not demonstrate clear benefits over fixed Top-K MoE routing. In particular, the key end-to-end throughput/latency evaluation is missing, making the practical impact of the proposed approach unclear.
- Limited novelty. The proposed method mainly modifies the reward signal of an existing adaptive routing framework, and the overall contribution appears incremental.
- Unusual and confusing writing style. The paper contains several presentation issues, such as citations appearing in the abstract and frequent references in the conclusion section, which reduce readability.
Questions for authors
- What is the practical motivation for adaptive expert count in modern MoE models? Can you provide evidence that dynamic K brings meaningful end-to-end benefits beyond fixed Top-K routing?
- Does any existing MoE inference framework support adaptive expert counts? If not, how can your method be efficiently integrated into practical serving stacks, such as vLLM/SGLang with NCCL/DeepEP-based expert parallelism, which are highly optimized for fixed Top-K routing?
- Have you evaluated the actual throughput/latency improvement under realistic serving settings? The current FLOP reduction does not clearly translate to system-level gains.
- How does your method compare with other adaptive routing or expert selection approaches beyond Ada-K?
- Can you evaluate your method on more MoE models and a broader range of tasks to verify whether the observed improvements generalize?
Response to Reviewer eVNf
We appreciate the careful reading and the clear grasp of our main contribution—dense MoE-δ credit and the gains over Ada-K—as well as the follow-up questions on Uniform-K and HumanEval / MATH-500. We respond with the verification results below; they will be included in the forthcoming revised manuscript.
On training cost. We agree that Uniform-K needs no training while an allocator does. As in Ada-K, that overhead is only small per-layer linear count heads on a frozen Top-K MoE (~0.68M parameters; backbone, router, and shared expert frozen). Whether this light cost is worthwhile is empirical — which the evaluations below help answer.
Primary Uniform-K control. Because Ada-K and delta-Credit operate near a mean k* of 5.5–6.2, we use Uniform-K=6 as the primary fixed-count baseline at a comparable expert budget. Appendix A.9 also reports Uniform-K=5 and Uniform-K=4.
On MMLU-Pro / MMMU-Pro (and GPQA vs. Uniform-K). Relative to Uniform-K=5 (Active 3.27B), delta-Credit (mean k∗≈5.60, Active 3.38B) matches or exceeds Uniform-K on all three primary benches: GPQA Diamond 82.8 vs. 78.8, MMLU-Pro 83.7 vs. 83.4, and MMMU-Pro tied at 63.1 (App. A.9). Against the higher-budget Uniform-K=6 (Active 3.45B), the picture is mixed by task type. On MMLU-Pro, delta-Credit stays essentially tied at lower routed cost (83.7 vs. 84.0). On GPQA Diamond, delta-Credit is ahead (82.8 vs. 80.8) despite using fewer activated experts on average. On MMMU-Pro, Uniform-K=6 remains ahead (65.0 vs. 63.1), so the current adaptive allocator does not yet give a consistent multimodal gain; we acknowledge this limitation in the revision and leave closing the vision gap to future work. In short, adaptive allocation is not uniformly dominant over every fixed K, but versus a nearby lower fixed budget (K=5) it is at least as strong on all three benches, and versus K=6 it improves the accuracy–compute trade-off on reasoning-heavy text (GPQA) while holding MMLU near parity.
Requested benchmarks. Following the reviewer’s question, we additionally evaluate HumanEval and MATH-500 on the same selected Gemma checkpoints against BF16 Top-8, Uniform-K=6, Ada-K (lambda=0.003), and delta-Credit. The table below summarizes these results and will appear in the forthcoming revised manuscript (planned as Table 4).
Protocol. HumanEval (Chen et al., 2021): official completion, pass@1, reasoning mode off, greedy T=0. MATH-500 (Hendrycks et al., 2021; Lightman et al., 2024, ICLR): same reasoning-mode stack as Table 2 (T=1.0, 64k).
delta-Credit vs. Uniform-K. Together with the paper’s GPQA Diamond result (delta-Credit 82.8 at k* ≈ 5.56 vs. Uniform-K=6 80.8 at k*=6.00; Table 2 / App. A.9), the pattern is consistent: at a similar or lower routed-expert budget near K=6, delta-Credit improves the accuracy–compute trade-off on reasoning-oriented downstream tasks (GPQA, MATH-500), and stays near parity with Uniform-K on HumanEval at lower k*. Relative to simply freezing K=6, this is the practical benefit obtained in exchange for delta-Credit’s small training cost.
In light of these strengths versus Uniform-K, we respectfully ask the reviewer for a more favorable assessment.