acceptodds
Under review as a conference paper at ICLR 2027

Dense Credit Assignment for Adaptive MoE Routing via Layer-wise MoE-

Abstract

MoE LLMs are widely used to scale capacity under limited inference compute. Their activated expert count is a primary lever for that cost. Because expert-count demand varies across tokens and layers, static Top- overallocates routed compute. Recent work therefore studies token- and layer-wise count allocation. Among these, Ada-K (Yue et al., 2025a) trains a thin -selector with reinforcement learning from next-token probability. That objective optimizes generation quality via a terminal likelihood signal; it does not directly score the count decision at each (token, layer). We argue that denser MoE-latent credit better matches optimization, and propose -Credit: a reward that scores each layer's count decision by how closely the resulting MoE-block update (the change in hidden state across that layer's expert computation) matches a full-capacity (Top-) teacher. Unlike Ada-K's single terminal likelihood signal broadcast to every depth, this credit is a dense per-layer score along the student trajectory. With this training target on Gemma 4 26B-A4B (reasoning-mode decoding, 64k context), -Credit outperforms Ada-K on GPQA Diamond / MMLU-Pro / MMMU-Pro at a lower mean activated count (82.8/83.7/63.1 vs. 76.3/82.0/61.7; vs. ), reaching 98.8%/100%/94.7% of BF16 Top-8 accuracy at 70% routed MoE cost. The same recipe on Qwen3-30B-A3B (Qwen Team, 2025) (text-only) is again ahead of Ada-K on both GPQA Diamond and MMLU-Pro at comparable routed cost. These results support MoE- as an effective action-sensitive credit signal for learning dynamic under our evaluation protocol, and show that when the training signal aligns with count allocation, adaptive routing can cut activated experts more aggressively while better preserving downstream accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of experts, adaptive routing, credit assignment, efficient inference, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ucAsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies adaptive expert-count routing for MoE models and proposes a dense credit assignment method to learn layer-wise expert counts. The paper replaces the terminal LM reward with a layer-wise MoE update alignment signal. However, the motivation for adaptive expert count and its practical benefits remain unclear, and the experimental evaluation is limited.

Critical strengths

The paper identifies a possible mismatch between the optimization signal and the expert-count decision in Ada-K. The proposed MoE-δ reward is a reasonable attempt to provide more localized feedback for this specific routing formulation.

Critical weaknesses
  1. Lack of motivation for adaptive expert count. The paper does not sufficiently justify why adaptive expert count is an important problem. To the best of my knowledge, current widely deployed large-scale MoE models rely on fixed expert counts, and the practical necessity of dynamic K remains unclear.
  2. Incomplete experimental evaluation. The experiments do not demonstrate clear benefits over fixed Top-K MoE routing. In particular, the key end-to-end throughput/latency evaluation is missing, making the practical impact of the proposed approach unclear.
  3. Limited novelty. The proposed method mainly modifies the reward signal of an existing adaptive routing framework, and the overall contribution appears incremental.
  4. Unusual and confusing writing style. The paper contains several presentation issues, such as citations appearing in the abstract and frequent references in the conclusion section, which reduce readability.
Questions for authors
  1. What is the practical motivation for adaptive expert count in modern MoE models? Can you provide evidence that dynamic K brings meaningful end-to-end benefits beyond fixed Top-K routing?
  2. Does any existing MoE inference framework support adaptive expert counts? If not, how can your method be efficiently integrated into practical serving stacks, such as vLLM/SGLang with NCCL/DeepEP-based expert parallelism, which are highly optimized for fixed Top-K routing?
  3. Have you evaluated the actual throughput/latency improvement under realistic serving settings? The current FLOP reduction does not clearly translate to system-level gains.
  4. How does your method compare with other adaptive routing or expert selection approaches beyond Ada-K?
  5. Can you evaluate your method on more MoE models and a broader range of tasks to verify whether the observed improvements generalize?
Reviewer_eVNfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. This paper investigate the activated expert count K in the MoE LLMs to improve the downstream accuracy while saving the inference cost.
  2. Differing from previous work that optimizes activated expert selection via learning based on next-token prediction, this work proposes a more fine-grained optimization strategy, which reward each layer count decision by measuring the cosine distance of the hidden state between the teacher(Top-K) and the predicted count decision.
  3. They conduct empirical experiments on GPQA, MMLU-Pro, and MMMU-Pro, and demonstrate great performance than Ada-K.
Critical strengths
  1. The idea of dense credit assignment based on hidden state is novel and have the potential for more fine-grained representation learning.
  2. Empirical results demonstrate clear superiority between the proposed method and the baseline method Ada-K, with both performance gain and inference cost saving.
Critical weaknesses

A critical weakness of this work is that, compared with the simple baseline uniform-K (K=6 or K=5), the benefit seems not obvious, in both MMLU-Pro and MMMU-Pro, the proposed methods perform worse than the simple baseline with K=6, and comparable performance but more cost compared with K=5.

This is a critical weakness, as a uniform-K baseline is simple and does not need training while the proposed method needs additional training.

Questions for authors

How does the method perform on other frequently used benchmarks, such as MATH, and HumanEval?

2 comments
AuthorsOpenReview ↗

Response to Reviewer eVNf

We appreciate the careful reading and the clear grasp of our main contribution—dense MoE-δ credit and the gains over Ada-K—as well as the follow-up questions on Uniform-K and HumanEval / MATH-500. We respond with the verification results below; they will be included in the forthcoming revised manuscript.

On training cost. We agree that Uniform-K needs no training while an allocator does. As in Ada-K, that overhead is only small per-layer linear count heads on a frozen Top-K MoE (~0.68M parameters; backbone, router, and shared expert frozen). Whether this light cost is worthwhile is empirical — which the evaluations below help answer.

Primary Uniform-K control. Because Ada-K and delta-Credit operate near a mean k* of 5.5–6.2, we use Uniform-K=6 as the primary fixed-count baseline at a comparable expert budget. Appendix A.9 also reports Uniform-K=5 and Uniform-K=4.

On MMLU-Pro / MMMU-Pro (and GPQA vs. Uniform-K). Relative to Uniform-K=5 (Active 3.27B), delta-Credit (mean k∗≈5.60k^{\ast}{\approx}5.60, Active 3.38B) matches or exceeds Uniform-K on all three primary benches: GPQA Diamond 82.8 vs. 78.8, MMLU-Pro 83.7 vs. 83.4, and MMMU-Pro tied at 63.1 (App. A.9). Against the higher-budget Uniform-K=6 (Active 3.45B), the picture is mixed by task type. On MMLU-Pro, delta-Credit stays essentially tied at lower routed cost (83.7 vs. 84.0). On GPQA Diamond, delta-Credit is ahead (82.8 vs. 80.8) despite using fewer activated experts on average. On MMMU-Pro, Uniform-K=6 remains ahead (65.0 vs. 63.1), so the current adaptive allocator does not yet give a consistent multimodal gain; we acknowledge this limitation in the revision and leave closing the vision gap to future work. In short, adaptive allocation is not uniformly dominant over every fixed K, but versus a nearby lower fixed budget (K=5) it is at least as strong on all three benches, and versus K=6 it improves the accuracy–compute trade-off on reasoning-heavy text (GPQA) while holding MMLU near parity.

Requested benchmarks. Following the reviewer’s question, we additionally evaluate HumanEval and MATH-500 on the same selected Gemma checkpoints against BF16 Top-8, Uniform-K=6, Ada-K (lambda=0.003), and delta-Credit. The table below summarizes these results and will appear in the forthcoming revised manuscript (planned as Table 4).

Protocol. HumanEval (Chen et al., 2021): official completion, pass@1, reasoning mode off, greedy T=0. MATH-500 (Hendrycks et al., 2021; Lightman et al., 2024, ICLR): same reasoning-mode stack as Table 2 (T=1.0, 64k).

MethodHumanEvalk*_HEMATH-500k*_M
BF16 Top-893.98.0080.88.00
Uniform-K=691.56.0080.66.00
Ada-K (lambda=0.003)90.26.2079.66.24
delta-Credit90.95.5881.45.52
  • HE (reasoning off): delta-Credit achieves 90.9, only 0.6 pp below Uniform-K=6 (91.5), while using fewer activated experts on average (5.58 vs. 6.00). It also outperforms Ada-K (90.2). Relative to BF16 Top-8 (93.9), there is a drop under compression, as expected.
  • MATH-500 (reasoning on): delta-Credit is best among the four — above Uniform-K=6 (81.4 vs. 80.6) and above Top-8 (80.8) — at the lowest k* (5.52).

delta-Credit vs. Uniform-K. Together with the paper’s GPQA Diamond result (delta-Credit 82.8 at k* ≈ 5.56 vs. Uniform-K=6 80.8 at k*=6.00; Table 2 / App. A.9), the pattern is consistent: at a similar or lower routed-expert budget near K=6, delta-Credit improves the accuracy–compute trade-off on reasoning-oriented downstream tasks (GPQA, MATH-500), and stays near parity with Uniform-K on HumanEval at lower k*. Relative to simply freezing K=6, this is the practical benefit obtained in exchange for delta-Credit’s small training cost.

In light of these strengths versus Uniform-K, we respectfully ask the reviewer for a more favorable assessment.

Thank you for your thoughtful response and the additional experiments. I have carefully considered your replies and appreciate your efforts to address my concerns. After considering the clarifications and results, I will maintain my original score. Thank you for your time and effort.

Reviewer_cTvoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes δ-Credit, a dense per-layer PPO reward for learning adaptive expert-count allocation in MoE LLMs. The key insight is that Ada-K's terminal next-token log P signal, broadcast via γ-discounting to all layers, provides indirect credit for the per-(token, layer) count decision k*. δ-Credit replaces this with a layer-local cosine similarity between the student and teacher MoE-block residual updates, giving each layer's allocator a dense, action-sensitive reward along the student trajectory. Extensive experiments prove the effectiveness of the proposed method.

Critical strengths
  1. The problem is clear and well-motivated. The mismatch between Ada-K's terminal LM credit and the per-layer count action is a genuine and underappreciated issue. The frozen probe experiment (Fig. 3) provides clean empirical support: MoE-δ's mean drop-per-k is ~1.6× larger than full-hidden cosine, demonstrating that δ is a more action-sensitive feature before any training occurs.

  2. Strong and extensive experimental results. Besides the main results on their primary Gemma model, they also provide the Qwen3-30B-A3B experiment (Tab. 5), which shows that the same recipe improves over Ada-K on a routed-only MoE without a shared expert. This provides meaningful evidence that δ-Credit is not narrowly tuned to Gemma's architecture.

Critical weaknesses
  1. Single training run in Table 2. All primary results in Table 2 are from a single PPO training run per method. No standard deviations, confidence intervals, or results across multiple seeds are reported. Given that PPO training is known to be sensitive to random initialization and rollout variance, it is difficult to assess whether the reported gains are reliable.

  2. Checkpoint selection relies on a training-time signal that is not method-neutral. The checkpoint gate uses LM-gap MA≥0, which is Ada-K's own training reward and only a logged monitor for δ-Credit. While the rule is applied uniformly, it implicitly favors methods that maintain a non-negative LM-gap during training. A truly method-neutral protocol would use a small held-out validation set for checkpoint selection, decoupled from either method's training signal.

Questions for authors
  1. Can the authors provide results from multiple training seeds (≥3) with mean and standard deviation for Table 2?

  2. How does the checkpoint selection rule affect results if applied at a fixed training step (e.g., end of epoch) for both methods? This would remove the asymmetric candidate pool issue raised in W2.

  3. How much of the observed δ-align signal at deeper layers reflects the current layer's k decision versus accumulated upstream drift?

2 comments
AuthorsOpenReview ↗

Response to Reviewer cTvo (Part 1/2)

Thank you for carefully reading our paper, for recognizing its core strengths (the credit-mismatch motivation and frozen probe evidence), and for the questions on multi-seed reliability, checkpoint selection, and upstream drift. We answer each point below and note the corresponding revision updates.

W1 / Q1. Multi-seed mean ± std for Table 2

Agreed. PPO allocators are sensitive to initialization and rollout sampling of k∗∼πθ(⋅∣h)k^{\ast} \sim \pi_{\theta}(\cdot\mid h), so a single run is not enough to judge reliability. Table 2 in the original submission reported one PPO run per method (training seed 42). We newly train and evaluate Ada-K and δ\delta-Credit at seeds 0, 1, and 2 under the same fixed-λ\lambda settings and the original Table 2 checkpoint-selection protocol (Ada-K λ=0.003\lambda{=}0.003, δ\delta-Credit λ=0.001\lambda{=}0.001), using the same reasoning-mode evaluation stack. As requested, we report mean ± std over these three new runs (seeds 0/1/2). We keep the original seed-42 Table 2 numbers separate rather than mixing the submission run with the new multi-seed block. The table below presents the requested mean ± std; per-seed raw scores will appear in the appendix of the forthcoming revised manuscript.

Protocol note for kˉ∗\bar{k}^{\ast}. Following Table 2, kˉ∗\bar{k}^{\ast} is the unweighted mean of the three benchmark request-mean k∗k^{\ast} values (GPQA / MMLU-Pro / MMMU-Pro). Accuracies are %; std is the sample standard deviation over seeds 0/1/2.

Methodkˉ∗\bar{k}^{\ast}GPQAMMLU-ProMMMU-Pro
Ada-K5.90±0.585.90 \pm 0.5876.8±2.676.8 \pm 2.680.2±0.680.2 \pm 0.662.0±1.362.0 \pm 1.3
δ\delta-Credit5.76±0.205.76 \pm 0.2080.0±1.280.0 \pm 1.282.8±0.282.8 \pm 0.264.0±1.264.0 \pm 1.2

Per-seed raw (seed 0 / 1 / 2):

  • Ada-K: kˉ∗\bar{k}^{\ast} 5.53 / 5.59 / 6.57; GPQA 75.3 / 79.8 / 75.3; MMLU-Pro 80.7 / 80.4 / 79.6; MMMU-Pro 60.5 / 62.4 / 63.1
  • δ\delta-Credit: kˉ∗\bar{k}^{\ast} 5.92 / 5.53 / 5.83; GPQA 81.3 / 79.3 / 79.3; MMLU-Pro 82.7 / 82.8 / 83.0; MMMU-Pro 62.8 / 64.1 / 65.1

Over seeds 0/1/2, δ\delta-Credit achieves a higher mean accuracy than Ada-K on GPQA, MMLU-Pro, and MMMU-Pro. The advantage is clearest on MMLU-Pro (+2.6 pp; 82.8±0.282.8 \pm 0.2 vs. 80.2±0.680.2 \pm 0.6) and MMMU-Pro (+2.0 pp; 64.0±1.264.0 \pm 1.2 vs. 62.0±1.362.0 \pm 1.3). The consistent advantage across all three seeds provides additional evidence that the observed gains are not driven solely by a single training run. Although three seeds do not establish statistical significance, they provide a more reliable estimate of the relative performance than the original single-seed comparison. On GPQA, δ\delta-Credit is also ahead in the mean (80.0±1.280.0 \pm 1.2 vs. 76.8±2.676.8 \pm 2.6), with a tighter spread than Ada-K. Mean activated expert count kˉ∗\bar{k}^{\ast} stays in a comparable operating regime for both methods (5.90±0.585.90 \pm 0.58 vs. 5.76±0.205.76 \pm 0.20), with δ\delta-Credit slightly lower on average and less variable across these three runs. This multi-seed block is the reliability check requested by the reviewer and will be added to the forthcoming revised paper (mean ± std next to Table 2; per-seed 0/1/2 in the appendix).

W2 / Q2. End-of-epoch (fixed last step) vs. LM-gap checkpoint gate

Beyond the selected checkpoints reported in Table 2, we also evaluate the models after a full one-epoch PPO run at a fixed last step, with no LM-gap qualifier and no activation-MA ranking (forthcoming revision, App. End-of-epoch / Tab. EOE; Ada-K λ=0.003\lambda{=}0.003 step 141; δ\delta-Credit λ=0.001\lambda{=}0.001 step 141; same BF16 reasoning-mode stack as Table 2). This is a method-neutral check that does not depend on the train-log selection gate.

CkptMethodkˉ∗\bar{k}^{\ast}GPQAMMLU-ProMMMU-Pro
Selected (Table 2)Ada-K6.1576.382.061.7
Selected (Table 2)δ\delta-Credit5.6082.883.763.1
EOE (step 141)Ada-K5.5074.779.061.2
EOE (step 141)δ\delta-Credit5.5877.382.063.2

At the fixed end-of-epoch checkpoint, δ\delta-Credit remains ahead of Ada-K on all three benchmarks at comparable kˉ∗\bar{k}^{\ast}. Both methods reach step 141 after the same one-epoch PPO schedule on the same mix (identical update count / training progress; only λ\lambda and the credit feature differ). We retain Table 2 as the original train-log-based checkpoint-selection protocol, rather than presenting it as held-out validation. The fixed-step evaluation provides an independent comparison that does not rely on this checkpoint-selection gate. Separately, relaxing Ada-K alone to LM-gap MA ≥−0.05\ge -0.05 yields a stronger Ada checkpoint (79.8/83.1/62.5) that still trails δ\delta-Credit on every benchmark (App. checkpoints).

(Continued in Part 2/2: Q3 and Closing.)

AuthorsOpenReview ↗

Response to Reviewer cTvo (Part 2/2)

Q3. Deep-layer δ-align: current k∗k^{\ast} vs. upstream drift

We agree that upstream drift is real: student and teacher run independent forwards, so hinh^{\mathrm{in}} at a deep layer ℓ\ell may already differ because of earlier counts k∗(t,1),…,k∗(t,ℓ−1)k^{\ast}(t,1),\ldots,k^{\ast}(t,\ell-1), and a drop in MoE-δ align can reflect that path disagreement as well as the current count k∗(t,ℓ)k^{\ast}(t,\ell).

We therefore do not claim that Rδ(t,ℓ)R^{\delta}(t,\ell) isolates the causal effect of k∗(t,ℓ)k^{\ast}(t,\ell). In the revision we state this explicitly: RδR^{\delta} is dense trajectory credit — teacher–student MoE-update agreement at (t,ℓ)(t,\ell) under the current student path — not a same-state counterfactual that freezes upstream hh and changes only k∗(t,ℓ)k^{\ast}(t,\ell).

To see how much of the deep-layer signal is local vs.\ path drift, we run two probes. In the trajectory probe, every layer uses the same forced kk, so upstream states can drift before we measure layer ℓ\ell. In the same-state probe, upstream layers stay at Top-8 (hinh^{\mathrm{in}} cosine ≈1.00{\approx}1.00 with the teacher) and only layer ℓ\ell changes k∗k^{\ast}. The trajectory drop-per-kk is larger than the same-state counterpart by 3.24×3.24\times, 4.91×4.91\times, and 4.27×4.27\times in the early, mid, and late bands, respectively. This larger response reflects the combined effects of local sparsification and accumulated upstream path changes, rather than an isolated estimate of upstream drift. Importantly, even under this controlled same-state setting, align remains sensitive to the local count (late-band drop-per-kk of 0.01660.0166):

k∗k^{\ast} at layer ℓ\ell only12345678
late-band mean cos⁡(δS,δT)\cos(\delta^{S},\delta^{T})0.8840.9390.9660.9810.9890.9940.9981.000

Thus the align signal that enters RδR^{\delta} responds to local k∗(t,ℓ)k^{\ast}(t,\ell) and is not purely an upstream-drift artifact.

Separately, two experiments in the original paper show why MoE-δ is a useful count objective under trajectory credit: (i) a frozen probe finds that MoE-δ has approximately 1.6×1.6\times larger mean drop-per-kk than full-hidden cosine; and (ii) under matched PPO settings, replacing only the credit feature (δ vs.\ full-hh) yields substantially different accuracy–compute trade-offs (82.8% GPQA at k∗≈5.6k^{\ast}\approx 5.6 vs.\ 65.2% at k∗≈4.6k^{\ast}\approx 4.6).

Closing

We hope these clarifications and additional results address your concerns and positively inform your assessment of the paper's contributions and empirical validity.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for recognizing the strengths of our work and for the constructive feedback and concerns. We have addressed all raised points in our individual responses (cTvo, eVNf, ucAs), and the corresponding updates are included in the revised PDF.

For quick navigation:

• Uniform-K controls: App. A.10 / Table 19

• HumanEval / MATH-500: Table 4

• Multi-seed mean ± std (seeds 0/1/2): Table 5; per-seed raw: App. A.18 / Table 24

• End-of-epoch vs. selected checkpoints: App. A.17 / Table 23

• Same-state vs. trajectory probe (local count vs. upstream drift): App. A.5 / Table 14 / Fig. 7

• Matched decode latency: App. A.13 / Table 22

• Presentation / scope clarifications (citations; matched decode vs. production serving): main text + App. A.13

We sincerely appreciate the reviewers’ help in making the paper clearer and more solid. We hope these revisions will be reflected positively in the final assessment. If further experiments would help substantiate our claims during the discussion period, we will do our best to provide them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.