acceptodds
Under review as a conference paper at ICLR 2027

ECCoMoE: Elastic Compute Conditioned Mixture of Experts

Abstract

Sparse mixture-of-experts (MoE) models run the same number of experts for every token, whether the token is easy or hard to predict. Learned methods that vary this number train it into the backbone or train one module per operating point; training-free rules learn no per-token decisions. We present ECCoMoE (Elastic Compute Conditioned Mixture of Experts), which adds a small allocator to each MoE layer of a frozen backbone. For each token, it decides how many of the router's top experts to run, but never which ones or with what weights. The compute budget is an input, so one checkpoint serves both a continuous budget dial and a free mode in which the allocator chooses on its own. It runs inside vLLM's routing stage as a fused Triton kernel. On three MoE backbones and four reasoning and coding benchmarks, at the same average number of experts per token, the dial gains most over interpolated fixed top- at tight budgets, up to 20 points near two experts per token. On Qwen3.6-35B-A3B at (), the dial beats fixed at matched average expert count on all four benchmarks and shows no detected difference from top-8 on three, including MATH-500 (92.3% versus 91.3%). We also find that renormalizing the kept experts' weights, as top- routers usually do, collapses fixed performance on two of the three backbones, and fewer experts per token do not guarantee a cheaper response, because the allocator changes how much the model writes.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Models, Mixture of Experts, Dynamic expert routing, adaptive compute, inference efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jgs7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes ECCoMoE, which inserts a MLP allocator into every MoE layer of a frozen MoE model. Given the layer input and a budget t, the allocator decides how many of the router's top-8 experts to keep. During training all model weights are frozen, the MLP is trained on text that the model itself generated at k=8. At inference the budget is folded into the allocator bias, k is taken as a per-token argmax, and only the first k experts are run. The main finding is that, compared to fixed top-k with the same average number of experts, accuracy improves under tight budgets.

Critical strengths
  1. The method is plug-and-play and can recover the original model exactly. The allocator only decides where to truncate along the router's ranking and never touches expert selection or weights.
  2. Only the allocator is trained, and for one backbone the whole pipeline from data generation to finished training takes under 10 GPU hours.
Critical weaknesses

Alloc-MoE and NAEE also freeze the backbone and allocate experts across layers and tokens, but the paper does not compare against this line of methods.

Questions for authors
  1. How was the hidden width of the allocator MLP chosen?
  2. What is the motivation for the two-stage training, and is there any evidence that it works better than a single-stage schedule in this experimental setting?
2 comments
AuthorsOpenReview ↗

Response to reviewer jgs7

Thank you for the review. We have now run all three of the comparisons you asked for, and we summarise the results below.

Alloc-MoE and NAEE

We ran both methods, together with a training-free top-p rule that keeps experts in router order until their cumulative router weight reaches p, and tuned each of them so that its measured E[k] is 2, 4 or 6. Alloc-MoE was run in two forms, once as published with renormalised weights and once with the router's native weights. At E[k]=2, ECCoMoE outperforms every method on every backbone and benchmark with two exceptions, since it ties top-p on Laguna MATH-500 and all of the competitors score higher on Gemma-4 MuSR. At E[k]=4 and 6 the methods are tied. The table below shows a selection of the E[k]=2 results, and the full tables will be added to the updated paper.

E[k]=2, accuracyECCoMoENAEEAlloc-MoE paperAlloc-MoE nativetop-p
Qwen3.6 GPQA-D74.455.40.857.663.8
Qwen3.6 MBPP+80.465.50.072.968.0
Gemma-4 MATH-50049.728.711.435.532.3
Gemma-4 MuSR57.160.461.758.260.1
Laguna MATH-50086.584.40.074.986.5
Laguna MBPP+76.166.00.069.060.2

Published Alloc-MoE collapses to near-zero accuracy on Qwen3.6 and Laguna at E[k]=2, and this is not an implementation error. Renormalising the kept weights makes top-k degenerate on these routers, whereas the same rule with native weights continues to work with only 1 to 2 experts. This degeneration is one of the findings of our paper, and the new comparison reproduces it.

Allocator width ablations

The width was not tuned; we set it to the number of routed experts, which gives the allocator's first layer the same shape as the router projection and lets it be computed in the same GEMM as the router (Table 10). To check whether this choice matters, we have now ablated widths of 128, 256 and 512 at budgets 2 and 4 on all four benchmarks (3 training × 3 eval seeds), and across these widths throughput changes by at most 2% on every backbone. Qwen3.6 is largely insensitive to width. At budget 2, every width stays within about 2 points of the paper's 256 on all four benchmarks, as the table below shows. At budget 4 the spread is under 1.5 points. Gemma-4 is more sensitive to width at budget 2, so we will add the full width table to the updated paper and state this clearly.

Qwen3.6, budget 2MATH-500GPQA-DMuSRMBPP+
width 12892.073.562.379.7
width 256 (paper)91.874.464.480.4
width 51292.475.164.479.7

Two-stage schedule

We compared the two-stage schedule with sampling the budget uniformly from the first training step. Accuracy is mixed between the two, with differences of up to about 3 points in either direction, but the schedules differ clearly in budget adherence, because single-stage training overspends at budget 2 on every backbone. The curriculum therefore buys adherence to the budget the user sets rather than higher accuracy, and we will make this point explicitly in the updated paper, where the full tables will also be added.

MATH-500, budget 2: acc (E[k])two-stage (paper)single-stage
Qwen3.691.8 (2.29)91.1 (2.40)
Gemma-449.7 (2.04)49.9 (2.11)
Laguna86.5 (2.09)84.8 (2.19)

I appreciate the authors' efforts in providing the results. While I still have reservations about the significance of the overall contribution and hold my current score at this stage, I will gladly keep an open mind during the reviewer discussion phase.

Reviewer_8eUUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ECCoMoE introduces a small, budget-conditioned allocator that chooses how many experts to activate for each token in a pretrained MoE. It keeps the backbone and router frozen, using the router's existing ranking and weights. One trained allocator can serve different compute budgets. Across three MoE backbones and four reasoning/coding benchmarks, the method improves over fixed top-k, particularly at low budgets. The paper also examines throughput, energy usage, and the effect of renormalizing expert weights.

Critical strengths
  • The allocator working with existing pretrained models offers practicality.
  • The paper provides useful analyses, e.g., throughput, energy, and negative results.
Critical weaknesses
  • However, I find the methodological contribution limited in nature. Basically, the method trains a small MLP to predict an expert count, then keeps that many experts from the original router's ranking. This seems like a fairly natural first approach to the problem. The training relaxation is sensible, but I struggled to identify a deeper conceptual or algorithmic contribution. Simplicity can be a strength, but then the experimental evidence needs to make a particularly strong case for why this approach is a substantial advance.
  • One related work is missing. MoSE [ICML 2026] also studies elastic, budget-adaptive MoE inference from a single model. It adapts to expert width rather than to the number of full-width experts, so the methods differ. Still, their goals overlap sufficiently that MoSE warrants discussion, particularly in motivating the novelty of flexible compute budgets.
  • Beating fixed top-k is encouraging, but it does not establish that ECCoMoE is competitive with existing adaptive methods at the same compute cost.
  • The efficiency comparison is not fully controlled. Some experiments use fused Triton routing for ECCoMoE and PyTorch routing for fixed-k, making it hard to separate the allocator's benefit from the implementation's. Please clarify if I am getting this wrong.
  • I would like to see more independent runs. The allocators appear to be trained once per backbone, with uncertainty mainly estimated from generation seeds. This matters especially for the smaller accuracy gains. Repeated training and serving measurements would make the conclusions more convincing.
Questions for authors

Please refer to the weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to reviewer 8eUU

Thank you for the review. You were right about the routing-path confound, so we have re-timed every setting on the same kernel.

Fused vs PyTorch routing

We reran fixed k through ECCoMoE's fused Triton routing with the allocator removed (on an H200 with 64 concurrent requests, 256 output tokens and 3 repeats). On this kernel, fused fixed k=8 matches stock vLLM to within about 1% on all three backbones, and ECCoMoE is 3 to 10% slower than fixed k at the same budget. Because ECCoMoE uses fewer than 4 experts at budget 4 and is still slower, the gap comes from allocator overhead. We will therefore re-time Tables 3 and 16 on the fused kernel and remove any claim that ECCoMoE is faster than fixed k.

Output tok/s (H200, 64 concurrent)Qwen3.6Gemma-4Laguna
stock vLLM (8 experts)3,7575,0813,999
fused fixed k=83,7515,1023,954
fused fixed k=25,4916,2816,075
ECCoMoE budget 25,2606,0615,444
fused fixed k=44,6515,8324,931
ECCoMoE budget 44,5065,6234,607

The Table 2 comparison against stock vLLM still holds, since budget 2 serves 1.19 to 1.40× stock throughput and budget 4 serves 1.11 to 1.20×. On Qwen3.6 and Laguna, budget 4 is within about a point of the full 8-expert model on all four benchmarks or better than it in terms of accuracy, while on Gemma-4 it trails by 2 to 5 points. What ECCoMoE offers over fixed k is higher accuracy at a given budget for a 3 to 10% throughput cost; at budget 2, for instance, Qwen3.6 GPQA-D reaches 74.4 against 62.6 and Gemma-4 MATH-500 reaches 49.7 against 31.5, while running 40% and 19% faster than stock vLLM respectively.

Energy and output length

Energy per correct answer also depends on how long the outputs are, and at budget 2 fixed k writes longer answers, so ECCoMoE uses about half the energy per correct answer. At mid budgets the reverse can happen, and we will report both cases in the paper.

SettingECCoMoE tokensfixed k tokensECCoMoE J/correctfixed k J/correct
Gemma-4 MATH-500, budget 216,61421,6056,39513,460
Qwen3.6 GPQA-D, budget 47,5395,5282,0691,504

Adaptive methods at the same compute

We added NAEE, Alloc-MoE and a training-free top-p rule, each tuned to the same E[k]. At E[k]=2, ECCoMoE beats all three on every backbone and benchmark with two exceptions, since it ties top-p on Laguna MATH-500 and every competitor scores higher on Gemma-4 MuSR, while at E[k]=4 and 6 the methods are tied. The table below shows a selection of the E[k]=2 results.

E[k]=2, accuracyECCoMoENAEEAlloc-MoE paperAlloc-MoE nativetop-p
Qwen3.6 GPQA-D74.455.40.857.663.8
Qwen3.6 MBPP+80.465.50.072.968.0
Gemma-4 MATH-50049.728.711.435.532.3
Gemma-4 MuSR57.160.461.758.260.1
Laguna MATH-50086.584.40.074.986.5
Laguna MBPP+76.166.00.069.060.2

Repeated training runs

Every backbone now has 3 training seeds × 3 eval seeds at every budget, and the resulting intervals are tight; Qwen3.6 MATH-500 at budget 2, for example, is 91.8 [91.4, 92.1].

MATH-500 [90% CI]ECCoMoE, budget 2fixed k=2ECCoMoE, budget 4fixed k=4
Qwen3.691.8 [91.4, 92.1]89.1 [87.6, 90.7]92.5 [92.1, 92.9]92.3 [91.5, 93.2]
Gemma-449.7 [47.1, 52.2]31.5 [27.8, 35.2]90.2 [89.3, 91.1]89.7 [88.4, 90.9]
Laguna86.5 [85.8, 87.1]86.1 [85.3, 87.0]90.6 [90.4, 90.9]90.7 [90.2, 91.3]

Novelty

The allocator is deliberately simple, and we see its value in two contributions that prior work does not cover.

  • A finding on renormalisation. Renormalising the kept weights makes top-k degenerate, which in the paper collapses fixed k on two of three backbones and in the new comparison drops published Alloc-MoE to near-zero accuracy on Qwen3.6 and Laguna at E[k]=2, while the same rule with native weights still works at 1 to 2 experts. We see this as a useful finding of the paper, because whether a method renormalises can change how it compares in this area.
  • A budget dial on a frozen backbone. The backbone stays exactly recoverable, and because the budget is an input, one checkpoint serves t = 2 to 8 as well as a free mode, while running inside vLLM with CUDA graphs. At budget 2, this single checkpoint beats both fixed k and the adaptive baselines above.

MoSE

Thank you for the pointer, which we will discuss in related work. MoSE shares our goal of a single elastic model but narrows the width of each expert, whereas ECCoMoE drops whole experts and leaves the kept experts and their weights untouched.

Thank you for the response and additional experiments, which address some of my concerns about the evaluation. I still find the overall contribution insufficient and will keep my current rating for now, but I consider this a borderline case and am open to revisiting it after discussion with the other reviewers.

Reviewer_3iZFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose ECCoMoE, a method that decides how many experts each token uses in a pretrained MoE model. The model itself stays frozen. At each MoE layer, a small two-layer MLP (the allocator) reads the token's hidden state and picks how many of the router's top-8 experts to run. The router still decides which experts run and how much each one counts, and the kept experts are not renormalized. Therefore, keeping all 8 experts gives back the original model exactly.

The compute budget t is an input to the allocator, so one checkpoint can run at any budget between 2 and 8 experts per token. The authors also drop the budget on 25% of training steps, which lets the same checkpoint run in a "free mode" where the allocator picks the count on its own. To train the allocator, they replace the hard count with a soft mask over the router's ranking. They train on about 20M tokens of each model's own reasoning outputs. At inference, the allocator runs inside vLLM's routing step as a single Triton kernel.

The authors test three models on four benchmarks with three seeds each. They compare against fixed top-k at the same average number of experts per token. The allocator wins 18 of 84 comparisons, 15 of them at budgets of 4 or fewer experts, with the largest gain of 20 points on Gemma-4 MATH-500. The authors also find two side results. First, renormalizing the kept expert weights breaks fixed top-k on two of the three models. Second, using fewer experts per token can increase energy per response, because the model writes longer answers.

Critical strengths

The evaluation is careful and honest. The authors run three seeds, define their error bars, compare at the same average number of experts, and report losses next to wins. They also test whether the per-token choices matter by swapping them out at inference.

One frozen checkpoint covers a range of budgets, and it runs in a real serving system. Prior learned methods such as Ada-K need a separate training run for each budget. Here, one checkpoint covers budgets from 2 to 8, and the allocator runs inside vLLM with CUDA graphs. At a budget of 2 with 64 requests, throughput gain rises by 1.20 to 1.37.

Critical weaknesses

The gains are small outside tight budgets, and the main comparison is weak. The allocator wins 18 of 84 comparisons, and 15 of those wins are at budgets of 4 or fewer experts. The comparison is against fixed top-k, with values between integer k filled in by a straight line. No learned baseline reaches these tight budgets, since BEAM runs at 3.7 to 6.5 experts.

Key design choices are untested, and the baselines may be weak. The authors train each allocator once and do not test the mutual-information bonus, the 25% budget dropout, or the soft mask. The BEAM and Ada-K baselines are the authors' own versions, with different data, initialization and kernels than the originals.

Questions for authors

How does the allocator compare to a training-free rule, such as keeping experts until their router probabilities reach a threshold, tuned to the same average number of experts?

How much do the results change if you train the allocator again with a different seed? One extra run on one model at budgets 2 and 4 would show whether the wins in Table 1 hold.

2 comments
AuthorsOpenReview ↗

Response to review 3iZF (1 of 2)

Thank you for the careful reading and the concrete questions. We ran both of the experiments you requested and also ablated the components you flagged, as described below.

Training-free threshold baseline

We added top-p routing, which keeps experts in router order until their cumulative router weight reaches p, and bisected p per backbone so that the measured E[k] is 2, 4 or 6, and we also added NAEE and Alloc-MoE to the comparison. At E[k]=2, ECCoMoE beats all three methods on 10 of the 12 cells below, ties top-p on Laguna MATH-500 and trails on Gemma-4 MuSR, where every competitor scores higher, while at E[k]=4 and 6 all methods are tied. The table uses Alloc-MoE with native weights, because the published version, which renormalises the kept weights, collapses to near-zero accuracy on Qwen3.6 and Laguna. The full tables will be added to the updated paper.

E[k]=2: ECCoMoE / NAEE / Alloc-MoE native / top-pMATH-500GPQA-DMuSRMBPP+
Qwen3.691.8 / 84.1 / 89.9 / 85.574.4 / 55.4 / 57.6 / 63.864.4 / 44.0 / 56.7 / 46.680.4 / 65.5 / 72.9 / 68.0
Gemma-449.7 / 28.7 / 35.5 / 32.344.9 / 38.0 / 39.1 / 39.157.1 / 60.4 / 58.2 / 60.149.1 / 30.1 / 21.0 / 38.2
Laguna86.5 / 84.4 / 74.9 / 86.548.5 / 34.0 / 42.3 / 37.260.8 / 56.8 / 59.8 / 55.876.1 / 66.0 / 69.0 / 60.2

Training variance

Every backbone now has 3 allocator training seeds × 3 eval seeds on all four benchmarks at every budget (with 90% CIs), and fixed k runs on the same fused kernel with the allocator removed. The MATH-500 results below cover both budgets you asked about, and because the intervals stay narrow across retrained allocators, the budget-2 wins on Qwen3.6 and Gemma-4 hold with intervals that do not overlap fixed k, whereas at budget 4, and on Laguna at both budgets, the two lie within each other's intervals.

MATH-500 [90% CI]ECCoMoE, budget 2fixed k=2ECCoMoE, budget 4fixed k=4
Qwen3.691.8 [91.4, 92.1]89.1 [87.6, 90.7]92.5 [92.1, 92.9]92.3 [91.5, 93.2]
Gemma-449.7 [47.1, 52.2]31.5 [27.8, 35.2]90.2 [89.3, 91.1]89.7 [88.4, 90.9]
Laguna86.5 [85.8, 87.1]86.1 [85.3, 87.0]90.6 [90.4, 90.9]90.7 [90.2, 91.3]

Gains above t=4

We agree that the gains are concentrated at tight budgets, since from budget 4 upward ECCoMoE matches fixed k within the confidence intervals. At budget 2, ECCoMoE leads on every Qwen3.6 and Gemma-4 cell, while on Laguna it ties fixed k except on GPQA-D, where there is a 2.8 point loss.

Budget 2: ECCoMoE / fixed kMATH-500GPQA-DMuSRMBPP+
Qwen3.691.8 / 89.174.4 / 62.664.4 / 59.780.4 / 74.2
Gemma-449.7 / 31.544.9 / 40.257.1 / 55.249.1 / 40.9
Laguna86.5 / 86.148.5 / 51.360.8 / 60.376.1 / 75.0

Fixed k does better in some cells, mostly at budget 3, namely Gemma-4 MBPP+ and MuSR and Laguna MATH-500, MuSR and MBPP+, and Gemma-4 MBPP+ also trails at budget 4 (78.6 vs 81.6). Part of the budget-3 gap is explained by the allocator spending only 2.6 to 2.9 experts there. We will report these cells in the paper and state the scope plainly, which is that a single checkpoint wins at tight budgets and matches fixed k elsewhere.

Budget 3: ECCoMoE / fixed kMATH-500GPQA-DMuSRMBPP+
Qwen3.692.3 / 92.375.6 / 76.665.1 / 65.081.0 / 81.0
Gemma-486.0 / 79.769.2 / 64.864.4 / 66.570.0 / 76.3
Laguna89.1 / 90.352.9 / 53.562.3 / 63.877.9 / 80.2
AuthorsOpenReview ↗

Response to reviewer 3iZF (2 of 2)

Continued from earlier response (see above)

Untested components:

  • MI bonus - This term is necessary, because without it the allocator underspends at budget 2 on every backbone and loses accuracy, with Gemma-4 MATH-500 at budget 2 dropping to 25.8. In free mode the allocator then collapses toward an on/off gate that spends more experts without any gain in accuracy.
  • Soft vs hard mask - A straight-through hard mask performs marginally better at budget 2, though within noise, and ties at budget 4, which shows that the soft relaxation is not what drives the results, so we will present the two as equally valid variants (see the table below).
  • Budget dropout - Budget dropout has no measurable effect on Qwen3.6 or Laguna, where accuracy stays within about a point with the same E[k] and the same free-mode spend, so we will describe it as an optional regulariser rather than a core component.
Modelpaper recipeno MI bonushard maskno budget dropout
Qwen3.6 GPQA-D, budget 274.4 (2.34)71.7 (1.94)75.9 (2.37)75.4 (2.29)
Laguna MATH-500, budget 286.5 (2.09)83.7 (1.86)88.0 (2.12)87.2 (2.04)
Laguna GPQA-D, budget 248.5 (2.13)45.6 (1.91)51.6 (2.17)47.2 (2.11)
Qwen3.6 MATH-500, free mode92.2 (4.56)92.2 (5.87)92.4 (4.58)92.1 (4.57)
Qwen3.6 free mode, tokens at 1 or 8 experts20% / 22%28% / 66%18% / 23%19% / 24%

Baselines as adaptations. As published, BEAM fine-tunes the whole model, and both BEAM and Ada-K fix one operating point per training run, so for a fair comparison we adapted both to a frozen backbone. Appendix E lists every departure from the original methods, and we will make this clearer in the main text. In the new comparison we also ran Alloc-MoE both as published, with the kept weights renormalised, and with native weights and no renormalisation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised paper uploaded

We have uploaded a revised version of the paper that incorporates the changes described in our responses. All ECCoMoE results are now reported over three training runs of each allocator, each evaluated with three sampling seeds. We've also included more detailed AllocMoE, NAEE and thresholded top-p comparisons against ECCoMoE. We thank the reviewers for their feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.