- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes ECCoMoE, which inserts a MLP allocator into every MoE layer of a frozen MoE model. Given the layer input and a budget t, the allocator decides how many of the router's top-8 experts to keep. During training all model weights are frozen, the MLP is trained on text that the model itself generated at k=8. At inference the budget is folded into the allocator bias, k is taken as a per-token argmax, and only the first k experts are run. The main finding is that, compared to fixed top-k with the same average number of experts, accuracy improves under tight budgets.
Critical strengths
- The method is plug-and-play and can recover the original model exactly. The allocator only decides where to truncate along the router's ranking and never touches expert selection or weights.
- Only the allocator is trained, and for one backbone the whole pipeline from data generation to finished training takes under 10 GPU hours.
Critical weaknesses
Alloc-MoE and NAEE also freeze the backbone and allocate experts across layers and tokens, but the paper does not compare against this line of methods.
Questions for authors
- How was the hidden width of the allocator MLP chosen?
- What is the motivation for the two-stage training, and is there any evidence that it works better than a single-stage schedule in this experimental setting?
2 comments
I appreciate the authors' efforts in providing the results. While I still have reservations about the significance of the overall contribution and hold my current score at this stage, I will gladly keep an open mind during the reviewer discussion phase.
Response to reviewer jgs7
Thank you for the review. We have now run all three of the comparisons you asked for, and we summarise the results below.
Alloc-MoE and NAEE
We ran both methods, together with a training-free top-p rule that keeps experts in router order until their cumulative router weight reaches p, and tuned each of them so that its measured E[k] is 2, 4 or 6. Alloc-MoE was run in two forms, once as published with renormalised weights and once with the router's native weights. At E[k]=2, ECCoMoE outperforms every method on every backbone and benchmark with two exceptions, since it ties top-p on Laguna MATH-500 and all of the competitors score higher on Gemma-4 MuSR. At E[k]=4 and 6 the methods are tied. The table below shows a selection of the E[k]=2 results, and the full tables will be added to the updated paper.
Published Alloc-MoE collapses to near-zero accuracy on Qwen3.6 and Laguna at E[k]=2, and this is not an implementation error. Renormalising the kept weights makes top-k degenerate on these routers, whereas the same rule with native weights continues to work with only 1 to 2 experts. This degeneration is one of the findings of our paper, and the new comparison reproduces it.
Allocator width ablations
The width was not tuned; we set it to the number of routed experts, which gives the allocator's first layer the same shape as the router projection and lets it be computed in the same GEMM as the router (Table 10). To check whether this choice matters, we have now ablated widths of 128, 256 and 512 at budgets 2 and 4 on all four benchmarks (3 training × 3 eval seeds), and across these widths throughput changes by at most 2% on every backbone. Qwen3.6 is largely insensitive to width. At budget 2, every width stays within about 2 points of the paper's 256 on all four benchmarks, as the table below shows. At budget 4 the spread is under 1.5 points. Gemma-4 is more sensitive to width at budget 2, so we will add the full width table to the updated paper and state this clearly.
Two-stage schedule
We compared the two-stage schedule with sampling the budget uniformly from the first training step. Accuracy is mixed between the two, with differences of up to about 3 points in either direction, but the schedules differ clearly in budget adherence, because single-stage training overspends at budget 2 on every backbone. The curriculum therefore buys adherence to the budget the user sets rather than higher accuracy, and we will make this point explicitly in the updated paper, where the full tables will also be added.