- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper examines token routing in Mixture-of-Experts (MoE) language models through a stochastic online linear programming (SOLP) framework. The framework maximizes token-expert affinity scores with linear constraints on per-token expert counts, pooled compute budgets, and balanced expert workloads. As the authors honestly disclose in a very scholarly related works section, viewing routing through linear assignment and primal-dual updates builds on precedent in Lewis et al. (2021) for BASE Layers and Han & Zhong (2025) for ALF-LB. However, the authors push this perspective further by unifying existing and new routing rules over five degrees of design freedoms. The resulting framework covers fixed-cardinality Top-k routing, Expert Choice routing, variable-cardinality score-threshold (VarK) and nucleus (TopP) routing, per-layer and cross-layer compute budget pooling, and causal sequence- or batch-level inference routing.
In the authors' formulation, the router subtracts two learned scalar ''prices'' from each expert's score, counts how many experts have a positive adjusted score, clips that count, and routes the token to that many top-scoring experts. Updating these prices via online gradient descent over update steps gives an inverse-square-root convergence rate on both expected score suboptimality and constraint violations. Additionally, the authors provide experiments across seven Gemma-4 MoE models of varying sizes (from 183 million to 6.59 billion parameters) demonstrating that relaxing fixed per-token expert counts and pooling the compute budget across all layers monotonically lowers validation loss.
Critical strengths
- Honest and scholarly discussion of related work surveying prior MoE routing schemes and theoretical analyses.
- Clear theoretical formulation (SOLP) and inverse-square-root convergence guarantee covering many commonly used MoE routing paradigms.
- Extensive experiments spanning seven Gemma-4 MoE model scales (183M to 6.59B parameters) and multiple routing methods.
Critical weaknesses
Writing is generally clear, but some places can be improved. Generally the work is already good and self-contained.
Questions for authors
-
Why is ''TopP-Layer'' missing in Figure 1? It seems to be the only one to be missing from that table?
-
Why are the 183M and 414M models excluded from the forward-compute scaling curve in Figure 1(c)?
-
For the tuning in Figure 2(a), are all parameters turned or just the dual variables?
-
Equation (1) allows raw logits, probabilities, or log-probabilities as router scores. However, it is unclear which choice was used in Corollary 1 and the experiments. Might the authors clarify?
-
Section 3 and Table 2 define the compute budget as an equality constraint, but Equations (3)–(4) formulate it as an inequality with a non-negative Lagrange multiplier. While it might turn out to be mathematically equivalent, which constraint is enforced in the experiments?
3 comments
I appreciate the authors' careful responses and the revisions they have made.
Since I am the odd-one-out here in terms of reviewer score, I will would like to give the following clarification for the AC and my fellow reviewers:
- This paper should be viewed as a conceptual clarification rather than an engineering contribution or realistic design proposition.
- In particular, the main concern of the other reviewers seem to be that the authors inadequately (in terms of experiments or design) justified that their SOLP framework could lead to the design of better/realistic MoE balancing updates. I agree with the other reviewers on this point, but I feel it is beside the point.
- Instead, the authors created a unified conceptual framework through which many of the modern already used-in-practice MoE balancing frameworks could be jointly reasoned about and analyzed. To my knowledge, does not currently have a precedent in the current literature (besides Han & Zhong (2025) which does it for ALF-LB in particular). Thus, I feel like it is worth sharing with the attendees of ICLR.
If this paper turns out unsuccessful for ICLR, I encourage the authors to improve the writing clarity of this paper (which I agree is rather confusing and disorganized at times, but not worse than usual quality of previously accepted ICLR papers) and submitting it to a more specialized journal like Operations Research, Mathematics of Operations Research, or Mathematical Programming.
Final thoughts for Reviewer TmAB
We sincerely thank Reviewer TmAB for the thoughtful follow-up and for clarifying the paper's framing for the AC and fellow reviewers. We have uploaded a revised manuscript incorporating all five of your technical clarifications along with a cleaner, self-contained Section 4.
We would like to add one brief, constructive nuance to your summary: while the unified primal–dual view is indeed a core contribution, the framework does not merely analyze existing methods - - it directly yields genuinely new routing algorithms with concrete practical gains:
- Cross-layer budget pooling (
VarK-Global/TopP-Global): Relaxing the budget constraint across depth () with mean-zero per-layer duals (Proposition 3) is a novel algorithm that spontaneously shifts more compute from early to late layers, roughly doubling the quality gain over per-layer adaptive routing. - Inference-time causal rebalancing (
Sequence-OLP/Batch-OLP, Algorithm 2): Decoupling training and serving constraints yields a novel zero-retraining serving controller. Following Reviewer5p72's helpful push, our new Expert-Parallel benchmarks onGemma 4 26B-A4B(Appendix E.4) show this delivers1.15x–3.5xfaster prefill and1.14x–1.45xfaster decode (which Reviewer5p72confirmed resolved their efficiency concern).
Thank you again for your careful reading and constructive guidance!
Author Response to Reviewer TmAB: Clarifications on Figures, Router Scores, and Budget Constraints
We sincerely thank Reviewer TmAB for the generous assessment, careful reading, and insightful technical questions, which we answer individually below.
1. Omission of
TopP-Layerin Figure 1TopP-Layerwas omitted from Figure 1(a–b) solely to avoid visual crowding with five overlapping curves; all seven scales are reported in Table 6, and we will includeTopP-Layerin Figure 1(b) and note this in the caption.2. Exclusion of 183M and 414M Models From Figure 1(c)
Scales S (183M) and M (414M) hold the shared dense MLP width fixed at 512 (rather than scaling with width d as in XL–4XL) and are dominated by the 262k-vocabulary output projection (3.2% and 8.1% routed FLOP share, Table 4), placing them outside the log-linear backbone scaling regime of 0.64B → 6.59B; we will clarify this in Section 6.1 and add the full 7-scale curve in Appendix E.
3. Parameter Tuning in Figure 2(a) Continuation Runs
Yes good catch here. Both model weights and the dual variables update simultaneously during continuation, though the dual threshold P=1−τ re-equilibrates to within ±0.005 experts/token inside the first 100 steps; we will state this explicitly in Section 6.2.
4. Router Score Representation Used in Corollary 1 and Experiments
This is an excellent question, and we apologize for the omission. Corollary 1 assumes bounded non-negative scores (directly satisfied by probabilities qt,e, or shifted bounded log-probabilities). In our experiments,
VarKapplies admission thresholding in centered log-probability space (logsoftmax(zt−λl)e>τl, Footnote 3) so the threshold is invariant to logit shifts, whereasTopPuses post-price probabilities qt=softmax(zt−λl) in the rank-slot tail-mass objective (Appendix C); in all arms, selected expert outputs are combined using renormalized raw softmax probabilities.5. Equality vs. Inequality Compute Budget Constraint
Our experiments enforce the equality constraint (=ktar/E): in
VarK, because log-probabilities are non-positive, the dual multiplier τ∈R is updated without non-negativity clipping, and inTopP, P=1−τ remains strictly inside (0,1), realizing kˉ∈[3.986,4.029] across all runs. Equations (3)–(4) wrote ≤ktar with τ≥0 under positive scores ct,e>0, where unconstrained demand exceeds ktar so the inequality binds at equality by complementary slackness; we will harmonize Sections 3 and 4 to make this explicit.