- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper aims to enable MoE to generate responses in batches with fewer experts and routes, while still maintaining the quality of the responses. The author first retains the most important routes for each token, then utilizes the reuse of experts among the same batch requests, and deletes the remaining routes. Subsequently, the weights of the retained routes are adjusted, and a dedicated implementation is used to skip the deleted parts. The experiment compared the quality of the responses and the generation speed.
Critical strengths
S1. Reducing the number of active experts and routes from tokens to experts is a clear idea. It takes into account both the cost of accessing expert weights and the actual execution cost.
S2. The comparison of deleting routes and replacing routes is convincing. The author fixed the expert set and the original weights, showing that deleting routes alone can bring benefits, not just the expert selection or weight adjustment.
S3. The comparison of execution optimization maintains consistency in routes, weights, and outputs. This makes it easier to confirm that the additional acceleration comes from skipping invalid calculations rather than further sacrificing the quality of the responses.
Critical weaknesses
W1. Accuracy is measured under fixed batches, and speed is measured under dynamic service batches. Other requests in the same batch can affect the expert selection. The appendix also shows that batch size affects quality, so it is necessary to confirm whether the same answer quality can still be maintained when actual acceleration is achieved.
W2. The trigger condition for weight correction is determined before supplementing experts. Some tokens may have no routes deleted in the end and may still have their weights reduced. Therefore, it is not clear whether the benefit comes from fixing the changes after deleting routes or from ordinary output scaling.
Questions for authors
Q1. Can the same set of evaluation requests be used to measure the quality and speed of answers simultaneously under dynamic service batching?
Q2. With the option for supplementary experts enabled, using the same calibration method, how does the current correction compare to "only correcting tokens that are ultimately missing a route" and "uniformly scaling all tokens"?
3 comments
Response to Reviewer ibXZ (Part 2 / 2)
W2: "The trigger condition for weight correction is determined before supplementing experts. Some tokens may have no routes deleted in the end and may still have their weights reduced. Therefore, it is not clear whether the benefit comes from fixing the changes after deleting routes or from ordinary output scaling."
Q2: "With the option for supplementary experts enabled, using the same calibration method, how does the current correction compare to "only correcting tokens that are ultimately missing a route" and "uniformly scaling all tokens"?"
Answer: Thanks for the detailed question. The gain requires rectifying every token that the deletion rule marks, and it is not ordinary output scaling. Calibration follows the paper's protocol (offline, , six held-out scores). Rectifying only tokens that finally lost a route coincides with the paper's rule at , so it receives the same coefficient. Uniform scaling, calibrated separately, selects 0.80 on Qwen1.5 and 0.95 on Qwen3. Results are eight-task averages, paired per question:
| Rule | Qwen1.5 k1b5 | Qwen3 k1b10 |
|---|---|---|
| No rectification (s=1) | 47.03 | 81.55 |
| Paper rule ( set before supplementation) | 49.16 (s=0.85) | 82.01 (s=0.95) |
| Only tokens finally missing a route | 47.34 (−1.82, p=0.001) | 81.25 (−0.76, p=0.032) |
| Uniform scaling, own calibration | 48.52 (s=0.80; −0.64) | 81.27 (s=0.95; −0.74, p=0.025) |
| Uniform scaling at the paper's s | 48.93 (−0.23, p=0.69) | — (same s) |
Rectifying only tokens that finally lost a route loses most of the gain on Qwen1.5 and all of it on Qwen3, where it falls below native weights. The tokens W2 asks about, whose routes were all restored by supplementation, matter: rectifying them alone gives +0.49 and rectifying only the other affected tokens gives +0.31, but rectifying both gives +2.13. Uniform scaling is never better than the paper rule. With its own calibration it is 0.64 points lower on Qwen1.5 and 0.74 points lower on Qwen3 (p=0.025), and even at the paper's s it does not improve on Qwen1.5 (−0.23, p=0.69). The paper rule has the highest accuracy on both models.
Rectification is also not an output-temperature adjustment. Dividing the logits by 1.25 or 1.5 at does not raise accuracy (−0.31 and −0.90) and stays 2.4–3.0 points below rectification (p≤0.0001). Rectification acts on the affected tokens' routed outputs inside the network.
Revision: We add this comparison and the output-temperature control to Section 4.4.3 (Table 7), with the full table in Appendix C.6 (Table C25).
Thank you again for your valuable feedback.
Thank you for the clarifications. Most of my questions have been addressed, and I will keep my current score unchanged.
Response to Reviewer ibXZ (Part 1 / 2)
We sincerely thank the reviewer for the careful reading and the encouraging feedback. We are glad that you found reducing both active experts and token–expert routes a clear idea (S1), the comparison between route deletion and route replacement convincing (S2), and the execution comparison consistent in routes, weights and outputs (S3).
Your suggestions were highly constructive and helped us strengthen the paper: they led us to measure accuracy and speed on the same requests under dynamic batching, and to compare the paper rule against the two alternatives you proposed. We hope these results further strengthen your support, and we would be grateful if you would consider raising your score. Below, we provide detailed responses to all the comments:
W1: "Accuracy is measured under fixed batches, and speed is measured under dynamic service batches … it is necessary to confirm whether the same answer quality can still be maintained when actual acceleration is achieved."
Q1: "Can the same set of evaluation requests be used to measure the quality and speed of answers simultaneously under dynamic service batching?"
Answer: Thanks for the constructive suggestion. Yes. Qwen3-30B-A3B serves a stratified panel of 4,384 questions from the eight paper tasks, replayed as Poisson-arrival requests with the paper's prompts and a fixed per-request seed, so accuracy and TPOT come from the same responses. We test QPS 1.68 and QPS 4.0, which is about 80% of Vanilla's capacity. Vanilla runs first and last, and the two runs differ by only −0.20 points (95% CI [−0.73, +0.33]). Rescored on this panel, the paper's fixed-batch predictions reproduce the Table 1 differences (k=1: −19.96 vs. −20.23; (1,10): −0.94 vs. −0.70).
Under realistic load, the near-saturation point keeps quality while accelerating: at QPS 4.0 it is within noise of Vanilla (−0.56) at 1.40× TPOT, with 9.1 ms lower TTFT. Both DelMoE points are faster than every baseline at both loads, and at QPS 1.68 Near-sat. is also far more accurate than SERE (−9.92) and XShare (−5.96).
At QPS 4.0, SERE, XShare and Near-sat. are all within noise of Vanilla in accuracy. Slower methods hold more concurrent requests and therefore decode larger batches, which retain a larger shared expert union and favor accuracy. XShare's 0.08-point edge over Near-sat. comes with a 50% larger decode batch (79.8 vs. 53.2) and a TPOT above Vanilla's. DelMoE reaches comparable accuracy while decoding faster with smaller batches.
Revision: We add this experiment to Section 4.3 (Table 3), with the full protocol and table in Appendix C.3 (Table C11).