- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes MarCap, a framework for reducing memory overhead in Mixture-of-Experts (MoE) inference while controlling buffer overflow risks. The key idea is to determine receive-buffer capacities using expert routing marginals and a user-specified overflow risk budget, rather than relying on worst-case memory reservation or heuristic capacity factors. The authors formulate a worst-case moment-generating function over all fixed top-k routing distributions consistent with the observed marginals. Under the IID token assumption, they derive probabilistic capacity bounds and introduce an exact linear programming reduction, further accelerated by an efficient lower convex-hull construction. To handle occasional overflows without dropping tokens, MarCap is integrated with a lossless multi-round replay kernel. Experiments using 2.737 billion token-level routing records from two MoE models validate the proposed capacity estimation approach and investigate its limitations under token dependence and routing distribution shifts. Overall, the paper combines theoretical capacity estimation with practical kernel implementation to improve memory efficiency in MoE inference.
Critical strengths
-
Clear motivation: The paper addresses an important systems challenge in MoE inference: allocating sufficient workspace for unpredictable expert routing loads without excessive memory reservation. The motivation is clear, and the proposed risk-aware formulation is practically relevant.
-
Solid theoretical contributions: The paper provides a mathematically grounded formulation for capacity provisioning based on routing marginals. The exact linear programming reduction and efficient lower convex-hull construction are technically interesting and substantially reduce the computational overhead of capacity estimation.
-
Practical system implementation: The integration of MarCap with the lossless replay kernel provides a practical mechanism for handling occasional buffer overflows without dropping tokens. This connects the theoretical formulation with an implementable system design.
Critical weaknesses
-
W1. Presentation and readability: Although the paper provides extensive theoretical analysis and experimental results, the overall presentation could be improved. In particular, the experimental section contains numerous configurations, metrics, and appendix results, making it difficult to identify the key findings and understand how the theoretical contributions translate into practical benefits. The authors could consider organizing the experiments around clearer research questions (RQs), highlighting the main conclusions, and providing more intuitive summaries of the theoretical and empirical results.
-
W2. Robustness of risk guarantees: The theoretical risk guarantees rely on the IID token assumption and accurate routing marginals, which may not always hold in real-world MoE serving scenarios. The paper reports substantial overflow risk violations under inter-token dependence and marginal drift, despite the specified risk budget. Although the authors investigate traffic mixing, sliding-window recalibration, and request-level dependence, the robustness of these mitigation strategies under abrupt distribution shifts remains unclear. Further analysis of capacity provisioning under marginal estimation errors and realistic routing dynamics would strengthen the practical applicability of the proposed framework.
-
W3. End-to-end system evaluation: Although the paper evaluates extensive routing traces from two MoE models, the hardware experiments are primarily conducted on Ascend 950PR with EP4 configurations and constructed inputs. Therefore, it remains unclear whether similar memory savings and latency performance can be achieved across different hardware platforms, larger expert-parallel configurations, and realistic serving workloads.
Questions for authors
-
Q1. Could the authors improve the organization of the experimental section by explicitly summarizing the key findings and explaining how the theoretical results translate into practical memory and latency benefits?
-
Q2. Could the authors extend the evaluation to more realistic end-to-end MoE serving scenarios?
2 comments
Thank you for the additional experiments and clarifications. They have addressed most of my concerns and strengthened the paper’s completeness and empirical support. Overall, I appreciate the contribution and remain supportive of acceptance. I will maintain my rating of weak acceptance.
We would first like to express our sincere appreciation for the time you dedicated to reviewing our manuscript, and we would like to address your questions and concerns below. The additional results mentioned will be incorporated into the final version of the paper submission.
Q1. On Organization
We thank the reviewer for this suggestion. We will improve the organization of the experimental section around clearer research questions with concise summarizations and make the connection between the theoretical results and the measured memory and latency benefits more explicit in the final manuscript.
Q2. On Broader Evaluation and and Portability
We have extended the hardware evaluation from EP4 on Ascend 950PR to EP8 on Ascend 950DT and added independent peak HBM memory measurements. Please refer to our response to Reviewer Zqt3's Q3 for the complete memory and latency results. At EP8 and local batch size 8192, scratch workspace decreases by 85.33% while the observed process memory peak decreases by 2952 MiB per rank, and the median paired mean latency change is -0.34%.
In practical full model serving, the released workspace can increase the memory budget available for resident requests when KV capacity becomes a constraint, reducing pressure to preempt, swap, or recompute. At a fixed batch without KV induced capacity pressure, the same expert computation is performed with lower memory reservation and similar kernel latency. The resulting serving benefits depend on the model's KV layout, parallel sharding, request lengths, allocator, and scheduler. Our controlled kernel measurements quantify the memory and latency effects of capacity selection while keeping these other conditions fixed within each experiment. The relationship to dropless and dynamic capacity systems is discussed in Section 5 and Appendix J.1 and in our response to Reviewer Zqt3's Q3.
On Risk Robustness
The contiguous prefill overflow issue is substantially mitigated by the block bound in Appendix C when the scheduler restricts each request's contribution to the allowed block length. We have evaluated this with both models and block lengths 64 and 1024. All reported Sliding configurations remain within the 1% overflow budget, while Frozen has one slight exceedance at 1.111%. The EP4 hardware experiments show zero observed overflow for block MarCap while retaining 14.52%-65.62% scratch workspace savings and similar execution latency. Please refer to our response to Reviewer Zqt3's Q2 for the complete setup and results.
For marginal estimation and shift, we use large history windows and sliding recalibration from completed routing observations, as discussed in the paper and our response to Reviewer kUGN's Q2. Abrupt changes can affect the validity of an existing capacity before the next update. As discussed in our response to Reviewer kUGN's Q3, per layer planning takes approximately 1-2 ms at the tested EP4/EP8 settings and can overlap with ongoing forward computation. Hence, the planning for an early recalibration triggered by a detected shift can run in the background without adding serving waiting time, provided that the updated capacity is ready before it is needed.