- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a smart compute–communication overlap technique for MoE models. Latency gains are demonstrated across multi-concurrency settings across popular open-source models with compute/communication ablations done. While analysis and intuition is properly documented, this work can be further improved by demonstrating any potential gains/tradeoffs in higher batch (up to 128) settings that are realistic in many production inference deployments.
Edit: Raising score from 3 to 4 after concerns are addressed. Paper has a lot of value for practical latency-sensitive deployments.
Critical strengths
- Throughput gains are shown in realistic experimental scenarios against strong baselines.
Critical weaknesses
- Experiments show gains only up to batch size 8.
Questions for authors
- Can the authors extend the experiments beyond batch size 8 to evaluate whether the reported speedups persist at higher batch sizes (128)?
3 comments
Thank you for the transparent analysis for this. No improvement is to be expected as high batch since it's more compute-bound at that regime. This is consistent with a lot of practical-industry decode settings. Paper is still very useful for latency sensitve regimes of serving in practical settings.. Raising score from 3->4 since all concerns are addressed.
Thank you
Thank you so much for the kind words and for raising your score! Your suggestion to look at larger batches was really helpful and made the paper's scope much clearer. The extended results are already in the revised PDF (App. O.6), and we will keep them in the final version. We truly appreciate your time and thoughtful feedback.
Response to Reviewer FJWv
We thank the reviewer for the positive assessment. We extended Table 10 to batch 128 on Qwen1.5-MoE (12 GB budget, caching disabled, 174 tokens per request; new rows are a mean of 3 runs):
The speedup declines smoothly to parity at batch ≥ 32; the large-batch values are within run-to-run variation.
Transfer volume sets this limit. At batch 128 the batch touches 51 of 60 experts per layer, more than any overlap window can hide. A deployment can disable prediction above a batch threshold and avoid CEPA's ~2% overhead there.
Batch 128 is also the edge of this memory regime. The KV cache already uses 4.4 GB of the 12 GB budget, and 512-token requests would need about 13 GB. CEPA targets latency-bound decoding on memory-constrained devices, where batch is typically below 8 and often 1. The revision adds the extended table to App. O.6 and states this scope.