- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EchoMoE reduces communication in expert-parallel MoE inference by maintaining a periodically refreshed full-width activation reference. Non-reference layers dispatch a compressed code of the change relative to the reference, while the combine path returns compressed expert outputs. Linear transformations are folded into expert projections, with reconstruction-based initialization followed by distillation adaptation.
On an adapted Qwen3-30B-A3B checkpoint, the (s=4, rd=512, rc=1280) configuration reduces estimated inter-node activation bytes by 46% and improves throughput by 17.2% on the reported EP32 workload, at a 2.0 percentage-point reduction in average score across 18 benchmarks. The paper includes ablations, FP8-quality tests, and throughput-only tests on three larger MoE architectures.
Critical strengths
-
A well-motivated, differentiated mechanism. Cross-layer references make narrow dispatch codes substantially more useful than independent activation compression. At width 512, the stride-4 reference dispatch loses 2.6 percentage points, versus 21.1 points for plain dispatch compression (Section 5.3). This provides direct evidence for the principal idea.
-
Strong systems implementation and evaluation. The paper addresses dispatch and combine, folds compression into expert projections, integrates with vLLM/DeepEP, and measures real throughput and latency on four H100 nodes. The extensive ablations and workload profiling strengthen its technical contribution.
Critical weaknesses
-
Limited cross-model task-quality evidence. All quality results come from one adapted Qwen3-30B-A3B checkpoint. The Qwen3-235B, DeepSeek-V3, and Kimi-K2.6 tests use random weights (Appendix G.4) and demonstrate runtime feasibility, not quality retention. This leaves the broad pretrained-model retrofit claim incompletely validated.
-
Incomplete comparison with an optimized deployment baseline. Table 14 examines quality under FP8 communication, but does not report matched end-to-end throughput and latency for an optimized uncompressed FP8 model versus EchoMoE+FP8. Given the 2.0-point average quality drop and workload-dependent decode gains, this is important for assessing the net practical benefit.
Questions for authors
-
Can the authors evaluate quality and throughput on a second pretrained MoE using real weights rather than random-weight execution tests? A limited task subset would help assess transferability.
-
On matched hardware and workloads, how do optimized FP8-only communication and EchoMoE+FP8 compare in throughput, time to first token, and time per output token? This would establish the incremental benefit.
4 comments
W2, Q2. End-to-end comparison with optimized FP8 communication
FP8 and EchoMoE reduce different components of communication volume: FP8 reduces bytes per element, while EchoMoE reduces the number of elements transmitted. Their benefits therefore compose, and the same adapted EchoMoE model supports BF16 and FP8 communication without retraining (Table 33). We added matched end-to-end experiments comparing uncompressed FP8 communication against EchoMoE with FP8 communication, using identical hardware, workloads, and serving implementations (Appendix G.6). This directly measures EchoMoE's incremental benefit over an optimized low-precision baseline. For Qwen3-30B-A3B, 4096 input / 256 output tokens, 128 requests per GPU (Table 30):
| Configuration | Overall score (%) | Throughput vs. uncompressed FP8 | Throughput vs. uncompressed BF16 |
|---|---|---|---|
| Uncompressed FP8 | 66.83 | 1.00× | 1.18× |
| (4,512,1280) + FP8 | 64.86 | 1.02× | 1.21× |
| (4,512,768) + FP8 | 62.99 | 1.12× | 1.33× |
| (8,256,512) + FP8 | 55.56 | 1.27× | 1.51× |
EchoMoE delivers additional throughput improvements beyond FP8 communication alone. Uncompressed FP8 improves throughput by 18% over uncompressed BF16. EchoMoE (4,512,768) with FP8 increases this improvement to 33%, achieving 12% higher throughput than the optimized FP8 baseline with a 3.8 pp task-performance drop. At (8,256,512), the incremental throughput improvement reaches 27%. Across evaluated configurations, FP8 communication changes overall task performance by at most 0.29 pp (Table 21). On bandwidth-constrained TCP, EchoMoE+FP8 achieves 1.86–3.24× higher throughput than uncompressed FP8 (Table 34).
EchoMoE's benefits extend across workloads and latency metrics (Appendix G.6, Table 31; Figures 13–15). Relative to uncompressed FP8, (4,512,768) improves throughput by 1.01–1.14× and TTFT by 1.03–1.17×. Stronger compression with (8,256,512) achieves 1.05–1.34× throughput and 1.08–1.41× TTFT. Decode speedups increase with concurrency, reaching up to 1.24×.
Online serving confirms the latency benefits over FP8. With Poisson arrivals and real prompts, both FP8 models sustain 384 requests/s under P99 TTFT ≤ 2 s and P99 TPOT ≤ 150 ms. At this load, EchoMoE further reduces median TPOT by 7–10% and P99 TPOT by 10–11% (Appendix G.8). The additional KV-cache memory overhead is negligible, with available capacity changing by only −0.8% to +2.3% (Table 32).
These matched experiments establish that EchoMoE provides additional end-to-end throughput and latency improvements beyond optimized FP8 communication, while retaining the flexibility to combine with existing low-precision serving optimizations.
We hope these results address the reviewer's concerns, and we would be glad to answer any further questions.
Dear Reviewer A1zp,
As the discussion period is nearing its end, we wanted to kindly check whether our response has addressed your concerns. We would be happy to clarify any pending concerns!
Dear Reviewer,
Thank you so much for your thoughtful initial reviews!
With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.
Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!
Warm regards
We thank the reviewer for recognizing the effectiveness of cross-layer reference compression and the strength of the systems evaluation. We have extended our adaptation and evaluation to four additional pretrained MoEs with real weights, and added matched comparisons against optimized FP8 communication. The new experiments address both concerns are included in the revised paper.
W1, Q1. Generalization to other pretrained MoE models
We retrofitted four additional pretrained MoEs using the same procedure and a common configuration strategy: stride 4, dispatch width approximately d/4, and combine width approximately d/2, without model-specific configuration searches. Updated Table 3 reports task performance across all five models.
Across all five pretrained models, EchoMoE reduces the RDMA communication by approximately 2x with task-performance drops of 2.4–5.9 percentage points. The variation broadly follows the measured cross-layer predictability (Section 3, Table 2). Qwen3 and Kanana-2, with 56.0% and 52.0% predictability at layer separation 4, incur smaller drops of 3.1 and 2.4 pp, respectively. The three shallower models, with lower predictability, incur drops of 4.4–5.9 pp.
We further evaluated end-to-end serving with real weights on the adapted Kanana-2 and Gemma 4 models, using the same four-node H100 setup as Qwen3-30B-A3B (Appendix G.11, Table 39) with speedups relative to each model's uncompressed BF16 baseline.
At 4096 input and 256 output tokens, EchoMoE improves throughput by 15% on Kanana-2 and 14% on Gemma 4, comparable to the 17% improvement on Qwen3-30B-A3B at the corresponding configuration. Across evaluated workloads with BF16 or FP8 communication, throughput improves by 10–16% on Kanana-2 and 7–14% on Gemma 4.
Together, the cross-model quality and serving experiments demonstrate that EchoMoE's benefits generalizes across models and architectures.