acceptodds
Under review as a conference paper at ICLR 2027

Communicate the Change: Efficient Expert Parallelism via Activation Compression

14 views

Abstract

Scaling has driven the success of large language models (LLMs), with sparse mixture-of-experts (MoE) architectures enabling greater parameter capacity without a proportional increase in computation. At large scales, MoE models rely on expert parallelism to distribute experts across devices. Each MoE layer requires two synchronous collective communication steps during inference: dispatching hidden states to the activated experts and combining the resulting expert outputs. As compute throughput has outpaced interconnect bandwidth, these synchronous collectives have become a major bottleneck to MoE scaling and efficiency. We address this challenge with EchoMoE, a lightweight architecture modification that retrofits existing MoE models, requiring about 8500 times fewer training tokens than full pretraining. Motivated by the observation that transformer blocks update the residual stream incrementally, we use a hidden state from an earlier layer as a reference and communicate only the compressed difference between it and the current state, rather than communicating each hidden state in full. EchoMoE periodically refreshes these reference states and transmits only the compressed differences synchronously, reducing the traffic on latency-critical paths without significant computational overhead. EchoMoE reduces inter-node communication volume by 46% and improve the throughput by 17.2%, while preserving average performance within 2 percentage points across 18 benchmarks. EchoMoE also tolerates quantization of the communicated representations without measurable quality loss. We release an efficient implementation for applying EchoMoE to pretrained MoE models, together with an inference integration for vLLM

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Expert Parallelism, Efficiency, Inference

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_A1zpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

EchoMoE reduces communication in expert-parallel MoE inference by maintaining a periodically refreshed full-width activation reference. Non-reference layers dispatch a compressed code of the change relative to the reference, while the combine path returns compressed expert outputs. Linear transformations are folded into expert projections, with reconstruction-based initialization followed by distillation adaptation.

On an adapted Qwen3-30B-A3B checkpoint, the (s=4, rd=512, rc=1280) configuration reduces estimated inter-node activation bytes by 46% and improves throughput by 17.2% on the reported EP32 workload, at a 2.0 percentage-point reduction in average score across 18 benchmarks. The paper includes ablations, FP8-quality tests, and throughput-only tests on three larger MoE architectures.

Critical strengths
  1. A well-motivated, differentiated mechanism. Cross-layer references make narrow dispatch codes substantially more useful than independent activation compression. At width 512, the stride-4 reference dispatch loses 2.6 percentage points, versus 21.1 points for plain dispatch compression (Section 5.3). This provides direct evidence for the principal idea.

  2. Strong systems implementation and evaluation. The paper addresses dispatch and combine, folds compression into expert projections, integrates with vLLM/DeepEP, and measures real throughput and latency on four H100 nodes. The extensive ablations and workload profiling strengthen its technical contribution.

Critical weaknesses
  1. Limited cross-model task-quality evidence. All quality results come from one adapted Qwen3-30B-A3B checkpoint. The Qwen3-235B, DeepSeek-V3, and Kimi-K2.6 tests use random weights (Appendix G.4) and demonstrate runtime feasibility, not quality retention. This leaves the broad pretrained-model retrofit claim incompletely validated.

  2. Incomplete comparison with an optimized deployment baseline. Table 14 examines quality under FP8 communication, but does not report matched end-to-end throughput and latency for an optimized uncompressed FP8 model versus EchoMoE+FP8. Given the 2.0-point average quality drop and workload-dependent decode gains, this is important for assessing the net practical benefit.

Questions for authors
  1. Can the authors evaluate quality and throughput on a second pretrained MoE using real weights rather than random-weight execution tests? A limited task subset would help assess transferability.

  2. On matched hardware and workloads, how do optimized FP8-only communication and EchoMoE+FP8 compare in throughput, time to first token, and time per output token? This would establish the incremental benefit.

4 comments

We thank the reviewer for recognizing the effectiveness of cross-layer reference compression and the strength of the systems evaluation. We have extended our adaptation and evaluation to four additional pretrained MoEs with real weights, and added matched comparisons against optimized FP8 communication. The new experiments address both concerns are included in the revised paper.

W1, Q1. Generalization to other pretrained MoE models

We retrofitted four additional pretrained MoEs using the same procedure and a common configuration strategy: stride 4, dispatch width approximately d/4, and combine width approximately d/2, without model-specific configuration searches. Updated Table 3 reports task performance across all five models.

ModelConfigurationRDMA reductionUncompressed performance (%)EchoMoE performance (%)
Qwen3-30B-A3B(4,512,1024)2.08×66.963.8 (−3.1 pp)
Kanana-2-30B-A3B(4,512,1024)2.01×58.556.1 (−2.4 pp)
Gemma 4 26B-A4B(4,768,1408)2.01×61.256.8 (−4.4 pp)
Nemotron-3.5-Lightning-30B-A3B(4,768,1408)1.91×62.858.3 (−4.6 pp)
Moonlight-16B-A3B(4,512,1024)1.99×53.047.1 (−5.9 pp)

Across all five pretrained models, EchoMoE reduces the RDMA communication by approximately 2x with task-performance drops of 2.4–5.9 percentage points. The variation broadly follows the measured cross-layer predictability (Section 3, Table 2). Qwen3 and Kanana-2, with 56.0% and 52.0% predictability at layer separation 4, incur smaller drops of 3.1 and 2.4 pp, respectively. The three shallower models, with lower predictability, incur drops of 4.4–5.9 pp.

We further evaluated end-to-end serving with real weights on the adapted Kanana-2 and Gemma 4 models, using the same four-node H100 setup as Qwen3-30B-A3B (Appendix G.11, Table 39) with speedups relative to each model's uncompressed BF16 baseline.

Workload (requests/GPU)Kanana-2: throughput / TTFT / TPOTGemma 4: throughput / TTFT / TPOT
1024/128 (512)1.15× / 1.16× / 1.13×1.13× / 1.15× / 1.07×
4096/256 (128)1.15× / 1.17× / 1.06×1.14× / 1.17× / 1.07×
16384/512 (32)1.11× / 1.15× / 1.00×1.12× / 1.16× / 1.01×

At 4096 input and 256 output tokens, EchoMoE improves throughput by 15% on Kanana-2 and 14% on Gemma 4, comparable to the 17% improvement on Qwen3-30B-A3B at the corresponding configuration. Across evaluated workloads with BF16 or FP8 communication, throughput improves by 10–16% on Kanana-2 and 7–14% on Gemma 4.

Together, the cross-model quality and serving experiments demonstrate that EchoMoE's benefits generalizes across models and architectures.

W2, Q2. End-to-end comparison with optimized FP8 communication

FP8 and EchoMoE reduce different components of communication volume: FP8 reduces bytes per element, while EchoMoE reduces the number of elements transmitted. Their benefits therefore compose, and the same adapted EchoMoE model supports BF16 and FP8 communication without retraining (Table 33). We added matched end-to-end experiments comparing uncompressed FP8 communication against EchoMoE with FP8 communication, using identical hardware, workloads, and serving implementations (Appendix G.6). This directly measures EchoMoE's incremental benefit over an optimized low-precision baseline. For Qwen3-30B-A3B, 4096 input / 256 output tokens, 128 requests per GPU (Table 30):

ConfigurationOverall score (%)Throughput vs. uncompressed FP8Throughput vs. uncompressed BF16
Uncompressed FP866.831.00×1.18×
(4,512,1280) + FP864.861.02×1.21×
(4,512,768) + FP862.991.12×1.33×
(8,256,512) + FP855.561.27×1.51×

EchoMoE delivers additional throughput improvements beyond FP8 communication alone. Uncompressed FP8 improves throughput by 18% over uncompressed BF16. EchoMoE (4,512,768) with FP8 increases this improvement to 33%, achieving 12% higher throughput than the optimized FP8 baseline with a 3.8 pp task-performance drop. At (8,256,512), the incremental throughput improvement reaches 27%. Across evaluated configurations, FP8 communication changes overall task performance by at most 0.29 pp (Table 21). On bandwidth-constrained TCP, EchoMoE+FP8 achieves 1.86–3.24× higher throughput than uncompressed FP8 (Table 34).

EchoMoE's benefits extend across workloads and latency metrics (Appendix G.6, Table 31; Figures 13–15). Relative to uncompressed FP8, (4,512,768) improves throughput by 1.01–1.14× and TTFT by 1.03–1.17×. Stronger compression with (8,256,512) achieves 1.05–1.34× throughput and 1.08–1.41× TTFT. Decode speedups increase with concurrency, reaching up to 1.24×.

Online serving confirms the latency benefits over FP8. With Poisson arrivals and real prompts, both FP8 models sustain 384 requests/s under P99 TTFT ≤ 2 s and P99 TPOT ≤ 150 ms. At this load, EchoMoE further reduces median TPOT by 7–10% and P99 TPOT by 10–11% (Appendix G.8). The additional KV-cache memory overhead is negligible, with available capacity changing by only −0.8% to +2.3% (Table 32).

These matched experiments establish that EchoMoE provides additional end-to-end throughput and latency improvements beyond optimized FP8 communication, while retaining the flexibility to combine with existing low-precision serving optimizations.

We hope these results address the reviewer's concerns, and we would be glad to answer any further questions.

Dear Reviewer A1zp,

As the discussion period is nearing its end, we wanted to kindly check whether our response has addressed your concerns. We would be happy to clarify any pending concerns!

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

Reviewer_iY4gOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EchoMoE, which reduces MoE expert-parallel communication by caching cross-layer reference activations, transmitting compressed differences, and compressing expert outputs, with distillation and full-model adaptation used to recover accuracy. Under a specific deployment setting for Qwen3-30B-A3B, the authors report approximately 46% lower estimated communication volume and 17.2% higher throughput, at the cost of a roughly 2-percentage-point drop in average task performance. The method demonstrates the practical value of exploiting cross-layer redundancy, but its gains primarily benefit prefill; its advantages at larger parallel scales, under realistic online workloads, and against strong low-precision communication baselines remain to be established.

Critical strengths

The paper provides a concrete implementation and useful ablations showing that cross-layer references improve the accuracy–communication trade-off over plain activation compression.

Critical weaknesses
  1. Inherent scalability limitations: Standard EP communicates only with nodes hosting selected experts, whereas EchoMoE broadcasts reference activations to all nodes. As parallelism increases, growing broadcast and cache costs may progressively offset the compression benefits.

  2. Insufficient baselines and resource controls: The paper lacks a complete end-to-end comparison against direct FP8 communication. Comparisons with plain compression also do not control for the additional parameters, computation, and cache required by the reference pathway, so better accuracy at a matched communication budget does not establish superior overall serving efficiency.

  3. Activation similarity does not guarantee preservation of task-critical information: High cosine similarity and low reconstruction error do not imply that discarded information is unimportant. The paper insufficiently analyzes how compression errors propagate through nonlinear experts, subsequent layers, and routing decisions, or how sensitivity varies across tasks and experts.

  4. Speedups depend on specific workloads: The experiments primarily measure batch throughput with random inputs, fixed output lengths, and simultaneously submitted requests, with gains concentrated in prefill. Limited decode improvements and the absence of evaluations under dynamic arrivals, mixed sequence lengths, and tail-latency constraints leave the practical value for online serving unclear.

  5. High training cost and a complex adaptation pipeline: The method requires activation collection, teacher-logit caching, configuration search, and full-parameter adaptation on 4.32B tokens per configuration. Comparing this cost with pretraining understates the deployment burden; without complete GPU-hour accounting and a break-even analysis, it remains unclear whether inference savings justify the adaptation expense.

  6. Missing comparisons with layer-adaptive compression: The method primarily uses a uniform reference stride and compression widths without sufficiently examining differences in layer sensitivity. Comparisons with adaptive reference refresh and layer-specific compression budgets are needed to establish whether the reported accuracy–communication trade-off is competitive.

Questions for authors
  1. At what EP scale do reference broadcasting and cache overhead outweigh the communication savings, and how does this threshold vary with routing locality and workload?

  2. When jointly accounting for accuracy, computation, memory, and tail latency, does EchoMoE retain an end-to-end advantage over direct FP8 communication and plain compression?

  3. What serving volume is required to amortize the complete adaptation cost, including configuration search, teacher-logit generation, and 4.32B-token training per configuration?

  4. Could reference refresh frequency and compression widths adapt to layer sensitivity and runtime network load, allowing a single adapted model to maintain a better accuracy–latency trade-off across deployment conditions?

Clear answers and supporting evidence could persuade me to raise my score.

6 comments

We thank the reviewer for the constructive feedback and for recognizing the implementation and the benefits of cross-layer references over plain compression. We have expanded the evaluation to address scalability, FP8 communication, error propagation, online serving, adaptation costs, and layer-adaptive compression. The new experiments address the concerns and questions are included in the revised paper.

W1, Q1. Scalability of reference communication and caching

EchoMoE maintains communication savings across EP16–EP128, even after accounting for reference broadcasts. We extended the communication analysis to larger EP degrees using measured routing where available (Qwen3-30B-A3B) and uniform routing otherwise (DeepSeek-V3) (Appendix F.1, Table 20). RDMA reduction relative to standard EP:

ConfigurationEP16EP32EP64EP128
Qwen3 (4,512,1280)1.88×1.84×1.69×1.49×
Qwen3 (4,512,768)2.46×2.39×2.15×1.82×
Qwen3 (8,256,512)4.13×4.03×3.66×3.04×
DeepSeek-V3 (4,1792,3584)2.12×2.06×1.87×1.61×

The reduction decreases with EP because references reach every node, while dispatch and combine communicate only with nodes hosting selected experts. Combine compression remains effective independently of reference communication, and larger strides can further reduce the reference overhead.

Routing locality is accounted. Equation (8) charges each reference as a broadcast to all mmax⁡m_{\max} remote nodes, while dispatch and combine reach only the mˉ\bar m remote nodes hosting a token's experts in that layer. Stronger locality lowers mˉ\bar m, which reduces both the baseline and EchoMoE's routed traffic but not the reference broadcast, so the reduction shrinks. Qwen3-30B-A3B's measured routing shows little node locality: each token's experts reach 2.732.73 remote nodes per layer at EP32, close to the 2.722.72 of uniform routing, so measured and uniform estimates in Table 20 differ by at most 0.01×0.01\times up to EP64.

Memory overhead is marginal. The reference cache occupies 1.00 GiB per GPU at EP32 and 2.00 GiB at EP64. Changes to expert weights largely offset this overhead, leaving available KV-cache capacity within −0.8% to +2.3% of the uncompressed model (Appendix F.3). Reference communication is included in our end-to-end serving measurements, with its runtime cost analyzed separately in Table 29.

W2, Q2. Comparison with FP8 communication and resource overhead

EchoMoE provides additional end-to-end throughput gains over direct FP8 communication. We added a comparison in which both the uncompressed model and EchoMoE use FP8 for dispatch and combine (Appendix G.6, Table 30).

ConfigurationOverall score (%)Throughput vs. uncompressed FP8
Uncompressed FP866.831.00×
EchoMoE (4,512,1280) FP864.861.02×
EchoMoE (4,512,768) FP862.991.12×
EchoMoE (8,256,512) FP855.561.27×

Qwen3-30B-A3B, 4096 input / 256 output tokens, 128 requests per GPU.

At (4,512,768), EchoMoE improves throughput by 12% over uncompressed FP8 and 33% over uncompressed BF16. Across the evaluated serving workloads, it improves throughput by up to 14% and TTFT by up to 17% relative to FP8. The two approaches are complementary: applying FP8 to EchoMoE changes overall task performance by at most 0.29 pp (Table 21). Under online FP8 serving, EchoMoE further reduces median TPOT by 7–10% and P99 TPOT by 10–11% at 384 requests/s (Appendix G.8). Serving-time overheads are included in our measurements. Folding reduces matrix MACs to 0.98× and 0.86× those of the uncompressed model for (4,512,768) and (8,256,512), respectively (Table 22).

We also expanded the resource-controlled comparison with plain compression. At matched RDMA budgets, EchoMoE preserves 2.1–5.1 pp more task performance, while at comparable task performance, it communicates 26% fewer bytes (Appendix C.2, Table 13). Plain compression requires 35–65% fewer matrix MACs and avoids the reference cache, but its lower computation comes with substantially greater performance degradation. Cross-layer references therefore improve the accuracy–communication trade-off rather than merely reducing activation dimensions.

W3. Preservation of task-critical information and error propagation

EchoMoE is designed to preserve model behavior, not simply minimize activation reconstruction error. Our initialization objective measures dispatch reconstruction error after the experts' gate and up projections, weighted by expert routing frequency (Section 4.4). This directly accounts for how reconstruction errors affect expert computation, and reduces the initial output KL divergence by 12–18% compared with minimizing raw activation reconstruction error (Appendix B.2). Subsequent end-to-end adaptation minimizes the divergence from the uncompressed model's final output distribution, accounting for errors propagated through nonlinear experts and later layers. Held-out KL continues to decrease throughout adaptation (Figure 7).

We further measured how compression affects downstream routing (Appendix C.3, Table 14). The router operates on the full-width activation at each layer, so routing differences arise only from errors propagated through preceding layers. At approximately 2× RDMA reduction, EchoMoE preserves 86.4–86.6% of the original top-8 expert assignments, compared with 84.7–84.8% for plain compression. Router KL is also lower, at 0.024–0.028 versus 0.033–0.040. The same pattern holds at approximately 4× RDMA reduction, showing EchoMoE better preserves downstream routing decisions.

W4. Online serving with dynamic arrivals and tail latency

EchoMoE sustains 1.5× the request rate of the uncompressed model under the same tail-latency constraints. We added online serving experiments using 5,017 held-out Dolci-Instruct prompts with variable input lengths, naturally terminated responses, and Poisson arrivals from 16 to 512 requests/s (Appendix G.8). Experiments use vLLM on 32 H100 GPUs (EP32) with each model's own routing decisions.

Arrival rateP99 TTFT (ms)Median TPOT (ms)P99 TPOT (ms)
16 req/s193 / 19729.6 / 30.161.1 / 58.8
128 req/s290 / 26070.3 / 66.888.3 / 79.5
256 req/s654 / 355107.8 / 88.8147.8 / 109.2
384 req/s19,255 / 1,584137.2 / 122.1181.8 / 141.7

Uncompressed / EchoMoE (4,512,768), BF16 communication (Table 35).

With P99 TTFT ≤ 2 s and P99 TPOT ≤ 150 ms, EchoMoE sustains 384 requests/s, compared with 256 requests/s for the uncompressed model. At 256 requests/s, it reduces P99 TTFT by 46%, median TPOT by 18%, and P99 TPOT by 26%. At 384 requests/s, the uncompressed model experiences severe queuing delays, while EchoMoE remains within the latency limits.

To isolate the effect of communication compression from differences in generated output lengths and routing load, we also evaluated balanced routing with identical response lengths. At 256 requests/s, EchoMoE reduces median TPOT from 117.8 to 88.6 ms and P99 TTFT from 501 to 314 ms. The gains persist with the same amount of generation per model.

Additional experiments on 16K–32K-token conversation histories show 1.15–1.54× improvements in median TPOT (Appendix G.9). Online serving experiments with retrofitted Kanana-2 and Gemma 4 models further demonstrate that the benefits extend beyond Qwen3 (Appendix G.11).

W5, Q3. Adaptation cost and serving break-even

The cost of full-model adaptation can be recovered within days of serving. We added measured GPU-hour accounting and a break-even analysis based on training and serving costs (Appendix B.3, Tables 9–10). For Qwen3-30B-A3B, initialization statistics require approximately 0.1 H100 GPU-hours, a 1.04B-token development run requires 67 GPU-hours, and full-model adaptation on 4.32B tokens requires 252 GPU-hours.

At EP32 on four H100 nodes, using 4096 input tokens, 256 output tokens, and 128 requests per GPU, the amortization costs are:

ConfigurationThroughput speedupGPU-hours saved per serving hourBreak-even
(4,512,1280)1.11×3.377 hours
(4,512,768)1.23×5.943 hours
(8,256,512)1.43×9.626 hours

For our main configuration (4,512,768), the 252 GPU-hour adaptation is recovered after approximately 43 hours of serving at the measured load. More aggressive compression shortens this to 26 hours.

Configuration search is a one-time research cost that can be reused across model deployments. Our study included 78 development runs. Conservatively charging each at the 67 GPU-hour cost of a joint configuration gives approximately 5,500 GPU-hours for the entire search and one full adaptation. At the measured savings of (4,512,768), this corresponds to approximately 39 days of serving on a single four-node deployment, with faster amortization across multiple deployments.

It is also worth noting that the breadth and width of the configuration search in our study is for research purposes and is not necessary for actual adaptation of new models. The resulting configuration principles transfer across architectures. As mentioned in Section 5, frequent references with narrow dispatch are preferred over sparse references with wider dispatch, combine should have more width than dispatch, and dispatch and combine drops are approximately additive. We retrofitted Gemma 4, Kanana-2, Nemotron-3.5-Lightning, and Moonlight using stride 4, dispatch width approximately d/4, and combine width approximately d/2, without repeating the configuration search (Table 3). Teacher targets are also generated once and reused across configurations, rather than recomputed for each adaptation.

W6, Q4. Layer-adaptive compression and runtime flexibility

We investigated both reference frequency and layer-specific compression budgets. At matched RDMA budgets, more frequent references with narrower dispatch codes consistently outperform less frequent references with wider codes (Appendix D.1, Table 18). Comparing stride 4 with dispatch width 512 against stride 8 with width 768 improves task performance by 1.3–2.0 pp across three combine widths.

Layer-adaptive compression is an extension to our design. We investigated allocating compression widths according to layer-specific reconstruction errors using the cached initialization activations (Appendix D.1). The optimized allocation retains 83.7% of signal energy, compared with 81.8% for uniform widths. This suggests that layer-specific budgets might further improve compression efficiency. Our analysis is offline; evaluating the resulting allocations after end-to-end adaptation would be a great extension for the camera-ready version.

Dynamic reference schedules at runtime pose significantly heavier overhead. Changing which layers serve as references during inference would require different adapted model parameters and potentially multiple weight configurations resident on the GPUs. This makes runtime switching substantially more expensive than changing the communication precision. We therefore use a fixed reference schedule and instead support runtime adaptation through BF16/FP8 communication. A single adapted model can switch precision without retraining, reducing communication bytes by a further 48.4% with at most 0.29 pp change in overall task performance. This yields an additional 5–9% throughput improvement on InfiniBand, with 1.87–2.28× throughput gains over TCP in the evaluated settings (Appendix G.6, Table 33).

We hope these results address the reviewer's concerns, and we would be glad to answer any further questions.

Reviewer_TQkxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors aim to reduce communication costs in MoE models by compressing token activations before dispatch and combine operations. Their approach is motivated by the observation that inputs to adjacent layers are similar, making the differences between their activations more compressible. Based on this observation, the authors propose to linearly project activation residuals into a low-rank representation and transmit only the corresponding low-rank coefficients. To avoid training the MoE model from scratch, they further introduce an initialization strategy that derives the compression and decompression matrices from pretrained weights.

Critical strengths
  • The paper is generally well written, clearly formulated, and easy to understand.
  • The proposed method is novel, particularly in its approach to retrofitting pretrained MoE models to reduce communication costs without requiring training from scratch.
  • The claims in the experiments seem accurate and solid. The evaluation covers a broad range of benchmarks, and the system-level metrics are carefully measured and reported.
Critical weaknesses
  • Although the similarity and predictability experiments motivate the existence of redundancy or shared structure in token features across adjacent layers, their connection to the proposed compression method remains unclear. Specifically, how do cosine similarity and affine predictability support the effectiveness of low-rank compression? Conversely, the paper lacks direct empirical evidence that the residual xl−a⊙zx_l - a \odot z exhibits a low-rank structure. It is recommended to make this connection explicit by either theoretical illustration or more empirical evidences.

  • Activation compression for reducing communication costs is not new for MoEs, as the authors also noted. To better highlight the contribution, I encourage the authors to clarify how their compression approach differs from existing methods and explain why the proposed design is particularly suitable for retrofitting pretrained MoE models.

  • Besides summarizing performance across different domains (e.g., math and coding), the authors may provide a per-benchmark performance breakdown. This would offer a more granular view of the method's performance and how such drop may relate to the choice of calibration datasets.

Questions for authors
  1. Modern MoE systems often rely on specialized CUDA kernels for efficient execution. Does the proposed method require modifications to existing system-level implementations?

  2. Do the reported RDMA communication results account for the cost of broadcasting zz across shared layers? What memory and runtime overheads are associated with caching zz?

  3. How should the tokens be allocated between the initialization stage and the end-to-end training stage?

4 comments

We thank the reviewer for the constructive feedback and for recognizing the novelty of retrofitting pretrained MoE models, the clarity of the presentation, and the strength of the experimental evaluation. We address the comments below and provide additional analyses in the revised manuscript.

W1. Connection between cross-layer predictability and low-rank compression

Cross-layer predictability and residual compressibility jointly explain the effectiveness of EchoMoE. Predictability determines how much information remains after subtracting the reference, while the residual spectrum determines how much of that information can be captured by a low-rank code. We now quantify both factors directly in Appendix A.1 (Table 4, Figure 5).

Cross-layer predictability reduces the energy that must be compressed. For Qwen3-30B-A3B at stride 4, the remaining part the reference misses contains only 15.4%, 25.9%, and 33.3% of the original input energy at the three successive layers following a reference.

The remaining retains a compressible spectrum. Its top 512 of 2048 principal directions capture 60% of its energy, increasing to 81% at width 1024. Although the raw activation has a more concentrated spectrum, much of its energy is already represented by the reference. Consequently, compressing the remaining preserves substantially more of the original input energy at the same code width.

Code widthPlain compression: input energy lostResidual compression: input energy lost
25644.5%13.4%
51233.0%9.6%
102417.7%4.6%

Residual compression thus discards 3.3–3.8× less input energy than plain compression. This advantage translates to task performance: at dispatch width 512, reference-based compression at stride 4 incurs only a 2.6 pp drop, compared with 21.1 pp for plain compression (Figure 3a).

W2. Distinction from prior work and suitability for retrofitting

EchoMoE differs from existing methods by exploiting redundancy across layers for the same token and, importantly, by making this compression applicable to pretrained MoE models.

Architectural approaches such as LatentMoE, BigMac, Multi-Head LatentMoE, ScMoE, and Federation of Experts reduce communication through designs that require training from scratch. Reduced-precision communication, such as FP8 in DeepEP, instead reduces bytes per element and is complementary to EchoMoE. Combining EchoMoE with FP8 improves throughput by 1.33–1.51× over uncompressed BF16, compared with 1.18× from FP8 alone (Table 30).

Among methods applicable to existing models, LSH-MoE exploits similarity between tokens within a layer, whereas EchoMoE exploits predictability across layers for the same token. Our analysis shows that cross-layer references provide substantially stronger predictability (Table 16). With the same adaptation recipe, EchoMoE achieves 63.8% overall performance versus 57.1% for LSH-MoE (Appendix C.4).

EchoMoE is designed to preserve the existing model. The router operates on the uncompressed activation, preserving the original routing computation. The linear reconstruction and compression operations are folded into expert weights, allowing existing fused expert kernels and DeepEP to be reused (Appendix G.2). To initialize the compressed model, we minimize reconstruction error weighted by the experts' input projections, reducing initial KL divergence by 12–18% relative to unweighted reconstruction (Appendix B.2).

This initialization enables efficient end-to-end adaptation using only 4.32B tokens, approximately 8,500× fewer than Qwen3 pretraining. We apply the same procedure to five pretrained MoE models from different families without architecture-specific changes (Table 3).

W3. Per-benchmark performance and calibration data

We thank the reviewer for this suggestion. Appendix H (Table 40) now reports all 27 evaluations across five adapted models. Performance degradation is distributed across benchmarks, with no single task dominating the overall loss. Calibration data has a modest effect. Across the 9 benchmarks evaluated in both chat and few-shot formats, chat evaluations generally show smaller losses (by approximately 1.5 pp on average for four of the five models). However, the overall degradation patterns remain consistent across formats, suggesting that calibration–evaluation mismatch is not the primary cause.

Q1. Does EchoMoE require modifications to existing system-level kernels?

EchoMoE requires no modifications to existing expert or communication kernels (Appendix G.2).

The linear projections are folded into expert weights, allowing existing fused MoE GEMMs to operate directly on the modified input and output dimensions. DeepEP handles the compressed representations without changes to its dispatch or combine kernels, while reference transfers use standard NCCL collectives.

The implementation adds projection, packing, and unpacking operations on the owner rank, costing 6.4–6.9 ms per prefill step. EchoMoE integrates through vLLM's plugin interface without forking vLLM. We also provide optional communication optimizations, but neither is required for the method.

Q2. Are reference transfers included in RDMA measurements, and what are the memory and runtime overheads?

All reported latency and throughput measurements include these costs. All reported RDMA reductions include reference transfers, as accounted for in Equation (8) and Appendix F.1.

Memory overhead. Each rank reuses one BF16 reference buffer, requiring 4096 bytes per cached token. At an 8192-token limit, this corresponds to 1.00 GiB per rank at EP32 and 2.00 GiB at EP64, plus staging buffers. The smaller expert weights offset much of this cost, leaving the available KV-cache allocation within −0.8% to +2.3% of the uncompressed model (Appendix F.3, Table 32).

Runtime overhead. Reference transfers take 14.9–35.8 ms per prefill step, while narrower dispatch and combine save 70.8–87.8 ms (Table 29). For the (4,512,768) configuration under the 4096-input/256-output workload with 128 requests per GPU, the reference collective costs 31.6 ms against 81.7 ms saved, yielding a net communication saving of 50.2 ms per prefill step.

Q3. How should tokens be allocated between initialization and end-to-end adaptation?

Initialization requires very little data, so nearly the entire training budget can be allocated to end-to-end adaptation.

Our closed-form initialization uses only 512K token positions and approximately 0.1 H100 GPU-hours (Table 9). End-to-end adaptation is valuable. Compared with the 1.04B-token frozen-backbone development setting, full-model adaptation using 4.32B tokens improves task performance by 1.5 pp at (4,512,1280), 4.7 pp at (4,512,768), and 10.4 pp at (8,256,512) (Appendix B.3). These comparisons reflect both additional training tokens and updating the backbone parameters. Held-out KL divergence continues to decrease by 0.6–1.0% over the final 13% of training updates across Qwen3-30B-A3B, Kanana-2, and Gemma 4 (Figure 7), suggesting that longer adaptation could further improve performance.

We hope these results address the reviewer's concerns, and we would be glad to answer any further questions.

Dear Reviewer TQkx,

As the discussion period is nearing its end, we wanted to kindly check whether our response has addressed your concerns. We would be happy to clarify any pending concerns!

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

Reviewer_suugOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EchoMoE, a method for reducing expert-parallel communication in pretrained MoE models. It uses cached activations from earlier layers to compress changes in subsequent expert inputs, compresses expert outputs, and folds projections into expert weights. The model is then adapted through distillation. Evaluation on Qwen3-30B-A3B across 18 benchmarks and a 32-GPU deployment shows a 46% reduction in inter-node communication and a 17.2% throughput improvement, with a 2-percentage-point drop in average task performance. Ablations examine compression choices and reference reuse, while additional experiments characterize workload-dependent speedups.

Critical strengths
  1. Effective use of cross-layer redundancy. Using earlier activations as references improves the quality–communication trade-off over plain compression. At approximately 2× communication reduction, EchoMoE preserves task performance better than the adapted plain-compression baseline by 2.2 percentage points, providing evidence that reference reuse contributes beyond compression alone.
  2. End-to-end implementation and validation. The paper integrates the method into vLLM and demonstrates a 17.2% throughput improvement on 32 GPUs with a 2-percentage-point average quality loss. Although workload-dependent, this result shows that the communication savings can translate into measurable serving benefits after accounting for implementation overhead.
Critical weaknesses
  1. Uncertain practical relevance of the targeted bottleneck. The paper does not sufficiently establish when MoE communication dominates realistic inference workloads. Its own profiling shows that attention increasingly limits gains at long contexts, while reference-transfer overhead nearly offsets communication savings during decode. A more comprehensive latency breakdown across representative workloads would help clarify how MoE communication savings translate into end-to-end performance improvements.

  2. Limited decode benefits and insufficient coverage of representative workloads. The headline 17.2% throughput improvement primarily reflects prefill savings under high concurrency. Decode gains are much smaller, and the appendix’s 128K-input, 256-output setting exhibits regressions, raising concerns about practical value for latency-sensitive generation. The authors should evaluate both end-to-end performance and task quality on long-context agent workflows and long-output reasoning workloads. Context length and concurrency should also be varied independently to isolate their effects and clarify when the method provides a net benefit.

  3. An uncertain quality–efficiency trade-off with limited generalization evidence. The headline acceleration costs 2 percentage points in average task performance, while aggressive compression loses substantially more. Quality preservation is demonstrated primarily on Qwen3-30B-A3B; larger-model experiments use random weights and therefore cannot establish that the trade-off generalizes to other pretrained MoEs.

  4. Incremental novelty and incomplete comparative validation. Cross-layer reference reuse improves over plain compression, but the methodological advance over existing activation-compression approaches appears modest. Stronger comparisons at matched task quality and deployment resources are needed to establish that this advance justifies model adaptation and additional serving complexity.

  5. Insufficient comparison with relevant communication-reduction methods. While the plain-compression baseline demonstrates the benefit of cross-layer references, comparisons with more closely related approaches remain limited. LSH-MoE [1] also exploits activation similarity and residual compensation to reduce communication. A controlled comparison, or an explanation of why it cannot be adapted fairly to the pretrained-model setting, would strengthen the novelty assessment. Evaluating the additional benefit over a communication–computation overlap baseline such as COMET [2] would also help establish the method’s value in an optimized serving system.

[1] LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing. NeurIPS 2024.

[2] COMET: Fine-grained Computation-communication Overlapping for Mixture-of-Experts. MLSys 2025.

Questions for authors
  1. When does MoE communication become the bottleneck? Could the authors provide latency breakdowns across context lengths and concurrency levels, including attention, expert computation, communication, and reference-management overhead, to explain when communication savings translate into meaningful end-to-end gains?
  2. What explains the long-context decode regressions? Can the authors characterize the conditions under which EchoMoE improves or worsens decode latency, varying context length and concurrency independently?
  3. Do the benefits extend to agent and reasoning workloads? Could the authors evaluate both task quality and end-to-end performance on long-context, short-output agent workloads and long-output reasoning workloads?
  4. Does the quality–efficiency trade-off generalize to larger pretrained models? Can the authors provide results using real pretrained weights for at least one additional architecture, rather than random-weight throughput measurements?
  5. How does EchoMoE compare with stronger communication-reduction baselines? Could the authors compare against an appropriately adapted LSH-MoE baseline, or explain why such a comparison is unsuitable, and measure the incremental benefit over a communication–computation overlap approach such as COMET?
5 comments

We thank the reviewer for the thoughtful and constructive feedback. We appreciate that the reviewer recognizes that cross-layer references improve the quality–communication trade-off beyond compression alone, and that our vLLM integration turns the byte savings into measured serving gains after implementation overhead. Below we address each weakness and question; all new results are in the revised paper.

W1, Q1: When does MoE communication become the bottleneck?

MoE communication is bandwidth-bound during prefill and at sufficiently high decode concurrency. We measured DeepEP dispatch and combine independently at EP32 across four nodes, varying tokens per rank and communication width (Appendix F.4, Figure 9). The table summarizes the result with the speedup per DeepEP call for a code with 4× fewer bytes than the full width (d/4), as dispatch / combine:

Tokens per rankd = 2048d = 7168
1281.05× / 1.07×1.52× / 1.46×
2561.10× / 1.05×2.27× / 1.74×
5121.21× / 1.07×3.01× / 2.02×
10241.79× / 1.31×3.06× / 2.49×
40962.51× / 2.11×3.62× / 3.31×

At d = 2048, calls of up to 128 tokens per rank are dominated by a fixed startup cost of about 65–74 µs, so the narrower code helps by at most 1.05× for dispatch and 1.10× for combine. The gain grows once calls become bandwidth-bound and approaches the 4× byte reduction. At d = 7168, the hidden size of DeepSeek-V3 and Kimi-K2.6, this happens about 4× earlier: from 256 tokens per rank instead of 1024.

The end-to-end profiles confirm this distinction (Figure 4 and Figure 12, Appendix G.5):

  • Prefill: Communication accounts for 48–57% of an uncompressed step across the profiled workloads. EchoMoE reduces this communication time by 31–63%, including reference transfers. At 4096/256, (4,512,768) reduces communication from 115 to 64 ms and expert computation from 37 to 31 ms per prefill step.
  • Decode: Communication accounts for 19–41% of an uncompressed step, increasing with requests per GPU. Attention occupies 26–56% of the step as context grows. Decode gains therefore depend primarily on concurrency, as detailed in W2.
  • End-to-end overhead: The profiles include reference transfers, compression kernels, synchronization delays, and CPU overhead. Compression kernels take 6.4–6.9 ms per prefill step and 0.8–1.4 ms per decode step.

Table 29 isolates the cost of references:

ConfigurationPrefill savedReferenceNet savingDecode savedReferenceNet saving
128/1024, 512 requests/GPU
(4,512,1280)71.435.835.68.68.00.6
(4,512,768)82.131.550.611.96.95.0
(8,256,512)87.814.972.913.53.410.1
4096/256, 128 requests/GPU
(4,512,1280)71.035.835.25.23.71.6
(4,512,768)81.731.650.26.03.52.4
(8,256,512)87.215.272.06.01.94.1

For (4,512,768) at 4096/256 and 128 requests per GPU, compressing dispatch and combine saves 81.7 ms per prefill step; the reference collective costs 31.6 ms, leaving a net 50.2 ms communication saving. During decode, the corresponding figures are 6.0, 3.5, and 2.4 ms.

The bandwidth-bound regime also depends on model width and network speed. Larger hidden widths of larger models increase communication payloads at a given token count, making communication compression effective at lower concurrency. At 256 tokens per rank, quarter-width DeepEP dispatch achieves 2.26× speedup at width 7168, compared with 1.10× at width 2048; the corresponding combine gains are 1.74× and 1.05× (Appendix G.10). Slower networks further amplify the benefits: for Qwen3-30B-A3B, reducing NIC bandwidth to approximately 160 Gb/s increases throughput speedup to 1.73×, while TCP serving reaches 2.38× (Appendix G.7). These results demonstrate that communication compression becomes effective at smaller token batches as model width increases or network bandwidth decreases.

W2, Q2: Decode gains and the long-context regression

Decode performance is governed primarily by concurrency, rather than context length alone. Appendix G.4 (Table 26) varies the two independently. For (4,512,768), decode TPOT speedup over the uncompressed model is:

Input/output tokens8 req/GPU163264128256512
128/10240.94×0.98×0.97×1.03×1.11×1.11×1.10×
4096/2560.95×0.98×0.99×1.04×1.08×——
16384/5120.96×1.00×1.01×————

At comparable concurrency, speedups remain similar across input lengths; they increase with requests per GPU. At 512 requests per GPU, (4,512,768) achieves 1.10–1.12× decode speedup across the measured workloads, while (8,256,512) reaches up to 1.27×.

The 128K-context decode regression follows from the KV-cache concurrency limit: only four 128K-token requests fit per GPU, where fixed communication costs dominate. Even in this setting, total throughput improves by 1–10% and TTFT by 3–13% across the evaluated configurations (Table 27).

EchoMoE also benefits latency-sensitive inference. At 4096/256 with eight requests per GPU, TTFT improves by 1.16–1.61× (Table 27). At higher concurrency, throughput and decode latency improve together; for (4,512,768), this occurs from 64 requests per GPU in the evaluated sweep (Table 28, Figure 11).

Q3: Do the benefits extend to agent and reasoning workloads?

We added targeted serving experiments for long-context agent turns and long-output reasoning (Appendix G.9, Table 36).

Agent turns (32K/256 and 64K/256). At 16 and 8 requests per GPU, respectively, EchoMoE (4,512,768) improves throughput by 1.10–1.16× with BF16 communication and 1.05–1.08× with FP8. TTFT improves by 1.13–1.20× with BF16; TPOT remains within 0.97–1.01× of the uncompressed model (as discussed in W1 Q1 for low concurrency). With an FP8 KV cache allowing twice the concurrency, BF16-communication throughput gains remain 1.17–1.18× at 32K/256 and 1.13–1.14× at 64K/256. We also evaluate agent-like online serving on real 16K- and 32K-token conversation histories with Poisson arrivals and model-specific routing (Appendix G.9, Figure 21).

Long-output reasoning (1K/8K and 1K/16K). Throughput is 0.99–1.03× with BF16 communication and 1.01× with FP8, while BF16 TTFT improves by 1.31–1.32×. Task quality is evaluated on long chain-of-thought benchmarks in the 18-benchmark suite. For (4,512,768), the changes include −3.6 pp on AIME 2024, −0.6 pp on AIME 2025, +0.2 pp on HMMT February 2025, and −3.0 pp on MATH-500 chat (Appendix H, Table 40). These experiments distinguish the substantial prefill gains in agent-style workloads from the smaller gains in decode-heavy reasoning.

W3 Q4: Generalization to pretrained MoE architectures

We retrofitted four additional pretrained models with real weights, using the same adaptation procedure and a single approximately 2× compression configuration per model, without a new configuration search (Table 3; Appendix B.1).

ModelConfigurationRDMA reductionUncompressed scoreEchoMoE scoreThroughput speedup
Qwen3-30B-A3B(4,512,1024)2.08×66.9%63.8% (−3.1 pp)1.17×
Kanana-2-30B-A3B(4,512,1024)2.01×58.5%56.1% (−2.4 pp)1.15×
Gemma 4 26B-A4B(4,768,1408)2.01×61.2%56.8% (−4.4 pp)1.14×
Nemotron-3.5-Lightning-30B-A3B(4,768,1408)1.91×62.8%58.3% (−4.6 pp)—
Moonlight-16B-A3B(4,512,1024)1.99×53.0%47.1% (−5.9 pp)—

Throughput uses BF16 communication at 4096/256 and 128 requests per GPU. On additional serving workloads, gains range from 1.10–1.16× for Kanana-2 and 1.07–1.14× for Gemma 4 with BF16 or FP8 communication (Appendix G.11). In online serving with each model's own routing at 128 requests/s, median TPOT falls by 10–17% for Kanana-2 and 20–22% for Gemma 4. These real-weight experiments demonstrate communication and serving gains beyond Qwen3-30B-A3B.

W4 Novelty and comparisons at matched quality and resources

The central distinction is the reuse of a cached cross-layer reference, rather than compressing each activation independently. We isolate this contribution by comparing EchoMoE with adapted plain compression at matched RDMA budgets (Appendix C.2, Table 13):

RDMA budgetMethodRDMA reductionOverall task scoreRelative matrix MACs
~2×EchoMoE (4,512,1024)2.08×63.8% (−3.1 pp)1.03×
~2×Plain (1024,1024)2.00×61.6% (−5.3 pp)0.61×
~4×EchoMoE (8,256,512)4.03×55.7% (−11.2 pp)0.86×
~4×Plain (512,512)4.00×51.4% (−15.5 pp)0.31×

At approximately 2× compression, the reference improves average task score by 2.2 pp; at approximately 4×, the improvement is 4.3 pp. Conversely, at comparable task quality, EchoMoE (4,512,768) transmits 26% fewer RDMA bytes than the least-compressed adapted plain baseline (Figure 1; Appendix C.2). This comparison directly measures the contribution of cross-layer reference reuse.

The serving results additionally account for the implementation's computation and communication overhead. With FP8 communication, EchoMoE (4,512,768) achieves 1.33× throughput over uncompressed BF16, compared with 1.18× from FP8 communication alone; (8,256,512) reaches 1.51× (Appendix G.6, Table 30). Retrofitting costs approximately 252 H100 GPU-hours, equivalent to 1.1–3.2 days of savings from a fully loaded four-node EP32 deployment under the paper's cost model (Appendix B.3, Table 10).

W5, Q5: LSH-MoE and communication–computation overlap

LSH-MoE. We adapted LSH-MoE to Qwen3-30B-A3B using the same 4.32B-token recipe and evaluated both methods on the full task suite (Appendix C.4, Table 15). We retained its residual-compensation term but learned its scale per layer and expert, initialized at zero. This avoids the large initial perturbation of the published compensation, whose token–centroid residuals are 6–8× larger than the expert outputs they correct.

MethodRDMA reductionOverall task scoreHeld-out KL: Dolmino / Dolci-Instruct
Uncompressed1.00×66.9%—
EchoMoE (4,512,1024)2.08×63.8% (−3.1 pp)0.126 / 0.067
LSH-MoE with learned compensation1.59×57.1% (−9.8 pp)0.389 / 0.264

EchoMoE scores 6.7 pp higher while communicating less. LSH-MoE sends 31% more RDMA bytes during training and scores lower on 26 of 27 tasks. The predictability analysis explains the difference (Table 16). On Qwen3-30B-A3B, Qwen3-235B-A22B, and DeepSeek-V3, a token's own state four MoE layers earlier predicts 55.7%, 70.0%, and 62.2% of the later input variance, respectively. The best other token routed to the same expert predicts 51.3%, 42.7%, and 36.2%, while actual LSH-MoE representatives predict 8.0%, 5.9%, and 5.8%. Reference reuse thus exploits a stronger source of predictability. It also remains effective in decode without relying on the number of tokens available for cross-token merging.

COMET and overlap. COMET reduces exposed communication time through overlap, whereas EchoMoE reduces the volume communicated; the two mechanisms are complementary. In our 4096/256 prefill profile, uncompressed communication and MoE computation take 115 ms and 37 ms. Even ideal overlap between these components would leave at least 78 ms of communication exposed. EchoMoE reduces them to 64 ms and 31 ms, respectively (Figure 4a), leaving less communication to hide.

Our serving experiments also include concurrent execution of shared experts or dense MLPs on a separate stream for Kanana-2 and Gemma 4, where EchoMoE still improves throughput by 1.07–1.16× (Appendix G.11). These results support the benefit of communication reduction even when computation and communication already overlap.

We hope these results address the reviewer's concerns, and we would be glad to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updates to our Submission

We thank the reviewers for their feedback. We have revised the paper and added experiments addressing communication overhead, serving performance, model generalization, and comparisons with related methods.

New evidenceReviewersLocation
Task quality on four additional pretrained MoEs; throughput on Kanana-2 and Gemma 4suug, A1zpTable 3; Appendix G.11
End-to-end FP8 baselines, including throughput, TTFT, TPOT, tail latency, and memoryA1zp, iY4gAppendices G.6, G.8
Online serving with real prompts, Poisson arrivals, model-specific routing, and P50/P99 latencyiY4g, suugAppendix G.8
Step-level latency breakdowns and DeepEP communication scalingsuugFigure 4; Appendices G.5, F.4
Independent context-length and concurrency sweeps; agent and long-reasoning workloadssuugAppendices G.4, G.9
Adapted LSH-MoE baseline and residual-spectrum analysissuug, TQkxAppendices C.4, A.1
Routing agreement, per-task scores, adaptation costs, EP topologies, and memoryiY4g, TQkxAppendices C.3, H, B.3, F.1, F.3

Updated throughput measurements. We now enforce balanced expert assignment and use identical per-call kernel selection for both the uncompressed model and EchoMoE (Appendix G.1). This eliminates the differences in load imbalance and kernel selection, providing a conservative estimate of the speedup attributable to communication compression. Under this protocol, we highlight (4,512,768), which reduces inter-node RDMA traffic by 58% and improves throughput by 22.6%, with a 3.8 pp reduction in average task score. Throughput is measured with BF16 communication, EP32 over four nodes, 4096 input and 256 output tokens, and 128 requests per GPU (Table 3; Section 5.4). We separately evaluate online serving with real prompts and each model's own routing (Appendix G.8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.