acceptodds
Under review as a conference paper at ICLR 2027

FactorEP: Accelerating Expert-Parallel MoE Inference through End-to-End Factor-Space Execution

16 views

Abstract

Mixture-of-Experts (MoE) models rely on expert parallelism (EP) to scale across GPUs, but communication can account for up to 47% of end-to-end execution time. Existing EP systems primarily optimize how activations are communicated, while redundancy in pretrained experts offers an opportunity to reduce what must be communicated and computed. We present FactorEP, a system that exploits this redundancy by co-designing compact representations and EP execution. Group-wise Tucker factorization with KFAC-weighted calibration produces group-shared input factors, expert-specific compact cores, and a layer-shared output factor. This asymmetric sharing enables input projection before dispatch and output reconstruction after combine, reducing both communication and expert computation. To turn these reductions into practical speedups, a three-stage factor-space pipeline amortizes shared projection costs and overlaps communication with compact expert execution. Route-compiled execution uses a packed token layout shared by communication and compact expert kernels to reduce capacity padding and per-expert launch overhead. On Qwen3-30B-A3B, FactorEP achieves higher average downstream accuracy than all evaluated compression baselines across all tested parameter budgets. Its deployment configuration reduces dispatch and combine feature widths by 20% while maintaining average downstream accuracy within 1.93 percentage points of the original model. The same configuration achieves end-to-end speedups over COMET of on eight H100 GPUs with NVSwitch and on eight RTX 6000 Ada GPUs over PCIe. These results, together with quality-latency sweeps on Qwen3-30B-A3B and OLMoE-1B-7B, demonstrate FactorEP's applicability across MoE models and hardware platforms.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Expert-Parallel, MoE Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_N2WaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FactorEP accelerates expert-parallel MoE inference by keeping routed activations in low-rank factor spaces. Group-shared input factors enable projection before dispatch, while a layer-shared output factor allows reconstruction after combine. A chunk-based pipeline and GPU-resident route compilation further reduce execution overhead. On Qwen3-30B-A3B, the deployment configuration reduces communication widths by 20%, lowers average downstream accuracy from 61.86% to 59.93%, and achieves end-to-end speedups of 1.23× over DeepEP+TE on H100 and 1.26× over COMET/Flux on RTX 6000 Ada. The paper also reports component ablations and quality–latency sweeps on OLMoE.

Critical strengths
  1. Group-shared input factors support projection reuse before dispatch, and a shared output factor enables reconstruction after combination. This directly connects compression to communication savings.

  2. The deployment configuration reports both downstream quality and end-to-end latency, achieving meaningful speedups with a 1.93-point accuracy loss.

  3. Paired ablations on NVSwitch and PCIe show different bottlenecks: compression matters more on PCIe, while routing and execution efficiency contribute more on H100.

Critical weaknesses
  1. The novelty over prior work is not clearly established. BigMac and LatentMoE already use low-dimensional communication, while TD-MoE and MoBE exploit structure shared across experts. FactorEP’s post-training setting is a useful distinction, but the paper should better explain what its asymmetric factorization adds beyond combining these ideas.

  2. The shared output basis may limit expert specialization. All routed outputs lie in one low-rank subspace. This saves reconstruction work, but the paper does not characterize the loss of expert-specific directions or justify this choice over group-specific output bases.

  3. Calibration does not directly target MoE output error. The objective fits individual projections using activation-based proxies, leaving error interactions through SiLU, gating, and weighted aggregation unmodeled. Its relevance to the full nonlinear computation needs stronger justification.

  4. Expert grouping overlooks execution costs. Groups are formed by weight similarity, although projection reuse depends on co-routing and execution efficiency depends on GPU placement. This leaves a key design choice disconnected from the system’s optimization goals.

  5. The quality–latency advantage remains unclear. Compression comparisons match parameter budgets, while latency comparisons use uncompressed baselines. Neither shows whether FactorEP is faster than competing compressed models at comparable quality.

Questions for authors
  1. What technical obstacle distinguishes FactorEP from adapting existing shared-factor methods to low-dimensional EP communication?

  2. How much quality is lost by sharing one output basis across all experts?

  3. Why should projection-wise calibration preserve the gated MoE output?

  4. How does weight-based grouping affect co-routing reuse and GPU placement?

  5. Does the speed advantage hold against compressed baselines at matched quality?

3 comments
AuthorsOpenReview ↗

Response to Reviewer N2Wa — (1/2)

We thank the reviewer for the detailed questions.

W1/Q1. What distinguishes FactorEP from prior work?

FactorEP converts pretrained MoE redundancy into end-to-end EP acceleration through algorithm–runtime co-design. In our Ada evaluation, D²-MoE and HC-SMoE increase latency, while FactorEP achieves a 1.28× speedup over the uncompressed model (W5/Q5).

BigMac and LatentMoE introduce new architectures requiring costly pretraining. TD-MoE and MoBE compress pretrained models, but their execution differs at the communication boundaries: TD-MoE’s reference implementation uses separate gate/up input factors and reconstructs outputs before aggregation, while MoBE retains the original down projections.

FactorEP jointly addresses communication, accuracy, and execution efficiency:

  1. Reducing both dispatch and combine traffic (C1). A group-shared input basis provides one compact representation for both gate and up before dispatch. A layer-shared output basis enables combine before reconstruction, keeping both communication phases in factor space while retaining expert-specific cores.

  2. Preserving quality without retraining (C1). KFAC-weighted calibration computes the factors in closed form to limit the accuracy loss introduced by sharing, without gradient-based training.

  3. Realizing the compression gains on GPUs (C2, C3). The three-stage pipeline overlaps projection and communication with expert computation. Route-compiled execution packs routed tokens into grouped kernels without capacity padding or per-expert launches, maintaining efficiency for the smaller factor-space GEMMs (Table 3).

W2/Q2. Cost of a shared output basis

A shared output basis restricts expert outputs to a common low-dimensional subspace, while expert-specific cores retain distinct mappings within it. We compare this design with group-specific output bases on Qwen3, giving each group basis the full rank.

Output basesKFAC-weighted output energy retainedWiki2 PPL7-task avg.Groups per tokenLatency change (Ada)
1, shared (FactorEP)97.3%9.6559.71Reference
2 groups97.8%9.4361.11.95+7–10%
4 groups98.3%9.3260.63.2+12–24%

Group-specific bases improve the 7-task average by 0.9–1.4 points, with a 7–24% latency increase. The shared basis retains 97.3% of KFAC-weighted output energy and enables partial expert outputs to be combined before a single reconstruction. With group-specific bases, partial sums in different bases cannot be added directly (Eq. 6). Each token needs a separate partial sum and reconstruction for each group its experts span, averaging 1.95–3.2 groups here. This increases reconstruction work and requires either more combine traffic or group-constrained placement.

The ablation quantifies the quality–latency trade-off behind the shared output basis. We will add it and the cost analysis to the paper.

AuthorsOpenReview ↗

Response to Reviewer N2Wa — (2/2)

W3/Q3. Does projection-wise calibration preserve the gated output?

The down-projection calibration statistics, Σ_e, are collected after SiLU and gating (Eq. 10), so calibration uses post-gating activations. We also evaluated directly fitting the MoE block output. In this ablation, held-out block error decreased by 5–7%, but Wiki2 PPL increased from 10.30 to 10.40. Fisher-weighted and route-conditioned variants changed PPL by +0.21 and −0.002, respectively.

These results show that reducing block reconstruction error does not necessarily improve model quality and support retaining the projection-wise objective. We will add this ablation to the appendix.

W4/Q4. Grouping, co-routing reuse, and placement

All experts in a factor group are placed on the same GPU. When a token selects multiple experts from one group, it is projected once with that group's shared input factor, and the experts reuse the compact row.

Our groups are formed by weight similarity. On C4/MATH routing, a token's eight selected experts span 6.83 groups on average, reducing projection count by about 15%. This reuse is modest. The three-stage pipeline overlaps projections with expert computation; in our H100 profile, projection and reconstruction together account for 4.3 ms of the 122.1 ms MoE critical path. Jointly considering weight similarity and co-routing may improve reuse further. We will add this analysis to App. A.1.

W5/Q5. Compressed baselines under EP

Figure 5 reports FactorEP's quality–latency trade-off across communication budgets on Qwen3 and OLMoE. We additionally evaluated HC-SMoE and D²-MoE, the two strongest compressed baselines from Table 1, in the same EP stack on 8× Ada using the Table 2 protocol. The table below reports BF16 results; FP8 results appear in our response to Reviewer Sh8N, W1.

Method (~20% budget)Latency (ms) / vs. uncompressedWiki2 PPLAvg. accuracy (%)
Original (COMET/Flux), uncompressed747.7 / 1.00×7.87361.86
D²-MoE, low-rank form921.8 / 0.81×10.61654.14
HC-SMoE, no route deduplication764.5 / 0.98×10.38257.69
FactorEP582.5 / 1.28×8.69759.93

In the evaluated configuration without route deduplication, HC-SMoE retains top-8 routing over same-size experts, reducing memory without reducing communication or compute. D²-MoE's low-rank form still communicates d-dimensional tokens and adds delta GEMMs, increasing latency by 23%.

FactorEP is 1.28× faster than the uncompressed model, 1.31× faster than HC-SMoE, and 1.58× faster than D²-MoE. At this budget, it achieves both lower latency and higher accuracy than the evaluated compressed baselines. We will add this comparison to §4.4.

I acknowledge and appreciate the responses. I tend to keep my scores due to the relatively shallow technical novelty.

Reviewer_Sh8NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FactorEP cuts expert-parallel (EP) communication by moving routed activations into a compact factor space before dispatch and keeping them there through combine. Authors achieve this using two systems components: a three-stage pipeline (project/dispatch, expert, combine) that advances destination chunks independently; and GPU-side route compilation into a packed layout consumed by grouped kernels, with no capacity padding. Authors proved the efficacy of their method on Qwen3-30B-A3B with end-to-end speedup of 1.23× over DeepEP on 8×H100 with NVSwitch and 1.26× over COMET on 8×RTX 6000 Ada over PCIe.

Critical strengths
  • Paper presents clear motivation. A layer-shared output factor plus linearity (Eq. 6) lets combine run in factor space
  • The method works without training unlike existing works. The closed-form weighted factorization is simple, and Figure 7 shows activation weighting clearly matters.
  • At 40% and 60% budgets, FactorEP leads D²-MoE and HC-SMoE by large margins in both perplexity and accuracy.
Critical weaknesses
  • Low-precision communication is cited as complementary (lines ~139–140) but never compared or combined. For example, DeepEP supports FP8 dispatch, which halves dispatch payload, usually at little accuracy cost. Authors should address this by adding results with FP8 dispatch
  • Table 2(b) shows the H100 speedup over COMET falls to about 1.0× at 1K tokens per rank. The decode regime, where per-rank token counts are small and EP latency is most critical, is therefore not evaluated and likely sees no gain.
  • End-to-end latency is reported for one model only.
Questions for authors

Please see above

2 comments
AuthorsOpenReview ↗

Response to Reviewer Sh8N — (1/2)

We thank the reviewer for the clear summary.

Existing MoE compression methods exploit redundancy to reduce model size. Both D²-MoE and HC-SMoE increase EP inference latency in our evaluation. FactorEP’s central contribution is to turn this redundancy into end-to-end EP acceleration through algorithm–runtime co-design. Conventional low-rank implementations can retain full-dimensional communication while fragmenting expert GEMMs. FactorEP places shared factors at the communication boundaries (C1), enabling dispatch, expert computation, and combine in a compact factor space. The three-stage pipeline (C2) overlaps projection and communication with expert computation, while route-compiled execution (C3) removes capacity padding and per-expert launch overhead.

W1. FP8 dispatch

FactorEP retains nearly the same speedup with FP8 dispatch

FP8 reduces bits per value, while FactorEP reduces the number of values communicated. We evaluated both BF16 and FP8 dispatch (DeepEP format, with BF16 combine) under the Table 2 protocol. The table reports results on 8× RTX 6000 Ada using the same COMET/Flux-based EP stack.

Method (~20% budget)BF16 (ms) / vs. uncompressedFP8 (ms) / vs. uncompressedWiki2 PPL (BF16 / FP8)
Original (COMET/Flux), uncompressed747.7 / 1.00×710.4 / 1.00×7.873 / 7.876
D²-MoE921.8 / 0.81×895.8 / 0.79×10.616 / 10.627
HC-SMoE764.5 / 0.98×730.1 / 0.97×10.382 / 10.395
FactorEP582.5 / 1.28×563.0 / 1.26×8.697 / 8.713

FactorEP retains a 1.26× speedup with FP8 dispatch, compared with 1.28× in BF16. FP8 further reduces FactorEP's latency from 582.5 to 563.0 ms, while both compressed baselines remain slower than the uncompressed model. These results demonstrate complementary benefits in the evaluated setting. Because Flux's fused all-gather feeds BF16 rows directly into GEMMs, the FP8 path uses a separate transport and dequantizes received rows before GEMM execution.

We also evaluated native DeepEP FP8 dispatch in the Table 2 DeepEP+TE baseline on 8× H100. Latency changes from 242.4 ms in BF16 to 245.6 ms with FP8 dispatch. In this implementation, TE 2.1's grouped GEMM does not directly accept DeepEP's FP8 format, requiring dequantization to BF16. Quantization and dequantization add about 2.6 ms per forward, limiting the benefit of reduced transfer volume in this path.

We will add these results to §4.3 and replace the qualitative “complementary” remark in §2.2.

AuthorsOpenReview ↗

Response to Reviewer Sh8N — (2/2)

W2. ~1.0× at 1K tokens per rank, and decode

After this optimization, FactorEP consistently achieves speedups over COMET with CUDA graphs across all evaluated token settings on both Ada and H100.

Profiling at small token counts identified host-side routing and launch overhead in the original FactorEP implementation. We evaluated FactorEP with per-layer CUDA graphs and compared it with COMET using its own kernels, with and without CUDA graphs.

The following table reports the cumulative latency of 48 MoE layers (ms) for small-batch prefill and decode, with token counts specified per rank.

tokens/rankprefill 16prefill 32decode 1decode 8decode 64decode 128
8× RTX 6000 Ada
COMET29.2232.2725.5327.6537.0448.86
COMET + CUDA graph24.8827.4916.0023.3132.5542.87
FactorEP, original code26.5928.0924.9824.7430.4137.35
FactorEP + CUDA graph14.1017.577.5012.3520.9728.47
Speedup vs. COMET + CUDA graph1.76×1.56×2.13×1.89×1.55×1.51×
8× H100 SXM
COMET34.4633.1133.8133.8135.7632.09
COMET + CUDA graph14.0314.339.2913.8914.6115.20
FactorEP + CUDA graph8.268.486.537.989.3811.23
Speedup vs. COMET + CUDA graph1.70×1.69×1.42×1.74×1.56×1.35×

In the CUDA-graph comparison, FactorEP is 1.51–2.13× faster on Ada and 1.35–1.74× faster on H100 across these settings. These results show gains for both small-batch prefill and decode. We will add them to §4.

W3. Only one model

Figure 5(b) reports FactorEP's quality–latency trade-off on OLMoE-1B-7B alongside Qwen3-30B-A3B. We additionally evaluated end-to-end latency on the shared-expert model Trinity-Mini against COMET/Flux, using the Table 2 protocol on 8× Ada.

  • OLMoE-1B-7B: 1.27× speedup at 20.3% communication reduction (235.5 → 186.2 ms; Wiki2 PPL 7.49 → 9.96).
  • Trinity-Mini: 1.19× speedup at the 20% budget (531.4 → 444.8 ms; 7-task accuracy 65.38 → 63.96). Its shared expert and dense layers remain unfactorized, limiting the overall gain.

We will add both results to the paper.

Reviewer_dLmbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FactorEP, a system for accelerating expert-parallel MoE inference by jointly optimizing model compression and distributed execution. The key idea is to use group-wise Tucker factorization to reduce activation dimensions during dispatch and combine, while allowing expert computation to operate on compact factorized weights. The factorization uses group-shared input bases, expert-specific compact cores, and a layer-shared output basis, with activation-aware calibration to preserve model quality.

To translate the reduced communication volume and computation into actual latency improvements, FactorEP introduces a three-stage pipeline to overlap communication and computation, along with route-compiled execution and ragged grouped kernels to reduce routing, padding, and kernel-launch overhead.

Experiments on Qwen3-30B-A3B show that FactorEP achieves better model quality than the evaluated compression baselines at matched parameter budgets. Its deployment configuration reduces dispatch and combine feature widths by 20%, with a 1.93-percentage-point drop in average downstream accuracy. It achieves end-to-end speedups of 1.23× on H100/NVSwitch and 1.26× on RTX 6000 Ada/PCIe over the strongest baseline on each platform.

Critical strengths

The paper proposes a meaningful approach to reducing both communication and computation costs. Rather than applying low-rank compression independently to each expert, FactorEP designs the factorization around the EP execution path. The group-shared input bases and layer-shared output basis allow activations to remain in compact representations during dispatch and combine, while reducing expert computation.

The authors consider practical execution challenges introduced by factorization, including small GEMMs, routing overhead, and communication-computation overlap. They propose several optimizations to address these issues and provide component ablations and profiling results to support their effectiveness. The evaluation also includes comparisons with strong EP implementations, such as COMET and DeepEP, and several compression baselines.

Critical weaknesses

Although the paper provides component ablations and detailed profiling measurements, it is still somewhat difficult to understand how the reductions in communication volume and expert computation translate into the reported end-to-end speedups. In particular, modern EP implementations already overlap substantial amounts of communication with computation, so reducing total communication volume does not necessarily translate into an equivalent reduction in latency. While the paper reports aggregate communication measurements and analyzes the route-metadata critical path, it remains unclear how much communication is exposed on the overall execution critical path in strong baselines such as DeepEP + TE and COMET/Flux, and how FactorEP changes this. A more direct critical-path breakdown comparing these baselines with FactorEP, including exposed communication, expert computation, routing overhead, and additional projection/reconstruction costs, would make the performance improvements more convincing.

Besides, I have some clarification questions as listed in the section below.

Questions for authors
  • The latency experiments use a sequence length of 2,048 and 4,096 tokens per rank, which appears to be a prefill-like workload. Could the authors clarify whether the reported end-to-end latency corresponds to prefill or includes autoregressive decoding? If the system primarily targets prefill, it would be helpful to state this explicitly. If decoding is also a target, additional latency results across different decoding batch sizes would be useful.

  • While FactorEP preserves quality better than the evaluated compression baselines, the deployment configuration still incurs some degradation in downstream accuracy and perplexity. It would be helpful to include additional evaluations on autoregressive generation tasks, particularly longer reasoning or code generation tasks, to understand whether the approximation errors introduced by compression could have a larger impact over longer generations.

2 comments
AuthorsOpenReview ↗

Response to Reviewer dLmb — (1/2)

We thank the reviewer for the positive assessment.

W1. How do the reductions translate into end-to-end speedup?

In our evaluated configurations, overlap leaves exposed communication, and fused communication–GEMM kernels can reduce GEMM efficiency. FactorEP reduces both transfer volume and computation, complementing overlap and shortening the MoE critical path.

We profiled all eight GPUs on H100 under the Table 2 setting. The breakdown below sums the critical-path contributions across 48 MoE layers; dispatch and combine report only exposed communication.

Metric / MoE stepDeepEP+TECOMET/FluxFactorEPFactorEP component
Communication per token (dispatch + combine), KB8.08.06.4C1
Expert GEMMs, TFLOP14.914.912.1C1
Routing and scheduling, ms49.137.427.8C3
Dispatch, ms27.315.39.0C2
Expert computation, ms62.578.062.1C1 + C3
Combine, ms36.658.318.9C1 + C2
Projection and reconstruction, ms––4.3Cost of C1
Total MoE critical path, ms175.5189.0122.1

Exposed communication decreases by 36.0–45.7 ms, accounting for about 70% of the MoE critical-path reduction. FactorEP's copy-engine dispatch and three-stage pipeline (C2) overlap transfers with expert GEMMs. Route compilation (C3) saves another 9.6–21.3 ms. Expert computation matches DeepEP+TE's tuned GEMMs and is 15.9 ms faster than COMET/Flux's fused kernels, while shared projection and reconstruction add 4.3 ms.

We will add this breakdown to §4.5 and App. D.4.

Q1. Prefill or decode?

After this optimization, FactorEP consistently achieves speedups over COMET with CUDA graphs across all evaluated prefill and decode settings on both Ada and H100.

We evaluated both small-batch prefill and decode. Profiling identified host-side routing and launch overhead in our original implementation. We therefore evaluated FactorEP with per-layer CUDA graphs and compared it with COMET using its own kernels, with and without CUDA graphs.

The table reports the cumulative latency of 48 MoE layers (ms), with token counts specified per rank.

tokens/rankprefill 16prefill 32decode 1decode 8decode 64decode 128
8× RTX 6000 Ada
COMET29.2232.2725.5327.6537.0448.86
COMET + CUDA graph24.8827.4916.0023.3132.5542.87
FactorEP, original code26.5928.0924.9824.7430.4137.35
FactorEP + CUDA graph14.1017.577.5012.3520.9728.47
Speedup vs. COMET + CUDA graph1.76×1.56×2.13×1.89×1.55×1.51×
8× H100 SXM
COMET34.4633.1133.8133.8135.7632.09
COMET + CUDA graph14.0314.339.2913.8914.6115.20
FactorEP + CUDA graph8.268.486.537.989.3811.23
Speedup vs. COMET + CUDA graph1.70×1.69×1.42×1.74×1.56×1.35×

In the CUDA-graph comparison, FactorEP is 1.51–2.13× faster on Ada and 1.35–1.74× faster on H100 across these settings. The improvement applies to both prefill and decode, where small token counts make host overhead significant relative to GPU work. We will add these results to §4.

AuthorsOpenReview ↗

Response to Reviewer dLmb — (2/2)

Q2. Generation, reasoning, and code

We evaluated GSM8K (CoT, 1,319 problems) and HumanEval (164 problems) at the ~20% budget, using greedy decoding and identical prompts across models.

Model (~20% budget)HumanEvalGSM8K7-task avg.
Original91.595.261.9
FactorEP70.793.159.9
FactorEP (25% code calibration)85.494.559.6
HC-SMoE53.093.757.7

Nearly half of FactorEP's additional HumanEval failures are syntax errors, such as unbalanced brackets, which were absent from the original model's evaluated outputs. The original calibration data contains no code.

Replacing 25% of the original natural-language calibration data with code raises HumanEval from 70.7 to 85.4 and GSM8K from 93.1 to 94.5. The 7-task average changes from 59.9 to 59.6. These results suggest that calibration coverage contributes to the code-generation degradation, although a 6.1-point HumanEval gap to the original model remains. This is an interesting problem, and we will work on it in the future.

We will add these results and the calibration-data analysis to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.