- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FactorEP accelerates expert-parallel MoE inference by keeping routed activations in low-rank factor spaces. Group-shared input factors enable projection before dispatch, while a layer-shared output factor allows reconstruction after combine. A chunk-based pipeline and GPU-resident route compilation further reduce execution overhead. On Qwen3-30B-A3B, the deployment configuration reduces communication widths by 20%, lowers average downstream accuracy from 61.86% to 59.93%, and achieves end-to-end speedups of 1.23× over DeepEP+TE on H100 and 1.26× over COMET/Flux on RTX 6000 Ada. The paper also reports component ablations and quality–latency sweeps on OLMoE.
Critical strengths
-
Group-shared input factors support projection reuse before dispatch, and a shared output factor enables reconstruction after combination. This directly connects compression to communication savings.
-
The deployment configuration reports both downstream quality and end-to-end latency, achieving meaningful speedups with a 1.93-point accuracy loss.
-
Paired ablations on NVSwitch and PCIe show different bottlenecks: compression matters more on PCIe, while routing and execution efficiency contribute more on H100.
Critical weaknesses
-
The novelty over prior work is not clearly established. BigMac and LatentMoE already use low-dimensional communication, while TD-MoE and MoBE exploit structure shared across experts. FactorEP’s post-training setting is a useful distinction, but the paper should better explain what its asymmetric factorization adds beyond combining these ideas.
-
The shared output basis may limit expert specialization. All routed outputs lie in one low-rank subspace. This saves reconstruction work, but the paper does not characterize the loss of expert-specific directions or justify this choice over group-specific output bases.
-
Calibration does not directly target MoE output error. The objective fits individual projections using activation-based proxies, leaving error interactions through SiLU, gating, and weighted aggregation unmodeled. Its relevance to the full nonlinear computation needs stronger justification.
-
Expert grouping overlooks execution costs. Groups are formed by weight similarity, although projection reuse depends on co-routing and execution efficiency depends on GPU placement. This leaves a key design choice disconnected from the system’s optimization goals.
-
The quality–latency advantage remains unclear. Compression comparisons match parameter budgets, while latency comparisons use uncompressed baselines. Neither shows whether FactorEP is faster than competing compressed models at comparable quality.
Questions for authors
-
What technical obstacle distinguishes FactorEP from adapting existing shared-factor methods to low-dimensional EP communication?
-
How much quality is lost by sharing one output basis across all experts?
-
Why should projection-wise calibration preserve the gated MoE output?
-
How does weight-based grouping affect co-routing reuse and GPU placement?
-
Does the speed advantage hold against compressed baselines at matched quality?
3 comments
Response to Reviewer N2Wa — (2/2)
W3/Q3. Does projection-wise calibration preserve the gated output?
The down-projection calibration statistics, Σ_e, are collected after SiLU and gating (Eq. 10), so calibration uses post-gating activations. We also evaluated directly fitting the MoE block output. In this ablation, held-out block error decreased by 5–7%, but Wiki2 PPL increased from 10.30 to 10.40. Fisher-weighted and route-conditioned variants changed PPL by +0.21 and −0.002, respectively.
These results show that reducing block reconstruction error does not necessarily improve model quality and support retaining the projection-wise objective. We will add this ablation to the appendix.
W4/Q4. Grouping, co-routing reuse, and placement
All experts in a factor group are placed on the same GPU. When a token selects multiple experts from one group, it is projected once with that group's shared input factor, and the experts reuse the compact row.
Our groups are formed by weight similarity. On C4/MATH routing, a token's eight selected experts span 6.83 groups on average, reducing projection count by about 15%. This reuse is modest. The three-stage pipeline overlaps projections with expert computation; in our H100 profile, projection and reconstruction together account for 4.3 ms of the 122.1 ms MoE critical path. Jointly considering weight similarity and co-routing may improve reuse further. We will add this analysis to App. A.1.
W5/Q5. Compressed baselines under EP
Figure 5 reports FactorEP's quality–latency trade-off across communication budgets on Qwen3 and OLMoE. We additionally evaluated HC-SMoE and D²-MoE, the two strongest compressed baselines from Table 1, in the same EP stack on 8× Ada using the Table 2 protocol. The table below reports BF16 results; FP8 results appear in our response to Reviewer Sh8N, W1.
| Method (~20% budget) | Latency (ms) / vs. uncompressed | Wiki2 PPL | Avg. accuracy (%) |
|---|---|---|---|
| Original (COMET/Flux), uncompressed | 747.7 / 1.00× | 7.873 | 61.86 |
| D²-MoE, low-rank form | 921.8 / 0.81× | 10.616 | 54.14 |
| HC-SMoE, no route deduplication | 764.5 / 0.98× | 10.382 | 57.69 |
| FactorEP | 582.5 / 1.28× | 8.697 | 59.93 |
In the evaluated configuration without route deduplication, HC-SMoE retains top-8 routing over same-size experts, reducing memory without reducing communication or compute. D²-MoE's low-rank form still communicates d-dimensional tokens and adds delta GEMMs, increasing latency by 23%.
FactorEP is 1.28× faster than the uncompressed model, 1.31× faster than HC-SMoE, and 1.58× faster than D²-MoE. At this budget, it achieves both lower latency and higher accuracy than the evaluated compressed baselines. We will add this comparison to §4.4.
I acknowledge and appreciate the responses. I tend to keep my scores due to the relatively shallow technical novelty.
Response to Reviewer N2Wa — (1/2)
We thank the reviewer for the detailed questions.
W1/Q1. What distinguishes FactorEP from prior work?
FactorEP converts pretrained MoE redundancy into end-to-end EP acceleration through algorithm–runtime co-design. In our Ada evaluation, D²-MoE and HC-SMoE increase latency, while FactorEP achieves a 1.28× speedup over the uncompressed model (W5/Q5).
BigMac and LatentMoE introduce new architectures requiring costly pretraining. TD-MoE and MoBE compress pretrained models, but their execution differs at the communication boundaries: TD-MoE’s reference implementation uses separate gate/up input factors and reconstructs outputs before aggregation, while MoBE retains the original down projections.
FactorEP jointly addresses communication, accuracy, and execution efficiency:
Reducing both dispatch and combine traffic (C1). A group-shared input basis provides one compact representation for both gate and up before dispatch. A layer-shared output basis enables combine before reconstruction, keeping both communication phases in factor space while retaining expert-specific cores.
Preserving quality without retraining (C1). KFAC-weighted calibration computes the factors in closed form to limit the accuracy loss introduced by sharing, without gradient-based training.
Realizing the compression gains on GPUs (C2, C3). The three-stage pipeline overlaps projection and communication with expert computation. Route-compiled execution packs routed tokens into grouped kernels without capacity padding or per-expert launches, maintaining efficiency for the smaller factor-space GEMMs (Table 3).
W2/Q2. Cost of a shared output basis
A shared output basis restricts expert outputs to a common low-dimensional subspace, while expert-specific cores retain distinct mappings within it. We compare this design with group-specific output bases on Qwen3, giving each group basis the full rank.
Group-specific bases improve the 7-task average by 0.9–1.4 points, with a 7–24% latency increase. The shared basis retains 97.3% of KFAC-weighted output energy and enables partial expert outputs to be combined before a single reconstruction. With group-specific bases, partial sums in different bases cannot be added directly (Eq. 6). Each token needs a separate partial sum and reconstruction for each group its experts span, averaging 1.95–3.2 groups here. This increases reconstruction work and requires either more combine traffic or group-constrained placement.
The ablation quantifies the quality–latency trade-off behind the shared output basis. We will add it and the cost analysis to the paper.