- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 1(−1)
Read review
Paper summary
This paper proposes MoESQ, a framework that compresses the expert weights of trillion-scale MoE models into the paired 4:8 sparse NVFP4 (W4A4) format natively supported by the Sparse Tensor Cores (SpTCs) of NVIDIA Blackwell GPUs. On the algorithmic side, MoESQ relaxes the selection of sparsity patterns using Gumbel-Softmax and optimizes the patterns jointly with the quantized weights, using an expert-wise reconstruction loss weighted by the router gating coefficients, which reduces the accuracy loss compared with existing methods. On the system side, the paper implements a grouped sparse GEMM kernel that executes this format across experts with varying token counts, and integrates it into vLLM. Experiments on Qwen3-30B-A3B, Qwen3.5-397B-A17B, and Kimi-K2.5 show higher accuracy than the compared compression baselines under the same paired 4:8 NVFP4 format, and on Kimi-K2.5, MoESQ achieves up to a 1.65× kernel speedup and 1.18× higher serving throughput compared with the dense NVFP4 grouped GEMM.
Critical strengths
1. Connecting hardware-compliant compression to end-to-end acceleration. Beyond compression, the paper provides a grouped sparse GEMM kernel and vLLM integration for the compressed experts, and demonstrates actual serving gains on a trillion-scale model.
2. Empirical support for accuracy improvements on large-scale models. MoESQ shows consistent accuracy improvements over existing sparse-quantization methods across MoE models of three different scales, and the paper reports the concrete compression cost for the trillion-scale model (about 27.6 hours on 8×B200 GPUs), which supports its practical applicability.
Critical weaknesses
1. The accuracy–efficiency trade-off is not sufficiently validated against simple alternatives. In Table 3, not only the compression method but also the weight/activation format and the execution backend differ across methods, so it is unclear where the serving advantage comes from. In particular, REAP achieves a higher OpenLLM average accuracy (80.77 vs. 79.21) and a lower storage cost (2.25 vs. 2.75 bits) than MoESQ, but it is evaluated only with the original INT4 W4A16 configuration. Although REAP removes half of the experts and thus halves the expert weight memory, its decode throughput is only 3,190 tok/s, less than half that of dense NVFP4 W4A4 (CuTeDSL, 7,619 tok/s; Table12b). This suggests that REAP's serving disadvantage may stem from the INT4 W4A16 execution format rather than from the pruning approach itself. Quantizing the remaining experts to dense NVFP4 W4A4 after REAP pruning would enable W4A4 computation using existing kernels under the same 2.25-bit budget.
Moreover, compared with dense NVFP4, MoESQ achieves 1.18× throughput on Kimi-K2.5 at the cost of a 2.69pp drop in OpenLLM average accuracy. This loss grows as the model becomes smaller (3.75pp on Qwen3.5-397B-A17B and 6.93pp on Qwen3-30B-A3B), yet the serving gains on these two models are not reported. I am particularly concerned that the trade-off seems less favorable where the accuracy loss is largest: Appendix D.2 (Figure 5) shows that the kernel-level speedup is smaller on Qwen3-30B-A3B (geometric means of 1.33× for gate/up and 1.20× for down, compared with 1.52× and 1.58× on Kimi-K2.5). Without a comparison against REAP + dense NVFP4 W4A4 and a trade-off analysis against dense NVFP4 across model scales, the "favorable balance" claim in Sec. 4.4 is not sufficiently supported.
2. Limited evidence for the algorithmic novelty and the necessity of joint optimization. The paper presents continuous reparameterization as its core algorithmic contribution, but this is close to applying MaskLLM's Gumbel-Softmax-based N:M mask learning to the paired 4:8 pattern, and the paper cites MaskLLM without discussing the differences. The STE-based QAT used for quantization is also a standard approach. Similarly, the router-weighted reconstruction objective (Eq. 4) closely resembles the affinity-guided quantization of MoEQuant (Chen et al., ICML 2025), which also weights each token's reconstruction error by its gating coefficient (applied per linear layer rather than at the expert-output level), but MoEQuant is neither cited nor discussed. The remaining algorithmic contribution therefore seems to be the joint optimization of the support selection and the quantized weights ("joint adaptation of sparse supports and weights"), but its effect is not clearly isolated.
Table 5 shows the benefit of learning the support (+1.00pp) by comparing against a variant that fixes the support to the SGPTQ solution and refines only the weight values. However, it does not show whether jointly optimizing the support and weight values is better than a sequential approach that first learns the support, fixes it, and then adapts the weight values. Rather, comparing Table 5 with the SGPTQ result in Table 1 shows that refining only the weight values already yields +3.35pp over SGPTQ, suggesting that most of the accuracy gain over SGPTQ comes from the additional refinement rather than from the proposed relaxation.
Questions for authors
Q1. Which backend was used to serve REAP in Table 3 (e.g., an INT4 W4A16 kernel)? How do the accuracy and serving throughput of MoESQ compare with a configuration in which the experts remaining after REAP pruning are quantized to dense NVFP4 W4A4 (REAP + dense NVFP4 W4A4)?
Q2. According to Table 1, the OpenLLM average accuracy loss of MoESQ relative to dense NVFP4 grows as the model becomes smaller (2.69pp on Kimi-K2.5, 3.75pp on Qwen3.5-397B-A17B, and 6.93pp on Qwen3-30B-A3B), whereas the serving throughput gain (1.18×) is reported only for Kimi-K2.5. Could you also measure the serving gains over dense NVFP4 on Qwen3.5-397B-A17B and Qwen3-30B-A3B, and present the accuracy–throughput trade-off across model scales in a table like Table 3, with dense NVFP4 included? An analysis of which MoE characteristics (e.g., the number or size of experts) cause this scale-dependent trend would also help assess whether the results generalize to other models.
Q3. Could you isolate the effect of jointly optimizing the support selection and the quantized weight values? Comparing with a sequential approach that first learns the support, fixes it, and then adapts the weight values under the same refinement budget would clarify the contribution of joint optimization.