acceptodds
Under review as a conference paper at ICLR 2027

Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts

Abstract

Mixture-of-Experts (MoE) architectures allow frontier language models to scale toward trillions of parameters, but their deployment is constrained by massive memory footprints and bandwidth limitations. While modern accelerators feature Sparse Tensor Cores (SpTCs) to reduce weight storage and boost throughput via low-precision semi-structured sparsity, exploiting them in MoEs is hindered by severe model-quality degradation and the lack of grouped sparse GEMM primitives. We present an end-to-end hardware–software co-design framework that compresses expert weights into hardware-native, low-precision sparse representations and accelerates their execution on SpTCs. Algorithmically, our framework relaxes discrete semi-structured support selection through continuous reparameterization, enabling differentiable joint optimization with quantized weights under a router-weighted reconstruction objective scaled through expert-parallel compression. Systemically, we implement a custom grouped sparse GEMM kernel tailored to sparse, low-precision MoE inference on SpTCs. Across MoE scales ranging from 30 billion to frontier-scale trillion-parameter models, our framework advances state-of-the-art joint sparse quantization by up to percentage points in task accuracy while preserving of the original models' performance. System-level benchmarks on NVIDIA B200 GPUs show that our kernel outpaces the vendor baseline by up to , delivering higher serving throughput and up to a reduction in end-to-end latency. These results establish hardware–software co-design as a practical and necessary pathway toward scalable, highly efficient MoE deployment.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Sparsity, Quantization, Model Compression, Inference Acceleration, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UU9fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
1(−1)
Read review
Paper summary

This paper proposes MoESQ, a framework that compresses the expert weights of trillion-scale MoE models into the paired 4:8 sparse NVFP4 (W4A4) format natively supported by the Sparse Tensor Cores (SpTCs) of NVIDIA Blackwell GPUs. On the algorithmic side, MoESQ relaxes the selection of sparsity patterns using Gumbel-Softmax and optimizes the patterns jointly with the quantized weights, using an expert-wise reconstruction loss weighted by the router gating coefficients, which reduces the accuracy loss compared with existing methods. On the system side, the paper implements a grouped sparse GEMM kernel that executes this format across experts with varying token counts, and integrates it into vLLM. Experiments on Qwen3-30B-A3B, Qwen3.5-397B-A17B, and Kimi-K2.5 show higher accuracy than the compared compression baselines under the same paired 4:8 NVFP4 format, and on Kimi-K2.5, MoESQ achieves up to a 1.65× kernel speedup and 1.18× higher serving throughput compared with the dense NVFP4 grouped GEMM.

Critical strengths

1. Connecting hardware-compliant compression to end-to-end acceleration. Beyond compression, the paper provides a grouped sparse GEMM kernel and vLLM integration for the compressed experts, and demonstrates actual serving gains on a trillion-scale model.

2. Empirical support for accuracy improvements on large-scale models. MoESQ shows consistent accuracy improvements over existing sparse-quantization methods across MoE models of three different scales, and the paper reports the concrete compression cost for the trillion-scale model (about 27.6 hours on 8×B200 GPUs), which supports its practical applicability.

Critical weaknesses

1. The accuracy–efficiency trade-off is not sufficiently validated against simple alternatives. In Table 3, not only the compression method but also the weight/activation format and the execution backend differ across methods, so it is unclear where the serving advantage comes from. In particular, REAP achieves a higher OpenLLM average accuracy (80.77 vs. 79.21) and a lower storage cost (2.25 vs. 2.75 bits) than MoESQ, but it is evaluated only with the original INT4 W4A16 configuration. Although REAP removes half of the experts and thus halves the expert weight memory, its decode throughput is only 3,190 tok/s, less than half that of dense NVFP4 W4A4 (CuTeDSL, 7,619 tok/s; Table12b). This suggests that REAP's serving disadvantage may stem from the INT4 W4A16 execution format rather than from the pruning approach itself. Quantizing the remaining experts to dense NVFP4 W4A4 after REAP pruning would enable W4A4 computation using existing kernels under the same 2.25-bit budget.

Moreover, compared with dense NVFP4, MoESQ achieves 1.18× throughput on Kimi-K2.5 at the cost of a 2.69pp drop in OpenLLM average accuracy. This loss grows as the model becomes smaller (3.75pp on Qwen3.5-397B-A17B and 6.93pp on Qwen3-30B-A3B), yet the serving gains on these two models are not reported. I am particularly concerned that the trade-off seems less favorable where the accuracy loss is largest: Appendix D.2 (Figure 5) shows that the kernel-level speedup is smaller on Qwen3-30B-A3B (geometric means of 1.33× for gate/up and 1.20× for down, compared with 1.52× and 1.58× on Kimi-K2.5). Without a comparison against REAP + dense NVFP4 W4A4 and a trade-off analysis against dense NVFP4 across model scales, the "favorable balance" claim in Sec. 4.4 is not sufficiently supported.

2. Limited evidence for the algorithmic novelty and the necessity of joint optimization. The paper presents continuous reparameterization as its core algorithmic contribution, but this is close to applying MaskLLM's Gumbel-Softmax-based N:M mask learning to the paired 4:8 pattern, and the paper cites MaskLLM without discussing the differences. The STE-based QAT used for quantization is also a standard approach. Similarly, the router-weighted reconstruction objective (Eq. 4) closely resembles the affinity-guided quantization of MoEQuant (Chen et al., ICML 2025), which also weights each token's reconstruction error by its gating coefficient (applied per linear layer rather than at the expert-output level), but MoEQuant is neither cited nor discussed. The remaining algorithmic contribution therefore seems to be the joint optimization of the support selection and the quantized weights ("joint adaptation of sparse supports and weights"), but its effect is not clearly isolated.

Table 5 shows the benefit of learning the support (+1.00pp) by comparing against a variant that fixes the support to the SGPTQ solution and refines only the weight values. However, it does not show whether jointly optimizing the support and weight values is better than a sequential approach that first learns the support, fixes it, and then adapts the weight values. Rather, comparing Table 5 with the SGPTQ result in Table 1 shows that refining only the weight values already yields +3.35pp over SGPTQ, suggesting that most of the accuracy gain over SGPTQ comes from the additional refinement rather than from the proposed relaxation.

Questions for authors

Q1. Which backend was used to serve REAP in Table 3 (e.g., an INT4 W4A16 kernel)? How do the accuracy and serving throughput of MoESQ compare with a configuration in which the experts remaining after REAP pruning are quantized to dense NVFP4 W4A4 (REAP + dense NVFP4 W4A4)?

Q2. According to Table 1, the OpenLLM average accuracy loss of MoESQ relative to dense NVFP4 grows as the model becomes smaller (2.69pp on Kimi-K2.5, 3.75pp on Qwen3.5-397B-A17B, and 6.93pp on Qwen3-30B-A3B), whereas the serving throughput gain (1.18×) is reported only for Kimi-K2.5. Could you also measure the serving gains over dense NVFP4 on Qwen3.5-397B-A17B and Qwen3-30B-A3B, and present the accuracy–throughput trade-off across model scales in a table like Table 3, with dense NVFP4 included? An analysis of which MoE characteristics (e.g., the number or size of experts) cause this scale-dependent trend would also help assess whether the results generalize to other models.

Q3. Could you isolate the effect of jointly optimizing the support selection and the quantized weight values? Comparing with a sequential approach that first learns the support, fixes it, and then adapts the weight values under the same refinement budget would clarify the contribution of joint optimization.

Reviewer_mPbROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents MoESQ, a framework for compressing MoE expert weights into paired 4:8 sparse NVFP4 and executing them with a custom grouped sparse GEMM kernel. It jointly optimizes sparse support and quantized weights using a router-weighted reconstruction objective.

Critical strengths
  1. The paper is clearly organized, and the motivation for combining compression with a hardware-native execution path is compelling.
  2. The evaluation covers three MoE scales, several accuracy benchmarks, and both kernel-level and end-to-end serving performance.
  3. Under the same paired 4:8 NVFP4 target, MoESQ achieves higher accuracy than the evaluated sparse-quantization baselines. The B200 serving results also show actual throughput and latency improvements over dense NVFP4.
Critical weaknesses
  1. The accuracy–memory–serving tradeoff against expert pruning remains unclear. In Table 3, REAP has higher accuracy and a smaller expert-weight footprint than MoESQ, but uses an INT4 W4A16 serving path. The differing numerical formats and kernels make it difficult to attribute the serving gap to the compression strategy alone.
  2. End-to-end serving is evaluated only on Kimi-K2.5 with B200 GPUs and a limited set of workload shapes, leaving the generality of the system-level gains uncertain.
Questions for authors
  1. Could you evaluate REAP expert pruning combined with dense NVFP4 W4A4, using the same serving stack as MoESQ? Results at 50% pruning and at a pruning level close to MoESQ’s 2.75-bit expert-weight budget would make the accuracy, memory, and throughput comparison more informative.
  2. Can MoESQ start directly from a natively quantized FP4 checkpoint, particularly an MXFP4 model such as Kimi K3 or gpt-oss?
Reviewer_rwqEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoESQ, a hardware–software co-design framework for efficient inference of large-scale Mixture-of-Experts (MoE) models using hardware-native sparse quantization.

On the compression side, MoESQ uses Gumbel-Softmax relaxation to jointly optimize semi-structured sparsity patterns and quantized weights under the paired 4:8 NVFP4 format supported by NVIDIA Blackwell Sparse Tensor Cores. The method also introduces router-weighted expert reconstruction and blockwise expert-parallel compression to scale to trillion-parameter models. On the systems side, the authors develop a sparse grouped GEMM kernel that supports dynamically routed experts and integrates it into vLLM serving.

The paper evaluates MoESQ on models ranging from Qwen3-30B to Kimi-K2.5 (1T parameters), showing good latency and accuracy results.

Critical strengths
  • The paper presents a well-integrated approach that takes advantage of the sparse and low-precision computation capabilities of NVIDIA Blackwell GPUs. The compression method produces hardware-compatible sparse weights, while the custom grouped sparse GEMM kernel enables efficient execution on Sparse Tensor Cores. This is a practical systems contribution that addresses both compression and execution challenges in large-scale MoE inference.

  • The evaluation covers MoE models ranging from 30B to 1T parameters. MoESQ achieves better accuracy than existing sparse-quantization baselines under aggressive compression constraints. The kernel benchmarks and end-to-end vLLM experiments also demonstrate meaningful improvements in both prefill and decode performance, along with reduced expert weight storage.

Critical weaknesses

I have a few clarification questions listed below.

Questions for authors
  • Table 2 does not include dense NVFP4 results on the reasoning benchmarks, where MoESQ shows noticeable accuracy degradation relative to the reference model. Could the authors provide these results to help clarify how much additional degradation is introduced by sparsity beyond quantization alone?
  • The paper reports effective TFLOP/s and speedups over dense grouped GEMM kernels. How close does the proposed sparse grouped GEMM kernel get to the theoretical peak throughput of Blackwell Sparse Tensor Cores? It would be helpful to understand its hardware utilization and the remaining performance headroom.
  • The paper reports a reduction in expert weight storage from 4.5 to 2.75 bits per weight, along with increased KV-cache capacity. Could the authors also quantify the actual GPU memory savings for the full model, including non-expert parameters and other memory overheads? This would help clarify the practical memory benefits of MoESQ at trillion-parameter scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.