- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ULMoE proposes an algorithm-system co-design for ultra-low-bit MoE compression and serving. It introduces a 2.5-bit quantization format with hierarchical integer scales and salient-channel protection, along with a specialized W2A16 grouped GEMM kernel for efficient GPU execution. Experiments across seven MoE models show around 6× expert weight compression with an average accuracy drop of 1.8 points, while improving serving throughput over BF16 and INT4 baselines.
Critical strengths
- The paper takes a practical approach to ultra-low-bit MoE quantization by designing the quantization format and GPU kernel together. The unified format allows all experts to use the same execution path, while the specialized kernel makes the compressed weights efficient to serve.
- The experiments are thorough, covering accuracy, kernel performance, and end-to-end serving across multiple MoE models. The results show substantial weight compression and throughput improvements with limited accuracy loss, supporting the practical value of the method.
Critical weaknesses
-
The novelty of the quantization method could be clearer. Fine-grained scaling and protecting important channels are related to existing quantization techniques, and the paper could better explain what is fundamentally different about its approach beyond adapting these ideas to MoE serving.
-
The paper includes ablations on the quantization format and metadata layout, but it remains unclear how much of the overall speedup comes from reduced weight traffic versus the optimized dequantization and GEMM kernel. A more direct breakdown would help clarify the contribution of the proposed co-design.
Questions for authors
- The paper combines hierarchical scaling and salient-channel protection in its quantization format. Could the authors clarify how these designs differ from existing quantization methods,?
- The paper provides ablations on the quantization format and metadata layout. Could the authors further clarify how much of the serving speedup comes from reduced weight traffic versus the optimized dequantization and GEMM kernel?
Response to Reviewer 8sDE
We thank the reviewer for recognizing our co-design and thorough experiments. Below are our detailed responses.
R1: We thank the reviewer for this insightful question. The novelty of ULMoE lies not in hierarchical scaling or salient-channel protection alone, both of which have precedent, but in a format in which both operate under the constraint that one grouped GEMM must execute every expert with one data type, one layout and one decode path. Hierarchical scales appear in the double quantization of QLoRA and in microscaling formats such as NVFP4. Salient channels are protected by activation-aware rescaling in AWQ, by sparse high-precision outliers in SpQR, and by higher-precision outlier channels in mixed-precision schemes.
Three properties follow from that constraint, and none of the formats above combines them.
First, there is no floating-point scale below the output channel. Each micro-group of 8 weights stores a 2-bit integer multiplier and a 1-bit zero, each macro-group of 64 stores a 7-bit integer scale code and a 1-bit ladder selector, and the only floating-point value is one FP16 anchor per output channel, shared by all E experts of a layer.
Second, the metadata sums to exactly 32 bits for 64 weights, so the whole decode state of a macro-group arrives in one aligned load and every field is extracted with a shift and a mask against immediates, without table lookup.
Third, salient channels are protected by 128 residual columns in the same 2-bit format under the same anchor, so protection extends K inside the same GEMM rather than adding a sparse or higher-precision side path. The down projection uses routing-weighted salience per expert, and its permutation is absorbed offline into the gate/up rows.
At the same 2.5-bit budget the format reaches 1.6× lower weight MSE than GPTQ Int2 (Figure 4). Against QuIP#, ULMoE is 1.2 to 3.1 points more accurate on the reasoning average, and our kernel is 3.6 to 4.0× faster than the grouped GEMM of QuIP# (R2 to Reviewer Gtka).
R2: In short, the reduced weight traffic is the source of the kernel speedup, and the co-design of format and kernel turns it into an actual speedup, since fewer bytes help only if the kernel dequantizes them as fast as it reads them. R3 to Reviewer Wp1m separates the two with a streaming-read ceiling, which gives the speedup from the byte reduction alone, and a variant of our kernel without dequantization. From 16 to 64 routed tokens the byte-only speedup over Marlin W4A16 is 1.61 to 1.62×, and our kernel achieves 1.53 to 1.61×, which is 95 to 99 percent of what the traffic reduction allows, while the dequantization takes only 2.6 to 3.4 percent of the kernel time. This efficiency comes from the format. Splitting the same bits into separate fields instead of one aligned 32-bit word makes the kernel 1.13× slower (Table 6). The grouped GEMMs of QuIP# and MiLo, which also read fewer weight bytes than Int4, are 2.1 to 4.0× slower than our kernel and slower than Marlin W4A16 at every token count (R2 to Reviewer Gtka).
At the serving level the gain has two parts, a per-step speedup from the expert GEMMs and a memory term once Int4 runs out of KV cache. Only the expert GEMMs become faster, while the rest of a decode step is identical in both formats, so the per-step speedup follows Amdahl's law, 1 / ((1 − f) + f / S_kernel), where f is the expert-GEMM share of the Int4 step and S_kernel is the kernel speedup. At batch 1 on Qwen3.5-122B on RTX PRO 6000, the routed expert GEMMs take 1.7 ms of the 10.5 ms Int4 step, and the kernels tie at one routed token, so the two formats tie (0.98×). As the batch grows, a step touches more experts while the shared work grows slowly, so both f and S_kernel rise.
The table gives the per-step decode time of Qwen3.5-122B on RTX PRO 6000, taken as the median TPOT of the real-request runs, and the expert share f of the Int4 step.
The long reasoning contexts of the real requests raise the share of attention in each step, so the per-step speedup stays below the kernel speedup of 1.36 to 1.58× at these token counts.
The memory term comes on top. ULMoE's smaller weights leave 2.2 to 4.5× more KV cache, so from about 64 requests in flight Int4 preempts and recomputes requests, and its running batch no longer grows with the load. The throughput ratio then rises to 1.45× offline and 1.74× at 20 tok/s/user (R3 to Reviewer Gtka).
We hope the clarifications above resolve your questions. Thank you for your helpful comments.