acceptodds
Under review as a conference paper at ICLR 2027

ULMoE: Algorithm-System Co-design for Efficient Ultra-Low-Bit MoE Compression and Serving

Abstract

Mixture-of-Experts (MoE) models now dominate frontier LLMs. However, the massive parameter count of experts makes them expensive to fit into GPU memory, and the traffic of reading expert weights on every decode step makes them expensive to serve. We show that ultra-low-bit weight quantization addresses both challenges, and that it pays off more in MoE than in dense models, because sparse routing makes the memory-bound region of grouped GEMM wider than dense GEMM, covering the practical serving batch sizes. Achieving practical serving speedup requires the quantization format and the kernel to be designed together. Methods such as vector quantization and expert-level mixed precision are designed for accuracy alone, and although they reach two bits, no grouped GEMM kernel can execute them efficiently. Methods designed around kernels achieve efficient serving performance but stop at four bits. We present ULMoE, an algorithm-system co-design for accurate and efficient ultra-low-bit MoE serving. Its format spends 2.5 bits per weight and places the quantization group scales on two integer levels. The metadata for 64 weights fits in one aligned 32-bit word for efficient memory access. On top of the format we build a fast W2A16 grouped GEMM kernel and integrate it into SGLang. Across seven MoE models from three families, ULMoE compresses expert weights at to effective bits and loses average accuracy points. At matched interactivity it delivers the serving throughput of BF16 on Qwen3.5-35B and that of Int4 on Qwen3.5-122B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Large Language Models, Model Compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8sDEOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ULMoE proposes an algorithm-system co-design for ultra-low-bit MoE compression and serving. It introduces a 2.5-bit quantization format with hierarchical integer scales and salient-channel protection, along with a specialized W2A16 grouped GEMM kernel for efficient GPU execution. Experiments across seven MoE models show around 6× expert weight compression with an average accuracy drop of 1.8 points, while improving serving throughput over BF16 and INT4 baselines.

Critical strengths
  1. The paper takes a practical approach to ultra-low-bit MoE quantization by designing the quantization format and GPU kernel together. The unified format allows all experts to use the same execution path, while the specialized kernel makes the compressed weights efficient to serve.
  2. The experiments are thorough, covering accuracy, kernel performance, and end-to-end serving across multiple MoE models. The results show substantial weight compression and throughput improvements with limited accuracy loss, supporting the practical value of the method.
Critical weaknesses
  1. The novelty of the quantization method could be clearer. Fine-grained scaling and protecting important channels are related to existing quantization techniques, and the paper could better explain what is fundamentally different about its approach beyond adapting these ideas to MoE serving.

  2. The paper includes ablations on the quantization format and metadata layout, but it remains unclear how much of the overall speedup comes from reduced weight traffic versus the optimized dequantization and GEMM kernel. A more direct breakdown would help clarify the contribution of the proposed co-design.

Questions for authors
  1. The paper combines hierarchical scaling and salient-channel protection in its quantization format. Could the authors clarify how these designs differ from existing quantization methods,?
  2. The paper provides ablations on the quantization format and metadata layout. Could the authors further clarify how much of the serving speedup comes from reduced weight traffic versus the optimized dequantization and GEMM kernel?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 8sDE

We thank the reviewer for recognizing our co-design and thorough experiments. Below are our detailed responses.


W1 & Q1: Difference from existing hierarchical scaling and salient-channel methods.

R1: We thank the reviewer for this insightful question. The novelty of ULMoE lies not in hierarchical scaling or salient-channel protection alone, both of which have precedent, but in a format in which both operate under the constraint that one grouped GEMM must execute every expert with one data type, one layout and one decode path. Hierarchical scales appear in the double quantization of QLoRA and in microscaling formats such as NVFP4. Salient channels are protected by activation-aware rescaling in AWQ, by sparse high-precision outliers in SpQR, and by higher-precision outlier channels in mixed-precision schemes.

Three properties follow from that constraint, and none of the formats above combines them.

  • First, there is no floating-point scale below the output channel. Each micro-group of 8 weights stores a 2-bit integer multiplier and a 1-bit zero, each macro-group of 64 stores a 7-bit integer scale code and a 1-bit ladder selector, and the only floating-point value is one FP16 anchor per output channel, shared by all E experts of a layer.

  • Second, the metadata sums to exactly 32 bits for 64 weights, so the whole decode state of a macro-group arrives in one aligned load and every field is extracted with a shift and a mask against immediates, without table lookup.

  • Third, salient channels are protected by 128 residual columns in the same 2-bit format under the same anchor, so protection extends K inside the same GEMM rather than adding a sparse or higher-precision side path. The down projection uses routing-weighted salience per expert, and its permutation is absorbed offline into the gate/up rows.

At the same 2.5-bit budget the format reaches 1.6× lower weight MSE than GPTQ Int2 (Figure 4). Against QuIP#, ULMoE is 1.2 to 3.1 points more accurate on the reasoning average, and our kernel is 3.6 to 4.0× faster than the grouped GEMM of QuIP# (R2 to Reviewer Gtka).


W2 & Q2: Speedup from reduced weight traffic versus the optimized kernel.

R2: In short, the reduced weight traffic is the source of the kernel speedup, and the co-design of format and kernel turns it into an actual speedup, since fewer bytes help only if the kernel dequantizes them as fast as it reads them. R3 to Reviewer Wp1m separates the two with a streaming-read ceiling, which gives the speedup from the byte reduction alone, and a variant of our kernel without dequantization. From 16 to 64 routed tokens the byte-only speedup over Marlin W4A16 is 1.61 to 1.62×, and our kernel achieves 1.53 to 1.61×, which is 95 to 99 percent of what the traffic reduction allows, while the dequantization takes only 2.6 to 3.4 percent of the kernel time. This efficiency comes from the format. Splitting the same bits into separate fields instead of one aligned 32-bit word makes the kernel 1.13× slower (Table 6). The grouped GEMMs of QuIP# and MiLo, which also read fewer weight bytes than Int4, are 2.1 to 4.0× slower than our kernel and slower than Marlin W4A16 at every token count (R2 to Reviewer Gtka).

At the serving level the gain has two parts, a per-step speedup from the expert GEMMs and a memory term once Int4 runs out of KV cache. Only the expert GEMMs become faster, while the rest of a decode step is identical in both formats, so the per-step speedup follows Amdahl's law, 1 / ((1 − f) + f / S_kernel), where f is the expert-GEMM share of the Int4 step and S_kernel is the kernel speedup. At batch 1 on Qwen3.5-122B on RTX PRO 6000, the routed expert GEMMs take 1.7 ms of the 10.5 ms Int4 step, and the kernels tie at one routed token, so the two formats tie (0.98×). As the batch grows, a step touches more experts while the shared work grows slowly, so both f and S_kernel rise.

The table gives the per-step decode time of Qwen3.5-122B on RTX PRO 6000, taken as the median TPOT of the real-request runs, and the expert share f of the Int4 step.

Requests in flight1481632
ULMoE step (ms)10.713.817.422.931.2
Int4 step (ms)10.514.919.927.438.6
Expert share f of the Int4 step0.160.280.400.460.53

The long reasoning contexts of the real requests raise the share of attention in each step, so the per-step speedup stays below the kernel speedup of 1.36 to 1.58× at these token counts.

The memory term comes on top. ULMoE's smaller weights leave 2.2 to 4.5× more KV cache, so from about 64 requests in flight Int4 preempts and recomputes requests, and its running batch no longer grows with the load. The throughput ratio then rises to 1.45× offline and 1.74× at 20 tok/s/user (R3 to Reviewer Gtka).

We hope the clarifications above resolve your questions. Thank you for your helpful comments.

Reviewer_Wp1mOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper proposes ULMoE, an algorithm-system co-design for ultra-low-bit MoE serving. It designs the weight and metadata precision to fit the 32-bit word size and implements a matching W2A16 grouped GEMM kernel, demonstrating practical throughput gains from expert quantization.
Critical strengths
  • The paper is well organized overall, and the algorithm-system co-design principle is easy to follow throughout.
  • The accuracy and throughput evaluations are systematically conducted across various models and benchmarks. In particular, the paper includes reasoning evaluations, which other papers often avoid.
Critical weaknesses
  • The output length distribution of the quantized models on reasoning tasks is not reported. A recent study [1] reported that quantized reasoning models tend to overthink, and longer outputs could offset the throughput gains.
  • Throughput is compared against GPTQ-Int4, but Table 1 does not report the accuracy of Int4, making it difficult to assess the accuracy–throughput trade-off relative to Int4.
  • Since ULMoE (2-bit) and the baseline (Int4) read different numbers of weight bytes, latency comparisons alone cannot reveal the efficiency of the kernel itself. The theoretical speedup expected from the byte reduction alone is about 1.5–1.6×, whereas the reported kernel speedup is 1.42×. The measured memory bandwidth utilization (as a fraction of peak) of each kernel should be reported across batch sizes to show the extent of the dequantization overhead.

[1] Lotfi, Sanae, et al. “Quantized reasoning models think they need to think longer, but they do not.” arXiv preprint arXiv:2606.00206 (2026).

Questions for authors

Please see the weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Wp1m

We thank the reviewer for recognizing the clarity of the paper and our systematic evaluation. Below are our detailed responses.


W1: Output length and overthinking on reasoning tasks.

R1: We thank the reviewer for raising this important point. We collect the output length on all five reasoning tasks for Gemma4-26B and Qwen3.5-35B with the sampling protocol and 64k cap of Table 1, and measure the output length of every format relative to the BF16 baseline. Lengths are paired per question and averaged over each format's seeds, and the table below reports the geometric mean of the per-question ratios over all questions of each model, averaged over the five reasoning tasks.

Length relative to BF16ULMoEGPTQ-Int4QuIP#MiLo
Gemma4-26B1.100.990.941.22
Qwen3.5-35B0.950.981.051.99

On Qwen3.5-35B ULMoE writes 5 percent less than BF16, and on Gemma4-26B it writes 10 percent more. MiLo, in contrast, shows the overthinking at 1.22× and 1.99× the BF16 length. GPTQ-Int4 matches BF16 on both models (0.98 to 0.99×). We use the default sampling parameters of Table 1 (the official recommended parameters for Qwen3.5-35B-A3B) for every format, which include a presence penalty of 1.5 alongside temperature 1.0, top-p 0.95 and top-k 20. The penalty discourages tokens that have already appeared, which could suppress the repetitive loops through which quantization error most often lengthens a reasoning trace.

The real-request serving runs in R3 to Reviewer Gtka stop every request naturally and have both formats answer the same MATH500 problems.


W2: Accuracy of GPTQ-Int4.

R2: We agree that the accuracy of Int4 is needed to assess this trade-off, and R1 to Reviewer Gtka adds GPTQ-Int4 to the accuracy comparison. Int4 stays within 1.1 points of BF16 on the reasoning average, and ULMoE trails Int4 by 1.5 to 2.1 points. ULMoE uses about 0.65× the bits of Int4 and delivers 1.27 to 1.74× its throughput on real reasoning requests at fixed interactivity (R3 to Reviewer Gtka).


W3: Bandwidth utilization of each kernel and the dequantization overhead.

R3: We thank the reviewer for this suggestion. Our kernel runs close to its bandwidth ceiling, and its dequantization overhead is small. To show this, we add two measurements to the benchmark of Figure 8 on RTX PRO 6000, with the same routing and the Qwen3.5-122B expert shapes.

  • The first measurement is a streaming kernel that reads exactly the weight bytes each grouped GEMM reads for the experts a batch touches, with no dequantization and no matrix multiply. Its time is the bandwidth ceiling for that byte count. The nominal peak bandwidth of the RTX PRO 6000 is 1792 GB/s, which is not reachable in practice. The highest read rate we measure on large contiguous buffers is about 1530 GB/s, 85 percent of the nominal peak, and the streaming kernel reaches 97 to 99 percent of that rate on large transfers.

  • The second is a variant of our kernel in which the dequantization is replaced by a single logical operation on the loaded words. The compiled kernel issues the same global loads and tensor-core instructions as the original, so the time difference between the two is the dequantization cost.

The table below sums the gate/up and down projections. Bandwidth is the weight bytes each kernel reads divided by its time, with the kernel's fraction of the streaming-read ceiling for the same bytes in parentheses. The byte-only speedup is the ratio of the two ceilings, and the dequantization share is the fraction of our kernel's time removed when its dequantization is removed.

Routed tokensULMoE (µs)Marlin W4A16 (µs)SpeedupByte-only speedupULMoE bandwidth, GB/s (of ceiling)Marlin bandwidth, GB/s (of ceiling)Dequant share
459.480.81.361.511027 (0.82)1244 (0.91)6.3%
16161.7247.81.531.611325 (0.93)1427 (0.97)3.4%
64313.3503.81.611.621412 (0.96)1449 (0.96)2.6%
256363.5582.71.601.631385 (0.93)1425 (0.94)2.8%

The measured speedup factors into a traffic term and an efficiency term, S = S_bytes × (η_ULMoE / η_Int4), where η is a kernel's fraction of its own ceiling. From 16 to 64 routed tokens, which covers decode at serving concurrency, both kernels run at 0.93 to 0.97 of their ceilings, our kernel streams weights at 1325 to 1412 GB/s (87 to 92 percent of the achievable 1530 GB/s), and the dequantization takes 2.6 to 3.4 percent of its time. The measured speedup of 1.53 to 1.61× is therefore within 0.08 of the byte-only 1.61 to 1.62×. At four routed tokens only a few experts are touched, the transfer is latency-bound, and our kernel reaches 0.82 of its ceiling.

Thank you for the detailed response. My concerns have been adequately addressed, and I will keep my score as is.

Reviewer_GtkaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that ultra-low-bit weight quantization addresses both the GPU memory footprint and the per-decode-step weight traffic of Mixture-of-Experts (MoE) models, and that it pays off more in MoE than in dense models because sparse routing makes the memory-bound region of grouped GEMM wider than dense GEMM. The paper presents ULMoE, an algorithm-system co-design whose quantization format spends 2.5 bits per weight, places the quantization group scales on two integer levels, and fits the metadata for 64 weights into one aligned 32-bit word. Across seven MoE models from three families, ULMoE compresses expert weights ∼6× at 2.6 to 2.8 effective bits with a 1.8 average accuracy point loss, and at matched interactivity it delivers 5.7× the serving throughput of BF16 on Qwen3.5-35B and 1.7× that of Int4 on Qwen3.5-122B.

Critical strengths
  1. The paper shows that the routed grouped GEMM stays memory-bound over a batch range E/kr times wider than a dense GEMM leading to speedup due to weight compression.
  2. The authors report a 1.8 point average drop vs. BF16 (AWQ and GEMQ lose 5.4 to 18.3 and 15.3 to 29.9 at the same budget) and 1.70× the Int4 throughput on Qwen3.5-122B at 30 tok/s/user.
Critical weaknesses
  1. Results for GPTQ-Int4 are missing.
  2. Results for Codebook/rotation methods, and MiLo are not presented but dismissed on serving-efficiency grounds without accuracy or kernel/serving measurements while GEMQ is evaluated for accuracy only. It would be good to have these results.
  3. The results in the paper use RTX PRO 6000 and L40S with random 128-token prompts, 1024 output tokens and context length 2048 and the roofline assumes ideal independent routing. It would be good to have HBM datacenter GPUs, real request traces and routing skew tested.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Gtka (1/2)

We thank the reviewer for recognizing our roofline analysis and results, and for the constructive suggestions. Below are our detailed responses.


W1: Results for GPTQ-Int4 are missing.

R1: We thank the reviewer for pointing out this missing baseline. We add GPTQ-Int4 to the accuracy evaluation, with only the routed experts quantized as for every other format. GPTQ-Int4 stays within 1.1 points of BF16 on the reasoning average of every model. ULMoE, at about 0.65× the bits of Int4, trails it by 1.5 to 2.1 points on the reasoning average and by at most 1.4 points on the non-reasoning average. At this cost ULMoE delivers 1.27 to 1.74× the throughput of Int4 on real reasoning requests at fixed interactivity (R3). The table below gives the average accuracy (%) over the five reasoning tasks of Table 1 (AIME24, AIME25, MATH500, GPQA and LiveCodeBench), and over the four non-reasoning tasks (MMLU, GSM8K, HumanEval and IFEval).

ModelMethodReasoning averageNon-reasoning average
Gemma4-26BBF1686.5390.11
GPTQ-Int485.9090.95
ULMoE83.7889.59
Qwen3.5-35BBF1687.4790.39
GPTQ-Int486.4389.20
ULMoE84.8989.27
Qwen3.5-122BBF1688.5090.64
GPTQ-Int488.6390.98
ULMoE86.8690.72

W2: Accuracy and kernel results for codebook and rotation methods, MiLo and GEMQ.

R2: We thank the reviewer for this suggestion. We add accuracy and kernel results for QuIP#, as the representative codebook and rotation method, and for MiLo, together with kernel results for GEMQ.

The table below gives the average accuracy (%) of QuIP# and MiLo. QuIP# is the strongest 2-bit baseline, and ULMoE leads it by 1.2 to 3.1 points on the reasoning average, mostly on LiveCodeBench and GPQA. We evaluate MiLo with a 2-bit base and a rank-32 compensator, which brings its average to 2.7 bits, the same as ULMoE for fair comparison. MiLo falls 13 to 15 points below ULMoE.

ModelMethodReasoning averageNon-reasoning average
Gemma4-26BULMoE83.7889.59
QuIP#80.6588.51
MiLo70.4180.25
AWQ72.0686.56
Qwen3.5-35BULMoE84.8989.27
QuIP#81.8788.38
MiLo70.1382.64
AWQ75.4886.76
Qwen3.5-122BULMoE86.8690.72
QuIP#85.6889.40
MiLo74.2088.79
AWQ80.6588.22

For kernel evaluation we add QuIP#, MiLo and GEMQ to the grouped GEMM benchmark on RTX PRO 6000, with the same Qwen3.5-122B expert shapes. The official repositories of QuIP# and MiLo provide only per-linear kernels, so we report each of them in two forms. The first is their original kernel, called once per touched expert. The second is a grouped GEMM version in which the method's own dequantization is placed inside the SGLang fused-MoE grouped GEMM.

  • The QuIP# implementation includes its input and output Hadamard transforms.
  • The MiLo grouped GEMM uses its own W3A16 weight packing, since the official MiLo kernel supports only 3-bit weights, and is reported with and without its rank-32 compensator, which we run as BF16 grouped GEMMs.
  • GEMQ is measured with its official kernels. At one token it runs its fused single-token path, and above one token it calls its dequantizing GEMM once per touched expert.

The table below gives the MoE experts GEMM latency in µs at 1 to 256 routed tokens, with the gate/up and down projections summed. Parentheses give the latency of each baseline divided by that of ULMoE.

Kernel, routed tokens141664256
ULMoE32.759.4161.7313.3363.5
Marlin W4A1632.5 (0.99×)80.8 (1.36×)247.8 (1.53×)503.8 (1.61×)582.7 (1.60×)
MiLo, grouped GEMM67.6 (2.07×)122.9 (2.07×)368.6 (2.28×)704.4 (2.25×)786.3 (2.16×)
MiLo, grouped GEMM with compensator100.1 (3.06×)159.6 (2.69×)422.2 (2.61×)782.8 (2.50×)887.7 (2.44×)
MiLo, official kernel176.0 (5.38×)646.7 (11×)2258 (14×)4667 (15×)5462 (15×)
QuIP#, grouped GEMM117.1 (3.58×)227.3 (3.83×)611.3 (3.78×)1210 (3.86×)1472 (4.05×)
QuIP#, official kernel881.2 (27×)3914 (66×)13617 (84×)28012 (89×)30831 (85×)
GEMQ54.9 (1.68×)6065 (102×)21160 (131×)42964 (137×)48384 (133×)

MiLo, QuIP#, and GEMQ are slower than Marlin W4A16 at every token count, and therefore their expert layers cost more per decode step than those of the Int4 baseline. Even when grouped, the codebook dequantization of QuIP# and its two Hadamard transforms cost 3.6 to 4.0× our latency, and MiLo's compensator adds an additional 11 to 48 percent latency to its own, most at small batches.

AuthorsOpenReview ↗

Response to Reviewer Gtka (2/2)


W3: Evaluations for HBM datacenter GPUs, real request traces and routing skew.

R3: We appreciate the reviewer's suggestion to test these broader settings. ULMoE keeps its advantage on an HBM datacenter GPU, on real reasoning requests and under routing skew. We extend the kernel and serving comparisons to NVIDIA B200. End to end in SGLang, under the revised offline protocol of the paper (General Response) for Qwen3.5-35B and Qwen3.5-122B, ULMoE delivers 1.25 to 1.43× the offline throughput of Int4 at every batch size from 1 to 256 (1.38× and 1.41× at batch 64). Averaged over 1 to 512 routed tokens, our grouped GEMM is 1.52× faster than Marlin W4A16. At 256 tokens the speedup exceeds the byte ratio because Marlin's weight streaming rate on B200 falls to 2.8 TB/s at 256 tokens while ours stays at 3.2 TB/s. The table below gives the grouped GEMM latency of Marlin W4A16 divided by that of ULMoE, with the gate/up and down projections summed.

Routed tokens41664256
Marlin W4A16 / ULMoE on B2001.321.541.611.86

On real reasoning requests, ULMoE delivers 1.27 to 1.74× the Int4 throughput at fixed interactivity. The revised paper serves real ShareGPT prompts throughout (General Response). To evaluate long reasoning requests in addition, we serve MATH500 prompts in thinking mode with natural stopping on Qwen3.5-122B on one RTX PRO 6000. The mean output is 8k to 10k tokens. The table below gives the throughput of ULMoE divided by that of Int4, read from the online Pareto envelopes of the two formats at fixed interactivity.

Interactivity (tok/s/user)603020
Throughput, ULMoE / Int41.271.491.74

With a fixed concurrency of 1, 4, 8, 16, 32, 64, 96 and 128, ULMoE delivers 0.98, 1.06, 1.08, 1.10, 1.15, 1.23, 1.33 and 1.45× the offline throughput of Int4. The gain grows with load for a memory reason as well as a traffic reason. The KV cache of Int4 holds 0.31 to 0.54 million tokens against 1.19 to 1.42 million for ULMoE, so over the runs at concurrency 64, 96 and 128, Int4 preempts and recomputes requests 36 times, while ULMoE never preempts.

The kernel speedup persists under routing skew. To study routing skew, we add skewed routing to the grouped GEMM benchmark on RTX PRO 6000. We model expert popularity with a Zipf law, a common model of popularity skew in which the item of popularity rank r is chosen with weight proportional to 1/r^s. We place the 256 experts in a random order, give the expert at rank r the weight 1/r^s, and let each token draw its 8 distinct experts with probability proportional to these weights. The exponent s sets the skew. At s = 0 every expert has the same weight, which is the independent routing of Eq. (2). As s grows, tokens concentrate on the top-ranked experts. The table below gives the speedup of ULMoE over Marlin W4A16 under each routing at 1 to 256 routed tokens, with the gate/up and down projections summed.

Routing, routed tokens1248163264128256
Independent (Eq. 2)0.991.201.361.471.531.581.611.631.60
Zipf s = 0.51.011.221.291.471.511.591.621.611.63
Zipf s = 1.01.001.211.231.471.371.511.601.491.48
Zipf s = 1.51.001.111.371.231.391.241.381.321.26

Independent routing gives the largest expected coverage, so Eq. (2) describes the high-traffic end of the range, and skew shifts work from weight traffic toward arithmetic on the hot experts. Under moderate skew, the speedup at 16 to 64 tokens stays at 1.37 to 1.62×, close to the 1.53 to 1.61× of independent routing. Under extreme skew (s = 1.5) it falls to 1.24 to 1.39×.

We hope these additional results address your concerns, and we would be glad to discuss any remaining questions. Thank you again for your detailed and constructive review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive feedback. In addition to the experiments in the individual responses below, we have revised the paper, and every change is marked in blue in the revised PDF. We audited the serving pipeline and re-measured every serving result with the following changes.

  • Workload. The prompts are 256 first-turn ShareGPT conversations whose chat-templated length is 120 to 140 tokens, in place of synthetic random-token prompts. Every request generates the target output length (1024 tokens, and 10240 for the long-context workload).

  • Engine settings. All formats run on the same engine (SGLang v0.5.20) with breakable prefill CUDA graphs and chunked prefill of 2048 tokens, within each comparison only the MoE differs.

The conclusions are unchanged, and ULMoE consistently shows serving improvements compared with GPTQ-Int4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.