acceptodds
Under review as a conference paper at ICLR 2027

BMoEQ: Binary-Coded MoE Quantization with Runtime-Adaptive Mixed-Precision Support

Abstract

Mixture-of-Experts large language models (MoE-LLMs) scale model capacity through sparse activation but retain a substantial memory footprint from expert weights. Existing expert-level mixed-precision methods alleviate this cost by fixing bit-widths according to calibration-based importance estimates, which may not generalize across inputs, domains, and tasks. We propose BMoEQ, a quantization method that combines uniform ultra-low-bit base precision across experts with input-adaptive execution precision through binary-coded quantization (BCQ). To improve quantization accuracy without static expert-level mixed-precision allocation, BMoEQ incorporates saliency-aware error reconstruction, assigning an additional bit plane to a small subset of salient weights while maintaining a common base bit-width across experts. At runtime, online importance scores guide the selective omission of bit planes for less important activated experts, reducing memory traffic and computation without removing their contributions entirely. A single stored representation thus supports multiple execution precisions without requantization or separate lower-precision copies, while preserving access to each expert’s full stored precision. We further develop hardware-efficient GPU kernels that selectively load and process the required bit planes and fuse the salient-weight correction path to minimize latency overhead. Experiments show that the saliency-aware quantization in BMoEQ achieves superior overall accuracy in both 2-bit and 3-bit settings, outperforming expert-level mixed-precision baselines at comparable effective bit-widths. With runtime precision adaptation and optimized GPU kernels, BMoEQ achieves up to 6.4 speedup over the FP16 cuBLAS baseline.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
mixture-of-experts, quantization, dynamic multi-precision, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kcYrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BMoEQ, a Binary-Coded Quantization framework for Mixture-of-Experts (MoE) large language models, aiming to achieve ultra-low-bit weight quantization and runtime-adaptive precision execution. First, the authors employ a BCQ bit-plane representation to quantize expert weights with a uniform base precision. They then introduce joint saliency estimation across the Up, Gate, and Down projections to assign an additional correction bit-plane to selected important channels, while applying offline channel permutation to improve memory access efficiency. Next, the authors propose Runtime-Adaptive Mixed-Precision (RMP), which dynamically selects the execution precision of each expert based on routing scores and token hidden-state norms. The authors further develop GPU kernels that integrate BCQ computation with salient-channel correction. Experiments on Mixtral, Qwen1.5-MoE, and Qwen3-30B-A3B evaluate model accuracy, performance on challenging generation tasks, and practical inference efficiency.

Critical strengths

Strength 1: The paper combines BCQ bit-plane representations with runtime-adaptive precision selection for MoE models, enabling a single quantized weight representation to support different execution precisions without storing multiple quantized model copies. Together with saliency-aware reconstruction and fused GPU kernels, the proposed framework offers a practical approach to low-bit MoE inference with potential deployment benefits.

Strength 2: The paper provides relatively comprehensive experimental evaluations, covering not only perplexity, zero-shot benchmarks, and MMLU, but also GSM8K, HumanEval, and MBPP. The authors further report matched effective-BPW comparisons, GPU kernel performance, and end-to-end inference results. In the 2-bit setting, BMoEQ demonstrates strong accuracy preservation and practical inference speedups.

Critical weaknesses

Weakness 1: I have concerns regarding the independent methodological novelty of BMoEQ. AnyBCQ [1] has already introduced BCQ bit-plane-based multi-precision execution and specialized GPU kernels, while MoBiQuant [2] explores token-adaptive precision routing. SERQ [3] also investigates saliency-aware error reconstruction and offline weight permutation. Although BMoEQ extends these ideas to MoE architectures, the paper does not sufficiently distinguish its new contributions from existing methods, particularly lacking targeted experiments that demonstrate the independent benefits of its MoE-specific designs.

Weakness 2: BMoEQ estimates expert importance using the product of the routing score and the input hidden-state norm. However, for the same token within a given layer, the hidden-state norm is identical across all routed experts and therefore does not change their relative ranking determined by routing scores. Consequently, this strategy is closely related to router-score-based selection with a token-dependent threshold, rather than directly estimating each expert's actual output contribution or quantization sensitivity.

Weakness 3: Although BMoEQ reports up to 6.4× kernel speedup and 2.58× end-to-end speedup, much of the improvement comes from low-bit BCQ execution rather than runtime adaptation itself. For example, on Mixtral E2, RMP increases throughput only from 125.88 to 131.60 tokens/s, while introducing some accuracy degradation. Furthermore, end-to-end performance is evaluated primarily on a single model with batch size 1, without sufficient analysis of dynamic precision distributions and inference efficiency across different batch sizes, models, and task domains. Therefore, the additional practical benefits of RMP remain insufficiently demonstrated.

[1] Gunho Park et al. "AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs." ICLR 2026. https://arxiv.org/abs/2510.10467

[2] Dongwei Wang et al. "MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Elastic LLMs." arXiv preprint arXiv:2602.20191, 2026. https://arxiv.org/abs/2602.20191

[3] Yeonsik Park, Hyeonseong Kim, and Seungkyu Choi. "SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization." arXiv preprint arXiv:2603.08185, 2026. https://arxiv.org/abs/2603.08185

Questions for authors

Q1: AnyBCQ [1] has already introduced multi-precision inference using BCQ bit-planes and specialized GPU kernels, while MoBiQuant [2] explores token-adaptive precision routing. Could the authors clarify the key technical contributions of BMoEQ beyond these existing methods and provide targeted experiments to demonstrate the independent benefits of its MoE-specific designs?

Q2: For the same token, the hidden-state norm does not change the relative ranking of experts determined by their routing scores. Could the authors explain the additional benefit of incorporating this term and compare routing-score-only, routing score multiplied by input norm, fixed-ratio, and random precision selection under matched active BPW budgets?

Q3: On Mixtral E2, RMP increases end-to-end throughput from 125.88 to 131.60 tokens/s, representing a relatively modest additional improvement. Could the authors provide inference efficiency results across different batch sizes, input lengths, and MoE architectures, and separately quantify the speedup attributable to RMP compared with standard BCQ execution?

Q4: Although the paper provides relatively comprehensive inference efficiency results, it lacks a comparison of offline quantization time. In the supplementary code, act_aware_quant_ER_moe_model defaults to 5,000 block reconstruction iterations per quantized layer and uses Adam to optimize BCQ scaling parameters. Could the authors confirm whether these settings were used in all main experiments and report the total quantization time of BMoEQ, GPTQ, BCQ, PMQ, and GEMQ under identical hardware conditions? Additionally, could the authors break down the computational costs of activation calibration, joint saliency estimation, channel permutation, BCQ optimization, and block reconstruction to provide a more complete assessment of the proposed method's deployment overhead?

[1] Gunho Park et al. "AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs." ICLR 2026. https://arxiv.org/abs/2510.10467

[2] Dongwei Wang et al. "MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Elastic LLMs." arXiv preprint arXiv:2602.20191, 2026. https://arxiv.org/abs/2602.20191

7 comments

Regarding W1 & Q1 (1)


We acknowledge that BCQ-based multi-precision execution, token-adaptive precision selection, and saliency-aware error reconstruction have each been explored in prior work. BMoEQ’s distinction lies in jointly designing the precision representation, saliency correction, and runtime execution around the structural properties of MoE and the requirements of efficient GPU inference, rather than simply applying these techniques to MoE. We have added a related-work discussion in Sec. 2 of the revised manuscript and clarify the distinctions below.

AnyBCQ supports multiple precisions within a single BCQ representation but uses separate scaling factors for each precision. The precision selected according to a request’s requirements remains fixed throughout that request. BMoEQ instead starts from a single base precision and dynamically adjusts execution precision by selectively skipping bit-planes for each token–expert pair. To mitigate severe accuracy degradation at ultra-low precision, it also introduces binary correction based on joint saliency and integrates this correction into the existing BCQ LUT computation.

MoBiQuant targets dense LLMs and uses a separately trained token-aware router to select residual bit slices according to token sensitivity. BMoEQ instead uses the existing MoE router’s gating scores and the input norm to determine token–expert-level precision without additional training. It can assign different bit widths to experts processing the same token and selectively skips only the final BCQ bit-plane, reducing memory access and computation while retaining each selected expert’s contribution.

SERQ performs offline weight permutation and low-rank compensation based on input-channel saliency within individual linear layers. In MoE, however, different input-channel permutations across experts may not be fully absorbed into the weights, potentially requiring routed token activations to be reordered during inference. BMoEQ instead estimates joint saliency along the intermediate dimension shared by the up, gate, and down projections, then exploits permutation symmetry to jointly reorder all three weight matrices. This preserves the original expert function and routing while placing salient channels in a contiguous region. As a result, the correction requires no runtime activation reordering, and its overhead is limited to the additional binary correction integrated into the existing BCQ LUT kernel.

Regarding W1 & Q1 (2)


BMoEQ combines two complementary components: SABCQ preserves accuracy under ultra-low-bit quantization, while RMP reduces execution cost through token–expert-specific precision selection.

To distinguish SABCQ’s benefit from that of additional storage, we compare it with asymmetric BCQ at matched effective BPW, including the correction plane and all quantization metadata. As shown below, SABCQ achieves substantially higher accuracy in the 2-bit setting under this matched budget. This supports allocating the correction budget according to joint saliency across an expert’s up, gate, and down projections. The selected intermediate channels share a coordinated memory layout, allowing the additional binary correction to be integrated into the existing BCQ LUT kernel.

Our kernel benchmarks further show that RMP’s savings exceed the overhead introduced by this correction, yielding lower GEMV latency than the optimized BCQ baseline. RMP selectively skips bit-plane accesses and computation according to runtime importance, while SABCQ compensates for the accuracy loss associated with ultra-low-bit execution. BMoEQ’s contribution therefore lies in coupling targeted accuracy correction with adaptive execution within a single BCQ representation and kernel path, improving both accuracy and GEMV efficiency over the underlying BCQ baseline.

Model#BitsMethodWiki (↓)0-shot₇ (↑)MMLU-AVG (↑)GSM8K (↑)HumanEval pass@1 (↑)MBPP pass@1 (↑)
Mixtral-8x7B-v0.1BF16baseline3.8474.5668.0159.5935.3748.40
E3BCQ (asym)4.3073.3166.1853.1532.9344.20
BMoEQ4.2674.0466.5155.1228.6640.20
E2BCQ (asym)6.0635.4623.600.000.000.00
BMoEQ5.2471.1559.2237.1523.1723.40
Qwen1.5-MoE-A2.7BBF16baseline7.2266.0860.8216.7633.5439.20
E3BCQ (asym)7.8964.5158.2022.9028.0533.00
BMoEQ7.7064.9858.3220.3227.4435.20
E2BCQ (asym)13.1359.6042.503.183.050.20
BMoEQ9.5562.2053.0520.5517.0720.20
Qwen3-30B-A3BBF16baseline8.7071.0177.8089.4631.7171.20
E3BCQ (asym)9.4869.6576.0089.1643.9070.80
BMoEQ9.3170.4476.6388.7047.5671.60
E2BCQ (asym)12.3368.3471.5183.0243.9064.60
BMoEQ11.4667.9672.5584.4631.1066.40

Regarding W2 & Q2


RMP computes its runtime importance score as si(l)=ri(l)⋅∥x(l)∥2s_i^{(l)} = r_i^{(l)} \cdot \lVert \mathbf{x}^{(l)} \rVert_2. For a given input token during decoding, the input L2 norm is indeed shared across all selected experts. However, the purpose of this term is to limit perturbations in the aggregated MoE layer output, rather than to distinguish individual experts within the same token.

The precision-selection rule can be rewritten as ri(l)≥τ/∥x(l)∥2r_i^{(l)} \ge \tau / \lVert \mathbf{x}^{(l)} \rVert_2. This makes clear that the input norm effectively adjusts the routing-score threshold according to the current token at each decoding step. For example, a token with a larger input norm has a lower effective threshold for high-precision execution. An expert that would be assigned lower precision for a token with a smaller norm may therefore be selected for higher precision at this step. Thus, although the input norm does not change the ranking of experts within a token, it changes their precision assignments and consequently the aggregated MoE output.

The table below compares WikiText-2 perplexity across low-precision ratios for Random, Fixed Top-k (fixed-ratio), Ours without the L2 norm (routing score only), and Ours (routing score × L2 norm). At E3, our full score achieves the lowest perplexity across all evaluated low-precision ratios. At E2, it provides particularly large gains over the alternatives at high low-precision ratios. These results show that incorporating the input norm improves the accuracy–precision trade-off and makes a meaningful contribution to overall model performance. These results are reported in Tab. 6 of the revised manuscript.

Model#BitsScore Method12.5%25.0%37.5%50.0%62.5%75.0%87.5%
Qwen3-30B-A3BE3Random10.7913.5017.4624.0434.9158.25113.70
Fixed Top-k9.409.549.7610.1310.7712.0415.49
Ours w/o ℓ29.339.379.419.499.649.8410.11
Ours w/ ℓ29.339.359.399.479.629.8310.06
E2Random12.3313.8115.8319.0723.6231.4744.65
Fixed Top-k11.5311.6311.8012.0912.5913.5415.79
Ours w/o ℓ211.5511.6711.8512.1512.6413.4615.17
Ours w/ ℓ211.5411.6411.8212.1112.5713.2814.63

Regarding W3 & Q3


We acknowledge that the end-to-end throughput gain in the Mixtral E2 setting you highlighted—from 125.88 to 131.60 tokens/s—is modest. However, BCQ alone suffers substantial accuracy degradation at 2-bit precision. The benefit of BMoEQ is that it mitigates this degradation through saliency-aware correction while using RMP to achieve higher throughput than baseline BCQ. We therefore view its contribution in terms of the combined accuracy–efficiency trade-off.

Our primary contribution is not the optimization of the underlying BCQ kernel itself, but we recognize that the original kernel-level evaluation was insufficient and apologize for this omission. To address it, we conducted additional GEMV and GEMM experiments, using the relatively large Mixtral model and the smaller Qwen1.5-MoE-A2.7B model. We compared GEMQ, which fuses dequantization and matrix multiplication into a single kernel, with the baseline BCQ LUT kernel, BMoEQ without RMP, and BMoEQ with RMP. For GEMV, we used optimized kernels for all methods. Since the optimized BCQ LUT implementation supports only batch-size-one GEMV, the GEMM measurements for BCQ and BMoEQ include the combined latency of a dequantization kernel followed by cuBLAS GEMM. For GEMQ, we used its custom GEMM kernel.

Across all evaluated models and precision settings, BMoEQ with RMP achieved the lowest GEMV latency, providing up to a 1.23× kernel speedup over BMoEQ without RMP. This comparison directly isolates the additional acceleration from RMP. For GEMM, BCQ and BMoEQ use the highly optimized cuBLAS implementation, with only a small additional overhead from dequantizing the weights and writing them back to memory before matrix multiplication.

Model#BitsMethodGEMV (μs)GEMM (μs)
Mixtral-8x7B-v0.1BF16baseline223.365915.7
E3GEMQ88.0719540.0
BCQ53.825801.4
BMoEQ58.075808.9
BMoEQ*51.665710.6
E2GEMQ76.5019840.0
BCQ39.775736.1
BMoEQ43.055750.1
BMoEQ*34.895652.6
Qwen1.5-MoE-A2.7BBF16baseline19.07283.1
E3GEMQ9.811098.1
BCQ10.47293.0
BMoEQ10.96289.6
BMoEQ*10.07287.8
E2GEMQ8.761114.5
BCQ8.85291.3
BMoEQ9.47288.4
BMoEQ*8.50286.4
Reviewer_5uMHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BMoEQ, a weight-only method for quantizing MoE expert weights to 2 or 3 bits. It builds on binary-coded quantization (BCQ) with a uniform base bit-width for all experts, and adds one extra bit plane to a small set of salient intermediate channels. These channels are grouped together by an offline permutation. At inference time, Runtime-Adaptive Mixed-Precision (RMP) scores each activated expert by its routing score times the input norm, and lets low-scoring experts skip their last bit plane. Fused GPU kernels load only the bit planes that are needed.

Critical strengths

The design is practical and hardware-friendly. The extra bit plane reuses the BCQ computation path, and the permutation is applied offline. At 2 bits, where existing MoE quantization methods degrade the most, the method clearly improves over uniform baselines and matches or beats expert-level mixed-precision methods at the same storage cost on general-domain metrics.

Critical weaknesses
  1. The novelty is unclear because prior work is not discussed. The paper has no related work section, yet the core ideas of RMP and the salient bit plane are already covered by existing work. This includes router-guided per-token expert precision (HOBBIT, Bandwidth-Efficient Adaptive MoE, DyMoE), dynamic MoE quantization (DynaMo, DynaExq), input-adaptive precision on nested or residual bit planes (DP-LLM, MoBiQuant, AnyBCQ), and extra binary residuals for salient weights (BiLLM, MoBiE). The authors should state clearly what is new and support it with experiments.
  2. The experimental evidence is not strong enough. In Table 2, several quantized models score higher than BF16, and some mixed-precision baselines almost fail on code tasks while GPTQ at the same bit-width does not. This suggests a problem with the evaluation setup, and no variance is reported. The baseline comparison is also incomplete: PMQ and GEMQ are allowed to use 1-bit experts, the matched-bit comparison leaves out code tasks, and important baselines (the online dynamic pruning in MC, MxMoE, and dynamic expert skipping) are missing.
Questions for authors
  1. What evaluation settings were used for GSM8K, HumanEval and MBPP, and why do some quantized models score higher than BF16?
  2. How do PMQ and GEMQ perform when limited to {2, 3, 4} bits, including on code tasks?
  3. What is the actual low-precision ratio of RMP on each task?
  4. Is x in the RMP score taken before or after RMSNorm? Which bit plane is skipped, and are the scaling factors re-optimized for the lower precision?
7 comments

Regarding W1 (1)


Thank you for raising this point. We acknowledge that the basic concepts of runtime-adaptive precision and saliency-aware quantization have been explored in prior work. BMoEQ combines MoE-specific saliency correction with token–expert-level precision adaptation in a unified BCQ framework. We have added a related-work discussion in Sec. 2 of the revised manuscript and clarify the distinctions below.

Runtime-Adaptive MoE Quantization HOBBIT, Bandwidth-Efficient Adaptive MoE, and DyMoE primarily address expert offloading and data transfer: HOBBIT loads lower-precision experts on cache misses; Bandwidth-Efficient Adaptive MoE selectively applies low-rank compensation based on routing scores; and DyMoE combines dynamic mixed precision with prefetching. BMoEQ targets GPU-resident experts, reducing weight traffic and arithmetic operations by executing only the required bit-planes, without offloading or a separate low-rank compensation path.

Flexible Multi-Precision Representation DP-LLM selects input-dependent layer precision, while MoBiQuant selects residual bit slices according to token sensitivity, both targeting dense LLMs. BMoEQ additionally uses MoE routing information to assign different precisions to experts activated by the same token. AnyBCQ shares bit-planes across precision levels with separate scale factors and supports LUT-based execution at a precision selected for each request. BMoEQ dynamically selects precision for individual tokens and experts within a request. It also incorporates saliency-aware binary correction into the existing BCQ LUT computation, preserving accuracy in ultra-low-bit MoE experts with limited correction overhead.

Saliency-Aware Bit Allocation BiLLM protects salient weights through binary residual approximation within individual weight matrices. BMoEQ estimates joint saliency across the intermediate channels shared by an expert’s up, gate, and down projections. Their permutation symmetry enables coordinated offline reordering that places salient channels contiguously, allowing binary correction within the BCQ kernel without runtime activation reordering or a separate compensation matrix multiplication. MoBiE addresses cross-expert redundancy, weight importance, and routing distortion through joint SVD decomposition and gradient- and Hessian-based importance estimation. BMoEQ focuses on correcting shared intermediate channels within each expert, with correction fused into the existing LUT computation. SABCQ’s contribution thus lies in jointly designing saliency selection across the three projections, memory layout, and correction execution to preserve accuracy at low runtime cost.

Regarding W1 (2)


BMoEQ’s contribution lies in jointly addressing accuracy preservation and execution efficiency in ultra-low-bit MoE inference through SABCQ and RMP. The comparison below isolates SABCQ’s accuracy benefit by matching its storage budget with asymmetric BCQ, accounting for the correction plane and all quantization metadata, including scales and offsets. We have added these results in Tables 3 and 4 of the revised manuscript.

At the same effective BPW, SABCQ substantially improves accuracy over BCQ in the 2-bit setting, demonstrating that its gains extend beyond the benefit of additional storage. Its joint saliency selection across an expert’s up, gate, and down projections concentrates the limited correction budget on shared intermediate channels, while the coordinated memory layout enables correction within the existing BCQ LUT kernel.

Separately, our kernel results show that RMP more than offsets the overhead of SABCQ’s additional binary correction, allowing BMoEQ to achieve lower GEMV latency than the optimized BCQ baseline. By selectively accessing and computing bit-planes for each token–expert pair, BMoEQ combines substantial ultra-low-bit accuracy recovery with faster execution. This joint design enables accuracy correction without sacrificing—and ultimately improving—the efficiency of the underlying BCQ kernel.

Model#BitsMethodWiki (↓)0-shot₇ (↑)MMLU-AVG (↑)GSM8K (↑)HumanEval pass@1 (↑)MBPP pass@1 (↑)
Mixtral-8x7B-v0.1BF16baseline3.8474.5668.0159.5935.3748.40
E3BCQ (asym)4.3073.3166.1853.1532.9344.20
BMoEQ4.2674.0466.5155.1228.6640.20
E2BCQ (asym)6.0635.4623.600.000.000.00
BMoEQ5.2471.1559.2237.1523.1723.40
Qwen1.5-MoE-A2.7BBF16baseline7.2266.0860.8216.7633.5439.20
E3BCQ (asym)7.8964.5158.2022.9028.0533.00
BMoEQ7.7064.9858.3220.3227.4435.20
E2BCQ (asym)13.1359.6042.503.183.050.20
BMoEQ9.5562.2053.0520.5517.0720.20
Qwen3-30B-A3BBF16baseline8.7071.0177.8089.4631.7171.20
E3BCQ (asym)9.4869.6576.0089.1643.9070.80
BMoEQ9.3170.4476.6388.7047.5671.60
E2BCQ (asym)12.3368.3471.5183.0243.9064.60
BMoEQ11.4667.9672.5584.4631.1066.40

Regarding W2(1) & Q1


Experimental settings. For each baseline, we followed the quantization settings provided by the corresponding method. All generation benchmarks were evaluated using the same version of lm-evaluation-harness (v0.4.12). Within each benchmark, we used identical evaluation data, few-shot counts, maximum generation lengths, and scoring procedures across all methods. The detailed settings are listed below.

BenchmarkSplit / Number of examplesFew-shotMaximum generated tokensMetric
GSM8Ktest / 1,3195256Exact match
HumanEvaltest / 16401,024Greedy pass@1
MBPP, fulltest / 5003256Greedy pass@1

Quantized models outperforming the BF16 baseline. This behavior is particularly noticeable on GSM8K for Qwen1.5-MoE and HumanEval for Qwen3, and we conducted additional analyses of both cases.

For GSM8K, the reported strict-match metric extracts answers using a prescribed output pattern, so the score depends on both answer correctness and adherence to the expected format. The table below reports results under both answer-extraction rules. With flexible-extract, the BF16 baseline achieves 61.18%, outperforming all quantized models. Among the models we evaluated, this reversal in rankings between extraction rules occurred only for Qwen1.5-MoE. These results highlight why a single extraction rule should not be treated as a complete measure of the model’s underlying task performance.

Model#BitsMethodGSM8K (flexible-extract)GSM8K (strict-match)
Qwen1.5-MoE-A2.7BBF16baseline61.1816.76
E3GPTQ49.2822.14
BCQ51.5517.89
MoEQ57.5427.45
PMQ47.9945.64
GEMQ53.4523.96
BMoEQ52.9220.32
E2GPTQ1.670.15
BCQ2.961.21
MoEQ22.9721.15
PMQ10.6910.39
GEMQ19.2618.42
BMoEQ28.5120.55

For HumanEval on Qwen3, we performed five additional evaluation runs for each method. Each method produced the same pass@1 score across all five runs. This is consistent with our fixed greedy-decoding setup, in which the generated Python code is executed against the benchmark’s tests to determine whether each solution passes or fails. We also inspected the model response logs. Compared with BF16, BCQ solved 38 additional problems but failed on 5 problems that BF16 solved. This shows that the improvement reflects changes in which problems were solved. Quantization can change the generated token sequence, and those changes can lead to more test-passing solutions on a particular benchmark. Thus, quantization does not necessarily reduce code-generation accuracy relative to BF16 in every setting.

Reviewer_7j9zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BMoEQ, a post-training weight-only quantization method for MoE LLMs based on binary-coded quantization (BCQ). It introduces two key techniques: (1) Saliency-Aware BCQ, which allocates an additional bit-plane to salient intermediate channels identified through quantization error and activation statistics, and (2) Runtime-Adaptive Mixed-Precision (RMP), which dynamically skips the final bit-plane of less important experts based on routing weights and input activation norms. The authors further develop fused LUT-based CUDA kernels for efficient inference.

Critical strengths
  1. The introduction of BCQ-based quantization to MoE LLMs is novel and effective, and the motivation for runtime-adaptive mixed-precision quantization is well justified.

  2. Experimental results show that BCQ-based non-uniform quantization outperforms existing INT quantization methods across zero-shot, math, and coding benchmarks in terms of both accuracy and speed (particularly at 2-bit), making it a promising alternative to conventional INT quantization schemes for MoE LLMs.

Critical weaknesses
  1. BMoEQ largely extends AnyBCQ to MoE LLMs. Its two main additions, saliency-aware bit allocation and runtime-adaptive precision, are relatively straightforward extensions. The former resembles existing channel-wise mixed-precision techniques (e.g., BiLLM, OWQ, and SliM-LLM), while the latter follows a similar idea to "Not All Experts Are Equal" but drops a single bit-plane instead of skipping the entire expert.

  2. Tables 1 and 2 suggest that much of BMoEQ's advantage over INT quantization methods comes from the BCQ base model, which is already competitive with or superior to these baselines. The additional gains from saliency-aware allocation appear marginal and are sometimes negative (e.g., HumanEval on Qwen3 E3). Moreover, the saliency-augmented models use more bits than plain BCQ, making the comparison unfair. A comparison under matched bit budgets is needed to demonstrate the effectiveness of the proposed saliency-aware allocation.

  3. The design choices and practical benefits of RMP are not sufficiently justified. First, the authors use the input activation norm as a lightweight proxy for expert output magnitude (Eq. 11), but this assumption lacks empirical validation. Second, the threshold relies on a heuristic layer-wise schedule calibrated on a specific dataset, without sufficient sensitivity analysis. As a result, the effective precision depends on the input distribution, and its generalizability across different data domains remains unverified.

Questions for authors

One aspect I find particularly interesting is the reported speed advantage of LUT-based BCQ over conventional INT quantization. As I am less familiar with BCQ-based inference, I find these results somewhat surprising, especially given the availability of highly optimized INT kernels. I would appreciate further clarification on what drives these speedups. My specific questions are as follows:

  1. In Table 1, GPTQ, MoEQuant, and PMQ achieve only 0.6× FP16 throughput, while GEMQ reaches 2.54×/2.92×. Given that all these methods use INT quantization, such a large performance gap is somewhat unexpected. Could the authors clarify which kernels were used for each baseline?

  2. To better understand the reported speed advantages of BCQ over INT quantization, could the authors provide kernel-level benchmarks comparing LUT-based BCQ with optimized INT implementations (e.g., Marlin)? In particular, comparisons of both linear kernels (GEMV/GEMM) and fused MoE kernels under matched settings would help clarify whether the speedups come from the BCQ representation itself or differences in kernel implementations.

  3. The quantization time does not appear to be reported. Could the authors provide the time required to quantize models such as Mixtral-8×7B and Qwen3-30B-A3B using BMoEQ?

7 comments

Regarding W1


Thank you for raising this point. We acknowledge that the original manuscript insufficiently explained BMoEQ’s relationship to prior work and have added a related-work discussion in Sec. 2 of the revised manuscript. Our contribution lies in jointly designing established saliency-aware allocation and bit-plane execution principles for ultra-low-bit MoE inference, preserving accuracy while reducing memory traffic and computation through token–expert-level precision adaptation. Below, we detail the distinctions from each of the works you mentioned.

First, BMoEQ differs in both saliency selection and correction execution. BiLLM applies binary residuals to salient weights within individual matrices, OWQ retains sensitive input-channel columns at high precision, and SliM-LLM allocates bits according to group-level saliency. BMoEQ jointly evaluates the shared intermediate channels across an expert’s up, gate, and down projections. A coordinated permutation places salient channels contiguously while preserving the expert’s input and output coordinates, avoiding runtime activation reordering. Additionally, their binary correction is fused into the BCQ kernel using the existing LUT, jointly addressing channel selection, memory layout, and correction overhead.

Second, BMoEQ reduces precision while retaining all selected experts. Not All Experts Are Equal skips experts based on relative routing weights, removing their contributions entirely. BMoEQ instead adjusts the precision of each selected expert: routing scores differentiate experts within a token, while the input norm adjusts the precision criterion across tokens. Reading and computing only the required BCQ bit-planes directly reduces weight traffic and LUT accumulation operations.

Finally, BMoEQ extends precision adaptation to individual token–expert computations while addressing ultra-low-bit storage. AnyBCQ shares binary bit-planes across precision levels with separately optimized scale factors, supporting request-level precision selection for dense LLMs. BMoEQ addresses both storing all experts and executing the active subset: SABCQ uses saliency-aware correction to eliminate the need for precision-specific scale factors, while RMP reduces memory traffic and computation by accessing and processing only the required bit-planes for each token–expert pair. These components jointly address the accuracy and hardware efficiency requirements of sparse MoE inference.

Regarding W2


We agree that BCQ is a strong baseline, particularly on Qwen3-30B-A3B, and accounts for part of BMoEQ’s advantage over INT-based methods. However, Saliency-aware BCQ provides substantial additional gains when ultra-low-bit BCQ incurs severe accuracy loss. For Mixtral-8×7B in E2, BCQ reduces zero-shot accuracy from 74.56 to 41.98 and MMLU accuracy from 68.01 to 39.19 relative to BF16. Saliency-aware BCQ recovers these scores from 41.98 to 71.15 and from 39.19 to 59.22, respectively, recovering a substantial portion of the lost accuracy. We nevertheless acknowledge the HumanEval degradation on Qwen3 in E3 and the smaller gains where BCQ already preserves accuracy well. Our claim is thus substantial accuracy recovery in challenging ultra-low-bit settings, rather than consistent improvements across every setting.

We also agree that the original comparison uses different storage budgets and therefore cannot isolate the benefit of saliency-based allocation from that of additional bits. To address this, we compare BMoEQ with asymmetric BCQ at the same effective bit-width, using zero-point parameters in BCQ that match the storage overhead of our additional correction bits. The results are reported in the table below. In E3, where most baselines preserve accuracy well, BMoEQ does not consistently outperform BCQ across all settings. In E2, however, the gains are more pronounced even under matched storage budgets, supporting the effectiveness of saliency-aware correction beyond the benefit of additional storage alone. These results are included in Tables 3 and 4 of the revised manuscript.

Model#BitsMethodWiki (↓)0-shot₇ (↑)MMLU-AVG (↑)GSM8K (↑)HumanEval pass@1 (↑)MBPP pass@1 (↑)
Mixtral-8x7B-v0.1BF16baseline3.8474.5668.0159.5935.3748.40
E3BCQ (asym)4.3073.3166.1853.1532.9344.20
BMoEQ4.2674.0466.5155.1228.6640.20
E2BCQ (asym)6.0635.4623.600.000.000.00
BMoEQ5.2471.1559.2237.1523.1723.40
Qwen1.5-MoE-A2.7BBF16baseline7.2266.0860.8216.7633.5439.20
E3BCQ (asym)7.8964.5158.2022.9028.0533.00
BMoEQ7.7064.9858.3220.3227.4435.20
E2BCQ (asym)13.1359.6042.503.183.050.20
BMoEQ9.5562.2053.0520.5517.0720.20
Qwen3-30B-A3BBF16baseline8.7071.0177.8089.4631.7171.20
E3BCQ (asym)9.4869.6576.0089.1643.9070.80
BMoEQ9.3170.4476.6388.7047.5671.60
E2BCQ (asym)12.3368.3471.5183.0243.9064.60
BMoEQ11.4667.9672.5584.4631.1066.40

Regarding W3 (1)


We agree with the reviewer that the original manuscript did not include sufficient ablation studies to substantiate the effectiveness of RMP.

1. Additional evaluation of the proxy for expert output magnitude

To account for token-level importance at runtime, RMP uses the input activation norm as a proxy for expert output magnitude. We evaluate this proxy on Mixtral and Qwen across three datasets by measuring the Spearman rank correlation between the input activation norm and the corresponding output norm obtained through full expert evaluation. This analysis uses 524,288 and 2,097,152 token–expert pairs per layer for Mixtral and Qwen, respectively. The results are presented in Fig. 6 and detailed in Appendix A.3 of the revised manuscript. As shown in Fig. 6, the input and output norms are strongly correlated across most layers, with weaker correlations limited to a few later layers. These findings provide empirical support for using the input norm as a proxy for expert output magnitude, enabling token-aware precision allocation without computing expert outputs in advance.

Regarding W3 (2)


2. Additional ablation study of layer-wise score scheduling

We thank the reviewer for highlighting the need to validate the threshold schedule and its generalization across input distributions. Our layer-wise schedule is motivated by the observation that deeper layers are more sensitive to low-precision execution, as shown in Appendix A.4 in the manuscript. The hyperparameter (R) controls the strength of this adjustment. Therefore, we have added a sensitivity analysis across three MoE models and two quantization budgets, reported in the first table below. Varying R from 0.1 to 0.3 changes the low-precision ratio by approximately 5–6 percentage points, while perplexity varies smoothly within a narrow range: at most 0.05 for E3 and 0.18 for E2. These results indicate that performance is relatively insensitive to R within the evaluated range.

#BitsRRatio (Mixtral)PPL (Mixtral)Ratio (Qwen1.5)PPL (Qwen1.5)Ratio (Qwen3)PPL (Qwen3)
E30.1045.86%4.3246.31%7.7145.44%9.43
0.1544.55%4.3244.96%7.7144.06%9.41
0.2043.34%4.3143.41%7.7142.09%9.40
0.2541.97%4.3141.87%7.7141.07%9.39
0.3040.68%4.3040.21%7.7139.49%9.38
E20.1048.05%5.5948.55%9.7046.55%11.98
0.1546.63%5.5647.16%9.6945.21%11.93
0.2045.21%5.5345.55%9.6843.30%11.88
0.2543.80%5.5043.95%9.6742.31%11.84
0.3042.39%5.4742.24%9.6640.77%11.80

To examine behavior across domains, we also report task-specific low-precision ratios in the second table below, keeping the calibration-derived thresholds fixed without task-specific re-tuning. The effective precision varies with the input distribution, as expected for runtime-adaptive allocation. Together with our previously reported task accuracy results, these measurements show that the thresholds support lower-precision execution across the evaluated models and tasks without substantial accuracy degradation. This provides empirical evidence of their applicability beyond the calibration distribution. Both analyses are included in the revised manuscript at Tables 7 and 9.

ModelWiki0-shot₇MMLU-AVGGSM8KHumanEval pass@1MBPP pass@1
Mixtral-8x7B-v0.145.21%45.14%45.72%46.24%47.32%47.29%
Qwen1.5-MoE-A2.7B45.55%42.49%44.83%51.09%75.34%56.22%
Qwen3-30B-A3B42.16%41.58%41.23%46.82%53.47%49.65%

Regarding Q1


We apologize for not clearly specifying the kernels used to measure each baseline’s speed.

For GEMQ, we used the authors’ custom kernel, which fuses weight dequantization and matrix multiplication and achieved speedups of 2.54$\times$ and 2.92$\times$ over BF16 in E3 and E2, respectively. For GPTQ, MoEQuant, and PMQ, we used the HQQ-based dequantization kernel adopted in the MCMoE implementation, followed by cuBLAS matrix multiplication. This execution path reconstructs the compressed weights before performing matrix multiplication in a separate kernel, incurring additional memory traffic for the reconstructed weights and an extra kernel launch. Consequently, INT weight storage alone does not guarantee faster execution than BF16. We executed both kernels sequentially on the same CUDA stream and measured their combined execution time using CUDA events, ensuring that the reported latency includes dequantization.

We will clarify the kernel source, fusion status, and timing scope for each baseline in Sec. 5.1 of the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary of the Rebuttal Discussion

First, we would like to sincerely thank all the reviewers for their constructive feedback, which has helped us improve our manuscript. To help the Area Chair with the final assessment, we would like to conclude the rebuttal discussion by briefly summarizing our key contributions and the major updates we have made to the manuscript.

 

Main Contributions


  • We observed that expert importance estimated from offline calibration may differ across downstream tasks. Based on this observation, we propose Saliency-aware BCQ (SABCQ), which achieves strong 2-bit quantization performance and generalization even with uniform precision.
  • To address the mismatch in expert importance between calibration and deployment, we propose Runtime-adaptive Mixed Precision (RMP), which uses a runtime score to dynamically adjust computation precision, improving computational efficiency while maintaining stable accuracy.

 

Major Updates


  • Related Works (Reviewers 7j9z, 5uMH, kcYr): We expanded Sec. 2 of the revised manuscript to cover relevant studies on MoE quantization, runtime-adaptive mixed precision, and saliency-aware bit allocation. We also clarified the specific differences between BMoEQ and the methods mentioned by the reviewers, highlighting the novelty of our proposed approach.

  • Additional Ablations on RMP (Reviewers 7j9z, kcYr): We added detailed ablation studies in Sec. 5.4 of the revised manuscript to validate the effectiveness of RMP. First, we compared different precision selection strategies and ablated the two components of our runtime score. The results demonstrate the importance of considering both the input norm and routing score. We also provide empirical evidence in Fig. 6 supporting the use of the input norm as a proxy for the expert output norm. To further evaluate robustness and generalization across models and tasks, Tab. 7 examines sensitivity to the hyperparameter (R), while Tab. 9 shows that RMP maintains a low-precision computation ratio above 40% across all evaluated models and tasks. These results support the robustness of RMP and its favorable accuracy–latency trade-off.

  • Additional Hardware Evaluations (Reviewers 7j9z, kcYr): We expanded our GPU performance evaluation beyond Mixtral to include the smaller Qwen1.5-MoE-A2.7B model. We also added comparisons with Marlin, a highly optimized INT kernel, in the 4-bit setting, and extended our kernel benchmarks from GEMV to GEMM. These additional experiments cover a wider range of hardware configurations and help clarify where the practical hardware performance gains of BMoEQ come from.

  • Matched Comparison with BCQ (Reviewer 7j9z): In addition to static mixed-precision baselines, we added BCQ to the matched-budget comparisons in Tab. 3 of the revised manuscript. The results show that BMoEQ's strong 2-bit quantization performance does not simply come from increasing the bit budget over BCQ. Instead, the substantial accuracy improvements are achieved through our proposed joint saliency-aware design.

  • Additional Matched Comparisons with Static Mixed-Precision Methods (Reviewer 5uMH): To provide more comprehensive comparisons under matched settings, we added experiments in the 3-bit setting where the candidate precision set for static mixed-precision methods is changed from {1, 2, 3, 4} to {2, 3, 4}, excluding the challenging 1-bit precision (Tab. 11). We also extended the evaluation to code generation tasks (Tab. 4). Even under these additional settings, the baselines still exhibit substantial accuracy degradation, whereas BMoEQ maintains strong performance. The advantages of BMoEQ are particularly pronounced in the 2-bit setting and on domain-specific tasks.

  • Additional Static Mixed-Precision Baseline (Reviewer 5uMH): We further included comparisons with MxMoE (Tab. 12), which applies mixed precision at the linear-layer level rather than the expert level. Although MxMoE uses finer-grained precision allocation, it still performs inference with fixed precision, similar to other static mixed-precision baselines. Our additional results show that MxMoE suffers substantial accuracy degradation on domain-specific tasks in the 2-bit setting, while BMoEQ maintains much stronger accuracy.

 

We sincerely thank the reviewers and Area Chair once again for their time and thoughtful consideration of our work.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.