- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes BMoEQ, a Binary-Coded Quantization framework for Mixture-of-Experts (MoE) large language models, aiming to achieve ultra-low-bit weight quantization and runtime-adaptive precision execution. First, the authors employ a BCQ bit-plane representation to quantize expert weights with a uniform base precision. They then introduce joint saliency estimation across the Up, Gate, and Down projections to assign an additional correction bit-plane to selected important channels, while applying offline channel permutation to improve memory access efficiency. Next, the authors propose Runtime-Adaptive Mixed-Precision (RMP), which dynamically selects the execution precision of each expert based on routing scores and token hidden-state norms. The authors further develop GPU kernels that integrate BCQ computation with salient-channel correction. Experiments on Mixtral, Qwen1.5-MoE, and Qwen3-30B-A3B evaluate model accuracy, performance on challenging generation tasks, and practical inference efficiency.
Critical strengths
Strength 1: The paper combines BCQ bit-plane representations with runtime-adaptive precision selection for MoE models, enabling a single quantized weight representation to support different execution precisions without storing multiple quantized model copies. Together with saliency-aware reconstruction and fused GPU kernels, the proposed framework offers a practical approach to low-bit MoE inference with potential deployment benefits.
Strength 2: The paper provides relatively comprehensive experimental evaluations, covering not only perplexity, zero-shot benchmarks, and MMLU, but also GSM8K, HumanEval, and MBPP. The authors further report matched effective-BPW comparisons, GPU kernel performance, and end-to-end inference results. In the 2-bit setting, BMoEQ demonstrates strong accuracy preservation and practical inference speedups.
Critical weaknesses
Weakness 1: I have concerns regarding the independent methodological novelty of BMoEQ. AnyBCQ [1] has already introduced BCQ bit-plane-based multi-precision execution and specialized GPU kernels, while MoBiQuant [2] explores token-adaptive precision routing. SERQ [3] also investigates saliency-aware error reconstruction and offline weight permutation. Although BMoEQ extends these ideas to MoE architectures, the paper does not sufficiently distinguish its new contributions from existing methods, particularly lacking targeted experiments that demonstrate the independent benefits of its MoE-specific designs.
Weakness 2: BMoEQ estimates expert importance using the product of the routing score and the input hidden-state norm. However, for the same token within a given layer, the hidden-state norm is identical across all routed experts and therefore does not change their relative ranking determined by routing scores. Consequently, this strategy is closely related to router-score-based selection with a token-dependent threshold, rather than directly estimating each expert's actual output contribution or quantization sensitivity.
Weakness 3: Although BMoEQ reports up to 6.4× kernel speedup and 2.58× end-to-end speedup, much of the improvement comes from low-bit BCQ execution rather than runtime adaptation itself. For example, on Mixtral E2, RMP increases throughput only from 125.88 to 131.60 tokens/s, while introducing some accuracy degradation. Furthermore, end-to-end performance is evaluated primarily on a single model with batch size 1, without sufficient analysis of dynamic precision distributions and inference efficiency across different batch sizes, models, and task domains. Therefore, the additional practical benefits of RMP remain insufficiently demonstrated.
[1] Gunho Park et al. "AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs." ICLR 2026. https://arxiv.org/abs/2510.10467
[2] Dongwei Wang et al. "MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Elastic LLMs." arXiv preprint arXiv:2602.20191, 2026. https://arxiv.org/abs/2602.20191
[3] Yeonsik Park, Hyeonseong Kim, and Seungkyu Choi. "SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization." arXiv preprint arXiv:2603.08185, 2026. https://arxiv.org/abs/2603.08185
Questions for authors
Q1: AnyBCQ [1] has already introduced multi-precision inference using BCQ bit-planes and specialized GPU kernels, while MoBiQuant [2] explores token-adaptive precision routing. Could the authors clarify the key technical contributions of BMoEQ beyond these existing methods and provide targeted experiments to demonstrate the independent benefits of its MoE-specific designs?
Q2: For the same token, the hidden-state norm does not change the relative ranking of experts determined by their routing scores. Could the authors explain the additional benefit of incorporating this term and compare routing-score-only, routing score multiplied by input norm, fixed-ratio, and random precision selection under matched active BPW budgets?
Q3: On Mixtral E2, RMP increases end-to-end throughput from 125.88 to 131.60 tokens/s, representing a relatively modest additional improvement. Could the authors provide inference efficiency results across different batch sizes, input lengths, and MoE architectures, and separately quantify the speedup attributable to RMP compared with standard BCQ execution?
Q4: Although the paper provides relatively comprehensive inference efficiency results, it lacks a comparison of offline quantization time. In the supplementary code, act_aware_quant_ER_moe_model defaults to 5,000 block reconstruction iterations per quantized layer and uses Adam to optimize BCQ scaling parameters. Could the authors confirm whether these settings were used in all main experiments and report the total quantization time of BMoEQ, GPTQ, BCQ, PMQ, and GEMQ under identical hardware conditions? Additionally, could the authors break down the computational costs of activation calibration, joint saliency estimation, channel permutation, BCQ optimization, and block reconstruction to provide a more complete assessment of the proposed method's deployment overhead?
[1] Gunho Park et al. "AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs." ICLR 2026. https://arxiv.org/abs/2510.10467
[2] Dongwei Wang et al. "MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Elastic LLMs." arXiv preprint arXiv:2602.20191, 2026. https://arxiv.org/abs/2602.20191
7 comments
Part2
Regarding W1 & Q1 (2)
BMoEQ combines two complementary components: SABCQ preserves accuracy under ultra-low-bit quantization, while RMP reduces execution cost through token–expert-specific precision selection.
To distinguish SABCQ’s benefit from that of additional storage, we compare it with asymmetric BCQ at matched effective BPW, including the correction plane and all quantization metadata. As shown below, SABCQ achieves substantially higher accuracy in the 2-bit setting under this matched budget. This supports allocating the correction budget according to joint saliency across an expert’s up, gate, and down projections. The selected intermediate channels share a coordinated memory layout, allowing the additional binary correction to be integrated into the existing BCQ LUT kernel.
Our kernel benchmarks further show that RMP’s savings exceed the overhead introduced by this correction, yielding lower GEMV latency than the optimized BCQ baseline. RMP selectively skips bit-plane accesses and computation according to runtime importance, while SABCQ compensates for the accuracy loss associated with ultra-low-bit execution. BMoEQ’s contribution therefore lies in coupling targeted accuracy correction with adaptive execution within a single BCQ representation and kernel path, improving both accuracy and GEMV efficiency over the underlying BCQ baseline.
| Model | #Bits | Method | Wiki (↓) | 0-shot₇ (↑) | MMLU-AVG (↑) | GSM8K (↑) | HumanEval pass@1 (↑) | MBPP pass@1 (↑) |
|---|---|---|---|---|---|---|---|---|
| Mixtral-8x7B-v0.1 | BF16 | baseline | 3.84 | 74.56 | 68.01 | 59.59 | 35.37 | 48.40 |
| E3 | BCQ (asym) | 4.30 | 73.31 | 66.18 | 53.15 | 32.93 | 44.20 | |
| BMoEQ | 4.26 | 74.04 | 66.51 | 55.12 | 28.66 | 40.20 | ||
| E2 | BCQ (asym) | 6.06 | 35.46 | 23.60 | 0.00 | 0.00 | 0.00 | |
| BMoEQ | 5.24 | 71.15 | 59.22 | 37.15 | 23.17 | 23.40 | ||
| Qwen1.5-MoE-A2.7B | BF16 | baseline | 7.22 | 66.08 | 60.82 | 16.76 | 33.54 | 39.20 |
| E3 | BCQ (asym) | 7.89 | 64.51 | 58.20 | 22.90 | 28.05 | 33.00 | |
| BMoEQ | 7.70 | 64.98 | 58.32 | 20.32 | 27.44 | 35.20 | ||
| E2 | BCQ (asym) | 13.13 | 59.60 | 42.50 | 3.18 | 3.05 | 0.20 | |
| BMoEQ | 9.55 | 62.20 | 53.05 | 20.55 | 17.07 | 20.20 | ||
| Qwen3-30B-A3B | BF16 | baseline | 8.70 | 71.01 | 77.80 | 89.46 | 31.71 | 71.20 |
| E3 | BCQ (asym) | 9.48 | 69.65 | 76.00 | 89.16 | 43.90 | 70.80 | |
| BMoEQ | 9.31 | 70.44 | 76.63 | 88.70 | 47.56 | 71.60 | ||
| E2 | BCQ (asym) | 12.33 | 68.34 | 71.51 | 83.02 | 43.90 | 64.60 | |
| BMoEQ | 11.46 | 67.96 | 72.55 | 84.46 | 31.10 | 66.40 |
Part3
Regarding W2 & Q2
RMP computes its runtime importance score as . For a given input token during decoding, the input L2 norm is indeed shared across all selected experts. However, the purpose of this term is to limit perturbations in the aggregated MoE layer output, rather than to distinguish individual experts within the same token.
The precision-selection rule can be rewritten as . This makes clear that the input norm effectively adjusts the routing-score threshold according to the current token at each decoding step. For example, a token with a larger input norm has a lower effective threshold for high-precision execution. An expert that would be assigned lower precision for a token with a smaller norm may therefore be selected for higher precision at this step. Thus, although the input norm does not change the ranking of experts within a token, it changes their precision assignments and consequently the aggregated MoE output.
The table below compares WikiText-2 perplexity across low-precision ratios for Random, Fixed Top-k (fixed-ratio), Ours without the L2 norm (routing score only), and Ours (routing score × L2 norm). At E3, our full score achieves the lowest perplexity across all evaluated low-precision ratios. At E2, it provides particularly large gains over the alternatives at high low-precision ratios. These results show that incorporating the input norm improves the accuracy–precision trade-off and makes a meaningful contribution to overall model performance. These results are reported in Tab. 6 of the revised manuscript.
| Model | #Bits | Score Method | 12.5% | 25.0% | 37.5% | 50.0% | 62.5% | 75.0% | 87.5% |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3-30B-A3B | E3 | Random | 10.79 | 13.50 | 17.46 | 24.04 | 34.91 | 58.25 | 113.70 |
| Fixed Top-k | 9.40 | 9.54 | 9.76 | 10.13 | 10.77 | 12.04 | 15.49 | ||
| Ours w/o ℓ2 | 9.33 | 9.37 | 9.41 | 9.49 | 9.64 | 9.84 | 10.11 | ||
| Ours w/ ℓ2 | 9.33 | 9.35 | 9.39 | 9.47 | 9.62 | 9.83 | 10.06 | ||
| E2 | Random | 12.33 | 13.81 | 15.83 | 19.07 | 23.62 | 31.47 | 44.65 | |
| Fixed Top-k | 11.53 | 11.63 | 11.80 | 12.09 | 12.59 | 13.54 | 15.79 | ||
| Ours w/o ℓ2 | 11.55 | 11.67 | 11.85 | 12.15 | 12.64 | 13.46 | 15.17 | ||
| Ours w/ ℓ2 | 11.54 | 11.64 | 11.82 | 12.11 | 12.57 | 13.28 | 14.63 |
Regarding W3 & Q3
We acknowledge that the end-to-end throughput gain in the Mixtral E2 setting you highlighted—from 125.88 to 131.60 tokens/s—is modest. However, BCQ alone suffers substantial accuracy degradation at 2-bit precision. The benefit of BMoEQ is that it mitigates this degradation through saliency-aware correction while using RMP to achieve higher throughput than baseline BCQ. We therefore view its contribution in terms of the combined accuracy–efficiency trade-off.
Our primary contribution is not the optimization of the underlying BCQ kernel itself, but we recognize that the original kernel-level evaluation was insufficient and apologize for this omission. To address it, we conducted additional GEMV and GEMM experiments, using the relatively large Mixtral model and the smaller Qwen1.5-MoE-A2.7B model. We compared GEMQ, which fuses dequantization and matrix multiplication into a single kernel, with the baseline BCQ LUT kernel, BMoEQ without RMP, and BMoEQ with RMP. For GEMV, we used optimized kernels for all methods. Since the optimized BCQ LUT implementation supports only batch-size-one GEMV, the GEMM measurements for BCQ and BMoEQ include the combined latency of a dequantization kernel followed by cuBLAS GEMM. For GEMQ, we used its custom GEMM kernel.
Across all evaluated models and precision settings, BMoEQ with RMP achieved the lowest GEMV latency, providing up to a 1.23× kernel speedup over BMoEQ without RMP. This comparison directly isolates the additional acceleration from RMP. For GEMM, BCQ and BMoEQ use the highly optimized cuBLAS implementation, with only a small additional overhead from dequantizing the weights and writing them back to memory before matrix multiplication.
| Model | #Bits | Method | GEMV (μs) | GEMM (μs) |
|---|---|---|---|---|
| Mixtral-8x7B-v0.1 | BF16 | baseline | 223.36 | 5915.7 |
| E3 | GEMQ | 88.07 | 19540.0 | |
| BCQ | 53.82 | 5801.4 | ||
| BMoEQ | 58.07 | 5808.9 | ||
| BMoEQ* | 51.66 | 5710.6 | ||
| E2 | GEMQ | 76.50 | 19840.0 | |
| BCQ | 39.77 | 5736.1 | ||
| BMoEQ | 43.05 | 5750.1 | ||
| BMoEQ* | 34.89 | 5652.6 | ||
| Qwen1.5-MoE-A2.7B | BF16 | baseline | 19.07 | 283.1 |
| E3 | GEMQ | 9.81 | 1098.1 | |
| BCQ | 10.47 | 293.0 | ||
| BMoEQ | 10.96 | 289.6 | ||
| BMoEQ* | 10.07 | 287.8 | ||
| E2 | GEMQ | 8.76 | 1114.5 | |
| BCQ | 8.85 | 291.3 | ||
| BMoEQ | 9.47 | 288.4 | ||
| BMoEQ* | 8.50 | 286.4 |
Part1
We acknowledge that BCQ-based multi-precision execution, token-adaptive precision selection, and saliency-aware error reconstruction have each been explored in prior work. BMoEQ’s distinction lies in jointly designing the precision representation, saliency correction, and runtime execution around the structural properties of MoE and the requirements of efficient GPU inference, rather than simply applying these techniques to MoE. We have added a related-work discussion in Sec. 2 of the revised manuscript and clarify the distinctions below.
AnyBCQ supports multiple precisions within a single BCQ representation but uses separate scaling factors for each precision. The precision selected according to a request’s requirements remains fixed throughout that request. BMoEQ instead starts from a single base precision and dynamically adjusts execution precision by selectively skipping bit-planes for each token–expert pair. To mitigate severe accuracy degradation at ultra-low precision, it also introduces binary correction based on joint saliency and integrates this correction into the existing BCQ LUT computation.
MoBiQuant targets dense LLMs and uses a separately trained token-aware router to select residual bit slices according to token sensitivity. BMoEQ instead uses the existing MoE router’s gating scores and the input norm to determine token–expert-level precision without additional training. It can assign different bit widths to experts processing the same token and selectively skips only the final BCQ bit-plane, reducing memory access and computation while retaining each selected expert’s contribution.
SERQ performs offline weight permutation and low-rank compensation based on input-channel saliency within individual linear layers. In MoE, however, different input-channel permutations across experts may not be fully absorbed into the weights, potentially requiring routed token activations to be reordered during inference. BMoEQ instead estimates joint saliency along the intermediate dimension shared by the up, gate, and down projections, then exploits permutation symmetry to jointly reorder all three weight matrices. This preserves the original expert function and routing while placing salient channels in a contiguous region. As a result, the correction requires no runtime activation reordering, and its overhead is limited to the additional binary correction integrated into the existing BCQ LUT kernel.