acceptodds
Under review as a conference paper at ICLR 2027

BitMoE: Post-Training Binarization for MoE-based Large Language Models

Abstract

Large language models (LLMs) with Mixture-of-Experts (MoE) architectures have achieved remarkable progress, yet their substantial weight storage and memory traffic hinder practical deployment. Weight binarization offers extreme compression, but transferring dense-model methods to MoE faces three coupled challenges: decomposition–binarization mismatch, interaction-dependent residual allocation, and quantization-induced expert-shift. To address these challenges, we propose BitMoE, a post-training binarization framework coordinating shared representation, residual capacity, and routing stability. First, Routing-Coupled Binary Factorization alternately refines higher-precision shared factors and expert-specific binary factors to reduce the reconstruction error of routed expert aggregation. Second, Conditional Expert Residual Allocation selects additional binary residual blocks by their conditional reconstruction gain per added storage bit after readapting the shared output factor, updating gains as corrections accumulate. Third, Margin-Aware Optimal Brain Routing uses boundary-aware curvature to preserve selected–unselected expert margins and mitigate expert-shift. BitMoE retains all experts and performs optimization offline under an explicit storage budget. Extensive experiments demonstrate that BitMoE consistently outperforms state-of-the-art binary baselines across multiple MoE models and benchmarks. For instance, on Qwen3-30B-A3B, BitMoE delivers 54.2% lower perplexity and improves average zero-shot accuracy by 19.71 percentage points over the strongest binary baseline, alongside an over 2 inference speedup over the BF16 model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Binarization, PTQ, MoE

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_PRjSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a three step post training binarization for mixture of experts LLMs. First, the expert weights of each projection are jointly factorized via SVD into a shared 8 bit factor and expert specific binary factors, refined against the routed aggregate output error. Second, binary residual blocks are added greedily by their conditional reconstruction gain per stored bit under a fixed storage budget. Last, the router is binarized and calibrated to preserve the margins between selected and unselected experts. Each step is ablated, and comparisons to binary and low bit PTQ baselines on multiple MoE models are shown.

Critical strengths
  • Principled design: three MoE specific binarization challenges are identified and each is addressed by a dedicated objective. The new elements are the routed aggregate loss with cross expert error terms (Eq. 5), residual selection by conditional gain per stored bit (Eq. 9), and margin preserving router calibration (Eq. 12).
  • Large and consistent gains over the binary baselines at similar resident storage, also on instruction tuned models, with ablations showing that each component contributes.
Critical weaknesses

Major:

  • Budget mismatch on the motivating metric. Tables 2 and 3 match the baselines at 1.11 BPW, but this amortizes a per layer 8 bit shared factor over all experts. On the per token active path, which motivates the paper (Section 1) and the speedup (Section 4.4), BitMoE sits at 1.72 BPW while BiLLM and ARB-LLM remain at 1.11. The headline gains are reported at unequal cost per token.
  • Baseline configuration is unreported and the GPTQ rows are hard to reconcile with the literature. No group size or quantized scope is given. MxMoE (ICML 2025, cited by the authors) reports GPTQ on DeepSeek-V2-Lite with the same calibration set at 6.18 perplexity for 3.25 bits and 8.49 for 2.25 bits, against 16.26 and 5e7 at 3 and 2 bits here. AlphaQ (arXiv 2606.04980) shows a similar gap on Qwen3-30B-A3B.

Minor:

  • MOBR is not isolated. The router is binarized although its storage is negligible, no full precision router control is given, expert shift in Table 1 is neither defined nor explained, and the cosine similarity metric of Figure 5 is not what MOBR optimizes.
  • Storage versus decode performance. The paper should either confine its claims to resident storage, where the reduction is real and large, or show how it translates into decode speed: a description of the kernel for the INT8 shared factor times binary expert product, bytes read per token, and a comparison to FP8 or NVFP4 on the same GPU. A single speedup against a BF16 reference at 42 tok/s does neither.
  • Presentation. Notation and terminology are not consistent: “shared experts” versus “shared factors”, the star denotes both teacher quantities and optima, and s^r and s^c, the “(m,1)” superscript, “planes” and the router’s “code and scale representation” are used before being defined. Figure 4 is too dense to serve as an overview. The statement “reconstruction errors are further combined through sparse routing” (Section 3.1) needs explanation. It should be stated that c_t is the always active branch on quantized inputs, what “deployed” routing weights p̄ means, and that all losses are evaluated on the requantized weights. The relation between q^(0) and q̂ should be clarified, and norm and vector notation unified throughout.
Questions for authors

Major:

  • How much of the gain over BiLLM and ARB-LLM remains at a matched active path budget?
  • Which group size and quantized scope were used for GPTQ and AWQ, and how do you reconcile the 3 bit rows with MxMoE’s numbers on the same model and calibration set?
  • Which data types do the row and column scales in Eq. 3 and the residual planes use, how are attention, always active experts, embeddings and norms treated, and are they counted in the 1.11 BPW?

Minor:

  • What does BitMoE achieve with a full precision router, and why is cosine similarity of routing probabilities the criterion for MOBR?
  • How does BitMoE compare to a ternary PTQ baseline at matched active bits, and to the cited but uncompared STBLLM, DBF, KBVQ-MoE and MxMoE?
3 comments
AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Revised Manuscript

Dear Reviewer PRjS,

Please forgive this brief follow-up. With fewer than 36 hours remaining in the author–reviewer discussion period, we wanted to kindly ask whether you have had an opportunity to review our rebuttal and the revised manuscript.

In response to your concerns, we clarified the static and active-path budget accounting and added an estimated active-budget comparison. We also corrected an issue in the AWQ/GPTQ calibration procedure and updated the corresponding results, added a BF16-router control to better assess MOBR, and provided further details on decoding performance and comparisons with related methods.

We hope these additions address your principal concerns. If you find that the new evidence resolves them, we would be sincerely grateful if you could reconsider your overall assessment. If any concern remains, even a brief indication would help us provide further clarification promptly.

We understand the demands on your time and sincerely appreciate your careful consideration.

Best regards,

The Authors

Thank you for the detailed response and the additional experiments. The manuscript has improved during the rebuttal, and some of my minor points have been clarified. The response is very extensive, with eight parts for my review alone. Since my time for this review is limited, I may have missed some details. Still, the need for such a long response suggests to me that the manuscript is not yet mature enough for acceptance, and the main concerns behind my initial assessment remain. The points below summarize these concerns, which in my view reflect conceptual shortcomings of the study.

  • Active path budget (major 1). I agree that the bitmaps and scales of the baselines should be counted. With this accounting BitMoE is more accurate at a lower active budget (about 1.7 versus 2.5 to 2.9 BPW), which strengthens the paper. However, the main result tables do not reflect this. Tables 2 and 3 report only the static BPW and still list BiLLM and ARB-LLM at 1.06 to 1.11 BPW, although the accounting of Table 7 would give them 2.5 to 2.9 BPW there as well. In addition, the storage format of the attention and dense projections of BitMoE is not stated. The reported budgets are therefore neither consistent across tables nor fully reproducible.
  • Comparison with baselines (major 2). Thank you for fixing the calibration issue and for revising the claim on 3 bit GPTQ. BitMoE improves on the binary baselines (see the previous point). However, only AWQ and GPTQ were rerun after the fix, so it remains unclear whether BiLLM, ARB-LLM, MoEQuant and NoWag are affected as well. Compared with 3 bit GPTQ, BitMoE now reaches similar accuracy at a lower budget. Compared with 4 bit methods, it needs less memory and bandwidth, but it is also clearly less accurate (59.85 versus 68.07 on Qwen3-30B-A3B). It is therefore unclear whether BitMoE improves the Pareto curve of accuracy versus memory and bandwidth. At the same budget, a smaller model at 4 bit may be the better choice.
  • Reliability of the efficiency numbers. Throughput is crucial for judging the method. In your response to Reviewer 7f1H you write: "The illustrative figures above require verification before they are presented as measured results." The same figures appear in Table 11, where the throughput is described as measured. It is therefore unclear which throughput numbers are measured. The model sizes in Table 11 are also about twice as large as the whole model BPW in Appendix A.4.2 implies. For Qwen3-30B-A3B, Table 11 reports 10.94 GB, while 1.41 BPW for about 30.5B parameters gives about 5.4 GB. The FP8 result in your response to me is not part of the revised PDF.

Thank you for the follow-up clarifications and the further corrections. My main concerns are now largely addressed. Tables 2 and 3 report the active path budget for all methods, the baseline configuration is documented, and the comparison with 3 bit GPTQ has been revised. In my view the paper now barely crosses the acceptance threshold, and I will raise my rating from 2 to 3. Two concerns remain.

  • Many numbers changed during the rebuttal. This includes the AWQ and GPTQ baselines after the calibration fix, the model sizes after the saving format fix, and the throughput note in the response to Reviewer 7f1H. This limits my confidence in the reported numbers. Please check all tables once more for the final version.
  • Changes to the manuscript are still pending. Table 10 still lists 26.96 GB for the 80B model. This is about 2.7 bits per parameter and does not match the corrected sizes in Table 11 (about 1.4 to 1.6 bits per parameter). The storage format of the attention and dense projections (Q1_0, 1.125 BPW) is stated only in the response, and the manuscript does not describe how these weights are binarized. The final version should contain all details needed to reproduce the reported budgets and results.
Reviewer_7f1HOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a post-training binarization framework for MoE LLMs named BitMOE, that integrates routing-coupled binary factorization (RCBF), conditional expert residual allocation (CERA), and margin-aware optimal brain routing (MoBR) to retain all experts under an explicit storage budget, while maintain accuracy and realize inference speedup.

Critical strengths
  • The three components in BitMoE are well ablation studied with supported results
  • The claimed storage budget controlling/reduction is not only illustrated with result numbers in tables & figures, but also given detailed reproducible theoretic analysis in Appendix-A2.
  • The technique sounds OK to me, and the experimental coverage on MoE models is sufficient.
Critical weaknesses
  • It is better adding the comparison to currently most-widely used 4bit quantization method as baseline.
  • Some related references are missing, for instance "QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models" and "BitNet: Scaling 1-bit Transformers for Large Language Models".
  • There are several citation references in main body starting with "Team". To be academically rigorous, in bibtex, you should use "{}" to put the team name together, for instance {Qwen Team}.
  • What is the framework the paper used to comparing decoding speedup? Is it huggingface transformers? What about transits to industrial inference engine like sglang?
Questions for authors

check weakness for my concerns.

Reviewer_JKhNOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

BitMoE is a post-training binary quantization method for MoE large language models that achieves approximately 1.11 bits . There are three main parts to this approach:

  • RCBF uses singular value decomposition to decompose each projection type across all experts into a common factor (INT8).
  • CERA generates potential binary residual candidates for the next level of quantization by creating residuals for every 16 columns. These candidates are chosen through a greedy process where the candidate's ability to improve reconstruction is evaluated based on its contribution per bit. After choosing the candidates, CERA will re-fit the common down factor used to generate the residuals.
  • MOBR calibrates the quantized router using a pairwise KL divergence between the selected/unselected margin of each pair of experts. A second order approximation is utilized to compute the calibration. This is done to help prevent expert shifting.
Critical strengths
  1. The model's performance at a constant 1.11 bits per word (BPW) is significantly better than all other binary baseline models. Perplexity was 12.26 on Qwen3-30B-A3B compared to 26.76 for ARB-LLM and the seven task average was 59.85 versus 40.14. This model outperformed 3-bit GPTQ on multiple models.
  2. There were three different types of ablations used in each of the three MoE specific designs: one where there were no cross expert terms; another using six different ways to allocate residuals; and finally MOBR which can be added to either BiLLM or ARB-LLM and will improve both.
  3. A careful accounting of how much space each type of data takes up (static, active path, entire model) when storing them.
Critical weaknesses
  1. Baseline credibility. 3-bit AWQ was shown to collapse while 3-bit GPTQ's performance on HumanEval using Qwen3-Next-80B dropped from 94.51 to 54.01, which does not represent typical behavior. The group size, calibration data, and expert calibration coverage will require verification.

  2. A weak argument exists for the use of a binary router. With approximately 12.6 million parameters, it would cost roughly 0.006 bits per parameter to keep the router in BF16. One possible reason that some of the benefits provided by MOBR were able to correct a preventable issue is because there was no BF16-router control used.

  3. Missing strong comparative studies were conducted. Studies comparing 2-bit techniques, nor studies comparing smaller models at equivalent amounts of memory have been performed.

  4. Incomplete efficiency evaluation. Efficiency was evaluated solely against BF16, only at a batch size of 4, and no description of the kernel was provided. Instruction-tuned models continue to experience a loss of approximately 30 points on HumanEval/GSM8K.

Questions for authors
  • What are each baseline’s group size, calibration data, and expert calibration coverage?
  • Do the baselines binarize the router?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.