- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
BitsMoE reduces the memory cost of MoE large language models by assigning different bit-widths to individual spectral components within expert weights. It introduces Shared-basis Spectral Decomposition (SSD) to separate shared structure across experts from expert-specific components, and Factorized Quantization Cost Modeling (FQCM) to estimate their quantization costs and guide bit allocation under a fixed memory budget. On Qwen3-30B-A3B at 2-bit, BitsMoE achieves 64.29% average accuracy over seven downstream tasks, exceeding GEMQ by 2.80 percentage points. It also achieves a 16.47× offline quantization speedup over GEMQ and up to 6.46× the decode throughput of GPTQ.
Critical strengths
1 BitsMoE defines fine-grained quantization units within MoE experts through Shared-basis Spectral Decomposition. It separates shared structure across experts from expert-specific spectral components and assigns heterogeneous bit-widths under a fixed bit budget.
2 FQCM combines intrinsic spectral importance, activation-dependent importance, and bit-width-dependent distortion to guide bit allocation. The ablation results show that incorporating activation and routing information substantially improves accuracy over allocation based only on spectral energy.
3 The experiments report both accuracy and efficiency gains. At 2-bit on Qwen3-30B-A3B, BitsMoE achieves 64.29% average accuracy over seven downstream tasks, exceeding GEMQ by 2.80 percentage points. It also achieves a 16.47× end-to-end offline quantization speedup over GEMQ and up to 6.46× the decode throughput of GPTQ in the evaluated settings.
Critical weaknesses
1、Table 4 shows a modest improvement from independent SVD to SSD under uniform allocation, from 24.86% to 25.27%. Could the authors also evaluate independent SVD with FQCM allocation under the same storage budget? Additional comparisons and implementation details would help clarify the contribution of SSD.
2、Could the authors provide more details on the ILP solution procedure, including the solver tolerances, time limits, and CPU and thread configuration?
3、Equation (8) combines intrinsic spectral importance, activation-dependent importance, and bit-width-dependent distortion through multiplication. How is the relative influence of these factors determined, and is any additional weighting or normalization needed?
4、In Table 3, the gains over GEMQ are concentrated in MQ and GS, while some other tasks show lower accuracy. Could the authors explain why the method benefits these mathematical tasks more than the others?
Questions for authors
The interpretation of Table 4 is my main concern.
3 comments
Response to Reviewer P3p6 Part 2
Weakness 3: Relative influence and normalization of the FQCM factors.
Thank you very much for your professional feedback.
The multiplicative form follows from the reconstruction-loss derivation in App. C under the stated approximations. The three factors are multiplied directly in Eq.(8) & Eq. (40), without additional weighting or normalization. The resulting component-wise quantization costs are summed to obtain the global FQCM loss in Eq. (9), with no further normalization.
Weakness 4: Larger gains on MathQA and GSM8K.
Thank you very much for your professional feedback.
GEMQ's expert-level allocation preserves some experts at higher precision but can perturb important directions in lower-precision experts. BitsMoE can retain higher precision for important spectral directions within each expert, which may help preserve the numerical distinctions needed for mathematical tasks. Although MathQA uses candidate-likelihood scoring, selecting the correct numerical answer still depends on the quantitative relationships in the question. GSM8K additionally requires generating a solution, where errors in key numerical values or intermediate operations can affect subsequent generation and the final answer.
We sincerely appreciate your professional feedback.
Thank you for your response. I realize I misunderstood this point earlier: Non-Basis refers to the SVD results. Most of my concerns have been addressed, and I will keep my positive rating.
Response to Reviewer P3p6 Part 1
We sincerely appreciate your constructive feedback. We provide point-by-point responses below.
Thank you very much for your professional feedback.
(1) Decomposition comparison
Non-Basis with FQCM achieves 49.98% average accuracy at a target bit-width of 2-bit on Qwen3-30B-A3B. Non-Basis applies SVD independently to each expert, without SSD. Average accuracy is computed over the seven downstream tasks.
(2) Task-level results
(3) Effectiveness of FQCM without SSD
In the Non-Basis setting, FQCM improves average accuracy from 24.86% under uniform allocation to 49.98%, demonstrating its effectiveness without SSD.
We have added the Non-Basis ablation to the revised manuscript in Section 4.3 (Tab. 4) and Appendix E.2 (Tab. 12).
Thank you very much for your professional feedback.
We use Gurobi 12.0.0 to solve one global ILP per projection type, allocating bits jointly across all MoE layers. The solver settings are listed below.
We use 48 solver threads for all three models. A representative host has two AMD EPYC 7763 CPUs (128 physical cores and 256 hardware threads) and NVIDIA A100 80GB PCIe GPUs.
We have added the solver settings and host configuration to the revised manuscript in Appendix A (Tab. 7).
We sincerely appreciate your professional feedback.