- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HAMoE, a heterogeneity-aware mixed-precision post-training quantization (PTQ) framework for Mixture-of-Experts (MoE) large language models. The authors investigate quantization sensitivity differences at both inter-expert and intra-expert levels. First, HAMoE estimates expert importance by combining expert activation frequency with part-of-speech (POS)-based token importance, and performs mixed-precision bit allocation under a fixed precision budget. Second, the authors estimate column-level sensitivity within each expert using the diagonal elements of the Hessian matrix and weight column norms, and replace the most sensitive 1% of weight columns with INT8-quantized representations. The method is evaluated on Qwen1.5-MoE, DeepSeek-V2-Lite, OLMoE, and MoE-Girl, with experiments covering WikiText2 perplexity, seven conventional zero-shot benchmarks, component ablations, and model storage analysis.
Critical strengths
Strength 1: The paper jointly considers variations in expert usage across MoE experts and differences in weight sensitivity within individual experts, incorporating information at both granularities into mixed-precision quantization. The proposed POS-based importance provides an additional signal to conventional expert importance estimation based on expert usage, offering a potentially useful direction for further exploration.
Strength 2: The paper evaluates the proposed method on four MoE models and compares it against several relevant baselines. The authors further investigate frequency-only, token-only, and combined importance metrics, as well as the effects of different sensitive-column protection ratios. The experimental results support the effectiveness of the proposed design in improving quantization performance on some models, although the gains over strong baselines remain relatively modest in certain settings.
Critical weaknesses
Weakness 1: I have substantial concerns regarding the methodological novelty of HAMoE. MC-MoE [1] has already incorporated expert activation frequency, routing scores, and reconstruction errors into mixed-precision allocation, while MxMoE [2] further considers expert activation dynamics, parameter sensitivity, and hardware constraints. D²MoE [3] also explores expert importance for dynamic precision selection and scheduling. For intra-expert quantization, OWQ [4] has introduced Hessian-based sensitive-column identification and high-precision retention, while QUIK [5] similarly preserves selected outlier columns at higher precision. Therefore, HAMoE's primary novelty lies in incorporating POS-based token importance into expert-level allocation. However, this metric relies on heuristic semantic weights, and its independent benefits over existing sensitivity measures remain insufficiently demonstrated. Moreover, the authors use a uniform 25% category proportion to justify expert specialization without accounting for the actual POS distribution of the calibration corpus. The column sensitivity metric also relies on original weight norms rather than actual quantization residuals, without sufficient theoretical justification or direct comparisons with OWQ-style alternatives. Overall, the core components substantially overlap with existing approaches, while the main new contribution lacks rigorous validation.
Weakness 2 The experiments cover only Qwen1.5-MoE, DeepSeek-V2-Lite, OLMoE, and MoE-Girl, without evaluating more recent architectures such as Qwen3-30B-A3B or Qwen3.5-35B-A3B. The benchmarks are also largely restricted to WikiText2 perplexity and conventional zero-shot tasks, lacking more challenging mathematical reasoning and code-generation evaluations such as MATH-500, AIME, GPQA Diamond, HumanEval, and LiveCodeBench. Given the emphasis on token semantic importance, its effectiveness on mathematical, coding, and cross-domain inputs remains unclear. Furthermore, the method replaces the most sensitive 1% of weight columns with INT8 after expert-level bit allocation, but does not clearly explain whether this additional precision is included in the reported bit budget. Complete effective BPW accounting and practical mixed-precision kernel performance are also missing. Section 5.4 additionally reports inconsistent quantized model sizes of 10.15 GB and 9.72 GB. These limitations weaken the evidence for HAMoE's generalizability and practical accuracy–efficiency advantages.
[1] Wei Huang et al. "MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More." arXiv preprint arXiv:2410.06270, 2024. https://arxiv.org/abs/2410.06270
[2] Haojie Duanmu et al. "MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design." ICML 2025. https://proceedings.mlr.press/v267/duanmu25a.html
[3] Haodong Wang, Qihua Zhou, Zicong Hong, and Song Guo. "D²MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving." arXiv preprint arXiv:2504.15299, 2025. https://arxiv.org/abs/2504.15299
[4] Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. "OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models." AAAI 2024. https://doi.org/10.1609/aaai.v38i12.29237
[5] Saleh Ashkboos et al. "QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models." EMNLP 2024. https://aclanthology.org/2024.emnlp-main.197/
Questions for authors
Q1: MC-MoE [1] and MxMoE [2] have already employed expert activation frequency for mixed-precision allocation, while OWQ [4] uses Hessian-based sensitivity to identify and retain high-precision weight columns. Could the authors clarify the distinct technical contributions of HAMoE's expert-level allocation and sensitive-column protection beyond these existing methods? In particular, could the authors provide a direct comparison with OWQ-style sensitive-column protection?
Q2: HAMoE uses |H_jj| × ||W[:,j]||_2 to measure quantization sensitivity, whereas second-order output reconstruction error is directly related to the actual quantization residual ΔW = W - Q(W). Why do the authors not adopt quantization-error-aware metrics such as H_jj × ||ΔW[:,j]||_2^2? Could the authors compare different sensitivity metrics in terms of selected columns and final quantization performance to justify the proposed heuristic?
Q3: The authors use a uniform 25% baseline across four token categories to demonstrate expert specialization, although actual POS distributions in natural language are generally non-uniform. Could the authors validate expert functional specialization using randomized controls that preserve the original category frequencies and expert activation statistics? Furthermore, how are word-level POS labels mapped to subword tokens, how is the importance weight for special symbols defined, and does the method generalize to mathematical and code tokens?
Q4: Could the authors extend their experiments to more recent MoE models, such as Qwen3-30B-A3B or Qwen3.5-35B-A3B, and evaluate more challenging mathematical reasoning, scientific reasoning, and code-generation benchmarks, including MATH-500, AIME, GPQA Diamond, HumanEval, and LiveCodeBench? In particular, does POS-based importance continue to outperform frequency-only allocation on these tasks?
Q5: When the top 1% most sensitive columns are upgraded to INT8, is the additional storage cost included in the reported 3.35-bit budget? Could the authors report the complete effective BPW, including scales, zero-points, protected columns, and metadata, and clarify how routed and shared experts of different sizes are accounted for? Furthermore, Section 5.4 reports inconsistent quantized model sizes of 10.15 GB and 9.72 GB. Could the authors explain whether these values correspond to different storage accounting conventions?
[1] Wei Huang et al. "MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More." arXiv preprint arXiv:2410.06270, 2024. https://arxiv.org/abs/2410.06270
[2] Haojie Duanmu et al. "MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design." ICML 2025. https://proceedings.mlr.press/v267/duanmu25a.html
[4] Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. "OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models." AAAI 2024. https://doi.org/10.1609/aaai.v38i12.29237