acceptodds
Withdrawn from ICLR 2027

HAMoE: Heterogeneity-Aware Mixed-Precision Quantization for Mixture-of-Experts Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixture-of-Experts (MoE) models face deployment challenges in resource-constrained devices due to their massive parameter scales. Quantization is recognized as one of the most effective approaches to address such challenges. In this work, we conduct a systematic quantization study of MoE models and identify two critical heterogeneity insights that have not been considered in prior work: 1) inter-expert heterogeneity, i.e., distinct activation frequencies and processed token types across experts, and 2) intra-expert heterogeneity, i.e., differential quantization sensitivity of weights within a single expert. To this end, we introduce HAMoE, a mixed-precision quantization framework that considers both heterogeneities. HAMoE assigns fine-grained mixed precision to inter-experts based on activation frequency and token type. In addition, it proposes an adaptive column replacement mechanism, which replaces the most quantization-sensitive intra-expert weight columns with INT8-quantized versions, thereby effectively reducing quantization loss. Evaluation results show that HAMoE achieves competitive results against strong baselines under equivalent bit budgets, reducing WikiText-2 perplexity by 4.0% relative to GPTQ at 3.35-bit, recovering about 45% of the quantization loss. HAMoE also consistently outperforms MxMoE in both average accuracy and perplexity across all four models.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, Model Quantization, Mixed-Precision Quantization, Large Language Models, Model Compression

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_RVqSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HAMoE, a heterogeneity-aware mixed-precision post-training quantization (PTQ) framework for Mixture-of-Experts (MoE) large language models. The authors investigate quantization sensitivity differences at both inter-expert and intra-expert levels. First, HAMoE estimates expert importance by combining expert activation frequency with part-of-speech (POS)-based token importance, and performs mixed-precision bit allocation under a fixed precision budget. Second, the authors estimate column-level sensitivity within each expert using the diagonal elements of the Hessian matrix and weight column norms, and replace the most sensitive 1% of weight columns with INT8-quantized representations. The method is evaluated on Qwen1.5-MoE, DeepSeek-V2-Lite, OLMoE, and MoE-Girl, with experiments covering WikiText2 perplexity, seven conventional zero-shot benchmarks, component ablations, and model storage analysis.

Critical strengths

Strength 1: The paper jointly considers variations in expert usage across MoE experts and differences in weight sensitivity within individual experts, incorporating information at both granularities into mixed-precision quantization. The proposed POS-based importance provides an additional signal to conventional expert importance estimation based on expert usage, offering a potentially useful direction for further exploration.

Strength 2: The paper evaluates the proposed method on four MoE models and compares it against several relevant baselines. The authors further investigate frequency-only, token-only, and combined importance metrics, as well as the effects of different sensitive-column protection ratios. The experimental results support the effectiveness of the proposed design in improving quantization performance on some models, although the gains over strong baselines remain relatively modest in certain settings.

Critical weaknesses

Weakness 1: I have substantial concerns regarding the methodological novelty of HAMoE. MC-MoE [1] has already incorporated expert activation frequency, routing scores, and reconstruction errors into mixed-precision allocation, while MxMoE [2] further considers expert activation dynamics, parameter sensitivity, and hardware constraints. D²MoE [3] also explores expert importance for dynamic precision selection and scheduling. For intra-expert quantization, OWQ [4] has introduced Hessian-based sensitive-column identification and high-precision retention, while QUIK [5] similarly preserves selected outlier columns at higher precision. Therefore, HAMoE's primary novelty lies in incorporating POS-based token importance into expert-level allocation. However, this metric relies on heuristic semantic weights, and its independent benefits over existing sensitivity measures remain insufficiently demonstrated. Moreover, the authors use a uniform 25% category proportion to justify expert specialization without accounting for the actual POS distribution of the calibration corpus. The column sensitivity metric also relies on original weight norms rather than actual quantization residuals, without sufficient theoretical justification or direct comparisons with OWQ-style alternatives. Overall, the core components substantially overlap with existing approaches, while the main new contribution lacks rigorous validation.

Weakness 2 The experiments cover only Qwen1.5-MoE, DeepSeek-V2-Lite, OLMoE, and MoE-Girl, without evaluating more recent architectures such as Qwen3-30B-A3B or Qwen3.5-35B-A3B. The benchmarks are also largely restricted to WikiText2 perplexity and conventional zero-shot tasks, lacking more challenging mathematical reasoning and code-generation evaluations such as MATH-500, AIME, GPQA Diamond, HumanEval, and LiveCodeBench. Given the emphasis on token semantic importance, its effectiveness on mathematical, coding, and cross-domain inputs remains unclear. Furthermore, the method replaces the most sensitive 1% of weight columns with INT8 after expert-level bit allocation, but does not clearly explain whether this additional precision is included in the reported bit budget. Complete effective BPW accounting and practical mixed-precision kernel performance are also missing. Section 5.4 additionally reports inconsistent quantized model sizes of 10.15 GB and 9.72 GB. These limitations weaken the evidence for HAMoE's generalizability and practical accuracy–efficiency advantages.

[1] Wei Huang et al. "MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More." arXiv preprint arXiv:2410.06270, 2024. https://arxiv.org/abs/2410.06270

[2] Haojie Duanmu et al. "MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design." ICML 2025. https://proceedings.mlr.press/v267/duanmu25a.html

[3] Haodong Wang, Qihua Zhou, Zicong Hong, and Song Guo. "D²MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving." arXiv preprint arXiv:2504.15299, 2025. https://arxiv.org/abs/2504.15299

[4] Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. "OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models." AAAI 2024. https://doi.org/10.1609/aaai.v38i12.29237

[5] Saleh Ashkboos et al. "QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models." EMNLP 2024. https://aclanthology.org/2024.emnlp-main.197/

Questions for authors

Q1: MC-MoE [1] and MxMoE [2] have already employed expert activation frequency for mixed-precision allocation, while OWQ [4] uses Hessian-based sensitivity to identify and retain high-precision weight columns. Could the authors clarify the distinct technical contributions of HAMoE's expert-level allocation and sensitive-column protection beyond these existing methods? In particular, could the authors provide a direct comparison with OWQ-style sensitive-column protection?

Q2: HAMoE uses |H_jj| × ||W[:,j]||_2 to measure quantization sensitivity, whereas second-order output reconstruction error is directly related to the actual quantization residual ΔW = W - Q(W). Why do the authors not adopt quantization-error-aware metrics such as H_jj × ||ΔW[:,j]||_2^2? Could the authors compare different sensitivity metrics in terms of selected columns and final quantization performance to justify the proposed heuristic?

Q3: The authors use a uniform 25% baseline across four token categories to demonstrate expert specialization, although actual POS distributions in natural language are generally non-uniform. Could the authors validate expert functional specialization using randomized controls that preserve the original category frequencies and expert activation statistics? Furthermore, how are word-level POS labels mapped to subword tokens, how is the importance weight for special symbols defined, and does the method generalize to mathematical and code tokens?

Q4: Could the authors extend their experiments to more recent MoE models, such as Qwen3-30B-A3B or Qwen3.5-35B-A3B, and evaluate more challenging mathematical reasoning, scientific reasoning, and code-generation benchmarks, including MATH-500, AIME, GPQA Diamond, HumanEval, and LiveCodeBench? In particular, does POS-based importance continue to outperform frequency-only allocation on these tasks?

Q5: When the top 1% most sensitive columns are upgraded to INT8, is the additional storage cost included in the reported 3.35-bit budget? Could the authors report the complete effective BPW, including scales, zero-points, protected columns, and metadata, and clarify how routed and shared experts of different sizes are accounted for? Furthermore, Section 5.4 reports inconsistent quantized model sizes of 10.15 GB and 9.72 GB. Could the authors explain whether these values correspond to different storage accounting conventions?

[1] Wei Huang et al. "MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More." arXiv preprint arXiv:2410.06270, 2024. https://arxiv.org/abs/2410.06270

[2] Haojie Duanmu et al. "MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design." ICML 2025. https://proceedings.mlr.press/v267/duanmu25a.html

[4] Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. "OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models." AAAI 2024. https://doi.org/10.1609/aaai.v38i12.29237

Reviewer_TDgjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

HAMoE studies mixed-precision weight quantization for MoE models at two levels: allocating precision across experts and protecting sensitive columns within each expert. It combines activation frequency with part-of-speech statistics to prioritize experts under a target bit budget, then stores the most sensitive 1% of columns in INT8. The central hypothesis is that the types of tokens an expert processes provide useful importance information beyond how often it is activated.

Experiments on 4 MoE models report improved perplexity and average zero-shot accuracy over the evaluated baselines. Component ablations suggest benefits from combining token-type and frequency signals and from protecting sensitive columns.

Critical strengths
  1. The paper explores an interpretable importance signal beyond activation frequency. Comparing experts with approximately matched frequencies is a useful step toward testing whether token specialization provides complementary information.

  2. Evaluation covers several architectures and both allocation levels. Four MoE checkpoints, perplexity and zero-shot benchmarks, and separate ablations of expert scoring and column protection provide useful evidence for the proposed combination.

Critical weaknesses

W1. The contribution beyond existing mixed-precision methods is insufficiently isolated — Major.

Protecting sensitive structured weights already has a direct precedent in OWQ[1], while MoQa[2] considers expert- and channel-level precision adjustment. The broad claim that prior methods overlook intra-expert heterogeneity is therefore difficult to support. The paper needs comparisons at matched final storage against alternative column selectors and a combination such as MxMoE[3] with the same column protection.

W3. The interpretation of token specialization exceeds the evidence

The 25% reference used to assess specialization ignores unequal category frequencies in the corpus and the selection of each expert's dominant category. Frequency-matched quantization probes also leave layer and intrinsic parameter sensitivity uncontrolled. Thus, the results suggest an association without establishing that token function directly determines quantization vulnerability. A shuffled control preserving corpus category frequencies and per-layer routing counts would strengthen the analysis. Appendix C additionally measures assignment agreement rather than downstream quality, which limits its evidence for robustness to semantic weights.

W4. Practical efficiency remains insufficiently validated

The paper reports model sizes and peak memory, but no end-to-end latency or throughput for the heterogeneous expert precisions and separate INT8 column path. Their execution overhead could affect deployment benefits, and the rough total GPU-hour estimate does not establish calibration cost relative to baselines. Unlike MxMoE[3], HAMoE leaves mixed-precision kernel support unexplored. The memory measurements also require clearer loading/offloading and measurement details, given the reported FP16 peaks of approximately 32–35 GB under the stated single RTX-4090 setup.

[1] Lee, Changhun, et al. "OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models." Proceedings of the AAAI Conference on Artificial Intelligence. 2024.

[2] Zheng, Zihao, et al. "MoQa: Rethinking MoE Quantization with Multi-stage Data-model Distribution Awareness." arXiv preprint arXiv:2503.21135v1. 2025.

[3] Duanmu, Haojie, et al. "MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design." Proceedings of the 42nd International Conference on Machine Learning. 2025.

Questions for authors
  1. What does the reported average bit budget include? Please reconcile the column-protection overhead (kernel latency) and conflicting model sizes, and verify comparisons at matched final storage.

  2. What does POS-aware allocation add beyond existing techniques? How does HAMoE compare with MxMoE plus identical column protection, and with alternative column selectors, at the same final size?

  3. Does token specialization predict sensitivity beyond corpus frequency, layer, and parameter vulnerability? Please provide a controlled analysis and downstream quality results for a substantially different semantic-weight assignment.

  4. What are the actual inference and calibration costs? Please report latency, throughput, calibration overhead, and a reproducible definition and measurement procedure for peak memory.

Reviewer_iSC5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HAMoE, a mixed-precision post-training quantization framework for Mixture-of-Experts (MoE) models. It is motivated by two types of heterogeneity: inter-expert and intra-expert.

First, experts are not equally important to the model (inter-expert heterogeneity): they differ in how frequently they are activated and in the types of tokens they process, which the authors claim leads to different sensitivity to quantization. To address this type of heterogeneity, the authors analyze token routing using POS (part-of-speech) tags and report that many experts specialize in particular token categories. The experts mainly process content words exhibiting larger quantization loss than those processing function words or punctuation at comparable activation frequencies. Based on this observation, HAMoE combines normalized activation activation frequency with a POS-based token importance score into a single expert importance score and assigns each expert a bit-width from {2,3,4,6,8} under an average bit budget by solving an integer program with Gurobi.

Second, weights within the same expert are not equally important either (intra-expert heterogeneity): a small subset of the weight column accounts for a disproportionate share of the quantization sensitivity. To address this type of heterogeneity, HAMoE introduces an adaptive column replacement mechanism. It ranks the columns of each expert weight matrix by the product of the Hessian diagonal estimated from calibration activations, and the column L2 norm, and replaces the top 1% of columns with INT8-quantized versions.

The authors quantize only the MoE layers, keeping attention in full precision, and evaluate HAMoE on four models (Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite, OLMoE-1B-7B, and MoE-Girl-1B-7B) against GPTQ, MoQE, MxMoE, SFMoE, and MoQa. The reported results show consistent improvements in WikiText-2 perplexity and average zero-shot accuracy across models and bit budgets; for example, at 3.35 bits on Qwen1.5-MoE, perplexity decreases from 7.46 (GPTQ) to 7.16. Ablations examine the contribution of each importance dimension and the column protection ratio, and quantizing the expert weights yields approximately 2.9–3.0× full-model compression.

Critical strengths

1. Linking functional expert specialization to quantization sensitivity.

While POS-based expert specialization has been observed in prior work, this paper connects it to quantization. Figure 2 shows that, among experts with similar activation frequencies, content-word-dominant experts tend to have higher INT2 quantization loss than those dominated by function words or punctuation, across four MoE models. The paper then turns this observation into a token-type importance signal for expert-level bit allocation, and the ablations show that combining it with activation frequency outperforms either signal alone. This is the main factor supporting acceptance in my assessment, as it introduces a functional dimension to MoE quantization that may inform future mixed-precision methods.

2. Complementary precision allocation at two granularities with consistent gains.

HAMoE combines expert-level mixed-precision allocation with fine-grained column protection within each expert. Table 2 shows that protecting only 1% of weight columns with INT8 reduces perplexity on all four evaluated models (approximately 1.0%–4.1% relative), and under the same reported bit budgets the full framework consistently outperforms several quantization baselines across MoE architectures with and without shared experts. This is consequential to my assessment because the consistency of the gains suggests that the design is not tailored to a single architecture, although the fairness of the bit-budget comparisons depends on the clarifications requested in Weakness 2.

Critical weaknesses

1. Incomplete related-work coverage and insufficiently established novelty

Several closely related prior works are not discussed. MC-MoE (Huang et al., ICLR 2025) already formulates expert-level mixed-precision bit allocation as a linear programming problem using activation frequency, routing scores, and quantization loss. Its quantization component would be a relevant baseline for HAMoE's inter-expert allocation. Antoine et al. (COLING 2025) have already investigated POS-based expert specialization across six MoE models, including Qwen1.5-MoE-A2.7B and OLMoE-1B-7B. This is particularly relevant to the specialization analysis in Section 3.1, although HAMoE additionally studies its relationship with quantization sensitivity. OWQ (Lee et al., AAAI 2024) protects quantization-sensitive weight columns in higher precision using a score based on the Hessian diagonal and squared quantization error. HAMoE adopts a similar protection strategy but uses the weight norm instead, without a direct comparison or a clear explanation of why this alternative metric is preferable. Xie et al. (ACL Findings 2025) also investigate channel-wise sensitivity and fine-grained quantization within MoE experts.

Although these works do not propose the same complete framework as HAMoE, they overlap with several of its underlying observations and design choices. Furthermore, while the Frequency-only ablation shows benefits from adding POS information, its additional value beyond existing Hessian-based expert sensitivity measures remains unclear. The functional specialization analysis also relies on a 25% random baseline without accounting for non-uniform POS distributions. These issues make it difficult to assess the paper's incremental novelty relative to prior work, which is important to my overall assessment.

Relevant Prior Work:

2. Unclear bit-budget accounting and experimental comparability

The paper claims improvements over existing methods under equivalent bit budgets, but the actual storage accounting and baseline configurations remain unclear. In particular, it is not clear whether the reported 3.35-bit budget includes the additional INT8-protected columns and associated metadata, or how baselines such as GPTQ achieve fractional average bit-widths such as 3.35 bits. The INT8/INT6 expert counts in Table 10 also appear to fall slightly below the minimum allocation constraints in Section 4.1. Furthermore, Section 5.4 states that metadata are included in a reported model size of 10.15 GB for Qwen1.5-MoE, while Table 3 reports 9.72 GB without explaining the difference. These issues are consequential because fair storage-budget comparisons are essential to establishing the claimed accuracy–compression advantages.

Questions for authors

Q1. Bit-budget accounting and baseline comparability.

Does the reported 3.35-bit budget include the additional INT8-protected columns, quantization metadata, and other relevant storage overhead? Could the authors provide the actual effective bits per weight or packed storage size for HAMoE and the compared baselines? It would also be helpful to clarify how fractional bit-widths are implemented for baselines such as GPTQ, and whether HAMoE's underlying low-bit quantizer uses round-to-nearest quantization or GPTQ-based error compensation. Finally, could the authors reconcile the expert allocation counts in Table 10 with the constraints in Section 4.1 and the two compressed model sizes reported in Section 5.4?

Q2. Positioning and independent contribution of POS-based expert importance.

How does HAMoE’s inter-expert allocation compare with MC-MoE (Huang et al., ICLR 2025), which also formulates expert-level bit allocation as a linear program over activation frequency, routing scores, and quantization loss? A direct comparison under the same bit budget would help clarify the benefit of the proposed allocation. Similarly, could the authors discuss how the specialization analysis in Section 3.1 relates to Antoine et al. (COLING 2025), who studied POS-based expert specialization in several of the same models? The Frequency-only ablation demonstrates benefits from adding POS information, but does POS provide additional improvements when combined with existing Hessian-based expert sensitivity measures, such as those used in MxMoE? Additionally, could the authors clarify the statistical basis for the 25% random baseline in the functional specialization analysis? A comparison against corpus-level POS proportions or a baseline that randomly shuffles POS labels while preserving the routing assignments would help determine whether the observed specialization exceeds what would be expected from the underlying token distribution.

Q3. Validation of the column sensitivity metric.

What motivates the proposed sensitivity score ∣Hjj∣⋅∥W:,j∥2|H_{jj}| \cdot \|W_{:,j}\|_2 rather than a metric incorporating the actual quantization residual, as in OWQ (Lee et al., AAAI 2024)? Could the authors compare the proposed selection rule against random selection, Hessian-only, weight-norm-only, or quantization-error-based selection under the same protection ratio and storage budget? Such comparisons would help determine whether the improvements come from the proposed sensitivity metric or selective INT8 protection alone. Additionally, are the group-wise quantization scales recalculated after excluding the protected columns, or are the original scales retained?

Q4. Practical inference efficiency and alternative channel protection strategies.

Given the heterogeneous expert bit-widths and additional INT8-protected columns, could the authors clarify the expected inference overhead of the proposed representation, particularly regarding memory access and kernel execution? Although the paper reports model size and peak memory, latency or throughput measurements would help assess its practical deployment benefits. It would also be useful to discuss or compare against alternative channel protection strategies, such as AWQ-style activation-aware scaling (or AWQ combined with GPTQ), which protects salient channels while keeping a uniform low-bit layout.

Reviewer_qJFXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HAMoE, a heterogeneity-aware mixed-precision quantization framework for Mixture-of-Experts (MoE) models. The method is motivated by two forms of heterogeneity: inter-expert differences in activation frequency and token-type preferences, and intra-expert differences in weight-column quantization sensitivity.

For inter-expert precision allocation, HAMoE constructs an importance score by combining normalized expert activation frequency with token-type importance derived from part-of-speech categories. The resulting scores are used in a constrained optimization formulation to assign different bit-widths to experts under a target average bit budget. For intra-expert quantization, the authors introduce an adaptive column replacement mechanism. A sensitivity score based on the diagonal Hessian approximation and weight-column norms is used to identify important columns, approximately 1% of which are preserved at INT8 precision.

The method is evaluated on four MoE models, including Qwen1.5-MoE, DeepSeek-V2-Lite, OLMoE, and MoE-Girl. The reported experiments demonstrate improvements in perplexity and average downstream accuracy over several mixed-precision quantization baselines, alongside substantial reductions in model storage requirements.

Critical strengths

S1. The method considers complementary sources of heterogeneity in MoE quantization.

A strength of HAMoE is its attempt to account for heterogeneity at both the inter-expert and intra-expert levels. In particular, incorporating token-type information alongside expert activation frequency introduces an additional signal for precision allocation that is not explicitly modeled by several existing usage-based approaches.

This is potentially consequential because experts with similar activation frequencies may differ in the characteristics of the tokens they process, suggesting that usage frequency alone may be insufficient for optimal precision allocation.

The proposed framework provides a practical mechanism for incorporating this additional information into a constrained mixed-precision allocation problem.

S2. Empirical improvements across multiple MoE architectures and quantization baselines.

The paper evaluates HAMoE on four MoE models with different expert configurations and compares against several relevant quantization methods, including GPTQ, MoQE, MxMoE, and SFMoE.

The reported results consistently favor HAMoE in average accuracy and perplexity under the evaluated bit-width settings. The ablation studies further provide evidence that combining token-type and activation-frequency information is more effective than using either signal alone.

The column replacement experiments also show that protecting a small fraction of sensitive columns can improve perplexity, with diminishing benefits beyond approximately 1% protection.

These findings provide useful empirical support for the proposed framework.

Critical weaknesses

W1. The evidence for token-type specialization and its independent contribution to quantization sensitivity is insufficient.

A central motivation of HAMoE is that different experts exhibit meaningful functional specialization according to linguistic token categories, and that experts processing semantically rich tokens are more vulnerable to quantization.

However, the analysis in Section 3.1 compares the maximum category proportions observed across experts against a 25% random expectation. This comparison is questionable because the four token categories are not necessarily equally represented in the underlying corpus.

Even under random routing, experts would be expected to reflect the potentially imbalanced token-type distribution of the calibration data. Therefore, observing a dominant category proportion substantially above 25% does not, by itself, establish expert specialization.

A more appropriate analysis would compare each expert's token-type distribution with the global token-type distribution, potentially using normalized mutual information, enrichment statistics, or a frequency-matched random-routing baseline.

Furthermore, the observed differences in quantization sensitivity across token categories may be confounded by activation magnitudes, token frequencies, or other characteristics of the corresponding expert inputs. Figure 2 controls for expert activation frequency but does not fully isolate token semantics from these alternative factors.

As the proposed semantic importance score is a primary distinguishing contribution, stronger evidence that token-type information provides independent predictive value beyond conventional quantization sensitivity indicators would materially strengthen the paper.

W2. The intra-expert sensitivity metric and mixed-precision efficiency claims lack sufficient justification.

The proposed column replacement mechanism ranks columns using the score

sj=∣Hjj∣∥W:,j∥2.s_j=|H_{jj}|\|W_{:,j}\|_2.

However, this formulation does not explicitly incorporate the actual quantization error of each column.

Under a diagonal approximation to the quadratic reconstruction objective, the quantization-induced loss is instead related to terms of the form

Hjj∥ΔW:,j∥22,H_{jj}\|\Delta W_{:,j}\|_2^2,

where ΔW:,j\Delta W_{:,j} denotes the error introduced by quantization.

Consequently, large Hessian-weighted parameter magnitudes do not necessarily imply large quantization-induced errors. The proposed metric may be a useful heuristic, but the paper does not sufficiently establish why it should outperform simpler magnitude-based, activation-based, or quantization-error-aware alternatives.

In addition, the method combines expert-level mixed precision with column-level INT8 protection, which may introduce nontrivial storage metadata, packing requirements, and heterogeneous execution overhead. The paper primarily reports nominal bit-widths, model sizes, and accuracy metrics, without demonstrating end-to-end inference latency or throughput benefits.

The storage accounting also requires clarification: Section 5.4 refers to a 10.15 GB full model size when discussing metadata, whereas Table 3 reports 9.72 GB for the quantized Qwen1.5-MoE model.

Without a clearer effective bit-budget analysis and practical execution measurements, it is difficult to fully assess the claimed compression–efficiency trade-off relative to existing mixed-precision approaches.

Questions for authors

Q1. Validity of the token-type specialization analysis (related to W1).

Could the authors report the global distribution of the four token categories in the calibration corpus and compare expert-level token-type proportions against an appropriate random-routing baseline preserving that distribution?

In particular, does the claimed specialization remain statistically significant after accounting for the natural imbalance among content words, function words, special symbols, and punctuation?

Additionally, how are word-level POS labels aligned with subword tokens used by the evaluated LLM tokenizers?

Q2. Independent contribution of semantic information (related to W1).

Could the authors investigate whether token-type importance predicts quantization sensitivity after controlling for expert activation frequency, activation magnitude, or measured reconstruction error?

A comparison against a precision allocation strategy using frequency and direct quantization sensitivity, rather than heuristic POS-based weights, would help establish whether linguistic information provides an independent and practically useful signal.

Q3. Justification of the column sensitivity criterion (related to W2).

Why is column sensitivity defined as the product of the Hessian diagonal element and the original weight-column norm, rather than a metric incorporating the actual quantization error?

Could the authors compare the proposed ranking against random column selection, weight magnitude, activation sensitivity, and Hessian-weighted quantization error under identical INT8 protection budgets?

Q4. Effective bit budget and deployment performance (related to W2).

Could the authors clarify whether the reported average bit-width includes the protected INT8 columns, scaling factors, zero points, and other precision-allocation metadata?

How should the difference between the reported 10.15 GB and 9.72 GB quantized model sizes be interpreted?

If available, measurements of actual inference latency, throughput, or memory bandwidth would help evaluate the practical implications of the heterogeneous bit-width assignments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.