acceptodds
Under review as a conference paper at ICLR 2027

HeteroQuant: Resolving Expert Heterogeneity in MoE Quantization via Targeted Calibration

Abstract

Mixture-of-Experts (MoE) architectures substantially expand model parameter capacity while maintaining a limited computational cost per token through sparse activation and dynamic routing mechanisms. However, during practical deployment, the complete set of expert parameters usually needs to remain in memory, and the resulting large storage overhead restricts the application of MoE models in resource-constrained environments. Post-training quantization (PTQ) is an important technique for reducing the storage and inference costs of large language models. Nevertheless, due to the significant expert heterogeneity, domain preferences, and imbalanced activation patterns in MoE models, existing quantization methods often fail to sufficiently calibrate critical experts, leading to severe performance degradation. To address these challenges, we propose HeteroQuant, an effective, training-free post-training quantization framework designed to resolve expert heterogeneity through targeted calibration. Specifically, HeteroQuant achieves this by first leveraging the DSPy framework to construct a highly diverse multi-domain candidate pool, thereby comprehensively activating all expert positions to collect their global routing statistics. Based on these statistics, we implement a core two-step selection mechanism to systematically optimize the calibration data. In the first stage, we perform AlphaQ-guided structural-targeted scoring to prioritize candidate samples that precisely target and activate structurally critical experts. In the second stage, we execute activation-redundancy-aware filtering to eliminate samples with overlapping routing footprints, thereby preventing calibration waste while maximizing activation diversity. Experimental results demonstrate that under the W4A4 quantization setting of the Qwen1.5-MoE-A2.7B model, HeteroQuant achieves an average improvement of 5.09 points over the current SOTA method across eight downstream tasks.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, Post-Training Quantization, Large Language Models, Model Compression, Calibration, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hcdrOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HeteroQuant, a training-free calibration data selection framework for post-training quantization of Mixture-of-Experts (MoE) models. The proposed approach consists of three components: (1) DSPy-based multi-domain candidate generation, (2) AlphaQ-guided sample importance scoring based on expert structural importance and routing probabilities, and (3) redundancy-aware sample filtering based on layer-wise routing similarity. Experiments on three MoE architectures show improvements over several W4A4 PTQ baselines.

Critical strengths

1.The paper addresses a practical problem in MoE quantization: heterogeneous expert activation and insufficient coverage of important experts during calibration.

2.The framework is conceptually simple and does not require gradient-based model fine-tuning.

3.The experiments cover three different MoE models and report improvements on multiple downstream benchmarks.

Critical weaknesses
  1. Limited methodological novelty.

The proposed framework largely combines existing techniques, including DSPy-based synthetic data generation, AlphaQ-based expert importance estimation, and cosine-similarity-based greedy sample filtering. The principal algorithmic contribution is a weighted aggregation of expert importance and routing probabilities, followed by a conventional redundancy filter. The paper does not sufficiently establish why these particular formulations are necessary or superior to simpler alternatives, such as routing-frequency sampling, activation-based importance metrics, or standard diversity-based subset selection.


  1. Insufficient isolation of calibration data selection from other factors.

The main experiments compare HeteroQuant, which uses a newly generated multi-domain calibration pool, against baselines calibrated with WikiText-2. As a result, the improvements may be partly attributed to the change in data distribution rather than the proposed expert-aware selection strategy. Moreover, the paper includes expert-aware smoothing and scale-merging operations whose roles in the performance gains are not sufficiently isolated. A controlled comparison using an identical PTQ implementation and calibration candidate pool is necessary to establish the incremental contribution of the proposed selection algorithm.


  1. A gap between expert structural importance and quantization calibration sensitivity.

The motivation relies on pruning experiments demonstrating that removing structurally important experts causes severe performance degradation. However, pruning sensitivity does not necessarily imply quantization sensitivity or a greater marginal benefit from additional calibration samples. The paper does not directly demonstrate that AlphaQ-derived expert importance correlates with actual quantization-induced errors or calibration requirements. This weakens the justification for the central sample scoring objective.


  1. Unclear mechanism through which calibration data influence the reported W4A4 configuration.

The implementation employs dynamic per-token activation quantization together with expert-aware smoothing. Since dynamic activation quantization generally computes activation scales at inference time, the paper should explicitly identify which quantization parameters or transformations are estimated from calibration data and how the proposed selection strategy improves them. The relationship between the generic calibration formulation in Section 3.1 and the actual quantization configuration in Section 4.1 requires clarification.


  1. Limited ablation evidence and incomplete cost analysis.

The improvements attributed to individual components in Table 2 are relatively modest, and no variance estimates are provided. The complete method also does not achieve the best WikiText-2 perplexity among the ablated configurations. Furthermore, although the final calibration set contains 128 sequences, the overall pipeline requires synthetic data generation, target-model inference on 512 candidate sequences, spectral analysis, and routing-statistics collection. These additional costs should be quantified.


  1. Insufficient evaluation of robustness and generality.

The evaluation primarily focuses on W4A4, with reduced router precision evaluated on only one model. Additional experiments across quantization bit-widths, modern MoE architectures, candidate pool sizes, and redundancy thresholds would strengthen the generality claims. Important implementation details, particularly the similarity threshold and synthetic data generation configurations, should also be clarified.a

Questions for authors

1.How does AlphaQ-derived expert importance correlate with actual expert-wise quantization error, output reconstruction error, and the marginal benefit of additional calibration tokens?

2.What are the results when random sampling, routing-balanced sampling, importance-only sampling, and diversity-only sampling are applied to the same DSPy-generated candidate pool under an identical PTQ implementation?

3.Can the authors clarify exactly which quantization scales, smoothing factors, or other parameters are derived from calibration samples under dynamic per-token activation quantization?

4.Does the proposed calibration set consistently improve multiple independent PTQ methods when the quantization pipeline is held fixed?

5.What are the complete offline costs of generating and selecting calibration data, and how sensitive are the results to the candidate pool size and similarity threshold?

Reviewer_xw6DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HeteroQuant, a framework for constructing and selecting calibration data for post-training W4A4 quantization of MoE models. It first uses DSPy/DeepSeek to generate 512 multi-domain candidate sequences, scores samples by AlphaQ-derived expert structural importance, and removes redundancy using cosine similarity between layer-wise routing vectors, ultimately selecting 128 calibration sequences. Experiments cover OLMoE, Qwen1.5-MoE, and Mixtral, and report perplexity and results on eight downstream tasks.

Critical strengths
  • The research problem is important: long-tailed expert activation and heterogeneous structural importance in MoE models can indeed make standard calibration data inadequate.
  • The “structural-importance prioritization + routing-coverage deduplication” design is intuitive, and Equations 7–16 provide an understandable two-stage selection rule.
  • Table 1 covers three distinct MoE architectures under an aggressive W4A4 setting; Table 2 ablates the candidate pool, AlphaQ scoring, and filtering.
Critical weaknesses
  1. The claim of comparison under the “same compute budget” is unsupported. The baselines use 128 WikiText-2 calibration sequences, whereas HeteroQuant first generates and processes 512×4096 tokens through the target model, collects routing information for all experts, computes AlphaQ, and performs pairwise similarity filtering. Section 3.4 claims an equal budget solely because the final sets both contain 128 sequences, ignoring DSPy/DeepSeek calls, target-model autoregressive generation, routing statistics over at least 4× as many candidate tokens, and selection costs. The gains may arise from a larger, more expensive candidate pool rather than the proposed selection criteria.
  2. Core implementation details are missing, preventing reproducibility. The paper does not specify the exact DeepSeek model/API version; the complete DSPy Signature/Module/Optimizer configuration; the domain list and prompt templates; generation temperature and random seeds; γ in Equation 8; threshold τ in Equation 16; ε; the procedure when fewer than 128 candidates survive; or a code link. An AI Use Statement does not substitute for a Reproducibility Statement.
  3. Fair selection baselines on the same candidate pool are missing. The most important controls would perform random selection, domain-balanced selection, AlphaQ-only selection, diversity-only selection, and EBSS/MoEQuant-style selection from exactly the same 512 candidates. The “without DSPy” condition in Table 2 does not explain what replaces the candidate pool, so the benefits of generation and selection cannot be separated.
  4. There are no repeated experiments or uncertainty estimates. API generation, candidate sampling, routing statistics, and greedy deduplication may all be stochastic, yet the main results and ablations report no multi-seed runs, error bars, or significance analyses.
  5. Several textual claims do not match the tables. The abstract claims an average improvement of 5.09 points on Qwen, but in Table 1 HeteroQuant's 64.14 exceeds EAQuant's 59.25 by 4.89 points and QuaRot by 3.11 points. The itemized Qwen comparisons against QuaRot and EAQuant in Section 4.2 are supported by Table 1; however, the broad claim of being best on “all benchmarks” across the three models is too strong. For example, OLMoE PIQA is below QuaRot; Mixtral BoolQ is below EAQuant; Mixtral C-Eval is below MoEQuant; and WikiText-2 PPL is also not the best. The conclusion should be limited to average metrics or a majority of tasks.
  6. SOTA coverage is incomplete. The related-work section discusses DynaMo, EBSS/MoEQuant, and other methods directly targeting MoE calibration, but the main table does not systematically compare these most relevant baselines across all models.
  7. There is no direct evidence linking AlphaQ to calibration value. AlphaQ measures weight-spectrum structure and is used for bit-width allocation. The paper directly equates “structurally important/pruning-sensitive” with “should receive more calibration samples,” but these concepts are not equivalent. Figure 2 only shows pruning behavior for Mixtral on two ARC tasks and does not report, across models, correlations between AlphaQ and per-expert quantization error, post-calibration reconstruction error, or marginal calibration benefit.
  8. The local normalization in the deduplication mechanism is largely ineffective. Equation 13 divides each layer's activation vector by a positive scalar, after which Equation 14 computes cosine similarity. Except for the numerical-stability term ε, cosine similarity is invariant to this scalar rescaling. Thus, “equal weighting of layers” actually comes from explicitly averaging per-layer cosine similarities in Equation 14, not from this normalization step. Moreover, a fixed threshold τ does not guarantee that 128 sequences can be selected from 512 candidates, and no fallback procedure is specified.
Questions for authors
  1. Under a shared pool of 512 candidates and a matched total forward-pass/API token budget, what are HeteroQuant's gains over random, domain-balanced, EBSS, AlphaQ-only, and diversity-only selection?
  2. Please provide complete values and configurations for τ, γ, and DSPy/DeepSeek, along with sensitivity analyses. Are results stable across generation and selection seeds?
  3. How much GPU time, how many API tokens and target-model forward passes, and how much peak memory are required for candidate-pool construction, AlphaQ computation, and similarity filtering, respectively?
  4. How is the 5.09 figure in the abstract derived from Table 1?
  5. Do AlphaQ scores correlate with expert quantization sensitivity and with the actual benefit of increased calibration coverage? How does the scalar normalization in Equation 13 change the cosine value in Equation 14?
Reviewer_zTkMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of highly heterogeneous expert activation and insufficient calibration-data coverage in post-training quantization (PTQ) of MoE models, and proposes HeteroQuant, a training-free calibration-data construction and selection framework. The method consists of three components: DSPy-driven multi-domain candidate-pool generation, AlphaQ-guided scoring of structurally sensitive experts, and activation-redundancy filtering based on routing-pattern similarity. Experiments are conducted on three MoE models—OLMoE, Qwen1.5-MoE, and Mixtral—under W4A4R16, with additional W4A4R8 results. HeteroQuant achieves better or comparable PPL and downstream average accuracy than QuaRot, EAQuant, MoEQuant, and other methods, and the ablation study supports the contribution of each component.

Critical strengths
  1. The paper provides a complete description of the specific challenges and the overall methodology.

  2. The paper treats calibration data as an optimizable object: it first constructs a multi-domain candidate pool, then uses structural-importance scoring and redundancy filtering to balance critical-expert activation and routing diversity, while remaining compatible with existing PTQ pipelines.

  3. The proposed method outperforms existing methods on multiple models and tasks.

Critical weaknesses
  1. The technical novelty is limited because DSPy-based generation, AlphaQ importance, and cosine-similarity-based redundancy removal are all existing techniques; the contribution mainly lies in their combination and adaptation. The authors should more clearly articulate the essential differences from routing-balance strategies such as EBSS/MoEQuant and from AlphaQ’s calibration-free bit allocation, and provide a mechanistic explanation or theoretical connection for why the proposed method reduces quantization error.

  2. Some functions in the equations are not explained, which hinders reader understanding—for example, the clip() and round() functions in Eq. (1), and the distinction between commas and semicolons in Eq. (3).

  3. Figure 4 measures textual domain diversity rather than expert activation coverage. The paper does not report the number of tokens activating each expert, expert activation entropy, long-tail expert coverage, or per-expert activation distribution comparisons with WikiText-2/C4.

  4. The correlation between AlphaQ scores and quantization sensitivity is not validated. Figure 2 concerns pruning sensitivity, not quantization-error sensitivity. AlphaQ was originally designed for bit allocation; mapping it to sample scoring lacks evidence that experts with high AlphaQ importance indeed exhibit larger activation outliers or quantization errors.

  5. The ablation in Table 2 is conducted only on OLMoE, and the average accuracy is computed over only six zero-shot tasks, excluding GSM8K and C-Eval. However, the largest improvements in Table 1 appear precisely on Qwen’s GSM8K (40.64 vs. 19.71) and C-Eval (71.47 vs. 65.08). The ablation therefore cannot explain which component contributes to these key gains.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.