acceptodds
Withdrawn from ICLR 2027

Rep-Q: Replaceability-Aware Mixed-Precision Quantization for MoE LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixture-of-Experts Large Language Models (MoE-LLMs) can expand model capacity while controlling computational costs, yet their massive expert parameters still introduce substantial memory overhead. Mixed-precision quantization mitigates this issue by assigning differentiated bit widths to distinct experts. Existing approaches mostly assess expert importance based on activation frequency or loss sensitivity but largely overlook expert replaceability, i.e., whether the functionality of one expert can be fulfilled by others. Highly replaceable experts encode knowledge already covered by other experts, but experts with low replaceability require higher precision. Accordingly, this paper proposes Rep-Q, a mixed-precision quantization framework that jointly models expert replaceability and importance for quantization bit-width adjustment. (1) To translate the directly unmeasurable replaceability among experts into a computable metric, we design a similarity score to quantify similarities between expert output representations, establishing a quantitative foundation for modeling information dependencies across experts. (2) Leveraging these similarity scores, we construct a unified expert uniqueness metric by aggregating representation similarities between each expert and all remaining experts. This metric identifies single-expert replaceability from pairwise expert correlations and quantifies the irreplaceability of each expert’s information, enabling the transition from relationship-level measurements to individual-level evaluations. (3) Subject to quantization budget constraints, we jointly model expert quantization sensitivity and uniqueness scores. This allocation ensures experts with high importance and low replaceability are assigned higher precision. Evaluations across multiple datasets on Mixtral 8×7B (pure routing experts) and DeepSeek-V2-Lite (with shared experts) demonstrate that under an identical quantization budget, Rep-Q yields consistent improvements in perplexity and downstream task accuracy over existing expert-level mixed-precision quantization baselines, and exhibits strong cross-architecture generalization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, LLM, Quantization, mixed-precision quantization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VDunOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Rep-Q, a mixed-precision quantization method for MoE models. It combines expert importance with a CKA-based estimation of expert replaceability to allocate bit widths. Experiments on two MoE models show improvements over several quantization baselines, including GEMQ and MC-MoE.

Critical strengths
  1. The paper is generally clear and easy to follow.
  2. The evaluation covers multiple benchmarks and average expert bit-width budgets. The selected baselines are relevant.
Critical weaknesses
  1. The evaluation is limited to old models like Mixtral 8×7B and DeepSeek-V2-Lite. Recent MoE models may have different patterns of expert redundancy due to changes in architecture and training. This limits the evidence for generalization.
  2. CKA has been used to study similarity and redundancy among MoE experts. [1] uses it to compare expert outputs for pruning and merging, and [2] applies it for expert weights merging.
  3. The gains over GEMQ on DeepSeek-V2-Lite are small at the 2.0 and 2.5 bit: Table 3 shows improvements of only 0.05 and 0.09 percentage points in average accuracy. The perplexity results are also nearly identical.
  4. The paper does not report system-level results, such as actual model size, peak GPU memory use, and inference latency.

[1] Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts

[2] Delta Decompression for MoE-based LLMs Compression

Questions for authors
  1. Could you evaluate Rep-Q on more recent MoE models, such as Qwen3.5-35B-A3B and Gemma4-26B-A4B, and include reasoning benchmarks such as AIME, GPQA Diamond, and LiveCodeBench?
  2. Could you report model size, peak GPU memory use, inference latency and throughput under matched hardware and inference settings?
Reviewer_o92pOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Rep-Q is an expert-level mixed-precision quantization method for MoE language models. It argues that bit allocation should consider expert replaceability in addition to activation frequency or loss sensitivity. Using linear CKA on routed expert outputs from common calibration tokens, it constructs pairwise representation similarities and an expert uniqueness score. The score modifies a baseline importance-driven ranking while preserving the expert bit-width budget; shared-expert and purely routed architectures use different reference sets and adjustment procedures. Experiments with GPTQ on Mixtral 8×7B and DeepSeek-V2-Lite show gains over selected mixed-precision baselines, especially at lower average expert bit widths; improvements on DeepSeek-V2-Lite are small at 2.0–2.5 bits.

Critical strengths
  1. Meaningful problem. Activation frequency or quantization sensitivity alone does not directly characterize functional overlap among experts. Incorporating inter-expert relationships into bit-width allocation is therefore a worthwhile research direction.
  2. Easy to integrate into existing pipelines. Rep-Q adjusts only the expert bit-width configuration without changing the underlying GPTQ quantizer. As designed, it also introduces no additional inference-time computation.
  3. Some empirical gains at low bit widths. On Mixtral 8x7B, the method outperforms the compared approaches at several bit-width budgets, with its benefits being particularly relevant under lower-bit settings. The paper also evaluates DeepSeek-V2-Lite, which includes shared experts.
Critical weaknesses
  1. CKA similarity does not directly establish expert replaceability. The paper provides no further theoretical analysis or mathematical derivation under quantization. The claim that “experts with lower uniqueness should receive lower bit-widths” therefore remains an empirical hypothesis rather than a conclusion derived from representation similarity.
  2. The bit-width refinement procedure is insufficiently specified for reproduction. Section 3.4 repeatedly relies on notions such as “significant differences,” a “preset threshold,” “bounded error increment,” and “sorting consistency,” but does not define the thresholds, the error-cost computation, layer-selection rule, or a complete algorithm. Moreover, the procedures differ between Mixtral and DeepSeek-V2-Lite. It is therefore difficult to determine which design choices are responsible for the reported gains, or to reproduce the method faithfully.
  3. Some results do not support the wording of stable, significant improvements. On DeepSeek-V2-Lite, Rep-Q improves the seven-task average over GEMQ by only 0.60, 0.05, and 0.09 percentage points at 1.5, 2.0, and 2.5 bits, respectively. At 1.5 bits, C4 perplexity also changes from 17.49 to 17.52. In Table 1, at 1.875 bits on Mixtral, Rep-Q obtains 60.58% average accuracy, below MC-MoE’s 60.91%, contradicting the text’s statement that Rep-Q is consistently higher than MC-MoE across the full budget range. Since no variation across calibration samples or random seeds is reported, it is difficult to assess whether the small differences are reliable.
  4. The current ablation does not sufficiently test the core hypothesis. Fully reversing the uniqueness ranking, especially at 1.5 bits, may create a clearly implausible allocation and does not establish that CKA-based uniqueness is better than simpler signals. Useful budget-matched baselines would include cosine similarity, maximum similarity rather than average similarity, and similarity weighted by co-routing frequency. A more direct experiment would quantify whether lowering one expert’s precision can actually be compensated for by other experts.
Questions for authors
  1. Could the authors directly validate the relationship between CKA-based uniqueness and the actual performance degradation caused by reducing an expert’s precision? Further theoretical analysis and mathematical derivations would also help substantiate the paper’s claims.
  2. Why does the pure routed-expert architecture use average similarity to all routed experts, whereas the shared-expert architecture compares a routed expert only to the shared expert(s)? Would maximum similarity, or similarity weighted by co-routing frequency, yield different conclusions? Please also clarify how the two shared experts in DeepSeek-V2-Lite are handled.
  3. Can the authors provide complete pseudocode for Section 3.4, including all thresholds, the definition and computation cost of each error increment, the layer-selection rule, and the definition of sorting consistency?
  4. For the small gains on DeepSeek-V2-Lite and the regression at Mixtral 1.875 bits, can the authors report means and standard deviations across multiple calibration sets and random seeds, and revise the statement that Rep-Q consistently outperforms MC-MoE over the full budget range?
Reviewer_ccw9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Rep-Q, which uses expert uniqueness as an additional signal for mixed-precision quantization of MoE models. It uses CKA-based output similarity to refine existing bit-width allocations under a fixed expert weight budget. Experiments on Mixtral and DeepSeek-V2-Lite show improvements in several configurations.

Critical strengths
  1. Considering the relationship of expert redundancy and importance is a useful and novel direction.
  2. The method integrates with existing quantization pipelines while preserving per-layer bit-width distributions.
  3. Results show promising improvements, particularly at low precision.
Critical weaknesses
  1. Equations (5) – (6) do not define how to aggregate outputs or similarities when multiple shared experts are present, leaving the method unclear for both the evaluated Deepseek and other architectures with multiple shared experts.
  2. For MoE models, shared experts remain active and unchanged during routed-expert bit reallocation. Therefore, similarity alone does not explain why they would mitigate a routed expert’s quantization error. This connection requires further validation.
  3. In Section 3.3, Rep-Q use average similarity for pure routing expert architecture. However, two nearly identical experts could both receive high uniqueness scores if they differ from the remaining experts. This conflicts with the intended interpretation of redundancy.
  4. In ablation study, forward and reverse ordering differ by only 0.23 and 0.09 percentage points at 2.0 and 2.5 bits. Comparisons against other similarity metrics would be more convincing.
  5. In Section 3.4, the correction procedure lacks explicit definitions and parameter settings for ranking mismatches, error-increment thresholds, and candidate selection, limiting reproducibility.
Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.