- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Rep-Q, a mixed-precision quantization method for MoE models. It combines expert importance with a CKA-based estimation of expert replaceability to allocate bit widths. Experiments on two MoE models show improvements over several quantization baselines, including GEMQ and MC-MoE.
Critical strengths
- The paper is generally clear and easy to follow.
- The evaluation covers multiple benchmarks and average expert bit-width budgets. The selected baselines are relevant.
Critical weaknesses
- The evaluation is limited to old models like Mixtral 8×7B and DeepSeek-V2-Lite. Recent MoE models may have different patterns of expert redundancy due to changes in architecture and training. This limits the evidence for generalization.
- CKA has been used to study similarity and redundancy among MoE experts. [1] uses it to compare expert outputs for pruning and merging, and [2] applies it for expert weights merging.
- The gains over GEMQ on DeepSeek-V2-Lite are small at the 2.0 and 2.5 bit: Table 3 shows improvements of only 0.05 and 0.09 percentage points in average accuracy. The perplexity results are also nearly identical.
- The paper does not report system-level results, such as actual model size, peak GPU memory use, and inference latency.
[1] Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
[2] Delta Decompression for MoE-based LLMs Compression
Questions for authors
- Could you evaluate Rep-Q on more recent MoE models, such as Qwen3.5-35B-A3B and Gemma4-26B-A4B, and include reasoning benchmarks such as AIME, GPQA Diamond, and LiveCodeBench?
- Could you report model size, peak GPU memory use, inference latency and throughput under matched hardware and inference settings?