- Rating
- 1
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SAGE-VQ, an ultra-low-bit mixed-precision vector quantization framework for Mixture-of-Experts (MoE) large language models. The authors argue that the output combination schemes of different MoE architectures affect the estimation of expert importance, and token-specific routing weights should be considered for the quantization error of expert outputs. To this end, the paper presents three main modules: Structure-Adaptive Policy (SAP) selects importance metrics from routing frequency, weight reconstruction error and output reconstruction loss according to the MoE output topology; Mixed Vector Quantization (MVQ) allocates 1–4 bits to different experts via a greedy strategy and adopts QuIP#-style (E₈) lattice VQ; Routing-aware Scale Correction (RSC) corrects quantized expert outputs by combining channel-wise norm correction and routing-weighted least squares. Experiments are conducted on four MoE models. On Qwen2-MoE, an average 71.14% accuracy across seven tasks is achieved with an average 2.25-bit expert precision, representing a 1.42 percentage point drop compared with FP16. A maximum full-model decode speedup of 1.30× is reported on Mixtral-8×7B.
Critical strengths
Strength 1: The paper investigates the impact of different MoE output topologies on expert quantization sensitivity, conducting experiments on three architectures: shared-gated, shared-ungated and routed-only. By comparing routing frequency, weight reconstruction error and output reconstruction loss, the authors demonstrate the performance differences of various importance signals across different models. These results offer valuable insights for understanding the relationship between MoE architectural variations and expert-wise mixed-precision allocation.
Strength 2: The paper achieves promising ultra-low bit quantization results on four MoE models. In particular, Qwen2-MoE achieves accuracy close to FP16 under 2.25-bit expert compression. Ablation studies of RSC also show that routing-aware correction can further reduce perplexity compared with plain scale correction without routing weights. In addition, the authors provide full-model decode latency tests, illustrating that the proposed method focuses not only on offline accuracy but also on practical inference efficiency.
Critical weaknesses
Weakness 1: Limited novelty of the core method, which is highly related to existing MoE quantization and output correction works. QuIP# [1] has proposed ultra-low-bit VQ based on Hadamard rotation and (E₈) lattice codebooks, while this paper directly adopts its rotation and VQ backend without proposing new codebooks or quantization reconstruction mechanisms. MxMoE [2] and GEMQ [3] have studied expert-wise importance estimation, mixed-precision allocation and global bit-width optimization, and MoEQuant [4] has introduced token–expert affinity into the quantization process. More importantly, KBVQ-MoE [5] has proposed channel-wise affine output correction for MoE VQ to mitigate output distribution shift caused by expert aggregation. Therefore, the main new contribution of SAGE-VQ appears to be selecting existing importance metrics according to the model structure, combined with heuristic bit-width allocation and routing-weighted scale correction.
Weakness 2: SAP is closer to fixed rules based on experimental observations rather than a structure-adaptive allocation mechanism with sufficient generalization grounds. In Eq. (7), SAP directly maps the three architectures, shared-gated, shared-ungated and routed-only, to routing frequency, output reconstruction loss and weight reconstruction error respectively. This rule is mainly based on the WikiText-2 PPL comparison of three representative models in Table 1, without deriving why specific metrics should be used for different architectures from the mathematical properties of MoE output topology. In particular, there are only a limited number of representative models for each of the three topologies, making it impossible to distinguish whether performance differences stem from output topology, or model scale, expert utilization, training distribution, or VQ sensitivity.
Weakness 3: The bit-width allocation strategy of MVQ is rather heuristic and lacks an explicit connection with the actual quantization loss. Algorithm 1 first initializes all routed experts to 1-bit, then gradually improves expert precision according to a fixed priority (s_{l,e}/(b_{l,e}+1)). This strategy adopts a simple diminishing marginal return assumption, but it does not prove that this priority aligns with the actual VQ reconstruction-loss reduction across different bit widths, nor does it establish corresponding optimization objectives or optimality guarantees. In addition, the authors predefine the budget for each layer and perform greedy allocation within each layer instead of jointly optimizing the quantization cost of all MoE layers. Compared with existing global allocation methods [3], the independent advantages of this strategy remain unclear. The paper also lacks direct comparisons with practical marginal distortion-based allocation, global ILP or other greedy rules, making it difficult to judge whether the current allocation mechanism is sufficiently effective.
Weakness 4: There is a discrepancy between the theoretical objective of RSC and the true MoE aggregate output reconstruction loss, and its independent innovation needs to be strengthened. Eq. (9) uses routing weights to weight the local output reconstruction error of each expert, and then obtains channel-wise scale factors via standard weighted least squares. However, the squared error of the true MoE aggregate output usually includes squared terms of routing weights as well as cross error terms between different routed experts. Therefore, the current objective is more like a routing-weighted per-expert reconstruction surrogate instead of the exact form of the full MoE output error.
Weakness 5: Evaluations on actual compression cost, execution efficiency and complex tasks remain insufficient. The reported 2.0–2.5-bit in the paper refers to the expert-weight average bit-width; shared experts are fixed at 4-bit, and non-expert parameters are usually kept in FP16. The paper does not provide the complete effective BPW and actual GPU memory footprint including VQ codebooks, scales, rotation metadata, correction metadata and unquantized parameters. Meanwhile, the specific codebook construction, encoding configuration and implementation details for 1/2/3/4-bit VQ are not fully presented.
Questions for authors
Q1: QuIP# [1] has provided (E₈) lattice VQ; MxMoE [2] and GEMQ [3] have studied expert-wise mixed-precision allocation for MoE; MoEQuant [4] and KBVQ-MoE [5] have explored routing-aware quantization and output correction respectively. Could the authors further clarify the independent technical innovations of SAP, MVQ and RSC compared with these existing methods? In particular, could they illustrate what new properties emerge from their combination that cannot be directly obtained by prior approaches, and verify the independent contribution of each component via matched-backend cross ablation?
Q2: SAP adopts a fixed topology-to-signal mapping in Eq. (7), yet this rule is mainly established based on WikiText-2 PPL results on three representative models. Could the authors further explain why shared-gated, shared-ungated and routed-only architectures are suited for these three specific importance metrics respectively, and provide corresponding theoretical or empirical evidence? In addition, is this rule determined on data independent of the final evaluation set? Could all candidate signals be compared on more MoE models excluded from rule selection to validate the cross-architecture generalization of SAP, rather than only performing empirical selection on tested models?
Q3: MVQ uses (s_{l,e}/(b_{l,e}+1)) as the priority to progressively increase expert precision, but the relationship between this rule and the actual reduction in VQ quantization loss remains unclear. Could the authors explain the design rationale of this formula, and conduct direct comparisons with actual marginal reconstruction-loss allocation, global ILP [3] and other importance allocation strategies under the same average bit width? Furthermore, could the codebook configuration, probe bit-width, residual encoding, scales and associated storage overhead of (E₈)-based VQ at different bit widths be specified to support algorithm reproducibility?
Q4: RSC constructs a per-expert least-squares objective using first-power routing weights in Eq. (9). However, the squared error of the full aggregated MoE output also involves squared routing weights and cross terms between different experts. Could the authors explain the theoretical basis for the current surrogate objective, and compare unweighted correction, routing-weighted correction, squared-routing-weighted correction and full-MoE-output correction? In addition, could the difference between fitting with FP16 and real quantized down-projection inputs be verified, and ablation comparisons with the affine output correction of KBVQ-MoE [5] be carried out under the same VQ backend?
Q5: Could the authors supplement complete effective BPW and practical GPU memory accounting, separately accounting for routed/shared expert weights, VQ codebooks, scales, rotation/correction metadata and non-expert parameters? Moreover, the current inference efficiency is only tested for Mixtral-8×7B batch-8 decode on two A100 80GB GPUs. Could tests be added for batch-1, prefill, different models and single-GPU deployment, with comparisons against efficient inference implementations using the same VQ backend? Finally, could more challenging tasks such as GSM8K, MATH-500, AIME or MBPP be supplemented to examine the retention of complex reasoning and code capabilities after ultra-low-bit quantization?