acceptodds
Withdrawn from ICLR 2027

SAGE-VQ: Structure-Adaptive Gating-Aware Expert Vector Quantization with Output Correction for MoE LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixture-of-Experts (MoE) large language models (LLMs) expand capacity through sparse activation, yet expert storage remains a bottleneck for single-GPU and on-device deployment. Post-training vector quantization (VQ) enables ultra-low-bit compression, but dense-LLM methods overlook heterogeneous expert sensitivity, architecture-dependent output composition, and routing-weighted errors. We introduce SAGE-VQ, a structure-adaptive, gating-aware expert VQ framework that aligns precision allocation and output correction with MoE structure. SAGE-VQ selects architecture-appropriate sensitivity signals to assign expert bit-widths under a fixed average bit budget. It incorporates routing weights into the correction objective to mitigate norm and scale mismatches in quantized expert outputs. Correction factors are fused before inference without additional runtime latency. On Qwen2-MoE, SAGE-VQ achieves 2.25-bit expert compression, reducing WikiText-2 perplexity by 1.21 relative to the MoE-aware mixed-precision baseline MxMoE while limiting average accuracy degradation across seven zero-shot benchmarks to 1.42 percentage points relative to FP16.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-experts, post-training quantization, vector quantization, ultra-low-bit compression, mixed-precision quantization, routing-aware error compensation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wpHrOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAGE-VQ, an ultra-low-bit mixed-precision vector quantization framework for Mixture-of-Experts (MoE) large language models. The authors argue that the output combination schemes of different MoE architectures affect the estimation of expert importance, and token-specific routing weights should be considered for the quantization error of expert outputs. To this end, the paper presents three main modules: Structure-Adaptive Policy (SAP) selects importance metrics from routing frequency, weight reconstruction error and output reconstruction loss according to the MoE output topology; Mixed Vector Quantization (MVQ) allocates 1–4 bits to different experts via a greedy strategy and adopts QuIP#-style (E₈) lattice VQ; Routing-aware Scale Correction (RSC) corrects quantized expert outputs by combining channel-wise norm correction and routing-weighted least squares. Experiments are conducted on four MoE models. On Qwen2-MoE, an average 71.14% accuracy across seven tasks is achieved with an average 2.25-bit expert precision, representing a 1.42 percentage point drop compared with FP16. A maximum full-model decode speedup of 1.30× is reported on Mixtral-8×7B.

Critical strengths

Strength 1: The paper investigates the impact of different MoE output topologies on expert quantization sensitivity, conducting experiments on three architectures: shared-gated, shared-ungated and routed-only. By comparing routing frequency, weight reconstruction error and output reconstruction loss, the authors demonstrate the performance differences of various importance signals across different models. These results offer valuable insights for understanding the relationship between MoE architectural variations and expert-wise mixed-precision allocation.

Strength 2: The paper achieves promising ultra-low bit quantization results on four MoE models. In particular, Qwen2-MoE achieves accuracy close to FP16 under 2.25-bit expert compression. Ablation studies of RSC also show that routing-aware correction can further reduce perplexity compared with plain scale correction without routing weights. In addition, the authors provide full-model decode latency tests, illustrating that the proposed method focuses not only on offline accuracy but also on practical inference efficiency.

Critical weaknesses

Weakness 1: Limited novelty of the core method, which is highly related to existing MoE quantization and output correction works. QuIP# [1] has proposed ultra-low-bit VQ based on Hadamard rotation and (E₈) lattice codebooks, while this paper directly adopts its rotation and VQ backend without proposing new codebooks or quantization reconstruction mechanisms. MxMoE [2] and GEMQ [3] have studied expert-wise importance estimation, mixed-precision allocation and global bit-width optimization, and MoEQuant [4] has introduced token–expert affinity into the quantization process. More importantly, KBVQ-MoE [5] has proposed channel-wise affine output correction for MoE VQ to mitigate output distribution shift caused by expert aggregation. Therefore, the main new contribution of SAGE-VQ appears to be selecting existing importance metrics according to the model structure, combined with heuristic bit-width allocation and routing-weighted scale correction.

Weakness 2: SAP is closer to fixed rules based on experimental observations rather than a structure-adaptive allocation mechanism with sufficient generalization grounds. In Eq. (7), SAP directly maps the three architectures, shared-gated, shared-ungated and routed-only, to routing frequency, output reconstruction loss and weight reconstruction error respectively. This rule is mainly based on the WikiText-2 PPL comparison of three representative models in Table 1, without deriving why specific metrics should be used for different architectures from the mathematical properties of MoE output topology. In particular, there are only a limited number of representative models for each of the three topologies, making it impossible to distinguish whether performance differences stem from output topology, or model scale, expert utilization, training distribution, or VQ sensitivity.

Weakness 3: The bit-width allocation strategy of MVQ is rather heuristic and lacks an explicit connection with the actual quantization loss. Algorithm 1 first initializes all routed experts to 1-bit, then gradually improves expert precision according to a fixed priority (s_{l,e}/(b_{l,e}+1)). This strategy adopts a simple diminishing marginal return assumption, but it does not prove that this priority aligns with the actual VQ reconstruction-loss reduction across different bit widths, nor does it establish corresponding optimization objectives or optimality guarantees. In addition, the authors predefine the budget for each layer and perform greedy allocation within each layer instead of jointly optimizing the quantization cost of all MoE layers. Compared with existing global allocation methods [3], the independent advantages of this strategy remain unclear. The paper also lacks direct comparisons with practical marginal distortion-based allocation, global ILP or other greedy rules, making it difficult to judge whether the current allocation mechanism is sufficiently effective.

Weakness 4: There is a discrepancy between the theoretical objective of RSC and the true MoE aggregate output reconstruction loss, and its independent innovation needs to be strengthened. Eq. (9) uses routing weights to weight the local output reconstruction error of each expert, and then obtains channel-wise scale factors via standard weighted least squares. However, the squared error of the true MoE aggregate output usually includes squared terms of routing weights as well as cross error terms between different routed experts. Therefore, the current objective is more like a routing-weighted per-expert reconstruction surrogate instead of the exact form of the full MoE output error.

Weakness 5: Evaluations on actual compression cost, execution efficiency and complex tasks remain insufficient. The reported 2.0–2.5-bit in the paper refers to the expert-weight average bit-width; shared experts are fixed at 4-bit, and non-expert parameters are usually kept in FP16. The paper does not provide the complete effective BPW and actual GPU memory footprint including VQ codebooks, scales, rotation metadata, correction metadata and unquantized parameters. Meanwhile, the specific codebook construction, encoding configuration and implementation details for 1/2/3/4-bit VQ are not fully presented.

Questions for authors

Q1: QuIP# [1] has provided (E₈) lattice VQ; MxMoE [2] and GEMQ [3] have studied expert-wise mixed-precision allocation for MoE; MoEQuant [4] and KBVQ-MoE [5] have explored routing-aware quantization and output correction respectively. Could the authors further clarify the independent technical innovations of SAP, MVQ and RSC compared with these existing methods? In particular, could they illustrate what new properties emerge from their combination that cannot be directly obtained by prior approaches, and verify the independent contribution of each component via matched-backend cross ablation?

Q2: SAP adopts a fixed topology-to-signal mapping in Eq. (7), yet this rule is mainly established based on WikiText-2 PPL results on three representative models. Could the authors further explain why shared-gated, shared-ungated and routed-only architectures are suited for these three specific importance metrics respectively, and provide corresponding theoretical or empirical evidence? In addition, is this rule determined on data independent of the final evaluation set? Could all candidate signals be compared on more MoE models excluded from rule selection to validate the cross-architecture generalization of SAP, rather than only performing empirical selection on tested models?

Q3: MVQ uses (s_{l,e}/(b_{l,e}+1)) as the priority to progressively increase expert precision, but the relationship between this rule and the actual reduction in VQ quantization loss remains unclear. Could the authors explain the design rationale of this formula, and conduct direct comparisons with actual marginal reconstruction-loss allocation, global ILP [3] and other importance allocation strategies under the same average bit width? Furthermore, could the codebook configuration, probe bit-width, residual encoding, scales and associated storage overhead of (E₈)-based VQ at different bit widths be specified to support algorithm reproducibility?

Q4: RSC constructs a per-expert least-squares objective using first-power routing weights in Eq. (9). However, the squared error of the full aggregated MoE output also involves squared routing weights and cross terms between different experts. Could the authors explain the theoretical basis for the current surrogate objective, and compare unweighted correction, routing-weighted correction, squared-routing-weighted correction and full-MoE-output correction? In addition, could the difference between fitting with FP16 and real quantized down-projection inputs be verified, and ablation comparisons with the affine output correction of KBVQ-MoE [5] be carried out under the same VQ backend?

Q5: Could the authors supplement complete effective BPW and practical GPU memory accounting, separately accounting for routed/shared expert weights, VQ codebooks, scales, rotation/correction metadata and non-expert parameters? Moreover, the current inference efficiency is only tested for Mixtral-8×7B batch-8 decode on two A100 80GB GPUs. Could tests be added for batch-1, prefill, different models and single-GPU deployment, with comparisons against efficient inference implementations using the same VQ backend? Finally, could more challenging tasks such as GSM8K, MATH-500, AIME or MBPP be supplemented to examine the retention of complex reasoning and code capabilities after ultra-low-bit quantization?

Reviewer_usf3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAGE-VQ, a PTQ framework for MoE LLMs. There are two contributions: 1) Architecture-dependent MPQ , that identify bit-width of each expert based on 3 different metrics on 3 different architecture correspondingly; 2) Routing-aware scale correction that reduce quantized model error by matching norm across channel, followed by the optimization of a weighted MSE loss. Experiments across 4 MoE over 3 architectures show consistent improvement over compared baselines.

Critical strengths
  1. The proposed method has significant improvements across 4 MoE models in perplexity and accuracy
  2. The paper targets low-bit MoE quantization (around 2 bit), an important problem for MoE quantization. It also provides inference speed using mixed-bit VQ kernels.
  3. The paper investigates a wide range of expert importance metrics and output correction strategies, with useful ablation studies
Critical weaknesses
  1. Limited technical novelty: The framework seems to utilize a combination of existing methods for MPQ and output correction; most components have been proposed in previous work. The idea of different precision per expert and output correction is not new. All three heuristics proposed to allocate the bit-width based on expert importance are simple heuristics widely used in quantization. The main novelty seems to be selecting different heuristics for different architectures.
  2. KBVQ-MoE does not use mixed-precision; it is not really surprising, as mixed-precision often beats uniform bit methods with similar settings. How is the proposed output correction compared to the heuristic proposed in KBVQ-MoE? Can you run some ablations on this?
  3. Lack of justification for the method design: There seems to be no intuitive explanation why MOE for 3 different architectures requires 3 different metrics according to how the author design, beyond empirical finding, making the current method not generalizable.
Questions for authors

See the weakness.

Reviewer_1tf9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies the imbalance between different experts and proposes two observations: different moe output topologies need different allocation signals and routing weights are important information to correct the quantization scales. Based on the two observations, this paper proposes a framework "SAGE-VQ". Firstly, SAGE-VQ selects the allocation signal according to different output structure of the MoE LLM, including shared gated MoE, shared ungated MoE and routed only MoE. The corresponding selected signals are routing frequency, output loss and weight error respectively (SAP). After SAP, the importance is calculated based on the allocation signal and the corresponding allocation strategy is applied. Secondly, two types of correction are applied to correct the norm and routing-weight importance of the quantized weights. Experiments and ablation studies show the effectiveness of proposed framework and SAGE-VQ achieves higher accuracy and lower ppl than the compared baselines, with lower decoding latency.

Critical strengths
  1. SAGE-VQ identifies that different MoE output topologies need different allocation signals, and proposes three types of allocation signals and their corresponding scores. This customized allocation strategy could allocate bit-width based on different topologies of MoE LLMs.
  2. SAGE-VQ uses routing-aware scaling and a norm correction factor to correct the quantization scales. This correction can mitigate quantization-induced errors in individual expert outputs and suppress their accumulation during routing-weighted aggregation within an MoE layer.
Critical weaknesses
  1. The contribution of SAP is ambiguous. Ablation of SAP (table 4) only discusses three models with three types of topology. But whether other models with the same output topology have the same results is unknown.
  2. Comparison is unfair. Table 2 compares SAGE-VQ/MxMoE under 2.25-bit with Uniform-VQ/RTN under 2-bit. Additionally, peak GPU memory usage during inference and a more comprehensive analysis of decoding latency and throughput across different batch sizes should be provided.
Questions for authors
  1. Could you report the main results of MxMoE and SAGE-VQ under 2-bit quantization, which would allow for a fair comparison?
  2. Could you provide more analysis of the SAP ablation study? For example, demonstrate consistency across different models within the same type. This would help to demonstrate the effectiveness of SAP and its generalizability.
  3. Could you provide additional inference latency results across different batch sizes (e.g. bs = 1, 16, 32, 64)? This will give a more comprehensive analysis of inference efficiency.
  4. Could you report the actual peak GPU memory usage of SAGE-VQ under 2-bit quantization and compare it with the baselines? This would help us to assess the deployment cost of SAGE-VQ.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.