acceptodds
Under review as a conference paper at ICLR 2027

MoE-VQ: A Model-Agnostic Gaussian Vector Codebook for 2-Bit Expert Weight Quantization

Abstract

Mixture-of-Experts (MoE) models concentrate most parameters in sparsely activated experts, making weight memory rather than computation the primary deployment bottleneck. However, scalar post-training quantization incurs severe and irreversible distortion at 2 bits. We show that, after row-group RMS normalization, expert MLP weights from six MoE architectures closely follow a shared standard Gaussian distribution. This observation enables a single, model-agnostic vector-quantization codebook: we fit an 8-dimensional, 2^16-entry codebook once to N(0, I) and reuse it across models, layers, and experts without model-specific codebook training. Building on this codebook, MoE-VQ combines block-wise LDLQ residual feedback with rank-14 low-rank error compensation. At 2.34 effective bits per weight, MoE-VQ improves average accuracy across seven tasks by 3.3–13.3 points over tuned 2-bit GPTQ on four MoE models, while removing 51% of the excess perplexity relative to FP16. It also better preserves expert routing and outperforms an equal-rate E8 lattice by 1.0–3.2% in perplexity. For Mixtral-8x7B, MoE-VQ reduces weight memory from 93.4 GB to 15.6 GB, allowing the validated weight configuration to fit on one 80 GB GPU instead of two. These results show that distribution-adaptive vector codebooks can be reusable rather than model-specific for low-bit MoE quantization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
moe, llm, quantiztion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_boaSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies 2-bit post-training quantization of MoE expert weights. Authors argue that a vector-quantization codebook need not see the target model at all—a codebook fitted once, offline, to a standard normal can be reused across every model, every layer and every expert. On top of this, the paper combines the codebook with two familiar components: a generalization of GPTQ-style sequential error compensation to multi-dimensional blocks, and a low-rank residual compensation. Experiments on four MoE models (plus two very large ones) report that, under a unified effective-bit-rate accounting accuracy gains concentrated mainly on generative tasks.

Critical strengths
  1. The observation itself is informative, and the authors do not take it for granted: they separate "constraints imposed by the normalization" from "genuine measurement," and control for selection effects through equal seed budgets and replicate spreads. What is rarer still is the overall honesty of the write-up—the paper volunteers cells where its own method loses, reports verdicts that flip under a like-for-like reading, and even points out a tokenization/scoring artifact in a public harness. This level of statistical self-discipline is above average and makes me willing to trust the numbers they do report.

  2. Every method is priced under the same effective bit rate (group scales and low-rank terms are all counted, and even the zero point of asymmetric 4-bit is charged in), baselines are re-run under one protocol, and robustness is probed across corpora, calibration sets and hardware.

Critical weaknesses
  1. Limited novelty: at bottom it is "one observation + a recombination of known parts." 2-bit vector quantization is not new (the QuIP#, AQLM, QTIP line), GPTQ/LDLQ-style error feedback is not new, and low-rank residual compensation is not new either (the MiLo family). The only genuinely new idea in the paper is that "normalized weights are Gaussian regardless of the model, so the codebook can be fitted to a Gaussian rather than to a model." But note that a model-agnostic, training-free code is exactly what a fixed lattice (the E8 lattice) already provides—and the paper's own tables show that the learned Gaussian codebook's advantage over E8 is not large. The conceptual increment over "just use a fixed code" is therefore thin, and the rest is assembly of existing techniques. I do not think this amounts to a sufficient new contribution.

  2. The field is already very crowded, and the gains here do not stand out within it. 2-bit MoE quantization is one of the most saturated directions right now (the paper's own related-work list is long). In such a field an incremental work needs a decisive and consistent win to qualify, and this paper does not have one: the codebook's advantage over the fixed lattice is modest; the default configuration does not take the best accuracy cell on any model; it loses to the closest VQ competitor on one of the four models; and on the model with few, wide experts a per-model codebook actually beats the supposedly "model-agnostic" one (which the authors cannot explain). This is precisely the pattern one expects from a recombination in a crowded area rather than from a new capability.

  3. The practical payoff that motivates the work is not delivered. The entire premise is memory-bound deployment, yet the reference kernel is far slower than fp16 at batch size 1, there is no batch>1 / prefill / end-to-end throughput, the memory figures are ledger arithmetic rather than a serving demonstration, and the quality of the configuration that actually fits on a single card (attention quantized as well) is never measured. So the reader is left with quality measured in a configuration that is not deployable, and memory computed in a configuration whose quality is unknown—while paying a large latency cost. For a systems-motivated paper sitting in a crowded field, this gap is decisive.

Questions for authors
  1. Compared with simply using a fixed, training-free lattice code (E8), what exactly does learning a codebook on a Gaussian buy? Given that the advantage over E8 in your own tables is small, could you reposition the contribution as the measurement rather than the codebook—or show a setting where the learned codebook's advantage is both large and consistent?

  2. Can you provide any end-to-end or batch>1 evidence that the memory saving is not cancelled out by the kernel's latency cost? If not, what concrete deployment scenario is this method actually meant for?

  3. On the model where a per-model codebook beats the shared one. What would it take to bound the model-agnosticism claim to a credible range—more models, an analysis of when per-model fitting helps, or simply a weaker claim?

Reviewer_4TCWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoE-VQ, a 2-bit vector quantization framework for Mixture-of-Experts (MoE) large language models. The authors observe that expert weights from different MoE architectures exhibit approximately Gaussian distributions after group-wise RMS normalization. Based on this observation, they construct a model-agnostic 8-dimensional vector codebook trained on synthetic Gaussian data using Lloyd iterations. The framework further incorporates block-LDLQ residual feedback and rank-14 low-rank compensation to improve quantization accuracy. The authors analyze weight distributions across six MoE architectures and evaluate the proposed method on four main models, as well as larger 236B and 250B models, covering perplexity, downstream task performance, codebook transferability, routing preservation, and storage efficiency.

Critical strengths

Strength 1: The paper systematically analyzes the distributional properties of group-normalized expert weights across six MoE architectures and validates the applicability of Gaussian codebooks through model-specific codebook fitting, cross-model transfer experiments, and statistical stability analyses. These empirical findings provide useful insights into the representational properties of ultra-low-bit quantization for MoE models.

Strength 2: The paper provides relatively comprehensive experiments across multiple MoE models, including evaluations on 236B/250B models, challenging reasoning and code-generation tasks, effective BPW accounting, and extensive ablation studies. The complete quantization pipeline demonstrates substantial improvements in 2-bit model quality compared with tuned GPTQ.

Critical weaknesses

Weakness 1: I have substantial concerns regarding the core methodological novelty of this paper. QLoRA [1] has already exploited approximately Gaussian weight distributions for quantization design, while QuIP# [2] not only employs 8-D vector quantization but also explicitly discusses and evaluates Gaussian K-means codebooks in Appendix C.3. More importantly, HIGGS [3] has introduced precomputed, model-independent Gaussian MSE-optimal multidimensional quantization grids. Therefore, the main novelty of MoE-VQ appears to lie in observing that RMS-normalized MoE expert weights can use a similar universal codebook without Hadamard rotation, rather than introducing a fundamentally new Gaussian VQ principle. The paper does not sufficiently clarify the independent technical contribution of this distinction or discuss its direct relationship with HIGGS.

Weakness 2: The major algorithmic components of MoE-VQ are closely related to established methods. Lloyd/K-means is a classical codebook optimization algorithm, BlockLDLQ has already been explored in QuIP# [2], and low-rank quantization error compensation has been investigated in MiLo [4] and related studies. More importantly, compared with the E8 lattice baseline using the same bit budget, residual feedback, and low-rank compensation, the Gaussian codebook provides only approximately 1.0%–3.2% perplexity improvements. Therefore, the current evidence is insufficient to establish substantial independent methodological novelty beyond combining existing techniques.

Weakness 3: Although the paper includes ablations involving E8 lattice quantization and some components of QuIP#, it does not directly compare against complete implementations of representative VQ methods such as QuIP# [2], HIGGS [3], UniSVQ [5], or QTIP [6] under a unified experimental protocol. For KBVQ-MoE [7], the most closely related MoE-specific VQ approach, the authors primarily rely on indirect comparisons with published results obtained under different quantization budgets and evaluation settings. Consequently, it remains difficult to assess whether MoE-VQ offers a meaningful accuracy–compression advantage over existing strong VQ methods.

Weakness 4: Despite achieving promising weight compression results, the proposed fused dequantization and GEMV kernel is slower than FP16 across all reported matrix shapes, achieving only approximately 0.17–0.62× the FP16 performance. Furthermore, the paper does not evaluate end-to-end model throughput, prefill performance, or inference efficiency at larger batch sizes. Considering that QuIP# [2], HIGGS [3], and UniSVQ [5] also emphasize efficient VQ execution, the substantial lookup overhead introduced by the large unstructured codebook significantly limits the practical deployment value of the proposed approach.

[1] Tim Dettmers et al. "QLoRA: Efficient Finetuning of Quantized LLMs." NeurIPS 2023. https://arxiv.org/abs/2305.14314

[2] Albert Tseng et al. "QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks." ICML 2024. https://arxiv.org/abs/2402.04396

[3] Vladimir Malinovskii et al. "HIGGS: Pushing the Limits of Large Language Model Quantization via the Linearity Theorem." NAACL 2025. https://aclanthology.org/2025.naacl-long.543/

[4] Beichen Huang et al. "MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators." MLSys 2025. https://arxiv.org/abs/2504.02658

[5] Haoyu Wang et al. "UniSVQ: 2-bit Unified Scalar-Vector Quantization." ICML 2026. https://proceedings.mlr.press/v306/wang26lg.html

[6] Albert Tseng et al. "QTIP: Quantization with Trellises and Incoherence Processing." NeurIPS 2024. https://arxiv.org/abs/2406.11235

[7] Zukang Xu et al. "KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Models." ICLR 2026. https://arxiv.org/abs/2602.11184

Questions for authors

Q1: HIGGS [3] has already introduced precomputed Gaussian MSE-optimal multidimensional grids for model-independent quantization, while QuIP# [2] also discusses Gaussian K-means codebooks. Could the authors clarify what new construction or optimization mechanisms MoE-VQ introduces beyond these methods? If the main distinction is avoiding Hadamard rotation, could the authors provide direct comparisons demonstrating significant accuracy or inference efficiency benefits?

Q2: Lloyd iterations, BlockLDLQ, and low-rank compensation are all established techniques. Could the authors clarify what new properties or capabilities emerge from their combination that cannot be directly obtained from existing methods? In particular, given the relatively modest improvement of the Gaussian codebook over the matched-BPW E8 lattice baseline, is this gain sufficient to support the paper's main methodological novelty claim?

Q3: Could the authors provide more direct comparisons with HIGGS [3], QuIP# [2], UniSVQ [5], and QTIP [6] under consistent quantization scopes, calibration datasets, effective BPW budgets, and evaluation protocols? Given that these methods employ different fine-tuning and inference strategies, could the authors separately report accuracy comparisons without additional fine-tuning and actual GPU inference efficiency?

Q4: The proposed kernel is slower than FP16 across all evaluated matrix shapes, and end-to-end throughput has not been reported. Could the authors provide complete MoE inference benchmarks at batch size 1 and larger batch sizes, including a breakdown of prefill, decoding, codebook lookup, and low-rank compensation overhead? Furthermore, compared with structured or computed codebooks, can the efficiency bottleneck introduced by the 65,536-entry lookup table be effectively addressed?

Q5: On Mixtral, the per-model codebook outperforms the shared Gaussian codebook in the complete quantization pipeline. Meanwhile, the paper reports that training the shared codebook takes only 51 seconds, whereas quantizing large models requires several hours. Could the authors report the actual fitting time and additional end-to-end computational cost of per-model codebooks? Under what conditions does avoiding model-specific codebook training provide a meaningful practical advantage that justifies the potential loss in quantization accuracy?

[2] Albert Tseng et al. "QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks." ICML 2024. https://arxiv.org/abs/2402.04396

[3] Vladimir Malinovskii et al. "HIGGS: Pushing the Limits of Large Language Model Quantization via the Linearity Theorem." NAACL 2025. https://aclanthology.org/2025.naacl-long.543/

[5] Haoyu Wang et al. "UniSVQ: 2-bit Unified Scalar-Vector Quantization." ICML 2026. https://proceedings.mlr.press/v306/wang26lg.html

[6] Albert Tseng et al. "QTIP: Quantization with Trellises and Incoherence Processing." NeurIPS 2024. https://arxiv.org/abs/2406.11235

Reviewer_2DAqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MoE-VQ, a post-training quantization method for compressing MoE expert weights. Based on the observation that group-normalized expert weights are approximately Gaussian across architectures, it trains a shared vector codebook on synthetic Gaussian samples and combines it with block-LDLQ error feedback and low-rank residual compensation. Experiments evaluate quantization quality, storage costs, routing preservation, and robustness across several MoE models.

Critical strengths
  1. The paper is well written and easy to follow, with clear motivation and a well-organized presentation of the quantization pipeline and experimental analysis.
  2. The paper identifies distributional similarities among normalized MoE expert weights and uses this observation to construct a Gaussian codebook shared across models, layers, and experts, avoiding model-specific codebook training (Sections 2.2 and 3.2).
  3. The experimental evaluation is extensive, including comparisons across model architectures and scales, component and bit-allocation ablations, routing analysis, calibration robustness checks, and kernel measurements.
Critical weaknesses
  1. The algorithmic novelty is limited. QuIP# [1] already investigated Gaussian K-means codebooks in Appendices C.3–C.4. Combining low-bit weights with high-precision low-rank components also resembles SVDQuant [2], although SVDQuant initially extracts the low-rank component before quantizing the residual. The contribution beyond adapting and combining these existing ideas for MoE needs clearer justification (Sections 3.2–3.4).
  2. The baseline comparison is insufficient. Important weight-only VQ baselines, including AQLM [3] and VQ models refined with PV-Tuning [4], are missing, as are scalar quantization approaches such as SpinQuant [5] and GPTAQ [6]. Comparisons should account for effective bit rates, quantized modules, and the additional optimization costs of fine-tuning or learned rotations.
  3. The method's broader applicability remains unclear. Table 9 reports similar weight statistics for dense MLPs, but no end-to-end quantization results on dense models are provided. Is the method specific to MoE models, or can it be equally effective for dense models?
  4. Accuracy improvements over stronger VQ baselines are limited. On Qwen1.5-MoE in Table 1, MoE-VQ improves C4 perplexity only from 9.63 to 9.54 relative to E8 lattice + low-rank, while Avg4 decreases from 61.29 to 60.59. The default configuration does not achieve the best Avg4 among quantized methods on any of the four models.
  5. The demonstrated efficiency benefit primarily concerns reduced weight memory. Although a reference kernel is provided, it is slower than FP16 at batch size one (Table 14). Further kernel optimization and end-to-end latency and throughput measurements are needed to establish practical inference acceleration. [1] Albert Tseng et al. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:48630–48656, 2024.

[2] Muyang Li et al. SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models. International Conference on Learning Representations, 2025.

[3] Vage Egiazarian et al. Extreme Compression of Large Language Models via Additive Quantization. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:12284–12303, 2024.

[4] Vladimir Malinovskii et al. PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression. Advances in Neural Information Processing Systems, 37, 2024.

[5] Zechun Liu et al. SpinQuant: LLM Quantization with Learned Rotations. International Conference on Learning Representations, 2025.

[6] Yuhang Li et al. GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration. Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:36690–36706, 2025.

Questions for authors

Please refer to the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.