- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies 2-bit post-training quantization of MoE expert weights. Authors argue that a vector-quantization codebook need not see the target model at all—a codebook fitted once, offline, to a standard normal can be reused across every model, every layer and every expert. On top of this, the paper combines the codebook with two familiar components: a generalization of GPTQ-style sequential error compensation to multi-dimensional blocks, and a low-rank residual compensation. Experiments on four MoE models (plus two very large ones) report that, under a unified effective-bit-rate accounting accuracy gains concentrated mainly on generative tasks.
Critical strengths
-
The observation itself is informative, and the authors do not take it for granted: they separate "constraints imposed by the normalization" from "genuine measurement," and control for selection effects through equal seed budgets and replicate spreads. What is rarer still is the overall honesty of the write-up—the paper volunteers cells where its own method loses, reports verdicts that flip under a like-for-like reading, and even points out a tokenization/scoring artifact in a public harness. This level of statistical self-discipline is above average and makes me willing to trust the numbers they do report.
-
Every method is priced under the same effective bit rate (group scales and low-rank terms are all counted, and even the zero point of asymmetric 4-bit is charged in), baselines are re-run under one protocol, and robustness is probed across corpora, calibration sets and hardware.
Critical weaknesses
-
Limited novelty: at bottom it is "one observation + a recombination of known parts." 2-bit vector quantization is not new (the QuIP#, AQLM, QTIP line), GPTQ/LDLQ-style error feedback is not new, and low-rank residual compensation is not new either (the MiLo family). The only genuinely new idea in the paper is that "normalized weights are Gaussian regardless of the model, so the codebook can be fitted to a Gaussian rather than to a model." But note that a model-agnostic, training-free code is exactly what a fixed lattice (the E8 lattice) already provides—and the paper's own tables show that the learned Gaussian codebook's advantage over E8 is not large. The conceptual increment over "just use a fixed code" is therefore thin, and the rest is assembly of existing techniques. I do not think this amounts to a sufficient new contribution.
-
The field is already very crowded, and the gains here do not stand out within it. 2-bit MoE quantization is one of the most saturated directions right now (the paper's own related-work list is long). In such a field an incremental work needs a decisive and consistent win to qualify, and this paper does not have one: the codebook's advantage over the fixed lattice is modest; the default configuration does not take the best accuracy cell on any model; it loses to the closest VQ competitor on one of the four models; and on the model with few, wide experts a per-model codebook actually beats the supposedly "model-agnostic" one (which the authors cannot explain). This is precisely the pattern one expects from a recombination in a crowded area rather than from a new capability.
-
The practical payoff that motivates the work is not delivered. The entire premise is memory-bound deployment, yet the reference kernel is far slower than fp16 at batch size 1, there is no batch>1 / prefill / end-to-end throughput, the memory figures are ledger arithmetic rather than a serving demonstration, and the quality of the configuration that actually fits on a single card (attention quantized as well) is never measured. So the reader is left with quality measured in a configuration that is not deployable, and memory computed in a configuration whose quality is unknown—while paying a large latency cost. For a systems-motivated paper sitting in a crowded field, this gap is decisive.
Questions for authors
-
Compared with simply using a fixed, training-free lattice code (E8), what exactly does learning a codebook on a Gaussian buy? Given that the advantage over E8 in your own tables is small, could you reposition the contribution as the measurement rather than the codebook—or show a setting where the learned codebook's advantage is both large and consistent?
-
Can you provide any end-to-end or batch>1 evidence that the memory saving is not cancelled out by the kernel's latency cost? If not, what concrete deployment scenario is this method actually meant for?
-
On the model where a per-model codebook beats the shared one. What would it take to bound the model-agnosticism claim to a credible range—more models, an analysis of when per-model fitting helps, or simply a weaker claim?