- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
MBVQ is a block-wise vector quantization method that uses small local codebooks, similarity-based block merging, selective residual quantization, and per-block scaling. The paper claims that these designs maintain competitive accuracy at approximately 1.5 bits per weight while providing a inference speedup. Experiments are a bit confined, covering LLaMA, Mistral, and Qwen models.
Critical strengths
None
Critical weaknesses
- The inference-speedup claim is not adequately supported.
The reported speedup is derived only from an analytical model, without an implemented kernel function. Measurements of latency, throughput, or memory traffic on real hardware are also missing, which is essential for such claim. Moreover, Eq. (16) appears to use rather than when calculating codebook size, potentially underestimating codebook-loading cost. Thus, the paper’s central hardware-efficiency claim remains unverified.
- The method is computationally heavy, and its storage accounting is unclear.
MBVQ involves all sorts of fine-tuning and optimization, such as repeated codebook fine-tuning, beam search, Transformer-block fine-tuning, and full-model end-to-end tuning, but none of their optimization cost and hyperparameters are reported. The claimed 1.51 bits per weight is also difficult to reconcile with the method: the Level-1 indices already require 1.5 bits per weight, while three additional residual levels add further indices, codebooks, scales, etc. The reviewer does not find the main method and claims of the paper sound enough due to such lack of important details.
- The evidence for performance superiority is insufficient.
Without correct effective-bit accounting, it is unclear whether Table 1 compares methods at equivalent compression ratios. Several relevant baselines, including hardware-oriented VQ methods, are missing, and the component analyses quantize only one layer of LLaMA2-7B rather than the full model. The claim of outperforming baselines across all benchmarks is also inconsistent with several results in Table 1.
Questions for authors
- Hardware performance
Can the authors report measured prefill and decoding latency, throughput, peak memory, and memory traffic using an implemented kernel on real hardware? Please also clarify Eq. (16) and provide the corrected speedup when the full codebook size is included.
- Effective compression ratio.
What is the actual model size and effective bits per weight after including all parameters such as indices, codebooks, scales, etc.? Does MBVQ retain its accuracy advantage when compared with the baselines under the same effective bit-width and optimization budget?