acceptodds
Under review as a conference paper at ICLR 2027

MBVQ: Multi-Level Block-Wise Vector Quantization for Hardware Friendly Post-Training Quantization of LLMs

Abstract

The rapid scaling of large language models (LLMs) has substantially increased their deployment cost. Post-training quantization (PTQ), especially vector quantization (VQ), focuses on model compression to below 2 bits per parameter, but usually overlooks the data movement bottleneck that dominates real-world inference latency. In this paper, we propose **M**ulti-level **B**lock-wise **V**ector **Q**uantization (MBVQ), a hardware-friendly VQ framework designed to address this issue. MBVQ introduces two key innovations: (1) capacity-constrained per-block codebooks for efficient memory residency during dequantization, and (2) multi-level residual quantization scheme with row-wise importance selection and per-block scaling to compensate for reconstruction error. Specifically, we first partition weight matrices into blocks to adapt large-scale codebooks to varying hardwares. Subsequently, we merge redundant adjacent blocks for quantizing residuals with selective codebooks and design codebook rescaling to adaptively adjust the block-wise distribution of values to simultaneously improve model accuracy and achieve substantial inference speedup. Experiments on a range of LLMs demonstrate that MBVQ consistently outperforms state-of-the-art low-bit quantization baselines under comparable compression ratios, delivering approximately inference speedup, while maintaining competitive perplexity and zero-shot accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-Training Quantization, Large Language Models, Model Compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RwhXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

MBVQ is a block-wise vector quantization method that uses small local codebooks, similarity-based block merging, selective residual quantization, and per-block scaling. The paper claims that these designs maintain competitive accuracy at approximately 1.5 bits per weight while providing a 6.62×6.62\times inference speedup. Experiments are a bit confined, covering LLaMA, Mistral, and Qwen models.

Critical strengths

None

Critical weaknesses
  1. The inference-speedup claim is not adequately supported.

The reported 6.62×6.62\times speedup is derived only from an analytical model, without an implemented kernel function. Measurements of latency, throughput, or memory traffic on real hardware are also missing, which is essential for such claim. Moreover, Eq. (16) appears to use dBd_B rather than 2dB2^{d_B} when calculating codebook size, potentially underestimating codebook-loading cost. Thus, the paper’s central hardware-efficiency claim remains unverified.

  1. The method is computationally heavy, and its storage accounting is unclear.

MBVQ involves all sorts of fine-tuning and optimization, such as repeated codebook fine-tuning, beam search, Transformer-block fine-tuning, and full-model end-to-end tuning, but none of their optimization cost and hyperparameters are reported. The claimed 1.51 bits per weight is also difficult to reconcile with the method: the Level-1 indices already require 1.5 bits per weight, while three additional residual levels add further indices, codebooks, scales, etc. The reviewer does not find the main method and claims of the paper sound enough due to such lack of important details.

  1. The evidence for performance superiority is insufficient.

Without correct effective-bit accounting, it is unclear whether Table 1 compares methods at equivalent compression ratios. Several relevant baselines, including hardware-oriented VQ methods, are missing, and the component analyses quantize only one layer of LLaMA2-7B rather than the full model. The claim of outperforming baselines across all benchmarks is also inconsistent with several results in Table 1.

Questions for authors
  1. Hardware performance

Can the authors report measured prefill and decoding latency, throughput, peak memory, and memory traffic using an implemented kernel on real hardware? Please also clarify Eq. (16) and provide the corrected speedup when the full codebook size is included.

  1. Effective compression ratio.

What is the actual model size and effective bits per weight after including all parameters such as indices, codebooks, scales, etc.? Does MBVQ retain its accuracy advantage when compared with the baselines under the same effective bit-width and optimization budget?

Reviewer_wMTCOpenReview ↗
Rating
1
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MBVQ combines small blockwise codebooks, merging of similar blocks, selective residual quantization, and per-block scaling for low-bit LLM compression. It reports perplexity and zero-shot results on six models at approximately 1.51 bits per weight, together with an analytical 6.62× speedup estimate.

Critical strengths

The focus on codebook access costs is practically relevant, and the local-codebook/residual design is plausible. Table 1 shows encouraging perplexity results across several model families, suggesting that the representation deserves further investigation.

Critical weaknesses
  1. Sections 4.3 and 5.1 specify a 6-bit index for each 4-weight vector and three additional residual levels selecting 5% of rows. Under this stated interpretation, indices alone require 1.7251.725 bits per weight, before codebooks and metadata, rather than Table 1's 1.51. A different encoding could explain this, but its accounting is missing. Section 6.3's 6.62× estimate is analytical, not measured inference performance; its codebook-loading term also uses dBd_B where the codebook definition requires 2dB2^{d_B}. These issues directly affect the matched-rate and deployment conclusions.
  2. Section 6's ablations concern Layer 1 of LLaMA2-7B rather than full-model quantization. Merging changes perplexity from 5.1490 to 5.1464, without uncertainty estimates, yet the text describes an 8.68% improvement in model quality using a relative-gap calculation. The appendix includes several optimization stages without establishing matched baseline optimization budgets. In addition, the claim of best zero-shot results across all benchmarks is contradicted by Table 1: QTIP scores 53.02 versus MBVQ's 52.61 on LLaMA3.2-3B. Full-model, matched-budget evidence is needed to substantiate the broad conclusion.
Questions for authors
  1. Can you provide a more comprehensive comparison on a fair bpw format?
  2. What speedup remains after correcting the codebook cardinality, and can an executable packed model validate it under a specified inference workload?
  3. Do the gains persist in full-model ablations with matched calibration and optimization budgets and reported uncertainty?
Reviewer_QsK4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MBVQ studies the trade-off between model quality and memory access costs in ultra-low-bit large language model quantization. It uses small codebooks for local weight blocks, then merges similar adjacent blocks to reduce codebook storage. Selective residual quantization and per-block scaling balance accuracy and memory access costs.The core idea is clear and has practical relevance: first control the size of the codebook working set, then recover model quality through selective refinement. Table 1 presents promising perplexity results on six models. However, the reported 6.62x speedup comes from an analytical estimate, rather than measured end-to-end inference latency. The effective bit width also needs some clarification. Thus, the current evidence is not sufficient to support the paper's full claims about model quality and efficiency.

Critical strengths
  1. The problem is fairly important, and the design motivation is clear. Smaller model files do not automatically lead to faster inference. Local codebooks are a reasonable way to address this issue, and adaptive sharing may also reduce their storage costs. Beyond simply reducing reconstruction error, this is a direction worth studying.
  2. The perplexity gains show some consistency across models. In Table 1, LLaMA2-7B has a WikiText2 perplexity of 7.46, compared with 7.69 for QTIP. Its average score across six tasks is 66.97, compared with 62.35 for QTIP. LLaMA3-8B has a WikiText2 perplexity of 11.83, compared with 16.29 for AQLM. Across all six models, MBVQ has better perplexity on both datasets than the available quantized baselines in the table. These results suggest some potential for the method. However, the storage accounting and optimization-budget concerns below still need to be addressed.
Critical weaknesses
  1. The central hardware-efficiency conclusion still lacks direct validation. The paper mainly relies on theoretical estimates and does not provide end-to-end inference latency measured on real hardware. The cost model also does not fully match the method description. It is therefore still difficult to judge whether the claimed speedup can be achieved in actual deployment.
  2. The compression ratio and comparison conditions need further clarification. The accounting for effective bit width and the calibration and fine-tuning budgets of each method are not explained clearly enough. It is therefore unclear whether the accuracy gains come from the method itself or partly from additional storage and optimization resources. The existing local ablations are also insufficient to fully establish the contribution of the core design to the model's experimental results.
Questions for authors
  1. Please clarify how the effective bit width is calculated and whether it includes the multilevel residuals and related storage overhead.
  2. Please explain how the theoretical speedup estimate corresponds to the actual implementation, and provide end-to-end latency results on real hardware.
  3. Did the baselines use comparable calibration data and fine-tuning budgets? Could you further demonstrate the gains from the core design under matched resource conditions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.