acceptodds
Under review as a conference paper at ICLR 2027

MatGPTQ: Efficient and Accurate Inference over Nested Quantized Models

17 views

Abstract

Matryoshka Quantization (MatQuant), Any-Precision-LLM (AP) and AnyBCQ (AB) are recent quantization approaches showing that a single integer-quantized model can be served across multiple precisions. In this paradigm, lower-precision models are extracted from a higher-precision model by simply reading fewer bits of the weights. This enables a single checkpoint to cover a wide range of memory and latency budgets, but makes both quantization and efficient execution substantially harder. Existing methods rely on expensive quantization-aware training (QAT) or gradient-based post-training quantization (PTQ) rather than fast one-shot PTQ, and offer limited system support: dedicated kernels are either missing or restricted to single- or small-batch decoding. We address these limitations with _Post-Training Matryoshka Quantization_ (MatGPTQ), an end-to-end pipeline for nested-model quantization and inference. MatGPTQ casts Matryoshka quantization as multi-precision error compensation, producing a single "sliceable" parent model jointly optimized for multiple target precisions in one pass over a small calibration set. We further refine the MatQuant representation so that an -bit model reads exactly bits, and introduce the first dedicated inference kernels for this format, supporting batch sizes beyond one and integrated into vLLM. Across standard LLMs and benchmarks, MatGPTQ outperforms MatQuant while remaining competitive with AP and AB at the smallest checkpoint size, and our kernels achieve end-to-end speedups of up to 3.5 over BF16 at the low-bit regime. Overall, MatGPTQ makes nested quantized models practical to serve from a single, compact checkpoint.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Nested LLMs, inference, serving, gpu, cuda, quantization, compression

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 2 reviews, above 99% of the others.
Reviewer_BS8JOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MatGPTQ extends GPTQ by jointly optimizing a single parent model to support multiple bit-widths within one unified quantized representation. It introduces true-prefix slicing, allowing smaller models to be obtained by selecting prefixes of the parent model while preserving compatibility across different compression levels. In addition, MatGPTQ provides dedicated inference kernels that are optimized for these nested structures, reducing deployment overhead and improving serving efficiency. Together, these techniques form a practical end-to-end pipeline for quantizing, extracting, and serving nested models with flexible accuracy–efficiency trade-offs.

Critical strengths
  • Coherent method design. The multi-precision objective and the corresponding changes to GPTQ are well motivated. The connection between the quantization algorithm, representation, and execution is clear.

  • Solid experimental evidence. The accuracy results and sliced-GPTQ ablation support the value of joint optimization. The kernel and end-to-end evaluations also demonstrate practical benefits across different workloads.

  • Clear presentation. I found the paper easy to follow, the authors explain the deployment problem and their design choices well, and the appendices provide useful implementation details and ablations.

Critical weaknesses
  1. The fastest execution setting lacks corresponding accuracy results, the performance evaluation includes peak speedups at 2 bits, but accuracy is only evaluated at 3 bits and above. This leaves the accuracy–throughput trade-off at the fastest setting unresolved. The practical speedup claim would be stronger if tied explicitly to configurations with demonstrated model quality.

  2. The evaluation scope is relatively narrow, accuracy is evaluated on two 8B models. I think these results are useful, but leave uncertainty about how well the approach scales to larger/smaller models and other hardware architectures, sometimes smaller model may more sensitive to quantization, more evaluations will be helpful for understand the method.

Questions for authors
  1. The designed kernels support 2-bit execution, while accuracy is evaluated at 3 bits and above. Could the authors clarify the intended use of the 2-bit setting and whether its accuracy is suitable for deployment?
  2. In this paper, you use different loss weights for MatQuant and MatGPTQ. Were both methods given comparable tuning opportunities, and how sensitive is the comparison to the MatQuant weighting and optimization budget?
Reviewer_dcmQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • This paper proposes MatGPTQ, an end-to-end pipeline for nested quantized models. It extends GPTQ to a multi-precision one-shot PTQ method, allowing multiple bit-width models to be extracted from a single checkpoint. Being based on uniform quantization, it yields a smaller checkpoint than state-of-the-art methods, and its dedicated kernels improve practical throughput.
Critical strengths
  • The paper is well written. The limitations of existing nested quantized models and the improvements made by the proposed method are clearly presented.
  • The kernel implementation is described in detail. In particular, it is interesting that different kernels are used depending on the batch size, distinguishing between memory-bound and compute-bound regimes.
Critical weaknesses
  • Although the work is well rounded from algorithm to kernel, the methodological contribution is somewhat incremental. Nested quantization is already an actively studied area, and the proposed method is a variant of the existing GPTQ.
  • The analysis of the experimental results is insufficient in some parts (see Questions below).
Questions for authors
  • In the 4-bit results of Figure 1, why does MatGPTQ show a larger performance drop than the Marlin kernel as the batch size increases?
  • My understanding is that the direct-load kernel is used for small batches (b ≤ 4) and the batched kernel for larger batches. Could you compare the performance of the two kernels across batch sizes?
  • Why are the bit-planes split into bits 1–2, 3, 4, 5–6, and 7–8? What are the trade-offs in performance or overhead compared to other splitting schemes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.