- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MatGPTQ extends GPTQ by jointly optimizing a single parent model to support multiple bit-widths within one unified quantized representation. It introduces true-prefix slicing, allowing smaller models to be obtained by selecting prefixes of the parent model while preserving compatibility across different compression levels. In addition, MatGPTQ provides dedicated inference kernels that are optimized for these nested structures, reducing deployment overhead and improving serving efficiency. Together, these techniques form a practical end-to-end pipeline for quantizing, extracting, and serving nested models with flexible accuracy–efficiency trade-offs.
Critical strengths
-
Coherent method design. The multi-precision objective and the corresponding changes to GPTQ are well motivated. The connection between the quantization algorithm, representation, and execution is clear.
-
Solid experimental evidence. The accuracy results and sliced-GPTQ ablation support the value of joint optimization. The kernel and end-to-end evaluations also demonstrate practical benefits across different workloads.
-
Clear presentation. I found the paper easy to follow, the authors explain the deployment problem and their design choices well, and the appendices provide useful implementation details and ablations.
Critical weaknesses
-
The fastest execution setting lacks corresponding accuracy results, the performance evaluation includes peak speedups at 2 bits, but accuracy is only evaluated at 3 bits and above. This leaves the accuracy–throughput trade-off at the fastest setting unresolved. The practical speedup claim would be stronger if tied explicitly to configurations with demonstrated model quality.
-
The evaluation scope is relatively narrow, accuracy is evaluated on two 8B models. I think these results are useful, but leave uncertainty about how well the approach scales to larger/smaller models and other hardware architectures, sometimes smaller model may more sensitive to quantization, more evaluations will be helpful for understand the method.
Questions for authors
- The designed kernels support 2-bit execution, while accuracy is evaluated at 3 bits and above. Could the authors clarify the intended use of the 2-bit setting and whether its accuracy is suitable for deployment?
- In this paper, you use different loss weights for MatQuant and MatGPTQ. Were both methods given comparable tuning opportunities, and how sensitive is the comparison to the MatQuant weighting and optimization budget?