- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Paper Summary
MixQuant proposes an adaptive mixed-precision quantization framework intended to support different deployment-time memory budgets after a single offline calibration procedure. It estimates layer distortion under randomly sampled quantized upstream configurations, introduces a tail regularizer to discourage excessively low-precision assignments, and calibrates quantizer parameters using solver-generated configurations. A greedy allocator then selects layer-wise bit-widths for a given budget. Experiments evaluate the framework with AWQ and GPTQ across three LLMs and multiple memory budgets
Critical strengths
Clear motivation addressing a meaningful calibration problem.
The paper clearly motivates the mismatch between evaluating a layer under full-precision upstream activations and deploying it within a fully quantized model. Accounting for changes induced by upstream quantization is meaningful, particularly when the final bit allocation is unknown during calibration. However, related concerns about global and cross-layer quantization effects have also been explored in previous works, so this general motivation should not be presented as entirely new.
Critical weaknesses
-
The practical advantage over a small set of pre-calibrated configurations is insufficiently established.
The paper emphasizes that the space of mixed-precision configurations is combinatorially large. However, practical deployment does not necessarily require calibrating every possible configuration. A straightforward alternative is to optimize and calibrate a small set of configurations targeting, for example, average precisions of 3, 4, and 5 bits, and select a feasible configuration at deployment. The paper does not sufficiently establish its advantage over this alternative in accuracy, offline calibration cost, storage requirements, or model-switching latency. Such a comparison would better justify the need for the proposed adaptive calibration framework.
-
The simple random-sampling strategy lacks sufficient validation.
Stage 1 estimates distortion using uniformly sampled upstream configurations, with a fixed sample count of for every layer and candidate bit-width. As network depth increases, the space of possible upstream configurations becomes substantially larger, yet the paper provides insufficient evidence that ten samples yield stable distortion estimates and reliable allocations. Sensitivity analyses over , variation across random seeds, and comparisons against estimates obtained with substantially more samples would strengthen this design choice. Furthermore, uniform random configurations may differ considerably from the budget-constrained configurations produced by the deployment-time allocator, leaving their representativeness insufficiently established.
-
The quality of the greedy allocation is insufficiently characterized.
The solver in Section 3.4 starts from the minimum-bit configuration and follows a sequence of irreversible local upgrades. Earlier decisions determine which subsequent upgrades remain feasible, and this procedure does not guarantee a globally optimal allocation under the given budget, even for the surrogate objective.
-
The presentation and reporting of quantization precision require improvement.
The manuscript contains writing and notation issues that hinder readability. For example, the sentence beginning with “bit-widths” on line 172 should start with a capital letter. Around line 214, should be clearly introduced in the main text and connected to the distortion estimates in Eq. (2), despite its definition in the caption of Figure 1. Furthermore, the experiments should report the achieved parameter-weighted average bit-width alongside memory consumption in GB. Absolute memory consumption depends on model size, whereas average bit-width makes the quantization level easier to interpret and facilitates comparisons across models. More thorough proofreading and clearer reporting would improve the presentation.
Questions for authors
See weakness