- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents AutoQS, which is a framework that allocates group sizes instead of bitwidths to each layer, by utilizing an LLM proposer.
Critical strengths
- Quantization strategy search for LLMs is a timely and important research question.
Critical weaknesses
Lack of motivation.
- The paper acknowledges prior work on layer-wise mixed-precision quantization, such as HAWQ-V3, which formulates quantization as a resource allocation problem. However, the paper does not sufficiently justify why group size should be the allocation target rather than bitwidth. A more in-depth analysis of prior work is needed to establish why quantization granularity allocation is a more desirable or complementary direction for resource allocation.
- The use of LLMs for searching the design space also lacks sufficient motivation. Prior work has utilized mathematical tools such as Integer Linear Programming (ILP) [1, 2] or Cooperative Game Theory [3] for resource allocation based on sensitivity metrics. An analysis of the limitations of such approaches should be provided to better motivate the use of LLMs.
Lack of novelty.
- The proposed framework appears to offer limited methodological novelty. The problem formulation, the use of an LLM as a proposer, and the validation mechanism are relatively generic, and their technical contributions beyond existing approaches are not clearly established.
- Prior work on resource allocation for LLM quantization generally shares two key components: a sensitivity metric for each layer and a resource allocation methodology that utilizes the sensitivity metric. In both aspects, the paper does not clearly demonstrate novel insights or mechanisms beyond existing approaches.
Limited applicability.
- The use of ARC-Challenge, PIQA, and BoolQ as quality metrics during the search process raises concerns regarding applicability to the PTQ scenario targeted by the paper. PTQ typically assumes a small calibration dataset, usually around 100–1,000 samples from a general-purpose corpus such as WikiText or C4. Reliance on three full downstream benchmark datasets during search may limit the practical applicability of the proposed framework in a typical PTQ setting.
Concerns about evaluation validity.
- The evaluation lacks comparisons with other resource allocation baselines, such as HAWQ-V3 and LLM-MQ. The current evaluation primarily compares AutoQS with other quantization methods, which does not directly establish the effectiveness of the proposed resource allocation strategy. Comparisons with other resource allocation approaches under a fixed quantization method would provide a more meaningful assessment of the proposed strategy.
- ARC-Challenge, PIQA, and BoolQ are used during the search process and therefore should not be considered held-out evaluation benchmarks. Although the paper acknowledges this overlap, including these datasets in the main evaluation results may overestimate the generalization performance. Results on benchmarks not used during search should be emphasized instead.
- AutoQS may have a different effective bitwidth from the other baselines, as the metadata size varies with group size. This difference could lead to an unfair comparison under unequal storage budgets. A comparison under matched effective bitwidth or checkpoint size would provide a fairer assessment.
[1] Li, Shiyao, et al., "LLM-MQ: Mixed-Precision Quantization for Efficient LLM Deployment," The Efficient Natural Language and Speech Processing Workshop with NeurIPS, 2023.
[2] Yao, Zhewei, et al., "HAWQ-V3: Dyadic Neural Network Quantization," ICML, 2021.
[3] Zhao, Junchen, et al., "CoopQ: Cooperative Game Inspired Layerwise Mixed Precision Quantization for LLMs," ACL Findings, 2026.
Questions for authors
Please refer to the Critical Weaknesses section and assess them.