acceptodds
Under review as a conference paper at ICLR 2027

AutoQS: Automated Quantization Strategy Design with Large Language Models

Abstract

Quantization reduces LLM deployment costs, but effective strategies must balance model quality and storage. LLM-driven search can automate strategy design through repeated proposal and evaluation. This process must identify useful strategies within a limited evaluation budget. The key challenge is that _actual quantization is expensive, while inexpensive proxies may not reflect quantized-model performance._ To address this challenge, we present AutoQS, an **Auto**mated **Q**uantization **S**trategy Design with LLMs. AutoQS keeps weight precision fixed and searches group-size allocations across layers and operators under deployment constraints. Evaluation history guides coordinated LLM proposals, while deterministic validation ensures compatible allocations. An inexpensive quality proxy evaluates candidates, and an candidate repository retains alternative allocations to inform subsequent proposals. Periodic actual quantization complements these proxy evaluations with measured quality and checkpoint storage, providing feedback for further search within the evaluation budget. Experiments on Qwen2.5-7B-Instruct show that AutoQS improves seven-task average accuracy over uniform-group GPTQ and AWQ by 0.70 and 1.12 percentage points, respectively, at four-bit weight precision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Quantization, Audo Quantization, LLM-Driven Algorithm

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fhNtOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents AutoQS, which is a framework that allocates group sizes instead of bitwidths to each layer, by utilizing an LLM proposer.

Critical strengths
  • Quantization strategy search for LLMs is a timely and important research question.
Critical weaknesses

Lack of motivation.

  • The paper acknowledges prior work on layer-wise mixed-precision quantization, such as HAWQ-V3, which formulates quantization as a resource allocation problem. However, the paper does not sufficiently justify why group size should be the allocation target rather than bitwidth. A more in-depth analysis of prior work is needed to establish why quantization granularity allocation is a more desirable or complementary direction for resource allocation.
  • The use of LLMs for searching the design space also lacks sufficient motivation. Prior work has utilized mathematical tools such as Integer Linear Programming (ILP) [1, 2] or Cooperative Game Theory [3] for resource allocation based on sensitivity metrics. An analysis of the limitations of such approaches should be provided to better motivate the use of LLMs.

Lack of novelty.

  • The proposed framework appears to offer limited methodological novelty. The problem formulation, the use of an LLM as a proposer, and the validation mechanism are relatively generic, and their technical contributions beyond existing approaches are not clearly established.
  • Prior work on resource allocation for LLM quantization generally shares two key components: a sensitivity metric for each layer and a resource allocation methodology that utilizes the sensitivity metric. In both aspects, the paper does not clearly demonstrate novel insights or mechanisms beyond existing approaches.

Limited applicability.

  • The use of ARC-Challenge, PIQA, and BoolQ as quality metrics during the search process raises concerns regarding applicability to the PTQ scenario targeted by the paper. PTQ typically assumes a small calibration dataset, usually around 100–1,000 samples from a general-purpose corpus such as WikiText or C4. Reliance on three full downstream benchmark datasets during search may limit the practical applicability of the proposed framework in a typical PTQ setting.

Concerns about evaluation validity.

  • The evaluation lacks comparisons with other resource allocation baselines, such as HAWQ-V3 and LLM-MQ. The current evaluation primarily compares AutoQS with other quantization methods, which does not directly establish the effectiveness of the proposed resource allocation strategy. Comparisons with other resource allocation approaches under a fixed quantization method would provide a more meaningful assessment of the proposed strategy.
  • ARC-Challenge, PIQA, and BoolQ are used during the search process and therefore should not be considered held-out evaluation benchmarks. Although the paper acknowledges this overlap, including these datasets in the main evaluation results may overestimate the generalization performance. Results on benchmarks not used during search should be emphasized instead.
  • AutoQS may have a different effective bitwidth from the other baselines, as the metadata size varies with group size. This difference could lead to an unfair comparison under unequal storage budgets. A comparison under matched effective bitwidth or checkpoint size would provide a fairer assessment.

[1] Li, Shiyao, et al., "LLM-MQ: Mixed-Precision Quantization for Efficient LLM Deployment," The Efficient Natural Language and Speech Processing Workshop with NeurIPS, 2023.

[2] Yao, Zhewei, et al., "HAWQ-V3: Dyadic Neural Network Quantization," ICML, 2021.

[3] Zhao, Junchen, et al., "CoopQ: Cooperative Game Inspired Layerwise Mixed Precision Quantization for LLMs," ACL Findings, 2026.

Questions for authors

Please refer to the Critical Weaknesses section and assess them.

Reviewer_LW1bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents AutoQS, an LLM-guided framework for searching heterogeneous quantization group-size allocations across Transformer layers and operator groups while keeping weight precision fixed. The framework combines LLM-proposed configuration edits, deterministic validity checks, low-cost proxy evaluation, and periodic GPTQ quantization with measured downstream accuracy and checkpoint storage. Evaluation includes Qwen and LLaMA-family models, proposer and feedback ablations, and cross-model policy transfer. On Qwen2.5-7B-Instruct at four-bit precision, AutoQS reports seven-task average accuracy improvements of 0.70 and 1.12 percentage points over uniform-group GPTQ and AWQ, respectively.

Critical strengths
  1. The paper addresses a practically relevant quantization configuration problem. Searching heterogeneous group sizes at a fixed weight precision provides a meaningful way to adapt quantization granularity across layers and operators. Incorporating operator-compatibility constraints connects the search to executable configurations, giving the proposed optimization a clear practical motivation.
  2. The search incorporates feedback from actual quantized checkpoints. Periodic GPTQ quantization provides measured downstream accuracy and checkpoint storage to inform subsequent search. Experimental results on Qwen2.5-7B-Instruct show improvements in seven-task average accuracy over uniform-group GPTQ and AWQ, providing empirical support for the potential value of heterogeneous group-size allocation.
Critical weaknesses
  1. Important implementation details are insufficiently specified for reproduction. Section 3.3 describes the signals recorded by the low-cost evaluator, but the computation of the scalar proxy score remains unclear: Equation 2 identifies equivalent names for this score rather than specifying how it is calculated. It is also unclear how each candidate allocation is incorporated into the NLL evaluation, which reference model is used to measure degradation, and how this evaluation avoids constructing a fresh deployment checkpoint. In addition, the manuscript describes the roles of parent selection and deterministic novelty repair, but does not sufficiently specify how parents are selected from the candidate repository or how repeated allocations are modified. I could not find an anonymous code repository link in the manuscript or a code attachment on the submission page, which makes these ambiguities harder to resolve. Since these choices can affect candidate ordering and the search trajectory, more explicit descriptions would help readers assess and reproduce the method.

  2. The experiments do not sufficiently isolate the benefit of the proposed allocation search. The main accuracy comparisons use uniform-G128 baselines, while AutoQS searches heterogeneous group sizes that can change metadata storage. Although Section 3.4 states that checkpoint sizes are measured, the manuscript does not report the corresponding sizes alongside the accuracy results. Reporting these sizes and comparisons at matched storage would help establish the accuracy–storage trade-off. The search ablations include random proposals and removal of physical feedback, but do not include additional conventional search baselines beyond random proposals, such as evolutionary or model-based optimization adapted to the same allocation space and evaluation budget. In addition, although the overlap is explicitly acknowledged, three tasks in Macro7 also participate in search. For Qwen2.5-7B-Instruct in Table 4, the average improvement over GPTQ, computed from the reported task scores, is approximately 1.53 percentage points on the three Macro3 tasks and 0.08 points on the remaining four tasks. Together, these limitations make it difficult to determine the benefit attributable to the proposed search procedure and its generalization beyond the search tasks.

  3. The methodological contribution of the search framework needs clearer positioning. Related search ideas, including LLM-guided candidate generation, feedback-driven optimization, and population management, have been explored in methods such as RZ-NAS, LLMatic, and EvoPress. AutoQS differs in searching heterogeneous group-size allocations at fixed weight precision, which is a relevant distinction. However, its current description appears to primarily adapt established search components to this setting. The manuscript would benefit from clarifying which aspects of candidate generation, candidate management, or evaluation feedback provide additional methodological contributions. Comparisons with conventional search methods under the same allocation space and evaluation budget would help substantiate these contributions.

Questions for authors
  1. Proxy evaluation. How is the scalar Stage-1 proxy score computed? How is each candidate allocation incorporated into the NLL evaluation, and which reference model is used to measure degradation? Please explain how this evaluation avoids constructing a fresh deployment checkpoint, and provide the relevant reference if an existing evaluator is used.

  2. Search rules. What are the concrete rules for selecting parents and retaining candidates in the repository? When a duplicate allocation is encountered, how does deterministic novelty repair choose which entries to modify and which replacement values to try?

  3. Storage comparability. What are the measured checkpoint sizes of the AutoQS configurations and the corresponding uniform-G128 baselines? Are the accuracy gains obtained at comparable storage costs, or do they involve additional metadata storage?

  4. Performance beyond search tasks. For the Qwen2.5-7B base-model ablations in Table 2, are results on the four tasks outside Macro3—MMLU, HellaSwag, ARC-Easy, and WinoGrande—available for the Full, random-proposer, and no-feedback variants, using checkpoints selected by the predefined search-time criterion? These results would help assess whether the benefits of the proposed search mechanism extend beyond the tasks supplying physical-evaluation feedback.

Reviewer_1gLzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

AutoQS is a framework that uses large language models to automatically design quantization strategies. With the weight bit-width fixed, it searches for heterogeneous group-size assignments across different Transformer layers and operator groups. Based on a parent policy, previous evaluation results, and a candidate archive, the LLM proposes sparse policy edits. AutoQS then uses deterministic validation, duplicate detection, and novelty repair to ensure that the proposed policies are valid. To reduce search cost, each candidate first goes through a low-cost proxy evaluation. Every five iterations, the system performs actual GPTQ quantization on one candidate and feeds the downstream accuracy and checkpoint size back into the search. The experiments show that the heterogeneous group-size policies found by AutoQS can match or outperform uniform-G128 quantization on some Qwen and LLaMA-family models, with some ability to transfer across related models.

Critical strengths

AutoQS may be the first work to apply an LLM-driven automated research framework to quantization strategy design.

Critical weaknesses
  1. Low-cost evaluation is a core part of AutoQS, as it is what allows the method to search under a limited budget for full quantization runs. However, the paper does not provide the necessary details. Section 3.3 does not define how the combined proxy score is computed or how its components are weighted. It also does not specify the dataset, number of samples, or exact procedure used for proxy evaluation. Neither the main paper nor the appendix reports the runtime of a single proxy evaluation, the total GPU hours required for a full search, or the cost ratio between proxy evaluation and actual GPTQ evaluation. Because important details are missing, making it difficult to assess the method’s engineering contribution.

  2. The policy-generation process in AutoQS is fairly black-box and stochastic. Important details that determine the search behavior, including the full prompt, exact model version, sampling parameters, and parent-selection procedure, are not provided. In addition, prior work on automated quantization has already used reinforcement learning, Hessian/Fisher/gradient-based sensitivity, differentiable search, integer programming, and training-free proxies to find quantization configurations. Although most of these methods search over bit-widths rather than group sizes under a fixed bit-width, they could still be adapted into strong baselines under the same search space and budget. It is currently unclear what advantage the LLM proposer has over these more transparent and reproducible alternatives, or whether the two types of methods could be combined. The paper also does not sufficiently explain why group size is chosen as the main search variable, or whether searching over group size provides larger gains than searching over other quantization parameters. Because important details are missing, making it difficult to assess the method’s engineering contribution.

3.The search-space calculation in Appendix E.1 appears inconsistent with the policy definition in Section 3.1. The main text defines only four independent decisions per layer—QKV, Out, GateUp, and Down—because Q/K/V and Gate/Up share their configurations. An LL-layer model should therefore have 4L4L independent variables. However, the appendix counts all seven physical projection matrices separately and reports search spaces of 51965^{196} for Qwen2.5-7B and 52245^{224} for LLaMA-7B. Based on the definition in the main text, the corresponding upper bounds should instead be 5112≈1078.35^{112}\approx10^{78.3} and 5128≈1089.55^{128}\approx10^{89.5}.

  1. The main figure is rather rough and overly simple, and its flow is difficult to follow. The writing also contains a number of issues that need careful revision.
Questions for authors

AutoQS makes several quantization-specific adaptations to the standard LLM-driven propose–evaluate–retain loop. However, the overall use of a candidate archive, historical feedback, and iterative proposal generation is very similar to FunSearch, EoH, ReEvo, and AlphaEvolve. Multi-fidelity evaluation is also not a new optimization paradigm. The paper does not demonstrate a clear advantage over more general automated search frameworks. Overall, the method appears closer to an application of existing frameworks to quantization than a substantially new search method, and its novelty is limited.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.