acceptodds
Withdrawn from ICLR 2027

MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixed-precision quantization improves post-training quantization by assigning higher precision to sensitive layers, but existing methods typically optimize bit allocation for a fixed memory budget. On edge devices, the memory available to an LLM can vary at runtime and is unknown during offline calibration. Adaptive quantization addresses this setting by calibrating once and deriving an allocation for the available budget at deployment. However, accurately calibrating before the deployment-time allocation is known is challenging because a layer's distortion depends on the bitwidths of its upstream layers. We show that this dependence substantially changes layer scores and the resulting allocations. We propose MixQuant, a quantizer-agnostic adaptive quantization framework that marginalizes over information unknown during offline calibration while exploiting the fixed quantizer and allocator. MixQuant estimates each layer's expected distortion over quantized upstream configurations, regularizes low-bit assignments that introduce disproportionate distortion, and calibrates quantizer parameters on plans produced by the deployment-time allocator. At deployment, a greedy solver produces an allocation for any available budget in time. Across three LLMs, two base quantizers, multiple memory budgets, and diverse downstream tasks, MixQuant consistently improves perplexity and average task accuracy. On Llama-3.2-3B at , it improves average accuracy over the strongest baseline by and percentage points with AWQ and GPTQ, respectively. Under tight memory constraints, competing methods incur – the degradation from FP16 of MixQuant across the evaluated task categories.

Primary area:
optimization
Keywords:
mixed-precision quantization, post-training quantization, adaptive quantization, large language models, LLM compression, AWQ, GPTQ, efficient deep learning, neural network compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_izKxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Paper Summary

MixQuant proposes an adaptive mixed-precision quantization framework intended to support different deployment-time memory budgets after a single offline calibration procedure. It estimates layer distortion under randomly sampled quantized upstream configurations, introduces a tail regularizer to discourage excessively low-precision assignments, and calibrates quantizer parameters using solver-generated configurations. A greedy allocator then selects layer-wise bit-widths for a given budget. Experiments evaluate the framework with AWQ and GPTQ across three LLMs and multiple memory budgets

Critical strengths

Clear motivation addressing a meaningful calibration problem.

The paper clearly motivates the mismatch between evaluating a layer under full-precision upstream activations and deploying it within a fully quantized model. Accounting for changes induced by upstream quantization is meaningful, particularly when the final bit allocation is unknown during calibration. However, related concerns about global and cross-layer quantization effects have also been explored in previous works, so this general motivation should not be presented as entirely new.

Critical weaknesses
  1. The practical advantage over a small set of pre-calibrated configurations is insufficiently established.

    The paper emphasizes that the space of mixed-precision configurations is combinatorially large. However, practical deployment does not necessarily require calibrating every possible configuration. A straightforward alternative is to optimize and calibrate a small set of configurations targeting, for example, average precisions of 3, 4, and 5 bits, and select a feasible configuration at deployment. The paper does not sufficiently establish its advantage over this alternative in accuracy, offline calibration cost, storage requirements, or model-switching latency. Such a comparison would better justify the need for the proposed adaptive calibration framework.

  2. The simple random-sampling strategy lacks sufficient validation.

    Stage 1 estimates distortion using uniformly sampled upstream configurations, with a fixed sample count of K=10K=10 for every layer and candidate bit-width. As network depth increases, the space of possible upstream configurations becomes substantially larger, yet the paper provides insufficient evidence that ten samples yield stable distortion estimates and reliable allocations. Sensitivity analyses over KK, variation across random seeds, and comparisons against estimates obtained with substantially more samples would strengthen this design choice. Furthermore, uniform random configurations may differ considerably from the budget-constrained configurations produced by the deployment-time allocator, leaving their representativeness insufficiently established.

  3. The quality of the greedy allocation is insufficiently characterized.

    The solver in Section 3.4 starts from the minimum-bit configuration and follows a sequence of irreversible local upgrades. Earlier decisions determine which subsequent upgrades remain feasible, and this procedure does not guarantee a globally optimal allocation under the given budget, even for the surrogate objective.

  4. The presentation and reporting of quantization precision require improvement.

    The manuscript contains writing and notation issues that hinder readability. For example, the sentence beginning with “bit-widths” on line 172 should start with a capital letter. Around line 214, DD should be clearly introduced in the main text and connected to the distortion estimates in Eq. (2), despite its definition in the caption of Figure 1. Furthermore, the experiments should report the achieved parameter-weighted average bit-width alongside memory consumption in GB. Absolute memory consumption depends on model size, whereas average bit-width makes the quantization level easier to interpret and facilitates comparisons across models. More thorough proofreading and clearer reporting would improve the presentation.

Questions for authors

See weakness

Reviewer_HqbnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MixQuant, a mixed-precision quantization scheme that adapts to varying memory budgets at deployment, which may not be known in advance. MixQuant scores each layer based on randomly selected upstream contexts, employs a tail penalty to reduce the lowest bit widths, and calibrates quantizer parameters using a solver.

Critical strengths
  • The problem is well-motivated for the target edge deployment scenario.
  • Deployment-time quantization is interesting
  • Shows significant performance gains
Critical weaknesses
  • At 1.25 GB, Random Memory in Table 5 has better performance than the adaptive baselines (CoopQ and LIM). This could be a configuration error. Also, CoopQ and LIM are evaluated at the block level, while MixQuant uses linear-layer granularity.
  • Comparison against a uniform or level-filling is not given
Questions for authors
  • Why do both adaptive baselines (CoopQ and LIM) perform poorly compared to random allocation? Are these baselines implemented/configured correctly?
  • The paper states that linear-layer CoopQ “performed substantially worse”. Please provide the numbers. Do they match the original paper’s?
  • How does MixQuant compare with a uniform or level-filling allocation under the same budget?
  • What is the total offline calibration time?
Reviewer_qSoiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents MixQuant, an adaptive mixed-precision quantization framework for LLMs under changing memory budgets. It combines context-aware distortion estimation, tail regularization, and solver-guided calibration to improve accuracy across three models with AWQ and GPTQ.

Critical strengths
  1. The paper addresses a practical problem. Available memory can change during deployment, and repeating calibration for every budget would be expensive.
  2. MixQuant combines expected-distortion estimation with solver-guided calibration to avoid recalibration for each deployment budget.
  3. the experiment result show substantial accuracy gains under tight memory budgets with both AWQ and GPTQ.
Critical weaknesses

1.The paper does not report measured online allocation time or model switching overhead, despite dynamic memory adaptation being its main motivation.

2.It is unclear how well the proposed distortion scores predict the quality of complete mixed-precision configurations. The current ablations do not cleanly isolate the benefit of sampling quantized upstream contexts.

3.The comparison with the ILP variant does not establish the greedy allocator’s optimality gap. Table 5 reports model quality, but not allocation objective values or solver runtimes.

Questions for authors

Overall, I think the problem is worth studying, but the current evaluation does not sufficiently demonstrate the practicality of the proposed approach.

First, the paper argues that calibrating every possible bit-width configuration offline is prohibitively expensive and proposes MixQuant to reduce online overhead. However, the experiments do not report the actual runtime of the online allocator, despite its stated complexity of O(L∣B∣log⁡L)O(L|\mathcal{B}|\log L)

Second, since MixQuant uses a greedy algorithm to select bit-widths, it would also be useful to compare its accuracy with that of a plan optimized and calibrated offline for the same fixed budget. This would clarify the accuracy trade-off introduced by supporting multiple budgets without budget-specific recalibration.

Third, the proposed distortion scores average activation errors over randomly sampled upstream configurations, but it is unclear how well these averages predict accuracy changes under a specific deployment configuration. For example, increasing a layer’s bit-width may have different effects on accuracy when upstream layers use 8 bits versus 3 bits. Averaging over these contexts may hide such differences and lead to inaccurate estimates of the benefit of a bit-width upgrade. The paper does not directly validate this relationship. At a minimum, the experiments should quantify how well the score-predicted benefits of upgrading a layer correlate with the actual accuracy gains across different upstream configurations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.