acceptodds
Under review as a conference paper at ICLR 2027

CoNestQ: Coordinated Nested Quantization for Low-Bit LLM Deployment

11 views

Abstract

Low-latency and data-locality requirements are shifting LLM inference from centralized clouds toward distributed, multi-tier architectures such as the Intelligence Delivery Network (IDN), which distributes intelligence across cloud, regional, edge, and local nodes on demand. Serving such heterogeneous nodes economically requires a single model representation from which multiple precision levels can be derived. Fixed-precision quantization cannot meet diverse node budgets without storing a separate model per bit width, while existing nested methods leave the cross-bit coupling induced by representation sharing largely unaddressed, with several bit-widths unusable on some models. To address these limitations, we propose CoNestQ (Coordinated Nested Quantization), a post-training quantization framework that derives all target precisions from a shared parent representation and coordinates the three coupled stages that prior nested methods address only partially: (i)joint quantization-parameter calibration across target bit-widths, (ii)balanced parent-code selection guided by normalized excess error, and (iii)branch-wise error compensation with periodic cross-precision synchronization. Across the full 3–8-bit range on four LLMs, CoNestQ attains a 100% validity rate with strong low-bit robustness. Relative to the non-linear nested baseline, it reduces multi-precision storage by 26–34%, lowers peak GPU memory by about 38% on average, and improves inference throughput by about 16%; relative to six independently quantized models, multi-precision storage shrinks by 5.5–6.2.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Post-Training Quantization, Multi-Precision Quantization, Nested Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s6ySOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoNestQ is a nested PTQ method supporting 3–8-bit inference through a shared parent representation. It combines joint parameter calibration, balanced parent-code selection, and branch-wise error compensation. Experiments across four LLMs report improved low-bit quality over MatGPTQ, with component ablations and deployment-cost measurements.

Critical strengths
  1. The paper clearly identifies cross-bit coupling in shared quantized representations and addresses calibration, code assignment, and error propagation within one framework. This is relevant to multi-precision deployment.
  2. Evaluation across all six target precisions demonstrates low-bit improvements over MatGPTQ, supported by component ablations.
Critical weaknesses
  1. The quality–cost advantage over relevant multi-precision methods remains unclear. Comparisons with AnyBCQ [1] and MoBiQuant [2] are missing. Against MatGPTQ, CoNestQ has comparable storage and inference costs but requires approximately 1.67× longer quantization time on Qwen3-8B-Base (Figure 2). Moreover, its Llama 8-bit PPL is 19.3, compared with approximately 11.0 for GPTQ and APL (Table 1). These results leave its deployment advantage and high-bit quality trade-off insufficiently established.

  2. The contributions beyond joint calibration are insufficiently isolated. Table 3 does show an improvement from Parent+Branch without calibration: mean PPL decreases from MatGPTQ's 43.18 to 39.60. However, the missing Para-only baseline—MatGPTQ with the same joint calibration—prevents assessing the additional benefit of these mechanisms once calibration is controlled. The ablation is also limited to Llama. At 4 bits, removing Branch improves PPL from 22.11 to 21.81, while removing Parent yields nearly identical PPL (22.14). More detailed, cross-model ablations are needed to establish consistent contributions.

Questions for authors
  1. Could you provide the Para-only baseline, namely MatGPTQ with the same joint parameter calibration? How much additional improvement comes from parent selection and branch compensation? Please also clarify whether these gains persist across models and provide objective-term ablations and sensitivity to the compensation window G.

  2. How does CoNestQ compare with AnyBCQ and MoBiQuant under matched hardware and comparable precision or effective-bit budgets? Please evaluate quality, checkpoint size, peak memory, and inference speed using the respective inference implementations.

  3. How are child weights obtained and stored at inference time? Eq. (1) uses rounding-based slicing rather than pure prefix truncation, requiring an additional rounding bit. Table 5 reports 4.60 GB peak memory for 3-bit CoNestQ, versus a 7.20 GB shared checkpoint in Table 4; these measurements do not establish that the full parent remains GPU-resident. Are child codes materialized and repacked before serving, or derived within the inference kernel? Please report precision-switching latency, including conversion/repacking and data transfer where applicable, and peak memory with the parent resident.

I would be willing to raise my score if these controlled ablations establish the independent contributions of the proposed mechanisms and the baseline comparisons and runtime measurements substantiate a competitive quality–cost trade-off.

References

[1] Gunho Park et al. AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs. ICLR, 2026. https://arxiv.org/abs/2510.10467

[2] Dongwei Wang et al. MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM. arXiv:2602.20191, 2026. https://arxiv.org/abs/2602.20191v2

4 comments

We thank the reviewer for the constructive feedback. We have revised the manuscript and conducted additional experiments to address the concerns.

1. R3-W2/Q1

We added a Para-only baseline (MatGPTQ with joint calibration) and extended the ablation study to Qwen3-8B-Base (Table 3). Para delivers the primary PPL gains, reducing Llama PPL from 43.18 to 24.31, while Parent and Branch provide complementary improvements in cross-bit consistency, reducing IM from 5.11% to 0.89% on Llama and from 1.87% to 0.30% on Qwen. We also added parameter and compensation-window sensitivity analyses (Appendix C.5, Tables 17–19), showing minimal sensitivity to λ\lambda and ϵ\epsilon, the importance of error-order regularization, and the effectiveness of moderate compensation windows (G=64G=64–128). Further details are provided in our responses to R2-W2/Q2. The ablation results across both models are summarized below:

MethodLlama PPL ↓Llama IM ↓Qwen PPL ↓Qwen IM ↓
MatGPTQ43.185.92%11.152.07%
Para only24.315.11%11.101.87%
w/o Parent22.511.55%11.040.49%
w/o Branch22.541.15%11.080.59%
CoNestQ21.600.89%10.990.30%

2. R3-W1/Q2

We added AnyBCQ comparisons under the same H100 evaluation environment. Although AnyBCQ achieves the best 3-bit results, CoNestQ delivers broadly comparable quality at other bitwidths, particularly in zero-shot accuracy, while offering better cross-bit consistency and resource efficiency. Specifically, CoNestQ reduces shared storage by 46% and peak GPU memory by 55%, with only 1% lower throughput, and cuts offline quantization time by 94%. MoBiQuant could not be reliably evaluated because its official implementation lacks the required executable operators; therefore, we do not report the relevant results. The detailed storage (Table 4), inference (Table 5), and offline quantization efficiency (Fig. 2) comparisons are summarized below

MetricAnyBCQCoNestQ
Shared storage (GB)13.467.20
Peak GPU memory (GB, 8bit)13.718.64
Average throughput (tokens/s)47.8947.33
Offline quantization time (s)251291449

3. R3-Q3

Before inference, the 8-bit parent weights are loaded into CPU memory, where the additional rounding bit is used to generate and repack the corresponding r-bit child weights according to Eq. (1). Only the packed r-bit weights are transferred to GPU memory; thus, the additional bit is used exclusively for offline child-weight extraction and does not participate in inference computation. On Qwen3-8B-Base, conversion, packing, and GPU loading take 23.9 s for 3-bit and 32.4 s for 8-bit weights (H100, 10-run average). During 8-bit inference, the full parent weights are loaded into GPU memory, resulting in a peak memory usage of 8.64 GB (Table 5).

Thank you for clarifying how CoNestQ extracts lower-precision weights. MoBiQuant derives its slice scales from a shared base scale and switches precision by changing which bit slices are active. When the packed slices are already GPU-resident, the switch itself has near-zero overhead because it requires no re-rounding, repacking, or host-to-device transfer. Could you clarify whether CoNestQ incurs the reported conversion, repacking, and transfer costs on each precision switch, or whether it caches the packed child weights? A comparison of the packed checkpoint sizes, including scale metadata, and the GPU memory needed to keep each method ready to switch would help clarify this trade-off even without a public MoBiQuant kernel. This would clarify the storage and switching trade-offs independently of GPU generation.

AuthorsOpenReview ↗

Comparison with AnyBCQ

We further compare AnyBCQ with our proposed CoNestQ. While AnyBCQ achieves the best results at 3 bits, CoNestQ delivers comparable quality at higher bitwidths, particularly in zero-shot accuracy, outperforming AnyBCQ in 12 of 16 configurations at 5–8 bits. Moreover, CoNestQ offers better cross-bit consistency and resource efficiency.

Cross-bit consistency. Cross-bit consistency is essential for multi-precision quantization, ensuring that increasing bitwidth does not degrade model quality and that each precision provides a meaningful accuracy–resource trade-off. We measure inconsistencies using the inversion rate (IR), the frequency of PPL increases with bitwidth, and inversion magnitude (IM), the average relative magnitude of such increases. Lower IR and IM indicate better consistency.

Quantization quality and cross-bit consistency. The results across four models are summarized in Table 1. CoNestQ substantially reduces average IR from 25% to 10% and IM from 3.39% to 0.78%, while maintaining competitive quantization quality, demonstrating improved cross-bit consistency.

ModelMethodPPL ↓ (Mean; 3/6/8-bit)Acc ↑ (Mean; 3/6/8-bit)IR ↓IM ↓
Llama3-8BAnyBCQ12.15 (14.9/11.5/12.0)69.00 (67.1/69.4/69.5)20%7.14%
CoNestQ21.62 (29.3/19.8/19.3)68.47 (62.0/69.8/70.4)20%1.05%
Qwen2.5-14BAnyBCQ9.32 (10.1/9.0/9.2)75.20 (73.7/75.5/75.5)20%3.33%
CoNestQ9.23 (10.7/8.8/8.8)75.58 (71.8/76.5/76.6)0%0%
Qwen3-8BAnyBCQ10.32 (11.1/10.2/10.1)70.53 (69.4/71.1/70.4)20%0.99%
CoNestQ11.00 (13.6/10.2/10.1)70.52 (66.1/71.9/72.0)0%0%
Qwen3-14BAnyBCQ9.65 (10.1/9.6/9.8)72.23 (70.3/72.6/72.4)40%2.11%
CoNestQ11.85 (22.9/9.6/9.6)71.68 (66.3/72.5/72.7)20%2.08%

Resource efficiency. CoNestQ further reduces shared storage by approximately 46% and peak GPU memory by up to 66% (at 3 bits), while maintaining comparable inference throughput. Notably, its offline quantization time is reduced by approximately 94% compared with AnyBCQ. The detailed resource comparisons are reported below.

MetricAnyBCQCoNestQ
Shared storage (GB, Llama3-8B-Instruct) ↓13.467.20
Peak GPU memory (GB, 3/4/8-bit) ↓13.71/13.71/13.714.60/5.39/8.64
Throughput (tokens/s, 3/4/8-bit) ↑47.49/48.31/47.8647.52/47.33/47.14
Offline quantization time (s, Qwen3-8B-Base) ↓25,1291,449

Thank you for this insightful question. In CoNestQ, child weights are not cached; each precision switch requires repacking and transferring the target-precision weights to GPU. While this incurs switching overhead, it avoids the expensive GPU memory overhead of keeping all precision-specific weights resident, as in AnyBCQ and MoBiQuant, even when only the lowest-bit configuration is used. In practice, requests can be grouped by precision, restricting switching to batch or inference-stage boundaries and amortizing the overhead across requests.

Due to time and hardware constraints, we are unlikely to complete a reliable evaluation of MoBiQuant within the rebuttal period. But we will include the corresponding packed checkpoint sizes and GPU memory requirements for precision switching in the camera-ready version.

Reviewer_sVHPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies nested quantization, where one shared model supports several bit widths. CoNestQ combines joint calibration, balanced parent-code selection, and separate error compensation for each precision with periodic synchronization. Experiments cover four LLMs at 3–8 bits. The method improves several low-bit results over MatGPTQ while keeping the same storage cost and similar inference performance.

Critical strengths
  1. Supporting several precisions from one checkpoint is useful for deployment. CoNestQ keeps the compact representation of MatGPTQ and improves quality in several settings without adding much inference overhead.
  2. Some low-bit gains are meaningful. On Qwen3-14B at 3 bits, average PPL drops from 33.1 to 22.9 and average accuracy rises from 63.1 to 66.3 compared with MatGPTQ. The dataset-level results in the appendix help assess these gains.
Critical weaknesses
  1. The contribution beyond MatGPTQ is not sufficiently established. Shared parent codes, joint code selection across bit widths, and cross-bit error compensation are already part of that framework. CoNestQ changes calibration, the code-selection objective, and the compensation strategy, but Table 3 suggests that calibration accounts for most of the improvement: removing it raises mean PPL from 21.60 to 39.60, while removing parent-code selection or branch compensation gives 22.51 and 22.54. A MatGPTQ baseline with the same joint calibration is needed. Ablations on only one model do not establish that the other two components provide consistent gains.
  2. Important parts of the method remain unclear. Equation (7) combines normalized error ratios with an unnormalized squared-error term, so their balance can change with weight scale. The value of epsilon and its effect are not reported, and the error-ordering penalty is not tested separately. There is also a mismatch between Equation (11), which updates all later weights, and Algorithm 1, line 9, which updates only weights within the current window. As written, the pseudocode does not show how error reaches later windows. These details matter for understanding and reproducing the method.
Questions for authors
  1. How does MatGPTQ perform with the same joint calibration? Please specify the baseline version and settings, and show whether the gains from the other two components also hold on a Qwen model.
  2. What epsilon is used in Equations (8) and (9)? How sensitive are the results to it and to the relative scales of the terms in Equation (7)? What happens when only the error-ordering penalty is removed?
  3. Does the implementation propagate error beyond the current window? Please resolve the mismatch between Equation (11) and Algorithm 1. How does periodic synchronization compare with synchronization at every step or no synchronization?
  4. APL shows severe degradation on all three Qwen models, even at 8 bits. Its official implementation warns that automatic adaptation to untested architectures may be incorrect. What architecture settings and correctness checks were used? The unusually high GPTQ C4 results on Llama also need explanation.
1 comment

We thank the reviewer for the constructive feedback. We have revised the manuscript and conducted additional experiments to address the concerns.

1. R2-W1/Q1

We added a Para-only baseline (MatGPTQ with joint calibration) and extended the ablation study to Qwen3-8B-Base (Table 3). Para provides the primary PPL gains, reducing Llama PPL from 43.18 to 24.31, while Parent and Branch further reduce IM from 5.11% to 0.89% on Llama and from 1.87% to 0.30% on Qwen. These results demonstrate that the proposed components make consistent and complementary contributions across different models.

MethodLlama PPL ↓Llama IM ↓Qwen PPL ↓Qwen IM ↓
MatGPTQ43.185.92%11.152.07%
Para only24.315.11%11.101.87%
w/o Parent22.511.55%11.040.49%
w/o Branch22.541.15%11.080.59%
CoNestQ21.600.89%10.990.30%

2. R2-W2/Q2

We corrected Eq. (7) to match the implementation by replacing the second term with the mean normalized excess error, rb(q)∣B∣\frac{r_b(q)}{|\mathcal B|}, aligning the scales of the first two terms. We set λ=0.1\lambda=0.1, μ=0.05\mu=0.05, and ϵ=10−10\epsilon=10^{-10}. Sensitivity analyses show negligible PPL variations across the tested λ\lambda and ϵ\epsilon ranges (Appendix C.5, Table 17). Removing the error-order penalty (μ=0\mu=0) keeps average PPL around 11.0 but introduces 6–7-bit order flips, confirming its role in improving cross-bit consistency (Appendix C.5, Table 18). The sensitivity results are summarized below:

ParameterTested valuesAverage PPL ↓
λ\lambda0.01 / 0.05 / 0.1 / 0.511.02 / 11.02 / 10.99 / 10.99
ϵ\epsilon10−610^{-6} / 10−1010^{-10} / 10−1410^{-14} / 10−1810^{-18}11.03 / 10.99 / 10.99 / 10.99

3. R2-W2/Q3

Eq. (11) propagates branch-specific errors within each compensation window, while Eq. (12) aggregates them at the window boundary and applies the shared update to all subsequent unquantized columns. We have clarified this procedure in Eqs. (11)–(12) and Algorithm 1. We further evaluated the sensitivity to compensation-window size (Appendix C.5, Table 19). Moderate windows (G=64G=64–128128) achieve the lowest average PPL, whereas G=1G=1 effectively disables branch retention and larger windows slightly degrade performance. The results are summarized below:

Window size (GG)164128256
PPL ↓11.0810.9910.9911.03

4. R2-Q4

We adapted the Qwen architecture configurations based on GPTQ and APL, specifying the target modules for quantization, and verified the inference pipeline through execution checks, representative outputs, and tensor-dimension validation. Nevertheless, rerunning the official APL implementation under the same evaluation settings (Appendix B.1–B.2) reproduced the abnormal results on both Qwen3 models, consistent with similar anomalies reported in MoBiQuant (Wang et al., 2026). For GPTQ, the degradation is localized to low-bit C4 evaluation on Llama3-8B-Instruct (3/4/5-bit PPL: 7016.31/1255.50/266.64), indicating dataset- and precision-specific quantization degradation rather than a general implementation error.

Reviewer_m3HtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper proposes CoNestQ, a PTQ method for nested quantization that jointly considers the errors of multiple precisions. It addresses the cross-bit coupling in three stages. By using a format similar to MatGPTQ, it can reuse the same inference implementation, and it shows improved accuracy.
Critical strengths
  • The paper identifies the cross-bit coupling problem in nested quantization and proposes a more effective optimization method to address it.
  • Experiments and ablations are conducted across various models and bit-widths.
Critical weaknesses
  • The novelty is limited. While the paper correctly identifies the problems in optimizing nested quantization, the improvement over MatGPTQ is marginal except at 3 bits. In addition, there is no kernel-level contribution, as the existing inference implementation is reused as is.
  • The nested mapping of CoNestQ (Eq. 1) uses rounding as in MatQuant, so, as pointed out in MatGPTQ, an r-bit model requires reading r+1 bits. However, CoNestQ uses the same inference implementation as MatGPTQ and shows identical peak memory (Table 5). Does the actual inference read r+1 bits, or does it decode via truncation?
  • Why were base models used for Qwen3? Moreover, the performance of Any-Precision LLM looks abnormal. On the Qwen models, its PPL collapses even at 8 bits, whereas in the Qwen3-8B experiments of the MatGPTQ paper, Any-Precision LLM does not break down. The correctness of the baseline implementation should be verified.
  • Recent models such as Qwen3 are reasoning models, yet no reasoning evaluation is provided. Furthermore, since a recent study [1] reported that quantized models tend to overthink, it would be helpful to also show the output token length distribution.
  • (Minor) In L099, the reference for Any-Precision LLM is incorrectly given as Ashkboos et al. (2024). It should be Park et al. (2024).

[1] Lotfi, Sanae, et al. “Quantized reasoning models think they need to think longer, but they do not.” arXiv preprint arXiv:2606.00206 (2026).

Questions for authors
  • Please see the weaknesses above.
2 comments

We thank the reviewer for the thoughtful comments and have refined the manuscript with additional experiments to address the raised concerns.

1. R1-W1

CoNestQ's improvements are not limited to 3-bit quantization. On Llama3-8B-Instruct, it consistently outperforms MatGPTQ at 4–8 bits, reducing PPL by 44.8%–59.1% while maintaining comparable or better accuracy (Table 1). On other models, the smaller high-bit PPL gaps are consistent with the inherent nested tax: deriving all precisions from a shared parent representation constrains independent optimization, while higher-bit quantization leaves less room for further improvement. More importantly, CoNestQ substantially improves cross-bit consistency, reducing average IR and IM from 25% and 2.60% to 10% and 0.78%, respectively (Table 1), corresponding to reductions of 60% and 70%.

2. R1-W2

Before inference, the 8-bit parent weights are loaded into CPU memory, where the additional rounding bit is used to generate and repack the corresponding r-bit child weights according to Eq. (1). Only the packed r-bit weights are transferred to GPU memory; thus, the additional bit is used exclusively for offline child-weight extraction and does not participate in inference computation. The conversion, packing, and GPU-loading times on Qwen3-8B-Base are:

Target precisionTime
3-bit23.9 s
8-bit32.4 s

3. R1-W3

We adapted the Qwen architecture configurations based on GPTQ and APL, specifying the target modules for quantization, and verified the inference pipeline through execution checks, representative outputs, and tensor-dimension validation. Nevertheless, rerunning the official APL implementation under the same evaluation settings (Appendix B.1–B.2) reproduced the abnormal results on both Qwen3 models, consistent with similar anomalies reported in MoBiQuant (Wang et al., 2026):

Model3-bit PPL8-bit PPL
Qwen3-8B-Base1.55×1031.55 \times 10^33.67×1033.67 \times 10^3
Qwen3-8B1.62×1041.62 \times 10^41.02×1041.02 \times 10^4

4. R1-W4

Our evaluation follows common practice in PTQ research, reporting PPL on WikiText-2, C4, and PTB, together with zero-shot accuracy on PIQA, ARC-Challenge, and HellaSwag. We will further include dedicated reasoning evaluation benchmarks and output token-length analyses under consistent generation settings in the camera-ready version.

5. R1-W5

We have corrected the Any-Precision LLM reference to Park et al. (2024) and fixed the related writing errors.

Thank you for the thorough response, which resolves several of my specific points. My remaining concern lies with the significance of the work rather than its correctness, as the gains over the strongest baselines are modest. I will keep my current score, but I would not object to acceptance if the other reviewers see the contribution more favorably.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.