- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CoNestQ is a nested PTQ method supporting 3–8-bit inference through a shared parent representation. It combines joint parameter calibration, balanced parent-code selection, and branch-wise error compensation. Experiments across four LLMs report improved low-bit quality over MatGPTQ, with component ablations and deployment-cost measurements.
Critical strengths
- The paper clearly identifies cross-bit coupling in shared quantized representations and addresses calibration, code assignment, and error propagation within one framework. This is relevant to multi-precision deployment.
- Evaluation across all six target precisions demonstrates low-bit improvements over MatGPTQ, supported by component ablations.
Critical weaknesses
-
The quality–cost advantage over relevant multi-precision methods remains unclear. Comparisons with AnyBCQ [1] and MoBiQuant [2] are missing. Against MatGPTQ, CoNestQ has comparable storage and inference costs but requires approximately 1.67× longer quantization time on Qwen3-8B-Base (Figure 2). Moreover, its Llama 8-bit PPL is 19.3, compared with approximately 11.0 for GPTQ and APL (Table 1). These results leave its deployment advantage and high-bit quality trade-off insufficiently established.
-
The contributions beyond joint calibration are insufficiently isolated. Table 3 does show an improvement from Parent+Branch without calibration: mean PPL decreases from MatGPTQ's 43.18 to 39.60. However, the missing Para-only baseline—MatGPTQ with the same joint calibration—prevents assessing the additional benefit of these mechanisms once calibration is controlled. The ablation is also limited to Llama. At 4 bits, removing Branch improves PPL from 22.11 to 21.81, while removing Parent yields nearly identical PPL (22.14). More detailed, cross-model ablations are needed to establish consistent contributions.
Questions for authors
-
Could you provide the Para-only baseline, namely MatGPTQ with the same joint parameter calibration? How much additional improvement comes from parent selection and branch compensation? Please also clarify whether these gains persist across models and provide objective-term ablations and sensitivity to the compensation window G.
-
How does CoNestQ compare with AnyBCQ and MoBiQuant under matched hardware and comparable precision or effective-bit budgets? Please evaluate quality, checkpoint size, peak memory, and inference speed using the respective inference implementations.
-
How are child weights obtained and stored at inference time? Eq. (1) uses rounding-based slicing rather than pure prefix truncation, requiring an additional rounding bit. Table 5 reports 4.60 GB peak memory for 3-bit CoNestQ, versus a 7.20 GB shared checkpoint in Table 4; these measurements do not establish that the full parent remains GPU-resident. Are child codes materialized and repacked before serving, or derived within the inference kernel? Please report precision-switching latency, including conversion/repacking and data transfer where applicable, and peak memory with the parent resident.
I would be willing to raise my score if these controlled ablations establish the independent contributions of the proposed mechanisms and the baseline comparisons and runtime measurements substantiate a competitive quality–cost trade-off.
References
[1] Gunho Park et al. AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs. ICLR, 2026. https://arxiv.org/abs/2510.10467
[2] Dongwei Wang et al. MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM. arXiv:2602.20191, 2026. https://arxiv.org/abs/2602.20191v2
4 comments
Thank you for clarifying how CoNestQ extracts lower-precision weights. MoBiQuant derives its slice scales from a shared base scale and switches precision by changing which bit slices are active. When the packed slices are already GPU-resident, the switch itself has near-zero overhead because it requires no re-rounding, repacking, or host-to-device transfer. Could you clarify whether CoNestQ incurs the reported conversion, repacking, and transfer costs on each precision switch, or whether it caches the packed child weights? A comparison of the packed checkpoint sizes, including scale metadata, and the GPU memory needed to keep each method ready to switch would help clarify this trade-off even without a public MoBiQuant kernel. This would clarify the storage and switching trade-offs independently of GPU generation.
Comparison with AnyBCQ
We further compare AnyBCQ with our proposed CoNestQ. While AnyBCQ achieves the best results at 3 bits, CoNestQ delivers comparable quality at higher bitwidths, particularly in zero-shot accuracy, outperforming AnyBCQ in 12 of 16 configurations at 5–8 bits. Moreover, CoNestQ offers better cross-bit consistency and resource efficiency.
Cross-bit consistency. Cross-bit consistency is essential for multi-precision quantization, ensuring that increasing bitwidth does not degrade model quality and that each precision provides a meaningful accuracy–resource trade-off. We measure inconsistencies using the inversion rate (IR), the frequency of PPL increases with bitwidth, and inversion magnitude (IM), the average relative magnitude of such increases. Lower IR and IM indicate better consistency.
Quantization quality and cross-bit consistency. The results across four models are summarized in Table 1. CoNestQ substantially reduces average IR from 25% to 10% and IM from 3.39% to 0.78%, while maintaining competitive quantization quality, demonstrating improved cross-bit consistency.
| Model | Method | PPL ↓ (Mean; 3/6/8-bit) | Acc ↑ (Mean; 3/6/8-bit) | IR ↓ | IM ↓ |
|---|---|---|---|---|---|
| Llama3-8B | AnyBCQ | 12.15 (14.9/11.5/12.0) | 69.00 (67.1/69.4/69.5) | 20% | 7.14% |
| CoNestQ | 21.62 (29.3/19.8/19.3) | 68.47 (62.0/69.8/70.4) | 20% | 1.05% | |
| Qwen2.5-14B | AnyBCQ | 9.32 (10.1/9.0/9.2) | 75.20 (73.7/75.5/75.5) | 20% | 3.33% |
| CoNestQ | 9.23 (10.7/8.8/8.8) | 75.58 (71.8/76.5/76.6) | 0% | 0% | |
| Qwen3-8B | AnyBCQ | 10.32 (11.1/10.2/10.1) | 70.53 (69.4/71.1/70.4) | 20% | 0.99% |
| CoNestQ | 11.00 (13.6/10.2/10.1) | 70.52 (66.1/71.9/72.0) | 0% | 0% | |
| Qwen3-14B | AnyBCQ | 9.65 (10.1/9.6/9.8) | 72.23 (70.3/72.6/72.4) | 40% | 2.11% |
| CoNestQ | 11.85 (22.9/9.6/9.6) | 71.68 (66.3/72.5/72.7) | 20% | 2.08% |
Resource efficiency. CoNestQ further reduces shared storage by approximately 46% and peak GPU memory by up to 66% (at 3 bits), while maintaining comparable inference throughput. Notably, its offline quantization time is reduced by approximately 94% compared with AnyBCQ. The detailed resource comparisons are reported below.
| Metric | AnyBCQ | CoNestQ |
|---|---|---|
| Shared storage (GB, Llama3-8B-Instruct) ↓ | 13.46 | 7.20 |
| Peak GPU memory (GB, 3/4/8-bit) ↓ | 13.71/13.71/13.71 | 4.60/5.39/8.64 |
| Throughput (tokens/s, 3/4/8-bit) ↑ | 47.49/48.31/47.86 | 47.52/47.33/47.14 |
| Offline quantization time (s, Qwen3-8B-Base) ↓ | 25,129 | 1,449 |
Thank you for this insightful question. In CoNestQ, child weights are not cached; each precision switch requires repacking and transferring the target-precision weights to GPU. While this incurs switching overhead, it avoids the expensive GPU memory overhead of keeping all precision-specific weights resident, as in AnyBCQ and MoBiQuant, even when only the lowest-bit configuration is used. In practice, requests can be grouped by precision, restricting switching to batch or inference-stage boundaries and amortizing the overhead across requests.
Due to time and hardware constraints, we are unlikely to complete a reliable evaluation of MoBiQuant within the rebuttal period. But we will include the corresponding packed checkpoint sizes and GPU memory requirements for precision switching in the camera-ready version.
We thank the reviewer for the constructive feedback. We have revised the manuscript and conducted additional experiments to address the concerns.
1. R3-W2/Q1
We added a Para-only baseline (MatGPTQ with joint calibration) and extended the ablation study to Qwen3-8B-Base (Table 3). Para delivers the primary PPL gains, reducing Llama PPL from 43.18 to 24.31, while Parent and Branch provide complementary improvements in cross-bit consistency, reducing IM from 5.11% to 0.89% on Llama and from 1.87% to 0.30% on Qwen. We also added parameter and compensation-window sensitivity analyses (Appendix C.5, Tables 17–19), showing minimal sensitivity to λ and ϵ, the importance of error-order regularization, and the effectiveness of moderate compensation windows (G=64–128). Further details are provided in our responses to R2-W2/Q2. The ablation results across both models are summarized below:
2. R3-W1/Q2
We added AnyBCQ comparisons under the same H100 evaluation environment. Although AnyBCQ achieves the best 3-bit results, CoNestQ delivers broadly comparable quality at other bitwidths, particularly in zero-shot accuracy, while offering better cross-bit consistency and resource efficiency. Specifically, CoNestQ reduces shared storage by 46% and peak GPU memory by 55%, with only 1% lower throughput, and cuts offline quantization time by 94%. MoBiQuant could not be reliably evaluated because its official implementation lacks the required executable operators; therefore, we do not report the relevant results. The detailed storage (Table 4), inference (Table 5), and offline quantization efficiency (Fig. 2) comparisons are summarized below
3. R3-Q3
Before inference, the 8-bit parent weights are loaded into CPU memory, where the additional rounding bit is used to generate and repack the corresponding r-bit child weights according to Eq. (1). Only the packed r-bit weights are transferred to GPU memory; thus, the additional bit is used exclusively for offline child-weight extraction and does not participate in inference computation. On Qwen3-8B-Base, conversion, packing, and GPU loading take 23.9 s for 3-bit and 32.4 s for 8-bit weights (H100, 10-run average). During 8-bit inference, the full parent weights are loaded into GPU memory, resulting in a peak memory usage of 8.64 GB (Table 5).