acceptodds
Under review as a conference paper at ICLR 2027

CostBit: Candidate-Cost-Informed Bit Allocation for Post-Training Quantization of LLMs

Abstract

Post-training quantization (PTQ) is a practical route to reducing the memory and bandwidth requirements of large language models (LLMs). However, uniform bit-width quantization becomes increasingly inefficient in the aggressive sub-3-bit regime due to heterogeneous quantization sensitivity across weight blocks. Although existing mixed-precision schemes alleviate this issue, their allocation rules fail to account for the quantization error variation of each weight block across candidate bit‑widths, resulting in suboptimal accuracy–compression trade-offs. To address this limitation, we introduce CostBit, a candidate-Cost-informed blockwise Bit allocation method for PTQ of LLMs. Specifically, CostBit evaluates the Hessian-weighted cost associated with each candidate bit-width for every weight block and leverages a Lagrangian allocator to closely match the target bit budget, supporting non‑integer values. Extensive experiments on OPT, LLaMA-2, and LLaMA-3 spanning 125M to 70B parameters demonstrate that CostBit yields improved performance at comparable target bit budgets, with significant gains in the sub-3-bit regime. Further, at an average bit-width of approximately 2.5 bits, it reduces WikiText-2 perplexity from 7.69 to 6.43 and improves the average zero-shot accuracy from 60.04% to 63.79% compared with OWQ on LLaMA-2-7B. CostBit also enables fully GPU-resident inference of LLaMA-2-70B on a single RTX 4090 (24 GB) at approximately 2.5 bits, achieving 75.70% average zero-shot accuracy, only 1.37% below the FP16 baseline. Our code is available at https://anonymous.4open.science/r/CostBit.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Post-Training Quantization, Mixed-Precision Quantization, Bit Allocation, Large Language Model Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YLUKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CostBit is a training-free block-wise mixed-precision allocation method for weight-only LLM quantization. It estimates Hessian-diagonal-weighted quantization costs for candidate bit widths, uses a Lagrangian search with local adjustment to approach non-integer per-layer budgets, protects sensitive columns in INT8, and applies GPTAQ reconstruction. The paper reports perplexity and zero-shot results across several model families, a mixed-precision CUDA implementation, and fully GPU-resident LLaMA-2-70B generation on an RTX 4090.

Critical strengths
  • This paper is practical motivation. Allocating a limited bit budget effectively matters for memory-constrained LLM deployment. Supporting non-integer budgets also offers useful flexibility when fixed 2-bit or 3-bit configurations cannot achieve the desired accuracy–memory balance.
  • Intuitive allocation criterion. Comparing each block’s quantization error across candidate bit-widths is a sensible way to estimate where an additional bit would help most. This is more informative than relying on a single sensitivity ranking.
  • Broad experimental coverage and useful ablations, the authors evaluates multiple model families and sizes using both perplexity and zero-shot tasks. They also examines allocation rules, cost definitions, reconstruction backends, and weak-column formats, providing useful evidence about the individual design choices.
  • Good to actual deployment. The custom mixed-bit CUDA kernel and single-GPU deployment results show that the method can be implemented beyond simulated quantization. The memory and throughput measurements add practical value to the accuracy results.
Critical weaknesses
  1. The methodological contribution needs clearer positioning. Hessian-aware allocation, constrained optimization, and group-wise mixed precision have been explored in prior work, including HAWQ-V3 and SliM-LLM. It is not yet clear how much CostBit adds beyond adapting these ideas to candidate-specific costs and non-integer budgets.
  2. My main concern is whether the block estimated costs predict actual layer-output distortion, this is a very important research question, the diagonal approximation ignores interactions between quantization errors across blocks. A lower sum of block costs therefore need not produce a better reconstructed layer. Figure 2b is illustrative, but I would prefer a systematic comparison of actual output distortion at matched bit budgets.
  3. The allocation gains need broader controlled validation. Table 3 helps isolate the allocation effect, but covers only two models. The simpler mean-Hessian-trace cost also performs comparably or slightly better in some settings. Comparisons on larger models, with clipping, weak-column protection, and reconstruction held fixed, would clarify the value of the proposed cost.
Questions for authors
  1. At the same bit budget, does CostBit reduce actual layer-output distortion more than alternative allocation strategies? How well do estimated costs predict measured errors?
  2. Do the allocation gains hold on larger models when the remaining PTQ pipeline is unchanged?
  3. What is the key methodological and innovation-level difference from previous hessian-aware, optimization-based allocation methods?
  4. Could the throughput evaluation include efficient 3-bit and 4-bit methods under the same hardware and workload? This would be more informative than comparing only with FP16.
9 comments
AuthorsOpenReview ↗

Validating CostBit’s distortion reduction and cost-error consistency

We sincerely thank you for the valuable suggestions, which help us improve the clarity and rigor of our work. We have carefully revised the manuscript, and all corresponding changes are highlighted in blue in the newly uploaded version. Our detailed responses are provided below.

  1. At the same bit budget, does CostBit reduce actual layer-output distortion more than alternative allocation strategies? How well do estimated costs predict measured errors?

Response: We have added systematic experiments to directly evaluate whether CostBit reduces actual layer-output distortion under matched bit budgets and whether its estimated costs reliably predict measured distortion. The new results demonstrate that CostBit consistently achieves lower layer-output distortion than alternative allocation strategies, while its estimated costs exhibit strong agreement with measured errors.

(1) Reduction of actual layer-output distortion. We compare CostBit with salience-based and random allocation strategies on 12 representative projections of LLaMA-2-7B, covering query, key, MLP up, and MLP down projections (see Page 19, Table 12). From the results, we find that CostBit obviously achieves lower measured layer-output NMSE in all 12 cases. The median reductions across projections were 16.75% relative to Salience and 10.03% relative to Random. These results directly confirm that candidate-cost-informed allocation reduces actual layer-output distortion.

Table a: Layer-output NMSE on LLaMA-2-7B at a 2.5-bit budget. Lower is better.

ProjectionLayerCostBitSalienceRandom
Query40.004110.004750.00452
Query160.010580.012910.01175
Query280.015350.018460.01684
Key40.002720.003190.00309
Key160.006850.008340.00763
Key280.011360.013680.01248
MLP up40.039360.046220.04291
MLP up160.034720.043410.03915
MLP up280.036840.044080.03998
MLP down40.047440.056980.05277
MLP down160.057490.069070.06481
MLP down280.059280.070700.06631

(2) Consistency between estimated costs and measured layer-output distortion. For each of the 12 layers, we uniformly sampled 30 distinct allocations at the same bit budget. We measured layer-output normalized mean squared error (NMSE) after GPTAQ reconstruction. For each layer, we computed the Spearman correlation between estimated total cost and NMSE. We also compared the 30 allocations two at a time, yielding 435 pairwise comparisons per layer. Pairwise ranking accuracy (PairAcc) is the percentage of comparisons in which the allocation with the lower estimated cost also yields lower measured NMSE. The estimated costs achieve a mean Spearman correlation of 0.835 and a pairwise ranking accuracy of 83.3% across the 12 projections. The related results are reported in Page 19, Table 13. In addition, the newly added Figures 5-7 in Pages 20-21 further show that allocations with lower estimated costs generally exhibit lower measured distortion. These results demonstrate that the estimated costs are able to predict measured layer-output distortion, despite relying on a diagonal Hessian approximation.

Table b: Rank agreement between estimated total cost and measured layer-output NMSE.

LayerProjectionSpearman ρ ↑PairAcc (%) ↑
4Query0.71276.6
4Key0.62071.5
4Down0.94690.6
4Up0.75377.5
16Query0.86884.4
16Key0.86184.4
16Down0.90286.9
16Up0.83382.3
28Query0.78880.9
28Key0.90987.1
28Down0.96192.2
28Up0.86885.3
OverallMean ± SD0.835 ± 0.10183.3 ± 6.0
AuthorsOpenReview ↗

Allocation ablation on larger models

  1. Do the allocation gains hold on larger models when the remaining PTQ pipeline is unchanged?

Response: We have extended the controlled allocation ablation to LLaMA-2-70B and LLaMA-3-70B to compare three blockwise bit allocation rules while keeping the rest of the PTQ pipeline fixed. All methods operate under approximately the same 2.5-bit budget, ensuring that performance differences arise from the allocation strategy itself. The related results are newly added in Page 16, Appendix B.3, Table 7.

Table c: Allocation ablation at 2.5 bits on LLaMA-2-70B/LLaMA-3-70B. We report WikiText-2 and C4 perplexity.

ComponentVariantActual bitsWikiText-2C4
AllocationCandidate Cost2.4987/2.49914.17/6.066.80/10.70
AllocationSensitivity2.4995/2.49964.39/6.757.18/11.60
AllocationRandom2.4982/2.49824.30/6.367.03/11.21

(1) Consistent improvements on 70B models. For instance, CostBit achieves the lowest WikiText-2 and C4 perplexity on both LLaMA-2-70B and LLaMA-3-70B. Specifically, on LLaMA-2-70B, it reduces WikiText-2 perplexity to 4.17, compared with 4.39 and 4.30 for sensitivity-based and random allocation. On LLaMA-3-70B, the corresponding results are 6.06 versus 6.75 and 6.36. Similar improvements are observed on C4. It indicates that the allocation strategy we propose still yields performance gains on larger models.

(2) Regarding the Mean Hessian Trace cost you mentioned. We firstly apologize that the previous version does not clearly distinguish the alternative cost formulations, leading to your misunderstanding. We have revised the descriptions of these three variants in Page 10, Table 3, i.e., Hessian diagonal weighted, Mean Hessian trace weighted, and Hessian spectral norm weighted. Note that these three variants can be all incorporated into CostBit to evaluate the cost associated with different block-bit configurations. We adopt the Hessian diagonal-weighted cost only as the default setting. Moreover, although Mean Hessian trace weighted variant performs comparably or slightly better in some settings, this only reflects differences in Hessian-based cost estimation rather than a limitation of CostBit’s allocation framework. Importantly, the core contribution lies in accounting for the varying MDGs of individual bit upgrades through candidate-specific costs, rather than relying on a particular Hessian-weighted formulation.

AuthorsOpenReview ↗

key methodological difference from prior methods

  1. What is the key methodological and innovation-level difference from previous hessian-aware, optimization-based allocation methods?

Response: We agree that Hessian-aware estimation, constrained optimization, and groupwise mixed precision are established techniques. However, the contribution of CostBit is not these individual components. We have clarified CostBit’s methodological contribution as follows.

(1) Core motivation. In the revised manuscript, we introduce the concept of marginal distortion gain (MDG) to better articulate our methodological contribution. Specifically, MDG refers to the quantization cost reduction achieved by a one-bit upgrade of a weight block (see Page 2, Rows 78-81). Rather than reflecting the absolute cost, MDG captures the cost difference between adjacent bit-widths. For instance, considering the cost shown in Eq. (6), the MDG of upgrading block kk from bb to (b+1)(b+1) bits is given by (costk,b−costk,b+1)(cost_{k,b}-cost_{k,b+1}) (see Page 6, Rows 279-282). Furthermore, Fig. 2(b) illustrates that upgrading a less sensitive block from 2 to 3 bits can yield a greater MDG than upgrading a more sensitive block from 3 to 4 bits. Consequently, effective blockwise bit allocation should account for the varying MDGs of individual bit upgrades (see Page 2, Rows 92-95; Page 3, Rows 125-129). It is particularly important in the sub-3-bit regime, where severe quantization errors and limited bit budgets make the benefit of each additional bit critical.

(2) Methodological distinction from prior work. For instance, HAWQ-V3 uses Hessian-based perturbation estimates to optimize layerwise precision, while SliM-LLM employs salience-guided group ranking and paired-bit configurations (Page 4, Rows 186-193). In contrast, CostBit estimates the quantization cost of each block-bit configuration, whose differences characterize the varying MDGs of individual bit upgrades (see Page 3, Rows 141-143). Further, it utilizes the resulting MDGs to guide bit allocation within individual LLM layers. Specifically, Eq. (7) guides blockwise bit allocation by balancing the MDG of each bit upgrade against its additional bit cost (see Page 6, Rows 300-302). Hence, the contribution of CostBit explicitly characterizes the varying MDGs of individual bit upgrades to guide budget-constrained blockwise allocation, rather than simply combining existing Hessian-aware estimation, constrained optimization, and groupwise mixed precision.

Importantly, the newly added Appendix D (Page 23) formally defines MDG as the cost reduction achieved by a one-bit upgrade and derives both the bit-width upgrade condition and the cross-block priority relation from Eq. (7), explaining why a less sensitive block may receive higher upgrade priority under equal additional bit costs.

Reviewer_KCRdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the author propose CostBit, a MP weight only method for PTQ that allocate 2-4 bit precision for each weight blocks that minimize quantization error, preserve sensitive columns in INT8, and applies GPTAQ for final weight reconstruction.

Critical strengths

1.The paper demonstrate consistent improvement in perplexity and zero-shot accuracy over OWQ across various architectures and scales.

2.The paper provide the efficienct on custom MPQ kernels and show that 70B model using the method can run on a single 24 GB GPU.

Critical weaknesses
  1. Limited novelty: This seems like a modified version of mixed-precision quantization using optimization objective. A lot of existing baselines also use similar objectives. For CNN MPQ, we even have stronger objective, that take into account cross-layer dependency. The main contribution seems to be groupwise MPQ.

  2. Insufficient experiments: The paper lacks comparisons against stronger mixed-precision allocation strategies, they mostly compare with M is OWQ, even among MPQ method that the paper list, the paper does not compare with most of them. The result for Slim LLM is only provided in Figure 2a for a single model.

  3. Efficiency: Fine-grained MPQ should increases implementation complexity compared to fixed-bit quantization, but the provided inference speed is compared only against FP16 rather than fixed-bit quantization kernels, nor other MPQ method.

Questions for authors

Please answer concerns in Weakness part

9 comments
AuthorsOpenReview ↗

Clarification of CostBit's novelty and distinctions from existing MPQ methods

We deeply appreciate your perceptive and enlightening comments that are highly helpful for improving this paper. We have revised the manuscript according to your comments and uploaded the revised version for your review. The corresponding changes are highlighted in blue. Please refer to the detailed responses below.

  1. Limited novelty: This seems like a modified version of mixed-precision quantization using optimization objective. A lot of existing baselines also use similar objectives. For CNN MPQ, we even have stronger objective, that take into account cross-layer dependency. The main contribution seems to be groupwise MPQ.

Response: We clarify CostBit’s novelty from the following aspects:

(1) Highlighting the central motivation. We newly introduce the concept of marginal distortion gain (MDG) in the introduction to further highlight the motivation of this paper. Specifically, MDG refers to the quantization cost reduction achieved by a one-bit upgrade of a weight block (see Page 2, Rows 78-81). It does not merely denote the absolute magnitude of the cost, but the difference between costs at adjacent bit-widths. For instance, considering the cost shown in Eq. (6), the MDG of upgrading block kk from bb to (b+1)(b+1) bits is given by (costk,b−costk,b+1)(cost_{k,b}-cost_{k,b+1}) (see Page 6, Rows 279-282). As illustrated in Fig. 2(b), upgrading a less sensitive block from 2 to 3 bits can yield a greater distortion reduction than upgrading a more sensitive block from 3 to 4 bits (see Page 2, Rows 92-95; Page 3, Rows 125-129). Hence, knowing which blocks are more sensitive does not necessarily reveal which bit upgrades are more beneficial under a limited budget.

(2) Core methodological contribution. We first clarify that CostBit mainly focuses on groupwise MPQ. However, existing groupwise MPQ methods fail to determine which bit upgrades are most beneficial for a certain group. This limitation becomes particularly critical in the sub-3-bit regime, where limited bit budgets require every additional bit to be allocated judiciously. CostBit addresses this limitation through candidate-specific cost modeling, where differences between adjacent candidate costs estimate the MDGs of individual bit upgrades (see Page 3, Rows 141-143). Moreover, Eq. (7) balances these estimated MDGs against their additional bit costs, favoring an upgrade when its cost reduction exceeds the corresponding bit penalty (see Page 6, Rows 300-302). Consequently, CostBit advances beyond existing groupwise MPQ by accounting for the varying MDGs of individual bit upgrades in budget-constrained allocation.

In the following content, we further clarify in detail the differences from existing methods:

(1) As you mentioned, CostBit is indeed an MPQ approach that adopts a constrained optimization objective. However, constrained optimization itself is not the core contribution of this work. Instead, its central contribution lies in modeling the varying MDGs of individual bit upgrades. Specifically, CostBit evaluates quantization costs for every block-bit configuration, allowing the benefits of different bit upgrades to be distinguished.

(2) CostBit also exhibits clear differences from existing methods. Unlike HAWQ-V3’s layerwise allocation for CNN, CostBit focuses on blockwise bit allocation within an LLM layer. More importantly, while HAWQ-V3 employs Hessian-based layerwise perturbation estimates, CostBit models candidate-specific quantization costs to characterize the varying MDGs of individual bit upgrades. Unlike OWQ’s sensitive-column protection and SliM-LLM’s salience-guided paired-bit allocation, the allocation decision of CostBit directly evaluates these candidate-specific costs to guide flexible and effective blockwise allocation without imposing a fixed importance ranking or paired-bit configuration (see Page 4, Rows 186-193). Hence, the key distinction is how quantization cost is modeled and used to guide bit-allocation decisions.

(3) Regarding cross-layer dependencies, we acknowledge that existing CNN MPQ methods like HAWQ-V3 consider cross-layer dependencies. Rather than proposing a stronger optimization objective with cross-layer dependencies, CostBit focuses on blockwise bit allocation within individual LLM layers, where the key challenge is to identify the varying distortion benefits of different bit upgrades under limited bit budgets. Further, inspired by your insightful suggestion, we plan to extend CostBit with cross-layer dependencies.

Importantly, the newly added Appendix D (Page 23) provides a formal definition and derivation of MDG, establishing its role in the Lagrangian allocation objective in Eq. (7) and explaining how it guides individual bit-upgrade decisions.

AuthorsOpenReview ↗

Expanded comparisons with strong MPQ baselines

  1. Insufficient experiments: The paper lacks comparisons against stronger mixed-precision allocation strategies, they mostly compare with M is OWQ, even among MPQ method that the paper list, the paper does not compare with most of them. The result for Slim LLM is only provided in Figure 2a for a single model.

Response: We have added three strong MPQ baselines for LLM, i.e., SliM-LLM, SliM-LLM+, and AMQ, covering multiple LLM architectures, model scales, and supported bit-widths. The expanded evaluation includes perplexity and six-task zero-shot accuracy. We have newly added these results to Page 8, Table 1 and Page 9, Table 2.

Table a: Perplexity Comparison with Additional MPQ Baselines. Lower is better.

MethodBitsLLaMA-2-7BLLaMA-2-13BLLaMA-2-70BLLaMA-3-8BLLaMA-3-70BOPT-125MOPT-350MOPT-1.3BOPT-2.7BOPT-6.7BOPT-13BOPT-30BOPT-66B
SliM-LLM2.016.019.416.2839.669.46338.293346.7230.7124.0814.4113.6811.3410.94
SliM-LLM+2.010.877.596.441100.04-136.9785.7424.5717.9814.2212.1611.2714.98
AMQ2.0113.228.575.58270.1510.23--------
CostBit2.017.636.534.8512.047.5152.5934.7120.5315.9712.8711.5810.5210.26
AMQ2.259.246.884.9517.997.70118.96-------
CostBit2.256.926.024.5010.106.8242.2930.4118.1614.9112.2711.1110.249.93
AMQ2.507.896.214.5911.766.7758.19-25.9919.5214.17-11.4911.20
CostBit2.56.435.644.178.776.0634.3427.7816.8814.0011.7510.759.979.68
AMQ2.756.835.684.299.545.8442.0953.4018.8316.0212.7211.6910.4410.48
CostBit2.756.085.363.887.845.1832.8825.5816.0013.4011.3710.479.789.53
SliM-LLM3.06.245.263.677.164.0844.0329.4115.9113.2611.2710.269.709.48
SliM-LLM+3.05.945.173.358.45-34.1026.5315.5812.8411.1810.449.679.51
AMQ3.06.205.364.058.055.4135.7732.3116.9113.9511.8211.019.999.79
CostBit3.05.805.143.627.164.2730.5623.8015.3612.9411.1210.289.689.42

Table b: Six-task average zero-Shot Accuracy (%) Comparison with Additional MPQ Baselines. Higher is better.

MethodBitsLLaMA-2-7BLLaMA-2-13BLLaMA-2-70BLLaMA-3-8BLLaMA-3-70B
SliM-LLM2.045.4652.1367.6731.8636.41
SliM-LLM+2.045.6247.60-29.42-
AMQ2.0152.3759.7571.3732.7162.93
CostBit2.0156.0260.9572.5747.2368.92
AMQ2.2559.0664.4973.4854.3072.74
CostBit2.2561.6365.0574.6054.5772.36
AMQ2.5062.2665.8774.0562.3274.24
CostBit2.563.7967.2975.7063.5474.76
AMQ2.7564.9669.4275.0764.6875.58
CostBit2.7566.2469.6976.3465.8777.16
SliM-LLM3.066.5870.3476.0566.8976.06
SliM-LLM+3.066.3470.06-65.11-
AMQ3.066.8770.3575.5768.7976.79
CostBit3.068.6571.1976.8369.9277.59

Note that one point requires clarification here. The symbol ’-’ denotes unreported result. Specifically, some SliM-LLM and SliM-LLM+ related results are not yet reported on LLaMA-2-70B and LLaMA-3-70B now due to their computationally expensive groupwise salience evaluation and salience-weighted quantizer calibration. We are continuing these experiments and will report the remaining results once available. For AMQ, some target bit-widths are unreachable with sensitive layers protected. From these available results, we still see that CostBit achieves lower perplexity and higher zero-shot accuracy than baselines under most settings. These improvements are especially evident in the sub-3-bit regime, demonstrating the effectiveness of CostBit in preserving model accuracy under limited bit budgets.

AuthorsOpenReview ↗

Deployment comparison with baselines

  1. Efficiency: Fine-grained MPQ should increase implementation complexity compared to fixed-bit quantization, but the provided inference speed is compared only against FP16 rather than fixed-bit quantization kernels, nor other MPQ methods.

Response: We have implemented a dedicated CUDA kernel and expanded the inference benchmarks to include fixed-bit (GPTQ) and MPQ baselines (SliM-LLM, SliM-LLM+, AMQ, OWQ). Specifically, the released CUDA kernels support only 3 and 4 bit decoding for OWQ and 2 and 3 bit decoding for SliM-LLM, SliM-LLM+. For AMQ, we follow the released implementation and use AutoGPTQ kernels for 2 and 3 bit decoding and a TensorRT-LLM-based kernel for 4 bit decoding.

Table c: Deployment comparison with baselines.

MethodActual bitsPPL ↓WM (MB)RM (MB)token/sCompression
FP1616.05.4712916.5213082.00131.941.00×
GPTQ4.05.613780.084190.0097.873.42×
GPTQ3.06.273002.053420.00117.194.30×
GPTQ2.024.162224.022458.00115.285.81×
SliM-LLM3.06.243005.553384.0032.334.30×
SliM-LLM2.016.012504.092768.0041.515.16×
SliM-LLM+3.05.943005.553206.6933.014.30×
SliM-LLM+2.010.872482.902705.1044.125.20×
AMQ4.05.603845.524058.29221.403.36×
AMQ2.99926.203252.673473.14170.003.97×
AMQ2.038213.222520.332679.72150.045.12×
OWQ4.00995.563668.584058.00127.053.52×
OWQ3.01075.862896.583254.00157.834.46×
CostBit3.99055.543732.353964.00170.473.46×
CostBit3.49255.653346.353624.00181.503.86×
CostBit2.99515.802960.353176.00181.744.36×
CostBit2.49616.432574.352808.00182.765.02×
CostBit2.00937.632190.342406.00176.625.90×

The above related results and corresponding descriptions are newly added to Pages 9-10, Section 5.5 and Table 4 of the revised manuscript.

From the results, we have the following findings:

(1) Although CostBit assigns different bit-widths across blocks, each block maintains uniform precision. This regular structure enables efficient weight packing and decoding through our dedicated CUDA kernel, reducing the overhead of fine-grained MPQ.

(2) Compared with baselines, CostBit achieves the highest decoding throughput among the evaluated methods except AMQ at 4 bits. These results demonstrate that CostBit’s flexible blockwise allocation can achieve competitive inference efficiency in practice.

Reviewer_wEAYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors present a mixed precision weight-only PTQ.Computing the Hessian diagonal with weighted quantization from 2 to 8 bits. The appriach is quite interesteina and the the results support the claims on on OPT, LLaMA-2 and LLaMA-3.

Critical strengths

main strenghts are method, perplexity and zero-shot on several models, a working kernel, measured weight memory, runtime memory and token/s, and a model compression. The deployment part is the strongest contribution. Ablations are clear.

Critical weaknesses

The contribution is not novel and is very common to a large typology of paper: an existing model, an existing dataset, and a small modification of one calibration or inference knob, followed by nice measurment set. This kind of paper in my opinion are largely automatize and they might even not require the human in the loop. The paper is saved by the engineering: a working kernel and a measured deployment of a 70B model on a single consumer GPU, which most PTQ papers do not provide. I weight the second part more than the first.

Questions for authors

I don't see any really question, the problem as I mentioned is the novelty and scientific contribution of this kind of papers.

1 comment
AuthorsOpenReview ↗

Clarifying CostBit’s methodological novelty and contribution

We sincerely thank you for recognizing the practical value of our CUDA implementation and single-GPU deployment of LLaMA-2-70B. Moreover, we appreciate your concern regarding scientific novelty and have revised the manuscript to further clarify the underlying research problem and methodological contribution. We have uploaded the revised version and would appreciate your review. The corresponding changes are highlighted in blue. We would also like to provide the following clarifications regarding your concern about the innovation.

(1) Refining of motivation. Under a limited bit budget, the key challenge is determining which individual bit upgrades provide the greatest marginal distortion gains (MDGs). Specifically, MDG means the quantization cost reduction achieved by a one-bit upgrade of a weight block (see Page 2, Rows 78-81). For instance, considering the cost shown in Eq. (6), the MDG of upgrading block kk from bb to (b+1)(b+1) bits is given by (costk,b−costk,b+1)(cost_{k,b}-cost_{k,b+1}) (see Page 6, Rows 279-282). Further, as illustrated in Figure 2(b), upgrading the less sensitive Block B from 2 to 3 bits yields a greater error reduction than upgrading the more sensitive Block A from 3 to 4 bits (see Page 2, Rows 92-95; Page 3, Rows 125-129). This raises a fundamental question under a limited bit budget: If we have one extra bit to allocate, which block should receive it?

(2) Clarification of methodological novelty. We have further analyzed the differences between CostBit and existing methods (see Page 4, Rows 174-177, 187-193). In general, the core contribution is that CostBit estimates candidate-specific Hessian-weighted quantization costs for each block-bit configuration, whose differences characterize the varying MDGs of individual upgrades (see Page 3, Rows 141-143). These costs guide budget-constrained allocation through Eq. (7) (see Page 6, Rows 300-302). The newly added derivation (see Page 23, Appendix D) further establishes how estimated MDGs are balanced against additional bit costs. Accordingly, CostBit introduces a more informed basis for bit allocation. Importantly, the core of CostBit lies in leveraging these cost differences to characterize the varying MDGs for bit allocation, rather than in the Hessian information itself.

(3) Strengthened experimental validation. As shown in Page 8, Table 1 and Page 9, Table 2, we have newly added more efficient mixed-precision quantization methods as baselines. CostBit remains the best-performing choice under most conditions. The advantage is particularly pronounced in the sub-3-bit regime, where quantization error is more severe and each bit upgrade must be carefully prioritized. In addition, as shown in Pages 19-21, Tables 12-13 and Figures 5-7, we have added experiments to systematically demonstrate that the estimated candidate-specific costs are consistent with measured layer-output distortion. These results provide evidence that CostBit offers meaningful guidance for allocation decisions.

We acknowledge that CostBit can be further refined and extended. We sincerely appreciate your constructive feedback, which has helped us clarify our methodological contributions and improve the manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Key Revisions

Dear Area Chairs and Reviewers,

We sincerely thank the Area Chairs and Reviewers for their time, careful evaluations, and constructive feedback throughout the review process. Your insightful comments have been invaluable in helping us clarify the methodological contributions and strengthen the experimental validation of our work. We have carefully addressed all concerns, revised the manuscript accordingly, and uploaded the updated version with changes highlighted in blue. We summarize the key improvements below.

1. Clarification of methodological novelty. We have clarified the central motivation of CostBit around marginal distortion gain (MDG), which captures the estimated quantization cost reduction from an individual bit upgrade. Rather than reflecting the absolute cost, MDG captures the cost difference between adjacent bit-widths. Hence, the key contribution is that CostBit utilizes candidate-specific quantization cost modeling that characterizes the varying MDGs of block-level bit upgrades. It enables more informed allocation under limited bit budgets, where each bit upgrade must be carefully prioritized. We have also added a formal derivation in Appendix D (Page 23) to establish the relationship between MDG and the proposed allocation rule.

2. Fundamental distinctions from existing methods. We have further clarified that CostBit differs from existing methods fundamentally in how bit allocation decisions are made. For instance, HAWQ-V3 optimizes precision across layers using Hessian-based perturbation estimates, OWQ preserves sensitive columns at higher precision, and SliM-LLM employs salience-guided paired-bit configurations. In contrast, CostBit explicitly evaluates the quantization cost reduction associated with each block-level bit upgrade. This distinction is important since these methods do not quantify the benefit of upgrading an individual block from one candidate bit-width to another. Consequently, CostBit provides an upgrade-specific decision mechanism that more directly accounts for the varying benefits of allocating additional bits. We have clarified these fundamental differences in the Introduction, Related Work, and Method sections.

3. Strengthened experimental validation. We have substantially strengthened the experimental evaluation in the following four aspects. (1) Expanded comparisons with strong MPQ baselines: we newly add strong baselines like SliM-LLM, SliM-LLM+, and AMQ. The additional perplexity and zero-shot results further demonstrate CostBit’s advantages, particularly in the sub-3-bit regime. (2) Allocation ablations on larger models: We extended controlled allocation ablations to LLaMA-2-70B and LLaMA-3-70B, confirming the effectiveness of CostBit's allocation strategy at larger scales. (3) Actual distortion reduction and cost-distortion consistency: CostBit consistently achieves lower measured layer-output NMSE than salience-based and random allocation strategies across all 12 representative projections. Furthermore, the estimated costs exhibit strong agreement with measured layer-output distortion. (4) Deployment comparisons with efficient quantization methods: we expanded throughput evaluations to include fixed-bit and MPQ baselines. CostBit achieves the highest decoding throughput among the evaluated baselines except AMQ at 4 bits, which demonstrates the practical efficiency of our mixed-precision CUDA implementation.

4. Clarification of layerwise versus model-level optimization. We have further strengthened the theoretical justification for CostBit’s allocation. Existing studies show that it is reasonable for CostBit to adopt layer-output reconstruction distortion as a tractable surrogate for mitigating model-level quantization degradation. Additionally, we further investigate whether redistributing bits across layers may improve model-level performance. The results show that although cross-layer allocation can further improve model-level performance under certain sensitivity settings, the improvements are slight and depend on the choice of layer sensitivity. Moreover, cross-layer allocation introduces substantially higher overhead, making it less flexible than CostBit for deployment on large-scale LLMs. These findings support CostBit as a practical balance between quantization effectiveness and optimization efficiency, without claiming global model-level optimality.

We sincerely appreciate your thoughtful feedback, which has helped us substantially strengthen the manuscript. We hope these clarifications and additional experimental results adequately address the concerns and provide a clearer understanding of CostBit’s methodological contribution and practical value. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.