- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CostBit is a training-free block-wise mixed-precision allocation method for weight-only LLM quantization. It estimates Hessian-diagonal-weighted quantization costs for candidate bit widths, uses a Lagrangian search with local adjustment to approach non-integer per-layer budgets, protects sensitive columns in INT8, and applies GPTAQ reconstruction. The paper reports perplexity and zero-shot results across several model families, a mixed-precision CUDA implementation, and fully GPU-resident LLaMA-2-70B generation on an RTX 4090.
Critical strengths
- This paper is practical motivation. Allocating a limited bit budget effectively matters for memory-constrained LLM deployment. Supporting non-integer budgets also offers useful flexibility when fixed 2-bit or 3-bit configurations cannot achieve the desired accuracy–memory balance.
- Intuitive allocation criterion. Comparing each block’s quantization error across candidate bit-widths is a sensible way to estimate where an additional bit would help most. This is more informative than relying on a single sensitivity ranking.
- Broad experimental coverage and useful ablations, the authors evaluates multiple model families and sizes using both perplexity and zero-shot tasks. They also examines allocation rules, cost definitions, reconstruction backends, and weak-column formats, providing useful evidence about the individual design choices.
- Good to actual deployment. The custom mixed-bit CUDA kernel and single-GPU deployment results show that the method can be implemented beyond simulated quantization. The memory and throughput measurements add practical value to the accuracy results.
Critical weaknesses
- The methodological contribution needs clearer positioning. Hessian-aware allocation, constrained optimization, and group-wise mixed precision have been explored in prior work, including HAWQ-V3 and SliM-LLM. It is not yet clear how much CostBit adds beyond adapting these ideas to candidate-specific costs and non-integer budgets.
- My main concern is whether the block estimated costs predict actual layer-output distortion, this is a very important research question, the diagonal approximation ignores interactions between quantization errors across blocks. A lower sum of block costs therefore need not produce a better reconstructed layer. Figure 2b is illustrative, but I would prefer a systematic comparison of actual output distortion at matched bit budgets.
- The allocation gains need broader controlled validation. Table 3 helps isolate the allocation effect, but covers only two models. The simpler mean-Hessian-trace cost also performs comparably or slightly better in some settings. Comparisons on larger models, with clipping, weak-column protection, and reconstruction held fixed, would clarify the value of the proposed cost.
Questions for authors
- At the same bit budget, does CostBit reduce actual layer-output distortion more than alternative allocation strategies? How well do estimated costs predict measured errors?
- Do the allocation gains hold on larger models when the remaining PTQ pipeline is unchanged?
- What is the key methodological and innovation-level difference from previous hessian-aware, optimization-based allocation methods?
- Could the throughput evaluation include efficient 3-bit and 4-bit methods under the same hardware and workload? This would be more informative than comparing only with FP16.
9 comments
Allocation ablation on larger models
- Do the allocation gains hold on larger models when the remaining PTQ pipeline is unchanged?
Response: We have extended the controlled allocation ablation to LLaMA-2-70B and LLaMA-3-70B to compare three blockwise bit allocation rules while keeping the rest of the PTQ pipeline fixed. All methods operate under approximately the same 2.5-bit budget, ensuring that performance differences arise from the allocation strategy itself. The related results are newly added in Page 16, Appendix B.3, Table 7.
Table c: Allocation ablation at 2.5 bits on LLaMA-2-70B/LLaMA-3-70B. We report WikiText-2 and C4 perplexity.
| Component | Variant | Actual bits | WikiText-2 | C4 |
|---|---|---|---|---|
| Allocation | Candidate Cost | 2.4987/2.4991 | 4.17/6.06 | 6.80/10.70 |
| Allocation | Sensitivity | 2.4995/2.4996 | 4.39/6.75 | 7.18/11.60 |
| Allocation | Random | 2.4982/2.4982 | 4.30/6.36 | 7.03/11.21 |
(1) Consistent improvements on 70B models. For instance, CostBit achieves the lowest WikiText-2 and C4 perplexity on both LLaMA-2-70B and LLaMA-3-70B. Specifically, on LLaMA-2-70B, it reduces WikiText-2 perplexity to 4.17, compared with 4.39 and 4.30 for sensitivity-based and random allocation. On LLaMA-3-70B, the corresponding results are 6.06 versus 6.75 and 6.36. Similar improvements are observed on C4. It indicates that the allocation strategy we propose still yields performance gains on larger models.
(2) Regarding the Mean Hessian Trace cost you mentioned. We firstly apologize that the previous version does not clearly distinguish the alternative cost formulations, leading to your misunderstanding. We have revised the descriptions of these three variants in Page 10, Table 3, i.e., Hessian diagonal weighted, Mean Hessian trace weighted, and Hessian spectral norm weighted. Note that these three variants can be all incorporated into CostBit to evaluate the cost associated with different block-bit configurations. We adopt the Hessian diagonal-weighted cost only as the default setting. Moreover, although Mean Hessian trace weighted variant performs comparably or slightly better in some settings, this only reflects differences in Hessian-based cost estimation rather than a limitation of CostBit’s allocation framework. Importantly, the core contribution lies in accounting for the varying MDGs of individual bit upgrades through candidate-specific costs, rather than relying on a particular Hessian-weighted formulation.
key methodological difference from prior methods
- What is the key methodological and innovation-level difference from previous hessian-aware, optimization-based allocation methods?
Response: We agree that Hessian-aware estimation, constrained optimization, and groupwise mixed precision are established techniques. However, the contribution of CostBit is not these individual components. We have clarified CostBit’s methodological contribution as follows.
(1) Core motivation. In the revised manuscript, we introduce the concept of marginal distortion gain (MDG) to better articulate our methodological contribution. Specifically, MDG refers to the quantization cost reduction achieved by a one-bit upgrade of a weight block (see Page 2, Rows 78-81). Rather than reflecting the absolute cost, MDG captures the cost difference between adjacent bit-widths. For instance, considering the cost shown in Eq. (6), the MDG of upgrading block from to bits is given by (see Page 6, Rows 279-282). Furthermore, Fig. 2(b) illustrates that upgrading a less sensitive block from 2 to 3 bits can yield a greater MDG than upgrading a more sensitive block from 3 to 4 bits. Consequently, effective blockwise bit allocation should account for the varying MDGs of individual bit upgrades (see Page 2, Rows 92-95; Page 3, Rows 125-129). It is particularly important in the sub-3-bit regime, where severe quantization errors and limited bit budgets make the benefit of each additional bit critical.
(2) Methodological distinction from prior work. For instance, HAWQ-V3 uses Hessian-based perturbation estimates to optimize layerwise precision, while SliM-LLM employs salience-guided group ranking and paired-bit configurations (Page 4, Rows 186-193). In contrast, CostBit estimates the quantization cost of each block-bit configuration, whose differences characterize the varying MDGs of individual bit upgrades (see Page 3, Rows 141-143). Further, it utilizes the resulting MDGs to guide bit allocation within individual LLM layers. Specifically, Eq. (7) guides blockwise bit allocation by balancing the MDG of each bit upgrade against its additional bit cost (see Page 6, Rows 300-302). Hence, the contribution of CostBit explicitly characterizes the varying MDGs of individual bit upgrades to guide budget-constrained blockwise allocation, rather than simply combining existing Hessian-aware estimation, constrained optimization, and groupwise mixed precision.
Importantly, the newly added Appendix D (Page 23) formally defines MDG as the cost reduction achieved by a one-bit upgrade and derives both the bit-width upgrade condition and the cross-block priority relation from Eq. (7), explaining why a less sensitive block may receive higher upgrade priority under equal additional bit costs.
Validating CostBit’s distortion reduction and cost-error consistency
We sincerely thank you for the valuable suggestions, which help us improve the clarity and rigor of our work. We have carefully revised the manuscript, and all corresponding changes are highlighted in blue in the newly uploaded version. Our detailed responses are provided below.
Response: We have added systematic experiments to directly evaluate whether CostBit reduces actual layer-output distortion under matched bit budgets and whether its estimated costs reliably predict measured distortion. The new results demonstrate that CostBit consistently achieves lower layer-output distortion than alternative allocation strategies, while its estimated costs exhibit strong agreement with measured errors.
(1) Reduction of actual layer-output distortion. We compare CostBit with salience-based and random allocation strategies on 12 representative projections of LLaMA-2-7B, covering query, key, MLP up, and MLP down projections (see Page 19, Table 12). From the results, we find that CostBit obviously achieves lower measured layer-output NMSE in all 12 cases. The median reductions across projections were 16.75% relative to Salience and 10.03% relative to Random. These results directly confirm that candidate-cost-informed allocation reduces actual layer-output distortion.
Table a: Layer-output NMSE on LLaMA-2-7B at a 2.5-bit budget. Lower is better.
(2) Consistency between estimated costs and measured layer-output distortion. For each of the 12 layers, we uniformly sampled 30 distinct allocations at the same bit budget. We measured layer-output normalized mean squared error (NMSE) after GPTAQ reconstruction. For each layer, we computed the Spearman correlation between estimated total cost and NMSE. We also compared the 30 allocations two at a time, yielding 435 pairwise comparisons per layer. Pairwise ranking accuracy (PairAcc) is the percentage of comparisons in which the allocation with the lower estimated cost also yields lower measured NMSE. The estimated costs achieve a mean Spearman correlation of 0.835 and a pairwise ranking accuracy of 83.3% across the 12 projections. The related results are reported in Page 19, Table 13. In addition, the newly added Figures 5-7 in Pages 20-21 further show that allocations with lower estimated costs generally exhibit lower measured distortion. These results demonstrate that the estimated costs are able to predict measured layer-output distortion, despite relying on a diagonal Hessian approximation.
Table b: Rank agreement between estimated total cost and measured layer-output NMSE.