- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SABRE, a score-guided anisotropic reconstruction approach for low-bit quantization of large language models. Conventional block-wise reconstruction objectives penalize all block-output perturbation directions equally, despite their different effects on the final predictive distribution. SABRE uses score vectors derived from the full-precision model to approximate predictive Fisher curvature and places additional penalties on errors aligned with these directions, while retaining standard reconstruction to control the overall perturbation magnitude. Gradient-norm normalization stabilizes directional weighting across blocks and tokens. The score vectors are computed once and cached, enabling block-local optimization without repeated end-to-end evaluation. SABRE modifies only the reconstruction objective and adds no inference-time computation. Experiments integrating SABRE into existing quantization methods demonstrate consistent perplexity improvements, substantial recovery under aggressive 2-bit quantization, and improvements across many downstream evaluations.
Critical strengths
The paper presents a clear motivation for accounting for the unequal effects of reconstruction errors on the final predictive distribution. The proposed objective combines score-guided directional penalties with standard reconstruction to control the overall error magnitude.
SABRE can be integrated into Block-AP, OmniQuant, and FlexRound by modifying only the reconstruction objective, without changing their optimization variables or adding inference-time computation.
The experiments cover multiple model families, quantization methods, and low-bit settings, showing consistent perplexity improvements and substantial performance recovery in several challenging 2-bit configurations.
The component ablations provide evidence for the main design choices. Additional experiments examine robustness across optimization seeds and calibration-data compositions, as well as performance on reasoning-oriented models.
Critical weaknesses
1 The novelty relative to existing methods that use first-order gradients or second-order curvature needs clearer justification. Section 2.2 discusses several related approaches, but additional analysis and direct experimental comparisons are needed to establish SABRE’s distinct contribution and effectiveness relative to these methods. In particular, the authors should clarify how token-specific score directions and gradient-norm normalization differ from existing gradient- or curvature-based objectives, and whether these differences lead to measurable benefits. Convincing results from these comparisons would strengthen the novelty claim and could justify a higher score.
2 The authors should explain the differences between the results in Fig. 3(a) and Tables 1–2. For Qwen3-4B-Instruct with Block-AP, Fig. 3(a) reports W3 PPL values of 13.15 for MSE and 12.04 for all-block SABRE, while Table 2 reports 15.61 and 14.96, respectively. Under W2g128, Fig. 3(a) reports 29.66 and 22.89, while Table 1 reports 40.82 and 29.30. The current manuscript does not clearly explain these differences. Please clarify whether they arise from different experimental settings.
3 The efficiency evaluation focuses primarily on runtime and provides limited information about memory and storage requirements. SABRE caches token-specific score vectors for every Transformer block, which introduces additional resource requirements. Reporting peak GPU memory and cache size would provide a more complete assessment of its practical cost and scalability.
Questions for authors
No further questions for the authors.
5 comments
Response to Reviewer Jcd7 (Part 2/3)
(continued from the previous comment)
Table R8. Controlled comparison of reconstruction objectives within the same Block-AP framework under W2g128. The coordinate-diagonal and token-shared Fisher variants are motivated by diagonal weighting and curvature aggregation, respectively, in the prior works listed in the second column; they are controlled variants in block-output space rather than full method re-implementations. Unlike Table 4, we tune each variant over and on Qwen3-4B-Instruct, and over nine settings with on Llama-3.1-8B-Instruct. For each variant, we report the setting with the lowest perplexity. All objectives follow the setup of Table 1, using the same initialization, calibration data, optimization seed, and number of iterations. Avg. denotes the average of CSR6, MMLU, and IFEval.
Qwen3-4B-Instruct
| Objective | Related prior structure | PPL↓ | CSR6 | MMLU | IFEval | Avg.↑ |
|---|---|---|---|---|---|---|
| MSE | – | 40.82 | 57.4 | 48.9 | 40.3 | 48.87 |
| Coordinate-diagonal (tuned) | SqueezeLLM, GuidedQuant | 37.42 | 58.5 | 49.9 | 42.5 | 50.30 |
| Token-shared Fisher (tuned) | GPTQ, GuidedQuant | 42.74 | 57.4 | 49.9 | 42.0 | 49.77 |
| SABRE (token-specific direction) | ours | 29.30 | 59.6 | 50.9 | 43.3 | 51.27 |
Llama-3.1-8B-Instruct
| Objective | Related prior structure | PPL↓ | CSR6 | MMLU | IFEval | Avg.↑ |
|---|---|---|---|---|---|---|
| MSE | – | 352.87 | 48.3 | 24.7 | 10.2 | 27.73 |
| Coordinate-diagonal (tuned) | SqueezeLLM, GuidedQuant | 269.27 | 47.8 | 24.7 | 8.9 | 27.13 |
| Token-shared Fisher (tuned) | GPTQ, GuidedQuant | 114.96 | 48.2 | 42.4 | 11.3 | 33.97 |
| SABRE (token-specific direction) | ours | 22.36 | 61.3 | 42.5 | 26.1 | 43.30 |
2. Inconsistent experimental results (W2)
-
We appreciate the reviewer's careful reading and apologize for the confusion. The discrepancy arises from an incorrect description in the caption of Figure 3 of the original submission and differences in its experimental settings. Specifically, the original caption states that Figure 3(a) uses Block-AP, whereas the reported results were obtained using FlexRound with different optimization seeds from those used in Tables 1 and 2.
-
To eliminate this inconsistency, we have replaced the original Figure 3 with a controlled comparison using the Block-AP protocol originally stated in its caption and aligned with Table 1. Table R9 reports the corresponding results across all models evaluated under W2g128. This broader comparison shows why applying predictive guidance throughout the network is important under aggressive quantization. On Qwen3-8B, Llama-3.1-8B-Instruct, and Qwen2.5-14B-Instruct, MSE reconstruction collapses under W2g128, and optimizing only the last block with either LFQ or SABRE does not recover the model. In contrast, applying SABRE to all blocks reduces perplexity to , , and , respectively, and improves IFEval to %, %, and %. Under W3, all three predictive variants improve over MSE reconstruction, and all-block SABRE gives the highest average accuracy. These results clarify the distinction between last-block predictive objectives and SABRE's all-block reconstruction strategy.
-
In the revised manuscript, we have replaced the original Figure 3 with the Block-AP W2g128 comparison consistent with the originally stated protocol (see Table 3), revised the caption and surrounding discussion accordingly, and included the additional W3 results from Table R9 in Appendix H.5.
Response to Reviewer Jcd7 (Part 3/3)
Table R9. Comparison of last-block and all-block reconstruction using Block-AP under W2g128 across four models and under W3 on Qwen3-4B-Instruct. MSE and all-block SABRE results are taken from Table 1 for W2g128 and Table 2 for W3. LFQ and last-block SABRE start from the corresponding MSE-reconstructed prefix and optimize only the final block for iterations. Last-block SABRE uses the configuration-specific in Table 5. Avg. denotes the average of CSR6, MMLU, and IFEval. Bold marks the best value of each column for each model.
| Model | Bits | Objective | PPL↓ | CSR6 | MMLU | IFEval | Avg.↑ |
|---|---|---|---|---|---|---|---|
| Qwen3-4B-Instruct | W2g128 | MSE (all blocks) | 40.82 | 57.4 | 48.9 | 40.3 | 48.87 |
| LFQ (last block) | 29.36 | 58.9 | 49.4 | 39.9 | 49.40 | ||
| SABRE (last block) | 29.28 | 59.7 | 49.5 | 41.4 | 50.20 | ||
| SABRE (all blocks) | 29.30 | 59.6 | 50.9 | 43.3 | 51.27 | ||
| Qwen3-8B | W2g128 | MSE (all blocks) | 51.4 | 46.0 | 21.1 | 39.50 | |
| LFQ (last block) | 51.7 | 45.7 | 24.0 | 40.47 | |||
| SABRE (last block) | 52.0 | 46.2 | 23.5 | 40.57 | |||
| SABRE (all blocks) | 26.05 | 64.1 | 50.3 | 28.5 | 47.63 | ||
| Llama-3.1-8B-Instruct | W2g128 | MSE (all blocks) | 352.87 | 48.3 | 24.7 | 10.2 | 27.73 |
| LFQ (last block) | 294.78 | 48.6 | 24.7 | 10.0 | 27.77 | ||
| SABRE (last block) | 284.97 | 48.8 | 24.7 | 9.8 | 27.77 | ||
| SABRE (all blocks) | 22.36 | 61.3 | 42.5 | 26.1 | 43.30 | ||
| Qwen2.5-14B-Instruct | W2g128 | MSE (all blocks) | 63.5 | 24.3 | 11.5 | 33.10 | |
| LFQ (last block) | 64.6 | 25.4 | 11.5 | 33.83 | |||
| SABRE (last block) | 64.4 | 25.0 | 11.5 | 33.63 | |||
| SABRE (all blocks) | 13.07 | 69.5 | 57.3 | 47.7 | 58.17 | ||
| Qwen3-4B-Instruct | W3 | MSE (all blocks) | 15.61 | 68.7 | 65.2 | 65.8 | 66.57 |
| LFQ (last block) | 14.23 | 69.4 | 65.3 | 67.3 | 67.33 | ||
| SABRE (last block) | 14.28 | 69.3 | 65.2 | 68.0 | 67.50 | ||
| SABRE (all blocks) | 14.96 | 69.1 | 66.2 | 69.3 | 68.20 |
3. Additional memory for score caching (W3)
-
We thank the reviewer for the suggestion. We profiled the peak GPU memory, host-memory overhead, and cache size of SABRE.
-
Cache size. SABRE stores the token-wise score vectors of all Transformer blocks in bfloat16 in a filesystem cache. For calibration sequences of length and hidden dimension , the cache of one block takes bytes. With and , the total cache size is:
Model #Blocks Per block Total Qwen3-4B-Instruct 2560 36 5 GiB 180 GiB Llama-3.1-8B-Instruct 4096 32 8 GiB 256 GiB Qwen3-8B 4096 36 8 GiB 288 GiB Qwen2.5-14B-Instruct 5120 48 10 GiB 480 GiB Llama-3.1-70B 8192 80 16 GiB 1.25 TiB The cache resides on disk. During block-wise quantization, only the score vectors of the block currently being optimized are loaded into a pinned host buffer that is reused across blocks, so the caches of all blocks are never resident in host memory at the same time.
-
Peak GPU memory. Rather than keeping the full score tensor of the current block on the GPU, we stream one minibatch of score vectors at a time using preallocated buffers. This reduces SABRE's GPU memory overhead from to , where is the reconstruction minibatch size. The table below reports the measured peak GPU memory, additional host memory, and per-block optimization time over iterations for Block-AP W3g128 with on a single NVIDIA H100 80GB GPU.
Model Method Peak GPU memory Add. host memory Time / block Qwen3-8B MSE 28.80 GiB – 327.2 s SABRE 29.17 GiB 8 GiB 338.6 s Llama-3.1-70B MSE 69.06 GiB – 980.4 s SABRE 69.81 GiB 16 GiB 1,006.5 s With minibatch-wise streaming, SABRE adds only GiB and GiB of peak GPU memory over MSE reconstruction for Qwen3-8B and Llama-3.1-70B, respectively. On the host side, the additional memory requirement is limited to one block of score vectors, corresponding to GiB for Qwen3-8B and GiB for Llama-3.1-70B. The host-to-device transfers are overlapped with the block forward pass using a separate CUDA stream, so the per-block optimization time increases by only % and %, respectively.
-
In summary, SABRE's disk-cache requirement grows linearly with , , , and the number of blocks, while minibatch-wise streaming makes its GPU memory overhead independent of for a fixed minibatch size . In our measurements, the additional peak GPU memory during block-wise quantization remains below GiB even for Llama-3.1-70B. We have added the cache sizes, peak GPU and host memory, and the score-streaming implementation details to the revised manuscript (see Section 4.3.3 and Appendix I).
Response to Reviewer Jcd7 (Part 1/3)
1. Unclear contribution over prior work (W1)
We appreciate the reviewer's comments. We conducted controlled ablations of SABRE's key design choices (e.g., token-specific score directions) within the same Block-AP framework, motivated by aggregation and diagonal weighting in prior gradient- and curvature-based approaches (see Table 4).
Replacing SABRE's token-specific metric with a shared metric obtained by averaging normalized score outer products across calibration tokens, motivated by curvature aggregation in prior approaches (e.g., GPTQ [1] and GuidedQuant [2]), increases W2g128 perplexity from 29.30 to 49.27 and decreases IFEval from 43.3% to 39.9% (see the ‘token-shared normalized Fisher’ row in Table 4). Applying a coordinate-diagonal approximation in block-output space, motivated by diagonal weight-space Fisher approximations (e.g., SqueezeLLM [3]) and feature-wise gradient-weighting (e.g., GuidedQuant [2]), increases W2g128 perplexity from 29.30 to 40.74 (see the ‘coordinate-diagonal approximation’ row in Table 4). Even after further tuning λ and ρ for the token-shared Fisher and coordinate-diagonal variants, both remain consistently behind SABRE in perplexity and average downstream accuracy on both evaluated models (see Table R8). These results show that both token-specific score directions and cross-feature directional structure contribute substantially to SABRE's performance gains.
We also validate the efficacy of gradient-norm normalization. SABRE bounds each token's additional directional weight, limiting the influence of large-gradient tokens while retaining relative gradient-magnitude information. Without normalization (λt=λ), W2g128 perplexity increases from 29.30 to 52.31 (see the ‘unnormalized, original strength’ row in Table 4). Using only block-wise scaling, which adjusts the block-level scale but leaves token-level directional weight unbounded, increases perplexity to 44.10 and lowers MMLU from 50.9% to 28.2% (see the ‘block-wise scaling only’ row in Table 4). These controlled ablations demonstrate that gradient-norm normalization provides substantial measurable benefits.
In addition to these controlled objective-level ablations, Table 9 in Appendix G provides a method-level comparison with GuidedQuant, which incorporates layer-output gradient information into GPTQ-style curvature-based quantization. The SABRE integrations generally outperform GuidedQuant across W2g128, W3, and W3g128. The improvement is particularly pronounced under W2g128, where FlexRound+SABRE reduces Qwen3-8B perplexity from 65.78 to 23.44 and improves IFEval from 25.9% to 45.5%. Under W3 and W3g128, FlexRound+SABRE also achieves the best performance across all reported metrics. These results demonstrate the effectiveness of SABRE integrated into block-wise quantization relative to GuidedQuant under the shared evaluation protocol.
In the revised manuscript, we have clarified the ablation and GuidedQuant comparisons in Section 4.3.1 and added the tuned objective-level results to Appendix E.
[1] E. Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", ICLR 2023.
[2] J. Kim et al., "GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance", ICML 2025.
[3] S. Kim et al., "SqueezeLLM: Dense-and-Sparse Quantization", ICML 2024.