acceptodds
Under review as a conference paper at ICLR 2027

Not All Reconstruction Errors Matter: Score-Guided Block-wise Reconstruction for Low-Bit LLM Quantization

Abstract

Block-wise reconstruction has been widely used for post-training quantization (PTQ) of large language models (LLMs), but conventional objectives penalize all block-output perturbation directions equally despite their different effects on the final predictive distribution. In this paper, we propose SABRE, a score-guided anisotropic reconstruction approach that accounts for these unequal predictive effects during block-wise reconstruction. To identify prediction-sensitive directions, SABRE approximates the Fisher curvature of the full-precision (FP) predictive distribution using rank-one outer products of sampled score vectors. Based on this approximation, SABRE assigns additional penalties to errors aligned with these directions while retaining isotropic reconstruction to control the overall perturbation magnitude. To ensure stable weighting across blocks and tokens, we further introduce gradient-norm normalization that bounds token-dependent directional weights, thereby reducing sensitivity to variations in score-vector scale. The FP-derived score vectors are computed once and cached, enabling efficient block-local reconstruction without repeated end-to-end evaluation. Because SABRE modifies only the reconstruction objective, it can be readily integrated into existing block-wise quantization methods without changing their optimization variables or inference-time computation. Extensive experiments across diverse quantization methods, LLM architectures, and low-bit settings demonstrate consistent perplexity improvements, substantial recovery under aggressive 2-bit quantization, and broad downstream gains.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Models, Post-training Quantization, Block-wise Reconstruction, Predictive Sensitivity, Fisher Information

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Jcd7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SABRE, a score-guided anisotropic reconstruction approach for low-bit quantization of large language models. Conventional block-wise reconstruction objectives penalize all block-output perturbation directions equally, despite their different effects on the final predictive distribution. SABRE uses score vectors derived from the full-precision model to approximate predictive Fisher curvature and places additional penalties on errors aligned with these directions, while retaining standard reconstruction to control the overall perturbation magnitude. Gradient-norm normalization stabilizes directional weighting across blocks and tokens. The score vectors are computed once and cached, enabling block-local optimization without repeated end-to-end evaluation. SABRE modifies only the reconstruction objective and adds no inference-time computation. Experiments integrating SABRE into existing quantization methods demonstrate consistent perplexity improvements, substantial recovery under aggressive 2-bit quantization, and improvements across many downstream evaluations.

Critical strengths

The paper presents a clear motivation for accounting for the unequal effects of reconstruction errors on the final predictive distribution. The proposed objective combines score-guided directional penalties with standard reconstruction to control the overall error magnitude.

SABRE can be integrated into Block-AP, OmniQuant, and FlexRound by modifying only the reconstruction objective, without changing their optimization variables or adding inference-time computation.

The experiments cover multiple model families, quantization methods, and low-bit settings, showing consistent perplexity improvements and substantial performance recovery in several challenging 2-bit configurations.

The component ablations provide evidence for the main design choices. Additional experiments examine robustness across optimization seeds and calibration-data compositions, as well as performance on reasoning-oriented models.

Critical weaknesses

1 The novelty relative to existing methods that use first-order gradients or second-order curvature needs clearer justification. Section 2.2 discusses several related approaches, but additional analysis and direct experimental comparisons are needed to establish SABRE’s distinct contribution and effectiveness relative to these methods. In particular, the authors should clarify how token-specific score directions and gradient-norm normalization differ from existing gradient- or curvature-based objectives, and whether these differences lead to measurable benefits. Convincing results from these comparisons would strengthen the novelty claim and could justify a higher score.

2 The authors should explain the differences between the results in Fig. 3(a) and Tables 1–2. For Qwen3-4B-Instruct with Block-AP, Fig. 3(a) reports W3 PPL values of 13.15 for MSE and 12.04 for all-block SABRE, while Table 2 reports 15.61 and 14.96, respectively. Under W2g128, Fig. 3(a) reports 29.66 and 22.89, while Table 1 reports 40.82 and 29.30. The current manuscript does not clearly explain these differences. Please clarify whether they arise from different experimental settings.

3 The efficiency evaluation focuses primarily on runtime and provides limited information about memory and storage requirements. SABRE caches token-specific score vectors for every Transformer block, which introduces additional resource requirements. Reporting peak GPU memory and cache size would provide a more complete assessment of its practical cost and scalability.

Questions for authors

No further questions for the authors.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Jcd7 (Part 1/3)

1. Unclear contribution over prior work (W1)

  • We appreciate the reviewer's comments. We conducted controlled ablations of SABRE's key design choices (e.g., token-specific score directions) within the same Block-AP framework, motivated by aggregation and diagonal weighting in prior gradient- and curvature-based approaches (see Table 4).

    • Replacing SABRE's token-specific metric with a shared metric obtained by averaging normalized score outer products across calibration tokens, motivated by curvature aggregation in prior approaches (e.g., GPTQ [1] and GuidedQuant [2]), increases W2g128 perplexity from 29.3029.30 to 49.2749.27 and decreases IFEval from 43.343.3% to 39.939.9% (see the ‘token-shared normalized Fisher’ row in Table 4). Applying a coordinate-diagonal approximation in block-output space, motivated by diagonal weight-space Fisher approximations (e.g., SqueezeLLM [3]) and feature-wise gradient-weighting (e.g., GuidedQuant [2]), increases W2g128 perplexity from 29.3029.30 to 40.7440.74 (see the ‘coordinate-diagonal approximation’ row in Table 4). Even after further tuning λ\lambda and ρ\rho for the token-shared Fisher and coordinate-diagonal variants, both remain consistently behind SABRE in perplexity and average downstream accuracy on both evaluated models (see Table R8). These results show that both token-specific score directions and cross-feature directional structure contribute substantially to SABRE's performance gains.

    • We also validate the efficacy of gradient-norm normalization. SABRE bounds each token's additional directional weight, limiting the influence of large-gradient tokens while retaining relative gradient-magnitude information. Without normalization (λt=λ\lambda_t=\lambda), W2g128 perplexity increases from 29.3029.30 to 52.3152.31 (see the ‘unnormalized, original strength’ row in Table 4). Using only block-wise scaling, which adjusts the block-level scale but leaves token-level directional weight unbounded, increases perplexity to 44.1044.10 and lowers MMLU from 50.950.9% to 28.228.2% (see the ‘block-wise scaling only’ row in Table 4). These controlled ablations demonstrate that gradient-norm normalization provides substantial measurable benefits.

  • In addition to these controlled objective-level ablations, Table 9 in Appendix G provides a method-level comparison with GuidedQuant, which incorporates layer-output gradient information into GPTQ-style curvature-based quantization. The SABRE integrations generally outperform GuidedQuant across W2g128, W3, and W3g128. The improvement is particularly pronounced under W2g128, where FlexRound+SABRE reduces Qwen3-8B perplexity from 65.7865.78 to 23.4423.44 and improves IFEval from 25.925.9% to 45.545.5%. Under W3 and W3g128, FlexRound+SABRE also achieves the best performance across all reported metrics. These results demonstrate the effectiveness of SABRE integrated into block-wise quantization relative to GuidedQuant under the shared evaluation protocol.

  • In the revised manuscript, we have clarified the ablation and GuidedQuant comparisons in Section 4.3.1 and added the tuned objective-level results to Appendix E.

[1] E. Frantar et al., "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", ICLR 2023.

[2] J. Kim et al., "GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance", ICML 2025.

[3] S. Kim et al., "SqueezeLLM: Dense-and-Sparse Quantization", ICML 2024.

AuthorsOpenReview ↗

Response to Reviewer Jcd7 (Part 2/3)

(continued from the previous comment)

Table R8. Controlled comparison of reconstruction objectives within the same Block-AP framework under W2g128. The coordinate-diagonal and token-shared Fisher variants are motivated by diagonal weighting and curvature aggregation, respectively, in the prior works listed in the second column; they are controlled variants in block-output space rather than full method re-implementations. Unlike Table 4, we tune each variant over λ∈300,1000,3000,10000\lambda\in\\{300,1000,3000,10000\\} and ρ∈0.94,0.97\rho\in\\{0.94,0.97\\} on Qwen3-4B-Instruct, and over nine (λ,ρ)(\lambda,\rho) settings with ρ∈0.93,0.94,0.965,1.0\rho\in\\{0.93,0.94,0.965,1.0\\} on Llama-3.1-8B-Instruct. For each variant, we report the setting with the lowest perplexity. All objectives follow the setup of Table 1, using the same initialization, calibration data, optimization seed, and number of iterations. Avg. denotes the average of CSR6, MMLU, and IFEval.

Qwen3-4B-Instruct

ObjectiveRelated prior structurePPL↓CSR6MMLUIFEvalAvg.↑
MSE–40.8257.448.940.348.87
Coordinate-diagonal (tuned)SqueezeLLM, GuidedQuant37.4258.549.942.550.30
Token-shared Fisher (tuned)GPTQ, GuidedQuant42.7457.449.942.049.77
SABRE (token-specific direction)ours29.3059.650.943.351.27

Llama-3.1-8B-Instruct

ObjectiveRelated prior structurePPL↓CSR6MMLUIFEvalAvg.↑
MSE–352.8748.324.710.227.73
Coordinate-diagonal (tuned)SqueezeLLM, GuidedQuant269.2747.824.78.927.13
Token-shared Fisher (tuned)GPTQ, GuidedQuant114.9648.242.411.333.97
SABRE (token-specific direction)ours22.3661.342.526.143.30

2. Inconsistent experimental results (W2)

  • We appreciate the reviewer's careful reading and apologize for the confusion. The discrepancy arises from an incorrect description in the caption of Figure 3 of the original submission and differences in its experimental settings. Specifically, the original caption states that Figure 3(a) uses Block-AP, whereas the reported results were obtained using FlexRound with different optimization seeds from those used in Tables 1 and 2.

  • To eliminate this inconsistency, we have replaced the original Figure 3 with a controlled comparison using the Block-AP protocol originally stated in its caption and aligned with Table 1. Table R9 reports the corresponding results across all models evaluated under W2g128. This broader comparison shows why applying predictive guidance throughout the network is important under aggressive quantization. On Qwen3-8B, Llama-3.1-8B-Instruct, and Qwen2.5-14B-Instruct, MSE reconstruction collapses under W2g128, and optimizing only the last block with either LFQ or SABRE does not recover the model. In contrast, applying SABRE to all blocks reduces perplexity to 26.0526.05, 22.3622.36, and 13.0713.07, respectively, and improves IFEval to 28.528.5%, 26.126.1%, and 47.747.7%. Under W3, all three predictive variants improve over MSE reconstruction, and all-block SABRE gives the highest average accuracy. These results clarify the distinction between last-block predictive objectives and SABRE's all-block reconstruction strategy.

  • In the revised manuscript, we have replaced the original Figure 3 with the Block-AP W2g128 comparison consistent with the originally stated protocol (see Table 3), revised the caption and surrounding discussion accordingly, and included the additional W3 results from Table R9 in Appendix H.5.

AuthorsOpenReview ↗

Response to Reviewer Jcd7 (Part 3/3)

Table R9. Comparison of last-block and all-block reconstruction using Block-AP under W2g128 across four models and under W3 on Qwen3-4B-Instruct. MSE and all-block SABRE results are taken from Table 1 for W2g128 and Table 2 for W3. LFQ and last-block SABRE start from the corresponding MSE-reconstructed prefix and optimize only the final block for 5,0005{,}000 iterations. Last-block SABRE uses the configuration-specific λ\lambda in Table 5. Avg. denotes the average of CSR6, MMLU, and IFEval. Bold marks the best value of each column for each model.

ModelBitsObjectivePPL↓CSR6MMLUIFEvalAvg.↑
Qwen3-4B-InstructW2g128MSE (all blocks)40.8257.448.940.348.87
LFQ (last block)29.3658.949.439.949.40
SABRE (last block)29.2859.749.541.450.20
SABRE (all blocks)29.3059.650.943.351.27
Qwen3-8BW2g128MSE (all blocks)4.5×1064.5{\times}10^{6}51.446.021.139.50
LFQ (last block)3.7×1063.7{\times}10^{6}51.745.724.040.47
SABRE (last block)3.7×1063.7{\times}10^{6}52.046.223.540.57
SABRE (all blocks)26.0564.150.328.547.63
Llama-3.1-8B-InstructW2g128MSE (all blocks)352.8748.324.710.227.73
LFQ (last block)294.7848.624.710.027.77
SABRE (last block)284.9748.824.79.827.77
SABRE (all blocks)22.3661.342.526.143.30
Qwen2.5-14B-InstructW2g128MSE (all blocks)4.5×1064.5{\times}10^{6}63.524.311.533.10
LFQ (last block)4.0×1064.0{\times}10^{6}64.625.411.533.83
SABRE (last block)4.2×1064.2{\times}10^{6}64.425.011.533.63
SABRE (all blocks)13.0769.557.347.758.17
Qwen3-4B-InstructW3MSE (all blocks)15.6168.765.265.866.57
LFQ (last block)14.2369.465.367.367.33
SABRE (last block)14.2869.365.268.067.50
SABRE (all blocks)14.9669.166.269.368.20

3. Additional memory for score caching (W3)

  • We thank the reviewer for the suggestion. We profiled the peak GPU memory, host-memory overhead, and cache size of SABRE.

  • Cache size. SABRE stores the token-wise score vectors of all Transformer blocks in bfloat16 in a filesystem cache. For NN calibration sequences of length TT and hidden dimension dd, the cache of one block takes N×T×d×2N \times T \times d \times 2 bytes. With N=512N=512 and T=2048T=2048, the total cache size is:

    Modeldd#BlocksPer blockTotal
    Qwen3-4B-Instruct2560365 GiB180 GiB
    Llama-3.1-8B-Instruct4096328 GiB256 GiB
    Qwen3-8B4096368 GiB288 GiB
    Qwen2.5-14B-Instruct51204810 GiB480 GiB
    Llama-3.1-70B81928016 GiB1.25 TiB

    The cache resides on disk. During block-wise quantization, only the score vectors of the block currently being optimized are loaded into a pinned host buffer that is reused across blocks, so the caches of all blocks are never resident in host memory at the same time.

  • Peak GPU memory. Rather than keeping the full score tensor of the current block on the GPU, we stream one minibatch of score vectors at a time using preallocated buffers. This reduces SABRE's GPU memory overhead from O(NTd)O(NTd) to O(BTd)O(BTd), where BB is the reconstruction minibatch size. The table below reports the measured peak GPU memory, additional host memory, and per-block optimization time over 5,0005{,}000 iterations for Block-AP W3g128 with B=4B=4 on a single NVIDIA H100 80GB GPU.

    ModelMethodPeak GPU memoryAdd. host memoryTime / block
    Qwen3-8BMSE28.80 GiB–327.2 s
    SABRE29.17 GiB8 GiB338.6 s
    Llama-3.1-70BMSE69.06 GiB–980.4 s
    SABRE69.81 GiB16 GiB1,006.5 s

    With minibatch-wise streaming, SABRE adds only 0.370.37 GiB and 0.750.75 GiB of peak GPU memory over MSE reconstruction for Qwen3-8B and Llama-3.1-70B, respectively. On the host side, the additional memory requirement is limited to one block of score vectors, corresponding to 88 GiB for Qwen3-8B and 1616 GiB for Llama-3.1-70B. The host-to-device transfers are overlapped with the block forward pass using a separate CUDA stream, so the per-block optimization time increases by only 3.53.5% and 2.72.7%, respectively.

  • In summary, SABRE's disk-cache requirement grows linearly with NN, TT, dd, and the number of blocks, while minibatch-wise streaming makes its GPU memory overhead independent of NN for a fixed minibatch size BB. In our measurements, the additional peak GPU memory during block-wise quantization remains below 11 GiB even for Llama-3.1-70B. We have added the cache sizes, peak GPU and host memory, and the score-streaming implementation details to the revised manuscript (see Section 4.3.3 and Appendix I).

Reviewer_Ghz4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SABRE, a score-guided block-wise reconstruction method for low-bit LLM quantization. It adds a gradient-normalized directional penalty to the standard reconstruction loss to better preserve the full-precision model’s predictive behavior. Experiments show improvements across multiple quantization methods, particularly under 2-bit quantization.

Critical strengths
  • Meaningful motivation: The paper considers the different effects of local reconstruction errors on final predictions, providing a reasonable motivation for improving reconstruction objectives.
  • Easy integration: SABRE modifies only the reconstruction objective and caches FP-derived score vectors to avoid repeated end-to-end evaluation, without changing inference-time computation.
  • Promising low-bit results: The method improves perplexity and downstream performance across multiple quantization frameworks and substantially reduces performance degradation in some 2-bit settings.
Critical weaknesses
  • Limited clarity in presentation and motivation: The key concepts and their connection to practical quantization problems lack intuitive explanations, making the need for the proposed design difficult to follow.
  • Limited validation of applicability: The experiments cover a narrow range of quantization settings, leaving the method’s effectiveness in broader settings unclear.
  • Limited evaluation of scalability and practical benefits: The experiments do not sufficiently assess larger models or show how the model-quality gains translate into practical deployment benefits.
  • Unclear contribution over prior work: The related-work discussion and experimental comparisons do not sufficiently clarify the method’s new contributions and their individual benefits.
Questions for authors

Q1. Could the authors explain more clearly how local reconstruction errors affect the final predictive distribution and why existing reconstruction objectives are insufficient? A simple example in the main text could help connect the problem, the key observation, and the proposed design.

Q2. The experiments mainly focus on weight-only quantization; can SABRE also support weight-and-activation quantization? Could the authors evaluate representative settings such as W4A4 or W4A8 and explain whether activation quantization requires changes to the score-vector computation or reconstruction objective?

Q3. Could the authors evaluate model quality, quantization time, and peak memory usage on Qwen3-32B or Llama-3.1-70B-Instruct? Please also report prefill latency and decoding throughput using vLLM under the same hardware, batch size, and input/output lengths, while specifying the quantization formats and kernel backends. In particular, can SABRE enable lower-bit quantization at comparable model quality and thereby reduce inference latency or GPU memory usage?

Q4. Could the authors clarify the main differences between SABRE and methods such as GuidedQuant and FIMA-Q, which also use gradient or Fisher information to guide quantization? Comparing different reconstruction objectives within the same quantization framework and under comparable optimization budgets would help separate the benefits of the proposed objective from those of the underlying quantization method.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Ghz4 (Part 1/4)

1. Limited clarity in presentation and motivation (Q1)

  • Thank you for pointing out that the motivation was not sufficiently clear in the main text. Appendix B provides a controlled perturbation experiment, which illustrates why the magnitude of a local block-wise reconstruction error alone does not fully characterize its effect on the final predictive distribution.

  • Specifically, reconstruction errors with the same magnitude can induce substantially different KL divergences depending on their direction. As the error becomes more aligned with the score vector (i.e., α→1\alpha \rightarrow 1 in Figure 4, where α\alpha is the cosine similarity between the error and the score direction), it induces a larger change in the predictive distribution and can alter the top-1 prediction even at a smaller error magnitude. This example demonstrates why an isotropic objective, which penalizes all error directions equally, can overlook prediction-sensitive errors and motivates SABRE's score-guided anisotropic objective.

  • We have added this explanation to the revised manuscript, making the connection between local reconstruction errors and the final predictive distribution more explicit (see Section 2.2).

2. Limited validation of applicability (Q2)

  • We appreciate the reviewer's question. SABRE's formulation remains unchanged under weight-and-activation quantization. Activation quantization changes only how the block-output perturbation Δzt(ℓ)\Delta\mathbf{z}^{(\ell)}_t is produced, by incorporating activation quantization error in addition to weight quantization error. The score vectors are still computed once from the full-precision model (see Eq. (4)), and the same SABRE objective in Eq. (12) is applied to the resulting combined perturbation. Consequently, the reconstruction objective directly accounts for errors induced by both weight and activation quantization, without requiring any changes to the score-vector computation or objective formulation.

  • We additionally evaluated SABRE under W4A4, W4A8, and W2g128A8 settings on Qwen3-4B-Instruct and Qwen3-8B. During block-wise reconstruction, the inputs to all linear layers are quantized using per-token dynamic symmetric round-to-nearest quantization with a straight-through estimator. The activation quantizers contain no learnable parameters and are identical for MSE and SABRE, ensuring that the comparison isolates the effect of the reconstruction objective. In all three settings, we apply QuaRot-style rotations to suppress activation outliers and compute the score vectors from the corresponding rotated full-precision model. The rotations change the weights that are quantized, so the W2g128A8 rows in Table R3 are not directly comparable with the W2g128 rows of Table 1, which use no rotation. The remaining implementation details are provided in the caption of Table R3.

  • As shown in Table R3, SABRE improves both perplexity and average downstream accuracy across all evaluated configurations. The gains are particularly pronounced under W4A4 and the more aggressive W2g128A8 setting. For example, on Qwen3-8B under W4A4, SABRE reduces perplexity from 13.4613.46 to 11.6711.67 and improves IFEval from 53.853.8% to 65.265.2%. Under W2g128A8, SABRE reduces perplexity by 2020–2626% on both models, e.g., from 29.3429.34 to 21.7621.76 on Qwen3-4B-Instruct, where IFEval improves from 38.438.4% to 47.547.5%.

  • We have added these weight-and-activation quantization results to the revised manuscript (see Section 4.2 and Appendix H.3).

Table R3 Weight-and-activation quantization. To suppress activation outliers, we applied QuaRot-style rotations. For SABRE, (λ,ρ)(\lambda, \rho) is selected from 1000,3000×0.94,0.97\\{1000, 3000\\} \times \\{0.94, 0.97\\}. All runs use optimization seed 42 and the calibration data described in the paper. Avg. denotes the average of CSR6, MMLU, IFEval, GSM8K, and MATH500 for W4A4 and W4A8, and the average of CSR6, MMLU, and IFEval for W2g128A8.

SettingModelMethod(λ,ρ)(\lambda,\rho)PPL↓CSR6MMLUIFEvalGSM8KMATH500Avg.↑
W4A4Qwen3-4B-InstructFlexRound–16.8365.061.764.068.948.861.68
+ SABRE(3000, 0.97)13.2364.562.568.970.750.863.48
Qwen3-8BFlexRound–13.4669.665.753.872.647.861.90
+ SABRE(3000, 0.94)11.6769.566.665.268.250.864.06
W4A8Qwen3-4B-InstructFlexRound–9.9572.671.480.890.272.277.44
+ SABRE(3000, 0.94)9.7872.771.582.490.573.278.06
Qwen3-8BFlexRound–9.2974.974.379.785.772.477.40
+ SABRE(3000, 0.94)9.2774.074.283.086.873.678.32
W2g128A8Qwen3-4B-InstructFlexRound–29.3453.843.338.4––45.17
+ SABRE(1000, 0.97)21.7656.145.747.5––49.77
Qwen3-8BFlexRound–23.0861.650.522.2––44.77
+ SABRE(1000, 0.97)18.4263.150.930.1––48.03
AuthorsOpenReview ↗

Response to Reviewer Ghz4 (Part 2/4)

3. Limited evaluation of scalability and practical benefits (Q3)

  • Peak memory. We thank the reviewer for the suggestion. SABRE stores the bfloat16 token-wise score vectors for all blocks in a filesystem cache. During block-wise quantization, only the score vectors for the block currently being optimized are loaded into a pinned host buffer (1616 GiB), and one minibatch at a time is streamed to the GPU. Consequently, only the score vectors of the current minibatch reside on the GPU. The measured peak GPU memory for Llama-3.1-70B is 69.8169.81 GiB, i.e., only 0.750.75 GiB more than MSE reconstruction (69.0669.06 GiB).

  • Quantization time. Although minibatch-wise streaming requires host-to-device transfers during optimization, we use a separate CUDA stream to overlap the transfer with the block forward pass. Consequently, for 5,000 optimization iterations, the optimization time per block is 1,006.51{,}006.5 s, compared with 980.4980.4 s for MSE reconstruction, i.e., only a 2.72.7% overhead. The full-model block-wise quantization time over all 8080 blocks, including full-precision block-output computation and score-vector loading, is 25.525.5 h for SABRE and 24.724.7 h for MSE reconstruction, corresponding to a 3.23.2% overhead. Computing the score cache requires an additional forward and backward pass through the full-precision model over the calibration set, which takes 3.53.5 h on 4 H100 GPUs. Since it depends only on the full-precision model and the calibration set, the same cache can be reused across bit-widths and quantization configurations (e.g., W3g128 and W2g128).

  • Model quality. As suggested, we additionally evaluate SABRE on Qwen3-32B and Llama-3.1-70B with Block-AP under W3g128. As shown in Table R4, SABRE improves WikiText perplexity from 7.747.74 to 7.607.60 on Qwen3-32B and from 5.3575.357 to 4.2074.207 on Llama-3.1-70B. SABRE also improves average downstream accuracy from 79.6079.60% to 80.6080.60% on Qwen3-32B and from 65.6365.63% to 67.3067.30% on Llama-3.1-70B. These results demonstrate that SABRE's model-quality improvements extend to both 32B- and 70B-scale models.

  • In summary, SABRE scales to a 70B-parameter model. In the revised manuscript, we have added the large-model quality results (see Section 4.2 and Appendix H.10) and the memory and runtime measurements (see Section 4.3.3 and Appendix I).

Table R4. Block-AP W3g128 quantization results on Qwen3-32B and Llama-3.1-70B. For Qwen3-32B, SABRE uses (λ,ρ)=(1000,0.94)(\lambda,\rho)=(1000,0.94); for Llama-3.1-70B, SABRE uses (1000,0.97)(1000,0.97). IFEval is omitted for Llama-3.1-70B because the evaluated model is the base model rather than an instruction-tuned model. Avg. denotes the average of CSR6, MMLU, IFEval, GSM8K, and MATH500 for Qwen3-32B, and the average of CSR6, MMLU, GSM8K, and MATH500 for Llama-3.1-70B. The best result among the quantized models is shown in bold.

ModelMethodWiki PPL↓CSR6↑MMLU↑IFEval↑GSM8K↑MATH500↑Avg.↑
Qwen3-32BBlock-AP7.7477.779.879.187.474.079.60
+ SABRE7.6077.680.279.990.375.080.60
Llama-3.1-70BBlock-AP5.35779.177.2–76.030.265.63
+ SABRE4.20780.077.1–76.735.467.30
AuthorsOpenReview ↗

Response to Reviewer Ghz4 (Part 3/4)

4. Prefill latency and decoding throughput (Q3)

  • We thank the reviewer for raising the deployment aspect. First, we clarify that SABRE affects only the block-wise reconstruction objective used during quantization. It does not change the quantization variables, bit width, group size, weight representation, or packed format of the underlying quantizer. Consequently, under the same bit width, packed representation, and kernel backend, a model quantized with SABRE executes exactly the same inference computation as the corresponding model quantized without SABRE. Thus, SABRE introduces no additional prefill latency, decoding overhead, or inference-time GPU memory usage. We have clarified this point in the revised manuscript (see Section 4.3.3).

  • To confirm this, we measure prefill latency, decoding throughput, and GPU memory using vLLM on a single NVIDIA A100 80GB GPU, with a batch size of 1, an input length of 256 tokens, and an output length of 128 tokens. The quantized models use the W2g128 format with the Humming kernel (see Table R5). Models reconstructed with MSE and SABRE exhibit similar prefill latency and decoding throughput, with identical reported model-loading GPU memory footprints. Compared with full precision, both achieve higher decoding throughput and lower GPU memory usage, while prefill latency remains comparable under the evaluated workload.

  • The practical benefit demonstrated by SABRE is improved model quality under the same quantization configuration and thus at the same inference cost. Under W2g128 with Block-AP, SABRE prevents the collapse observed with MSE reconstruction on Qwen2.5-14B-Instruct: SABRE reduces perplexity from above 10610^6 to 13.0713.07, while improving IFEval from 11.511.5% to 47.747.7% (see Table 1). SABRE also provides consistent quality gains under W3 and W3g128 settings without changing their inference cost (see Table 2). Accordingly, SABRE improves the quality–efficiency trade-off by providing better model quality at the same deployment cost.

  • We have added the vLLM latency, throughput, and GPU-memory measurements to the revised manuscript and clarified SABRE's practical benefit of improving model quality without changing the underlying inference format or deployment cost (see Section 4.3.3 and Appendix J).

Table R5. Model quality and vLLM inference performance for BF16 full-precision models and W2g128 models reconstructed with Block-AP using MSE or SABRE. All inference measurements use a single NVIDIA A100 80GB GPU with batch size 1, input length 256, and output length 128. The W2g128 models use the Humming kernel. Decode reports output-token throughput, and prefill reports average latency. GPU mem denotes the per-GPU memory footprint reported by vLLM for model loading, excluding the pre-allocated KV-cache pool. The pool size depends on the GPU-memory budget configured via gpu_memory_utilization. The quality metrics are from Table 1.

ModelMethodPPL↓CSR6↑MMLU↑IFEval↑Decode↑ (tok/s)Prefill↓ (ms)GPU mem↓ (GiB)
Qwen3-4B-InstructFP9.4073.072.682.6138.5916.387.83
Block-AP40.857.448.940.3224.4517.412.72
+ SABRE29.359.650.943.3220.5916.912.72
Qwen3-8BFP9.0074.274.881.088.9022.4015.46
Block-AP4.5e651.446.021.1172.0323.564.27
+ SABRE26.064.150.328.5178.2724.614.27
Llama-3.1-8B-InstructFP6.7575.968.273.884.7724.2015.17
Block-AP352.948.324.710.2185.4624.003.92
+ SABRE22.461.342.526.1189.8023.183.92
Qwen2.5-14B-InstructFP5.2578.979.978.248.4237.9827.61
Block-AP4.5e663.524.311.5111.6340.116.22
+ SABRE13.169.557.347.7112.4738.246.22

5. Unclear contribution over prior work (Q4)

  • We appreciate the reviewer's comments. GuidedQuant uses feature-wise layer-output gradient weighting to construct curvature matrices for quantization. In contrast, SABRE preserves each token's score direction and the corresponding cross-feature directional structure during block-output reconstruction.

  • Table 4 provides controlled comparisons within the same Block-AP framework, motivated by aggregation and diagonal weighting in prior approaches (e.g., GuidedQuant). Replacing SABRE's token-specific metric with a shared metric obtained by averaging normalized score outer products across calibration tokens degrades W2g128 perplexity from 29.3029.30 to 49.2749.27 and IFEval from 43.343.3% to 39.939.9%. Applying a coordinate-diagonal approximation results in a perplexity of 40.7440.74. Even after further tuning λ\lambda and ρ\rho for the token-shared Fisher and coordinate-diagonal variants, both remain consistently behind SABRE in perplexity and average downstream accuracy on both evaluated models (see Table R6). These results show that both token-specific score directions and cross-feature directional structure contribute substantially to SABRE's performance gains.

Reviewer_BjroOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the limitation of isotropic mean squared error (MSE) objectives in block-wise PTQ (e.g., OmniQuant, FlexRound, Block-AP), which treat all output perturbation directions equally regardless of predictive impact. The authors propose SABRE (Score-guided Anisotropic Block-wise REconstruction). SABRE uses FP-model score vectors (gradients of sampled token log-probabilities) cached from a single backward pass to approximate the Fisher information matrix as rank-one outer products. It penalizes block-output errors aligned with prediction-sensitive directions, regularized by an isotropic MSE term and stabilized via token-wise gradient-norm normalization. Tested across multiple model families (LLaMA-3.1, Qwen3, Qwen2.5) and quantizers, SABRE shows consistent gains in perplexity and downstream tasks at 2-bit and 3-bit regimes.

Critical strengths
  1. Clean Integration as a Plug-and-Play Objective: SABRE requires no changes to parameter parameterizations or inference runtimes; it acts purely as a loss function replacement for block-wise reconstruction, making it broadly applicable across Block-AP, FlexRound, and OmniQuant.
  2. Principled Fisher Formulation and Normalization: Deriving anisotropic sensitivity from the Fisher curvature of the output distribution and stabilizing it with median-based gradient norm normalization effectively prevents optimization instability caused by depth-wise gradient decay.
Critical weaknesses
  1. Additional Memory for Score Caching: Storing token-wise score vectors gt(l)g_t^{(l)} across all layers for long calibration sequences requires non-negligible accelerator memory or host-device transfer overhead, which is not thoroughly profiled for 70B+ scale models.
  2. Dependence on Hyperparameter Tuning: The depth-decay factor ρ\rho and final strength λ\lambda vary across models and bit-widths (e.g., λ∈[200,10000]\lambda \in [200, 10000] in Table 4), suggesting that finding the optimal weighting schedule requires calibration-set validation.
Questions for authors
  1. What is the exact peak host/GPU memory overhead of caching the bfloat16 score vectors across all blocks for larger models (e.g., 70B parameters) when using 512 sequences of 2048 tokens?
  2. How robust is the method if λ\lambda and ρ\rho are fixed to a single default setting across all models and bit-widths, rather than tuned per configuration in Table 4?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Bjro (Part 1/2)

1. Additional Memory for Score Caching (W1, Q1)

  • We appreciate the reviewer's question. SABRE stores the token-wise score vectors of all Transformer blocks in bfloat16 in a filesystem cache. During block-wise quantization, only the score vectors for the block currently being optimized are loaded into a pinned host buffer, and one minibatch of them at a time is streamed to the GPU using preallocated buffers. Consequently, only the score vectors of the current minibatch reside on the GPU.

  • For NN calibration sequences of length TT and hidden dimension dd, keeping the full score tensor of the current block on the GPU requires N×T×d×2 bytesN \times T \times d \times 2\ \text{bytes} in bfloat16. On Llama-3.1-70B (d=8192d=8192) with N=512N=512 and T=2048T=2048, this requires 1616 GiB of additional GPU memory, which, together with the 69.0669.06 GiB required by MSE reconstruction, exceeds the capacity of a single NVIDIA H100 80GB GPU. Minibatch-wise score streaming reduces SABRE's GPU memory overhead from O(NTd)O(NTd) to O(BTd)O(BTd), where BB denotes the reconstruction minibatch size. With B=4B=4, the measured peak GPU memory is 69.8169.81 GiB, i.e., only 0.750.75 GiB more than MSE reconstruction. On the host side, the pinned buffer is reused across blocks, so the additional host-memory requirement is limited to a single block of score vectors (i.e., 1616 GiB for Llama-3.1-70B), while the complete filesystem cache across all Transformer blocks occupies 1.251.25 TiB.

  • Although minibatch-wise streaming requires host-to-device transfers during optimization, we overlap these transfers with the block forward pass using a separate CUDA stream. Consequently, on Llama-3.1-70B, the per-block optimization time increases by only 2.72.7% (1,006.51,006.5 s versus 980.4980.4 s for MSE over 5,0005,000 iterations).

  • We have added the score-streaming implementation details, peak GPU and host memory overhead, and runtime measurements to the revised manuscript (see Section 4.3.3 and Appendix I).

AuthorsOpenReview ↗

Response to Reviewer Bjro (Part 2/2)

2. Dependence on Hyperparameter Tuning (W2, Q2)

  • We appreciate the reviewer's question. Section 4.3.2 and Figure 3(a) evaluate SABRE's sensitivity to both λ\lambda and ρ\rho on Qwen3-8B with OmniQuant under W3. Across the evaluated range, every plotted (λ,ρ)(\lambda,\rho) pair outperforms the MSE baseline in perplexity and all five downstream metrics. Although the best configuration varies across individual benchmarks, these results show that SABRE remains effective over a broad range of λ\lambda and ρ\rho and does not require precise hyperparameter tuning to improve over isotropic reconstruction.

  • Furthermore, to directly answer the reviewer's question, we applied a single fixed setting across all configurations reported in Tables R1 and R2 and additionally evaluated a second fixed setting, (3000,1.0)(3000,1.0), under W2g128. Across the configurations in Tables R1 and R2, SABRE improves perplexity and average downstream accuracy over MSE with the fixed setting (1000,0.94)(1000,0.94), without configuration-specific tuning. In addition, SABRE with (3000,1.0)(3000,1.0) lowers perplexity on all four models and improves average accuracy on the three larger ones. These results demonstrate that SABRE provides robust improvements over MSE reconstruction even with a single fixed setting, i.e., without the need for precise, configuration-specific hyperparameter tuning.

  • We have included the results of Tables R1 and R2 to emphasize SABRE's robustness to hyperparameter selection in the revised manuscript (see Section 4.3.2 and Appendix H.7).

Table R1. Block-AP results under W2g128 with two fixed settings. All rows are means over three optimization seeds (seed 42, as in Table 1, and two more). All other settings follow Table 1. Avg. denotes the average of CSR6, MMLU, and IFEval. Bold marks the best perplexity and the best Avg. for each model.

ModelMethodPPL↓CSR6MMLUIFEvalAvg.↑
Qwen3-4B-InstructBlock-AP48.9658.349.340.449.33
+ SABRE (1000, 0.94)31.4959.450.644.051.33
+ SABRE (3000, 1.0)28.5259.046.737.247.63
Qwen3-8BBlock-AP1.9×1061.9{\times}10^{6}50.747.119.239.00
+ SABRE (1000, 0.94)4.0×1034.0{\times}10^{3}56.951.826.144.93
+ SABRE (3000, 1.0)24.7663.549.427.246.70
Llama-3.1-8B-InstructBlock-AP465.4247.524.710.927.70
+ SABRE (1000, 0.94)76.0152.024.721.932.87
+ SABRE (3000, 1.0)24.4860.537.026.741.40
Qwen2.5-14B-InstructBlock-AP7.9×1067.9{\times}10^{6}54.624.111.530.07
+ SABRE (1000, 0.94)4.3×1044.3{\times}10^{4}69.535.538.447.80
+ SABRE (3000, 1.0)13.3669.957.935.754.50

Table R2. OmniQuant results under W3 and W3g128 quantization with the fixed setting (λ,ρ)=(1000,0.94)(\lambda,\rho)=(1000,0.94). All rows are means over three optimization seeds (seed 42, as in Table 2, and two more). All other settings follow Table 2. Avg. denotes the average of CSR6, MMLU, IFEval, GSM8K, and MATH500.

ModelBitsMethodPPL↓CSR6MMLUIFEvalGSM8KMATH500Avg.↑
Qwen3-4B-InstructW3OmniQuant20.5665.659.460.959.438.156.68
+ SABRE15.3466.360.863.561.539.858.38
W3g128OmniQuant13.4366.764.672.079.059.168.28
+ SABRE11.5366.966.072.281.459.869.26
Qwen3-8BW3OmniQuant15.1567.664.861.151.338.356.62
+ SABRE12.3769.064.266.455.943.759.84
W3g128OmniQuant11.8471.169.171.574.556.768.58
+ SABRE10.5471.769.476.175.360.970.68

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response

We sincerely thank all reviewers for the time and effort spent evaluating our manuscript. To address the concerns raised across the reviews, we conducted additional experiments and strengthened several analyses. We have also revised the manuscript accordingly, with changes highlighted in blue. The corresponding locations in the revised manuscript are given in brackets. Specifically:

  • Memory and runtime. We profiled the cache size, peak GPU memory, additional host memory, and per-block runtime for a 70B model (Reviewers Bjro, Ghz4, and Jcd7) [Section 4.3.3; Appendix I, Tables 21–22].

  • Controlled comparison of reconstruction objectives. Within the same Block-AP framework, we compared SABRE with tuned curvature-based variants (Reviewers Ghz4 and Jcd7) and with the reconstruction objective of FIMA-Q (Reviewer Ghz4) [Section 4.3.1; Appendix E, Tables 6–7].

  • Results with a fixed hyperparameter setting. We applied a single fixed setting (λ,ρ)=(1000,0.94)(\lambda, \rho) = (1000, 0.94) across all configurations reported in Tables R1 and R2 without any per-configuration selection (Reviewer Bjro) [Section 4.3.2; Appendix H.7, Tables 16–17].

  • Motivation and related work. We clarified how local reconstruction errors affect the final predictive distribution and expanded the discussion of prior gradient- and Fisher-based methods (Reviewer Ghz4) [Section 2.2].

  • Weight-and-activation quantization. We evaluated SABRE under W4A4, W4A8, and W2g128A8 (Reviewer Ghz4) [Section 4.2; Appendix H.3, Table 12].

  • Larger models. We extended our evaluation to larger-scale models such as Qwen3-32B and Llama-3.1-70B (Reviewer Ghz4) [Section 4.2; Appendix H.10, Table 20].

  • Inference efficiency. We measured prefill latency, decoding throughput, and GPU memory with vLLM (Reviewer Ghz4) [Section 4.3.3; Appendix J, Table 23].

  • Clarification on the original Fig. 3(a). We corrected the experimental-setting description and provide an updated comparison consistent with Tables 1 and 2 (Reviewer Jcd7) [Section 4.2, Table 3; Appendix H.5, Table 14].

Unless explicitly referring to the original submission, all table and figure numbers refer to the revised manuscript; tables prefixed with “R” belong to this response. Detailed responses to each reviewer are provided below. We hope our additional experiments and clarifications sufficiently address the reviewers' concerns, and we welcome any further questions or suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.