acceptodds
Under review as a conference paper at ICLR 2027

SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability

Abstract

Extreme low-bit weight-only quantization is an effective approach to reducing the deployment costs of large language models. Below 3 bits, scalar quantization often suffers severe accuracy loss, while vector quantization (VQ) better preserves accuracy through more expressive joint encoding. However, existing VQ methods still exhibit substantial degradation on real-world tasks, and two design challenges remain underexplored: (1) protecting sensitive weight channels within jointly encoded vectors; and (2) allocating a fixed bit budget among representation components, such as direction and radius, to suit different weight matrices. To address these challenges, we propose : plit daptation and rouping by nergy for weight-only vector quantization. First, Energy-Stratified Grouping (ESG) couples channel grouping with activation-weighted fitting, grouping channels from different energy tiers within each subvector to prioritize sensitive channels. Second, Adaptive Bit Split (ABS) balances direction-codebook capacity and radius precision per matrix, selecting the split with the lowest activation-weighted reconstruction error under a fixed budget. Across benchmarks covering knowledge, mathematics, code generation, instruction following, and tool use, SAGE achieves state-of-the-art performance in the 2-bit quantization setting. On Qwen3-30B-A3B, it retains 95.3% of the BF16 model's average benchmark score. With our custom-designed CUDA kernel, SAGE delivers up to a 2.55 non-streaming end-to-end speedup over FP16. We will release our code, models, and kernels.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, Extreme Low-Bit Quantization, Weight-Only Quantization, Vector Quantization, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_iphkOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper build upon SPHQuant: a weight-only vector quantization method that represents each group of weights with a vector, sign, and radius. The paper improves it by applying 2 techniques:

  • Energy-Stratified Grouping: permutes channels into groups of similar importance (sorted within each group). This ensures that we don't represent multiple channels with wide range of importance using the same vector look up table (LUT)
  • Adaptive Bit Split: finding the best bit width for radius (i.e., precision of radius) and vector LUT (i.e., how many vectors)

The authors apply the approach on a wide range of tasks and models and show that they beat SoTA in 2-bit quantization

Critical strengths
  • Clear Figure 2 summarizing the solution
  • To the best of my knowledge: permuting channel weights (done in ESG) is novel
  • Evaluated a wide range of model sizes and 2 model families on a wide range of tasks.
  • Compared with a wide range of tasks and consistently beats them on most tasks.
  • Ablation studies that clearly show the contribution of each component of the solution
  • detailed systems analysis: TTFT, TPS, Speedup, Model Size, even showing when the model is slower on TTFT compared to some techniques like AWQ but faster than all other 2-bit VQ techniques
  • highest end-to-end speedup after comparing with wide range of quantization approaches
Critical weaknesses
  • Don't really have a critical weakness in mind.
Questions for authors

Suggestions:

  • Figure 1: Please add to each plot, a horizontal dashed line showing FP16 accuracy
  • Figure 2: Inside the "Reconstruction" box and under "Three Candidates", the font of "w" is consistent with the rest of the Figure, and "cz" should be "c_{z}"
  • Line 196: Please define v
  • Line 198: here, |w|, denotes absolute values, rather than norm, right? Clarifying would be helpful.
  • Line 202: please add citation to the paper that proposes spherical k-means as it seems to be a seminal paper. when I searched online, it seems to be this:

Dhillon, Inderjit S., and Dharmendra S. Modha. "Concept Decompositions for Large Sparse Text Data Using Clustering." Machine Learning, vol. 42, no. 1/2, 2001, pp. 143–175. SpringerLink, doi:10.1023/A:1007612920971.

  • Line 205: can you clarify how many bits are used to store offset and step?
  • Equation 3: "v" here is to store a sign bit for each element in a subvector
  • Line 300: for those who are not familiar with SPHQuant, please briefly describe how you optimize. Do you do some training?
3 comments

Response to Reviewer iphk (denoted as R3)

Q3-1: Figure 1: Please add to each plot, a horizontal dashed line showing FP16 accuracy

A3-1: Thank you for the suggestion. We have added a horizontal dashed line indicating the full-precision (BF16) reference score to each accuracy plot in Figure 1 in the revised PDF.

Q3-2: Figure 2: Inside the "Reconstruction" box and under "Three Candidates", the font of "w" is consistent with the rest of the Figure, and "cz" should be "czc_z "

A3-2: We have standardized the font of the weight symbols and corrected the notation to czc_z in Figure 2 in the revised PDF.

Q3-3: Line 196: Please define v

A3-3: We have explicitly defined vv as the subvector dimension, with v=8v=8 in our experiments, at its first occurrence in the revised PDF.

Q3-4: Line 198: here, |w|, denotes absolute values, rather than norm, right? Clarifying would be helpful.

A3-4: Yes, ∣w∣|\mathbf{w}| denotes the elementwise absolute value, rather than a norm. We have explicitly clarified this distinction in Section 3.1 of the revised PDF.

Q3-5: Line 202: please add citation to the paper that proposes spherical k-means as it seems to be a seminal paper.

A3-5: We have added the citation to Dhillon and Modha (2001) [1] when introducing spherical k-means in Section 3.1 of the revised PDF.

Q3-6: Line 205: can you clarify how many bits are used to store offset and step?

A3-6: The offset and step are each stored in FP16 (16 bits each), contributing an additional 0.25 BPW at a group size of 128 weights. We have explicitly stated this in Section 3.1 of the revised PDF.

Q3-7: Equation 3: "v" here is to store a sign bit for each element in a subvector

A3-7: Yes, that is correct. Each weight in a subvector has one sign bit, requiring vv sign bits in total. We have clarified this alongside Equation (3) in the revised PDF.

Q3-8: Line 300: for those who are not familiar with SPHQuant, please briefly describe how you optimize. Do you do some training?

A3-8: Yes, following SPHQuant, we perform blockwise optimization on calibration data, updating radius and codebook parameters to minimize the MSE between quantized and full-precision block outputs while keeping discrete assignments fixed. We have expanded Section 3.4 in the revised PDF to make this procedure clearer and self-contained for readers unfamiliar with SPHQuant.

[1] Dhillon, Inderjit S., and Dharmendra S. Modha. "Concept Decompositions for Large Sparse Text Data Using Clustering." Machine Learning, vol. 42, no. 1/2, 2001, pp. 143–175. SpringerLink, doi:10.1023/A:1007612920971.

I have read all the reviews and the rebuttals. I believe that the authors have addressed the reviewers concerns. The results in the rebuttal for larger models or applying ESG on other quantization approaches are impressive. I would like to keep my score.

Dear Reviewer iphk,

Thank you for your thoughtful review and positive feedback on our rebuttal and additional experiments. We sincerely appreciate your time and careful consideration throughout the review process.

Best regards, The Authors

Reviewer_Z5X3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAGE, a 2-bit weight-only vector quantization framework built on spherical VQ. SAGE introduces Energy-Stratified Grouping (ESG) to protect activation-sensitive channels through energy-aware grouping and weighted fitting, and Adaptive Bit Split (ABS) to allocate the fixed bit budget between direction and radius on a per-matrix basis. Additionally, the optimized CUDA kernels improve the end-to-end inference speed of spherical VQ by up to 2.55$\times$ over FP16.

Critical strengths
  • SAGE effectively extends spherical vector quantization with activation-aware ESG and matrix-specific ABS, yielding substantial accuracy improvements across diverse benchmarks.

  • The evaluation is broad and practically relevant, covering not only perplexity but also code generation, mathematical reasoning, instruction following, and tool use across multiple LLMs.

  • Beyond the quantization algorithm, the paper carefully optimizes inference through ESG permutation fusion, shared-memory codebook reuse, and specialized GEMV/GEMM kernels for spherical VQ, with detailed descriptions of the kernel implementation.

Critical weaknesses
  • Effective bit budget: QuIP#, QTIP, and UniSVQ operate at roughly 2 BPW, while SAGE requires about 2.5 BPW once the radius-grid parameters are included. It would be more convincing to compare methods under a similar effective storage budget.

  • Calibration sensitivity: Fig. 4(b) shows noticeable gains as the calibration set grows. It would be helpful to clarify how much of the improvement comes from the proposed ESG/ABS design itself versus the use of high-quality, multi-domain calibration data.

  • Portability beyond GPUs: ESG introduces runtime activation permutation, while ABS adds matrix-specific bit configurations. These overheads are hidden through CUDA-specific fusion and kernel optimizations, but it is unclear how well the method transfers to other computing platforms or distributed settings such as tensor parallelism.

  • Scalability to larger models: Since quantization becomes increasingly important for very large models, results on models larger than 70B would further strengthen the scalability. Similarly, the inference-efficiency evaluation is mainly conducted on the small model(4B), so results on larger models or a wider range of linear-layer shapes would better demonstrate the hardware efficiency.

Questions for authors

Please address the concerns raised in Weaknesses.

4 comments

Response to Reviewer Z5X3 (denoted as R2)

Q2-1: Effective bit budget

A2-1: Thank you for raising this concern. We conducted additional storage-matched experiments on Qwen3-4B-Instruct-2507 by increasing the storage budgets of UniSVQ, QuIP#, and QTIP to achieve approximately the same model size as SAGE. Specifically, we retained all seven attention and MLP projections in the 36th Transformer layer, along with mlp.down_proj and self_attn.k_proj in the 35th layer, in BF16. For UniSVQ, we additionally retained self_attn.v_proj in the 35th layer in BF16. The results are shown below.

MethodModel Size (GB)MMLU-RGSM8KMATH-500IFEvalHumanEval+BFCL-v3Average
UniSVQ1.788140.1416.074.4028.284.8829.7320.58
QuIP#1.787663.8973.1636.6065.9929.2741.6851.77
QTIP1.788572.5681.0565.6074.4953.6651.0466.40
SAGE1.786773.4788.7078.8082.8159.7653.3372.81

Under these matched storage budgets, SAGE outperforms all three baselines on every benchmark, achieving an average score of 72.81 and maintaining a 6.41-percentage-point advantage over the strongest baseline, QTIP. These results demonstrate that SAGE's accuracy advantage persists when actual model sizes are closely matched.

Q2-2: Calibration sensitivity

A2-2: To ensure a fair comparison, we use exactly the same calibration data and dataset size as SPHQuant: 1,920 training sequences and 128 validation sequences, each containing 2,048 tokens. With both data content and size held fixed, the improvements over SPHQuant in Table 1 reflect our algorithmic changes, ESG and ABS, as summarized below.

ModelSPHQuantSAGEImprovement
Qwen3-4B-Instruct-250765.2572.81+7.56
Qwen3-30B-A3B-Instruct-250778.4382.48+4.05
Qwen3.5-4B-Thinking51.4467.68+16.24
Ministral-3-8B-Instruct-251251.5763.21+11.64
Ministral-3-14B-Instruct-251265.5771.96+6.39

Scores are averaged across the six benchmarks, with improvements reported in percentage points. Figure 4(b) separately shows that increasing the calibration set size can further improve SAGE's performance. However, for the main comparisons, we keep the calibration data content and size identical to those used for SPHQuant to fairly evaluate the algorithmic improvements.

Q2-3: Portability beyond GPUs

A2-3: Thank you for raising this concern. We did not have access to non-CUDA hardware during the rebuttal period. However, we broadened our evaluation across GPU architectures by adding experiments on the RTX PRO 6000 (Blackwell), complementing the original RTX 4090 (Ada Lovelace) results. The additional Qwen3-4B timing results are reported below, with decoding throughput speedups measured relative to FP16.

Prefill 128 / Decode 1024 tokens:

MethodFirst token (ms)Output (tokens/s)Decode throughput speedup
FP1611.91181.00×
AWQ13.82782.36×
UniSVQ16.02402.03×
QuIP#36.01821.55×
QTIP36.21771.50×
SPHQuant95.92652.25×
SAGE17.02822.39×

Prefill 1024 / Decode 128 tokens:

MethodFirst token (ms)Output (tokens/s)Decode throughput speedup
FP1663.71121.00×
AWQ66.22432.17×
UniSVQ68.22121.89×
QuIP#80.31601.42×
QTIP78.31581.41×
SPHQuant726.01070.95×
SAGE65.42442.19×

SAGE achieves decoding throughputs of 282 and 244 tokens/s under the two settings, supporting its efficiency across the evaluated GPU architectures. Regarding distributed deployment, ESG and ABS operate within individual linear layers and are conceptually orthogonal to pipeline parallelism (PP) and expert parallelism (EP). ABS configurations are determined offline and require no runtime search. For tensor parallelism (TP), the 8-element quantization groups can be preserved through aligned shard boundaries, while ESG permutations and quantization metadata must be coordinated with the distributed tensor layout.

Q2-4: Scalability to larger models

A2-4: We added accuracy results on Qwen3-Next-80B-A3B-Instruct, an MoE model with 80B total parameters and 3B activated parameters, to further evaluate SAGE at a larger model scale. The results are shown below. Average denotes the arithmetic mean across the six benchmarks, and bold indicates the best result among quantized methods.

MethodMMLU-ReduxGSM8KMATH-500IFEvalHumanEval+BFCL-v3Average
BF1692.3396.4497.0087.2590.8567.8288.62
UniSVQ87.5194.0988.2082.8179.8860.6082.18
QuIP#88.4095.9194.4084.1089.0362.9685.80
QTIP89.1295.5393.4084.6685.9863.9185.43
SPHQuant88.5194.9291.6083.9284.1561.2784.06
SAGE89.9696.2995.0086.3289.1563.5386.71

SAGE achieves the highest average score among quantized methods, reaching 86.71 compared with 88.62 for BF16, a gap of only 1.91 percentage points. It outperforms SPHQuant and QuIP# by 2.65 and 0.91 percentage points, respectively. These results further validate SAGE's effectiveness at a larger model scale while maintaining performance close to full precision.

We also evaluated hardware efficiency on Ministral 8B using an RTX 4090, extending the evaluation beyond the original 4B model to a larger model configuration and additional linear-layer shapes. These shapes (output × input) include 4096 × 4096 for q/o projections, 1024 × 4096 for k/v projections, 14336 × 4096 for gate/up projections, and 4096 × 14336 for down projections. Results under two input/output-length settings are reported below, with decoding throughput speedups measured relative to the dense baseline.

Prefill 128 / Decode 1024 tokens:

MethodFirst token (ms)Output (tokens/s)Decode throughput speedup
Dense25.41854.4561.00×
UniSVQ44.473171.4363.15×
QuIP#104.230143.2272.63×
QTIP97.781147.7852.71×
SPHQuant233.796165.2303.03×
SAGE27.281165.1613.03×

Prefill 1024 / Decode 128 tokens:

MethodFirst token (ms)Output (tokens/s)Decode throughput speedup
Dense108.44552.1881.00×
UniSVQ132.854148.2502.84×
QuIP#229.669115.9432.22×
QTIP224.746119.8042.30×
SPHQuant1840.00349.6040.95×
SAGE141.651141.4692.71×

SAGE achieves 3.03× and 2.71× decoding throughput speedups over the dense baseline under the two settings. These measurements extend our hardware-efficiency evidence beyond the original 4B setting to a larger model configuration and additional linear-layer shapes.

Reviewer_Z5X3OpenReview ↗

Response to the Authors' Rebuttal

Thank you for the detailed response and additional experiments.

First, the results on a relatively large model demonstrate that the proposed method can maintain competitive performance even at 2-bit precision, which sufficiently addresses my concern. The additional hardware evaluations are also informative, although the performance improvements over other methods appear somewhat limited.

Regarding the other responses on EPW and calibration, while the experimental settings are not exactly aligned for a direct comparison, I think they address my concerns to some extent. As for the group-wise runtime activation overhead, I still see some potential hardware inefficiencies, but I find the approach reasonable given the GPU-based inference environment considered in this work.

Overall, while some limitations remain, the rebuttal has addressed several of my concerns. Given the limited time available for further experiments and the fact that such experiments are unlikely to significantly change my overall assessment, I will maintain my original score.

Dear Reviewer Z5X3,

Thank you for carefully reviewing our rebuttal and additional experiments. We sincerely appreciate your recognition of the larger-model results and the value of the additional hardware evaluations. Your feedback on the comparison settings and runtime overhead is also valuable, and we will continue working to broaden our evaluation across different inference environments and hardware platforms.

Thank you again for your time and constructive feedback throughout the review process.

Best regards, The Authors

Reviewer_4HKbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAGE improves extreme low-bit weight-only vector quantization through Energy-Stratified Grouping (ESG) and Adaptive Bit Split (ABS). Building on SPHQuant, it uses activation statistics to guide channel grouping and codebook fitting and selects direction-radius bit allocations per weight matrix.

Critical strengths
  1. Substantial improvements over SPHQuant, with large benchmark gains across five models (as shown in Table 1).
  2. Comprehensive evaluation with meaningful component ablations (Table 2). Custom CUDA kernels also demonstrate practical inference acceleration, reporting end-to-end speedup over FP16 (Figure 5c).
Critical weaknesses
  1. My main concern is that SAGE builds heavily on SPHQuant, which already provides the underlying spherical representation, codebook optimization, and direction-radius bit-allocation tradeoff (ABS is more general, of course, but the idea is the same in my understanding). ESG is the more interesting contribution, but its energy-stratified interleaving lacks a clear justification. It is unclear why mixing channels from different activation-energy tiers should outperform alternative groupings. The method works well but the generality of ESG beyond this SPHQuant quantization framework is unclear.
  2. SAGE reports 2.25 bits per weight for its encoding payload, but radius-grid metadata adds another 0.25 bits per weight (the authors also discussed this in Appendix A.6). However, several competing methods use 2-bit representations. Although the comparison with SPHQuant is meaningful, the broader SOTA claims would benefit from quality comparisons at matched actual model sizes.
Questions for authors
  1. What motivates energy-stratified interleaving over alternative grouping strategies? Can the authors compare ESG with other permutations under matched codebook fitting and optimization budgets? Also, I feel like ESG is the main contribution of this work. If so, the authors should think about whether they can apply this to other quantization methods. And show that the energy-stratified interleaving is good across these scenarios.
  2. How much benefit does ABS provide over the best fixed direction-radius split? I feel like it’s hard to claim this point as a technical contribution to be honest if we already have something similar in SPHQuant. Consider reframing the contribution part.
  3. Can the authors clarify the relationship to prior work on optimized vector partitioning and provide storage-matched comparisons?
5 comments

Response to Reviewer 4HKb (denoted as R1)

Q1-1: What are SAGE’s novel contributions beyond SPHQuant, and what justifies ESG’s energy-stratified interleaving and its generalizability beyond this framework?

A1-1: Thank you for this question. We retain SPHQuant's spherical representation because of its advantages in decoding efficiency and inference speed. Our novel contributions are:

  • ESG: We address how to protect activation-sensitive channels when multiple weights are jointly encoded. Scalar quantization methods often protect sensitive channels through rescaling or group similar channels together to better fit shared scalar quantization parameters. These design principles do not directly carry over to VQ, where weights share a codeword and their reconstruction errors are coupled. ESG introduces a different perspective: it couples grouping across activation-energy tiers with activation-weighted fitting to prio ritize sensitive positions within each jointly encoded subvector.

  • ABS: We introduce the perspective that the best direction–radius trade-off can vary across weight matrices. Different weight distributions and activation sensitivities create different requirements for direction-codebook capacity and radius precision. Instead of imposing a single fixed split throughout the model, ABS selects the allocation with the lowest activation-weighted reconstruction error for each matrix under a fixed encoding budget, adapting the representation to matrix-specific demands.

To evaluate whether ESG's benefits extend beyond SPHQuant, we applied ESG channel reordering to AQLM [1] on Qwen3-4B-Instruct-2507. The results are shown below.

MethodMMLU-RGSM8KMATH-500IFEvalHumanEval+BFCL-v3Average
AQLM71.7483.0964.0070.6146.9547.8464.04
AQLM + ESG72.9684.6065.2073.1651.3450.0766.22

ESG improves AQLM on all six benchmarks, increasing the average score by 2.18 percentage points. These results demonstrate that ESG's benefits extend to a quantization framework beyond SPHQuant.

[1] Extreme Compression of Large Language Models via Additive Quantization, ICML, 2024.

Q1-2: How does SAGE compare with competing methods at matched actual model sizes, accounting for the additional radius-grid metadata?

A1-2: Thank you for raising this concern. SAGE and SPHQuant share the same encoding budget and radius-grid storage overhead, making their comparison appropriate. However, UniSVQ, QuIP#, and QTIP use smaller storage budgets. We therefore conducted additional experiments at approximately matched actual model sizes on Qwen3-4B-Instruct-2507. We retained all seven attention and MLP projections in the 36th Transformer layer, along with mlp.down_proj and self_attn.k_proj in the 35th layer, in BF16 for these baselines. UniSVQ additionally retains self_attn.v_proj in the 35th layer in BF16.

MethodModel Size (GB)MMLU-ReduxGSM8KMATH-500IFEvalHumanEval+BFCL-v3Average
UniSVQ1.788140.1416.074.4028.284.8829.7320.58
QuIP#1.787663.8973.1636.6065.9929.2741.6851.77
QTIP1.788572.5681.0565.6074.4953.6651.0466.40
SAGE1.786773.4788.7078.8082.8159.7653.3372.81

At approximately the same model size, SAGE outperforms all three baselines on every benchmark, maintaining a 6.41-percentage-point average advantage over the strongest baseline, QTIP.

Q1-3: What motivates energy-stratified interleaving over alternative grouping strategies? Can the authors compare ESG with other permutations under matched codebook fitting and optimization budgets?

A1-3: Thank you for this question. ESG couples energy-stratified interleaving with activation-weighted fitting to prioritize sensitive positions within jointly encoded subvectors. In spherical VQ, weights within a subvector share a direction codeword and radius, coupling their reconstruction errors. Placing channels from different activation-energy tiers together allows weighted fitting to prioritize higher-energy positions. ESG also maintains a consistent tier ordering across subvectors, so that each codeword position corresponds to a consistent importance rank.

We compare four permutation strategies on Qwen3-4B-Instruct-2507 under matched codebook-fitting and optimization budgets. No permutation preserves the original channel order; Random randomly permutes channels; Energy-sorted groups adjacent channels after sorting by activation energy; and ESG interleaves channels across energy tiers. The table below reports the average score across MMLU-Redux, GSM8K, MATH-500, IFEval, HumanEval+, and BFCL-v3.

Permutation strategyNo permutationRandomEnergy-sortedESG
Average68.8368.9268.2472.81

ESG achieves the highest score on all six benchmarks, improving the average over No permutation by 3.98 percentage points. Random permutation performs similarly to No permutation, while Energy-sorted grouping does not improve performance. These results support energy-stratified interleaving over random permutation or grouping channels with similar activation energies.

Q1-4: How much benefit does ABS provide over the best fixed direction-radius split?

A1-4: Thank you for this question. Table 2 compares ABS with the best-performing fixed split: a 6-bit direction index (64 codewords) and a 4-bit radius code (tested in SPHQuant). With ESG enabled in both settings and the same encoding budget, the results are as follows.

Bit allocationMMLU-ReduxGSM8KMATH-500IFEvalHumanEval+BFCL-v3Average
Fixed (6-bit direction / 4-bit radius)69.0582.1868.4080.0454.8845.1766.62
ABS73.4788.7078.8082.8159.7653.3372.81

ABS improves the average score by 6.19 percentage points over this fixed split, with gains across all six benchmarks. Different weight matrices have different requirements for direction-codebook capacity and radius precision. By selecting the split according to each matrix's activation-weighted reconstruction error, ABS adapts this trade-off to matrix-specific characteristics rather than imposing the same allocation throughout the model.

Q1-5: Can the authors clarify the relationship to prior work on optimized vector partitioning and provide storage-matched comparisons?

A1-5: Thank you for this question. Previous permutation and grouping methods pursue different objectives. RPTQ [2] groups activation channels with similar numerical ranges to improve shared scalar quantization, while PT²-LLM [3] groups structurally similar weight columns to facilitate ternarization. These strategies are designed around scalar quantization grids. CRVQ [4] considers VQ, grouping important channels together and assigning additional codebooks to these groups.

ESG introduces a different grouping perspective for jointly encoded weights: it places channels from different activation-energy tiers within each subvector and uses activation-weighted fitting to prioritize sensitive positions. Please see A1-3 for the detailed motivation and controlled permutation comparisons, and A1-1 for the extension to AQLM.

For storage-matched comparisons, please see A1-2. At approximately matched actual model sizes, SAGE outperforms the strongest matched baseline, QTIP, by 6.41 percentage points in average score.

[2] RPTQ: Reorder-based Post-training Quantization for Large Language Models, arXiv:2304.01089, 2023.

[3] PT²-LLM: Post-Training Ternarization for Large Language Models, ICLR, 2026.

[4] CRVQ: Channel-Relaxed Vector Quantization for Extreme Compression of LLMs, TACL, 2025.

Dear Reviewer 4HKb,

Thank you again for your thoughtful review. As the discussion period approaches its end, we would appreciate hearing whether any concerns remain after our rebuttal. We briefly summarize our responses below:

  • Contributions and prior work: We clarified the components inherited from SPHQuant and the contributions of ESG and ABS, including their relationship to prior grouping and permutation methods.

  • ESG motivation and permutation comparisons: We explained the rationale for energy-stratified interleaving and compared it with alternative permutations under matched fitting and optimization budgets. ESG improves the average score over no permutation by 3.98 percentage points.

  • Generality beyond SPHQuant: Adding ESG channel reordering to AQLM improves all six benchmarks, with an average gain of 2.18 percentage points.

  • ABS effectiveness: With ESG enabled in both settings, ABS improves the average score by 6.19 percentage points over SPHQuant's fixed direction–radius split.

  • Storage-matched comparisons: At approximately matched model sizes, SAGE outperforms UniSVQ, QuIP#, and QTIP on all six benchmarks, maintaining a 6.41-percentage-point average advantage over QTIP.

We welcome any further questions and would be happy to clarify any remaining concerns. Thank you for your time and consideration.

Best regards, The Authors

Comments on the paper

2 comments
Area_Chair_EMGfOpenReview ↗

Please check author responses

Dear reviewers,

We’re about halfway through the discussion period. This is a friendly reminder to check other's reviews and the author responses. If the authors have answered questions or responded to critiques in reviews, please leave a comment for the authors!

Best, AC

AuthorsOpenReview ↗

Summary of Rebuttal and Discussion

Dear Program Chairs, Senior Area Chairs, Area Chair, and Reviewers,

We sincerely thank you for your valuable time and constructive feedback. To facilitate the final assessment, we summarize the strengths recognized by the reviewers and the additional evidence provided during the discussion.

We are pleased that the reviewers recognize the following strengths of our work:

  1. Novel technical design. Reviewer iphk (R3) recognizes the novelty of ESG’s channel permutation strategy, which groups channels from different activation-energy tiers into jointly encoded subvectors.

  2. Strong empirical performance. Reviewer 4HKb (R1) highlights substantial improvements over SPHQuant across five models, Reviewer Z5X3 (R2) recognizes substantial accuracy gains across diverse benchmarks, and Reviewer iphk (R3) notes that SAGE outperforms the compared methods on most tasks.

  3. Comprehensive evaluation and meaningful ablations. Reviewers 4HKb (R1) and iphk (R3) appreciate the component ablations, while Reviewer Z5X3 (R2) highlights the broad evaluation across models and diverse generative tasks.

  4. Practical inference acceleration. Reviewers 4HKb (R1) and iphk (R3) recognize the demonstrated end-to-end speedup, while Reviewer Z5X3 (R2) appreciates the detailed CUDA kernel optimizations.

In response to the reviewers’ concerns, we provided the following clarifications and additional evidence:

  1. Clarifying our contributions and relationship to prior methods. We clarified the spherical representation inherited from SPHQuant and explained the specific contributions of ESG and ABS relative to prior grouping and bit-allocation methods. (R1: Q1-1, Q1-4, and Q1-5)

  2. ESG motivation and generality. Under matched codebook-fitting and optimization budgets, ESG improves the average score over no permutation by 3.98 percentage points. Applying ESG channel reordering to AQLM improves all six benchmarks, with an average gain of 2.18 percentage points, supporting its effectiveness beyond SPHQuant. (R1: Q1-1 and Q1-3)

  3. Storage-matched comparisons. We added comparisons at approximately matched actual model sizes, where SAGE outperforms UniSVQ, QuIP#, and QTIP on all six benchmarks, maintaining a 6.41-percentage-point average advantage over the strongest matched baseline, QTIP. (R1: Q1-2 and Q1-5; R2: Q2-1)

  4. Larger-model evaluation and broader hardware coverage. On Qwen3-Next-80B-A3B-Instruct, SAGE achieves the highest average score among the evaluated quantized methods, within 1.91 percentage points of BF16. Additional timing experiments on the RTX PRO 6000 and the Ministral 8B configuration extend the evaluation across GPU architectures and additional linear-layer shapes. (R2: Q2-3 and Q2-4)

  5. Presentation and implementation clarity. We revised the figures, clarified notation and radius-grid storage precision, added the spherical k-means reference, and expanded the explanation of blockwise optimization. (R3: Q3-1–Q3-8)

Following the discussion, Reviewer 4HKb (R1) raised the score from weak rejection to weak acceptance, while Reviewers Z5X3 (R2) and iphk (R3) maintained weak acceptance and clear acceptance, respectively. We sincerely thank all reviewers and chairs for their time, careful consideration, and constructive feedback throughout the review process.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.