acceptodds
Under review as a conference paper at ICLR 2027

SignRoundV2: Toward Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs

Abstract

Extremely low-bit quantization is critical for efficiently deploying Large Language Models (LLMs), yet it often leads to severe performance degradation at 2 bits and even at 4 bits (e.g., MXFP4). We present SignRoundV2, a post-training quantization framework designed to maintain high performance even under aggressive compression. SignRoundV2 introduces (1) a simple yet efficient adaptive mixed-precision strategy that leverages gradient information and quantization-induced reconstruction errors to guide layer-wise bit allocation, and (2) a set of lightweight techniques, including loss filtering and a pre-tuning scale search, to improve tuning effectiveness in extremely low-bit regimes. Our approach takes a significant step toward closing the performance gap between quantized and full-precision models. Experimental results across diverse LLMs demonstrate that SignRoundV2 achieves near-lossless performance in mixed MXFP settings, narrowing the gap to 1% at an average of 4.5 bits, while substantially improving accuracy in challenging 2-bit weight-only quantization.

Primary area:
optimization
Keywords:
post-training quantization, Extremely low-bit quantization, adaptive mixed-precision, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QB4jOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SignRoundV2, a post-training quantization framework designed to maintain model performance under extremely low-bit quantization. Building on SignRoundV1, it combines gradient information and quantization-induced errors to guide layer-wise mixed-precision allocation, and introduces loss filtering and a pre-tuning scale search to improve tuning effectiveness. Experiments on Llama and Qwen models show improved accuracy in 2-bit weight-only quantization and average task performance close to full precision at an average of 4.5 bits under mixed MXFP4/6 quantization.

Critical strengths

1 The method combines gradient-based sensitivity estimation and layer-wise bit allocation with lightweight scale initialization and loss filtering. These components are simple and use a small calibration dataset.

2 SignRoundV2 improves over SignRoundV1 on all models reported in Table 1 under the same 2-bit weight-only quantization setting, providing clear evidence of its effectiveness in this regime.

3 The reported results show competitive accuracy on large models at substantially lower quantization cost than the compared QAT and vector-quantization methods, while keeping the cost close to that of SignRoundV1.

Critical weaknesses

1 The paper would benefit from a clearer account of its new contributions. The method builds on SignRoundV1, uses an established approach for bit allocation, and draws on the importance matrix in llama.cpp for scale initialization. In particular, please clarify how DeltaLoss differs from prior sensitivity metrics and what advantages these differences bring.

2 Section 3.2 defines the discrete optimization objective and constraints, but states only that the problem can be solved by dynamic programming or integer linear programming. Please specify the solver used in the experiments and provide sufficient implementation detail or pseudocode to reproduce the bit assignments.

3 The ablation results need further discussion. In Table 4, V1+Filtering outperforms the full method on four tasks for Qwen3-8B. The ablation results need further discussion.

Questions for authors

No further questions for the authors.

3 comments
AuthorsOpenReview ↗

Thank you!

Thank you for taking the time to review our work. We hope you have a great weekend!

1. In particular, please clarify how DeltaLoss differs from prior sensitivity metrics and what advantages these differences bring.

We have added a new ablation study in Table 7 comparing DeltaLoss with LLM-mq, as well as an ablation study on Hessian information in Table 37. Both studies demonstrate the advantages of our method and provide further insights into the importance of first-order information, which has potentially been overlooked by the community.

2. Dynamic programming reproducibility

We have added pseudocode in Appendix J (line 1532) to clarify the dynamic programming procedure. The implementation is relatively straightforward, and we will open-source the complete code to ensure reproducibility.

3. In Table 4, V1+Filtering outperforms the full method on four tasks for Qwen3-8B.

Similar to how a quantized model can sometimes outperform its full-precision counterpart on individual tasks, such performance variations across tasks are expected. This is precisely why we evaluate our algorithm across multiple tasks rather than drawing conclusions from individual results. Therefore, we do not believe this observation requires further discussion.

Thank you for your response. The ablation results show that DeltaLoss performs better, but they do not explain why. Its formulation looks very similar to that of LLM-mq. Please explain how the two methods differ in principle and why these differences lead to better performance.

AuthorsOpenReview ↗

Thank you for the quick reply!

Thanks for the quick reply, especially over the weekend.

The major difference between our method and LLM-MQ lies in the point at which the gradients are computed.

LLM-MQ computes gradients with respect to the full-precision weights. This raises two theoretical concerns. First, when a model is well-trained and close to a local optimum, the gradients with respect to the full-precision weights are expected to be close to zero, making them less informative for sensitivity estimation. Second, the gradients computed using full-precision weights are shared across different bit-widths, and therefore cannot adequately reflect the loss associated with different quantization bit-widths.

In contrast, our method computes gradients with respect to the quantized weights, avoiding the above issues.

We have explained these differences in Lines 213–226. Please let us know if any part of our explanation is unclear or if you have further concerns.

Reviewer_XfZ4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SignRoundV2 extends SignRound in two ways:

  • It proposes DeltaLoss, a gradient-based sensitivity metric, and allocates layer-wise bit-widths via dynamic programming.
  • It adds lightweight tuning techniques, including a pre-tuning scale search and loss filtering.

The method is evaluated under both weight-only and weight-activation quantization. It shows strong results, particularly against existing mixed-precision methods.

Critical strengths
  1. It outperforms existing mixed-precision methods, especially at low bit-widths. Also, it achieves comparable performance to QAT and vector quantization (VQ) methods at much lower quantization cost.

  2. Fig. 2 supports allocating precision per-weight matrix rather than per-transformer block. Even within a block, down_proj is notably more sensitive than other weights.

Critical weaknesses

I currently lean toward rejection, mainly because (1) DeltaLoss closely resembles LLM-MQ (Li et al., 2023) and its design choices are not sufficiently ablated, and (2) its compatibility and comparision with rotation and Hessian-based rounding methods is only partially demonstrated.

  1. Novelty of DeltaLoss design. DeltaLoss is very similar to LLM-MQ. LLM-MQ also allocates bit-widths using a first-order sensitivity |gᵀ(W − Q_b(W))|, and integer programming.
  • Section 3.1 argues that Hessian-based metrics are unreliable, but no comparison is provided. Comparing first-order and second-order error terms would support this claim.
  • The element-wise L1 norm is motivated by sign cancellation, but it is not ablated against LLM-MQ’s absolute inner product. The L1 form also bounds the first-order error rather than estimating it.
  • The reason for computing gradients at the quantized weights instead of the full-precision weights is not empirically validated.

DeltaLoss differs from LLM-MQ mainly in L1 aggregation, gradient location, and the activation term. Each choice should be ablated against LLM-MQ to demonstrate the novelty of DeltaLoss.

  1. Compatibility with Hessian and rotation-based methods. The tuning stage of SignRoundV2 is an improved quantizer built on SignRoundV1. Its combination with rotation is tested only with Hadamard rotation at MXFP4 (Table 5). Its combination with Hessian-based rounding, such as LDLQ (Chee et al., 2023) or YAQA (Tseng et al., 2026), is not evaluated, and YAQA with INT quantizer is not compared.
  • Please compare Ours+H with MR-GPTQ in the MXFP4 format (Table 9).
  • In Table 5, Hadamard rotation improves RTN by 1.3–3.3 points but changes V2 by at most 0.2 points. Please explain this gap.
Questions for authors
  1. Could you compare DeltaLoss with Hessian-based metrics under the same quantizer and solver?

  2. Could you ablate DeltaLoss against LLM-MQ under the same quantizer and solver? Please vary the aggregation (L1 and absolute inner product) and gradient location (W_q and W_f).

  3. How much does Hadamard rotation improve SignRoundV2 under INT and MXFP4 quantization? Please compare V2+H with MR-GPTQ at MXFP4, and explain the small gain over V2 in Table 5.

  4. Could you demonstrate compatibility and compare with Hessian and rotation-based methods such as QuIP and YAQA?

6 comments
AuthorsOpenReview ↗

Initial Response

Thank you for your constructive comments, which have helped us further refine our manuscript. We have added an initial response in case you have any further questions. We apologize in advance for any typos.

Q1 Comparison with Hessian(Work In Progress)

We analyze the magnitudes of the first- and second-order contributions in Appendix K, showing that the magnitude of the first-order loss contribution is 1.4--21$\times$ larger than that of the second-order contribution. This suggests that first-order information cannot be safely ignored in practice. Although the initial accuracy results show only a slight improvement with the first-order term, we maintain that first-order information provides a more meaningful basis for sensitivity estimation from the perspective of contribution magnitude. Moreover, second-order information is computationally expensive to obtain and can be numerically unstable. For example, as observed in GPTQ, the Hessian estimated from a calibration dataset is not necessarily positive semidefinite. These considerations further motivate our choice of first-order information as the primary sensitivity metric.

Q2 DeltaLoss differs from LLM-MQ

LLM-MQ has a theoretical limitation, as explained in Lines 214–215: the gradients evaluated at full-precision weights are expected to be close to zero when the model is well-trained and near a local optimum. We have added new ablation studies in Table 7, where our method outperforms LLM-MQ in 8 out of 12 scenarios. To be honest, the performance of LLM-MQ exceeds our expectations. One possible explanation is that mixed-precision allocation is primarily a selection problem rather than a regression problem, and therefore does not require highly accurate loss estimation. Nevertheless, we maintain our position that using gradients at full-precision weights has a theoretical limitation, as discussed above.

Regarding the choice between element-wise L1 norm and absolute inner product, we provide an ablation study of both variants in Table 7. The results show that the L1 norm performs better on MXFP data types, while the absolute inner product has an advantage on INT data types. We acknowledge that the absolute inner product is more closely aligned with the original formulation. We have updated Lines 230–248 accordingly.

Q3 How much does Hadamard rotation improve SignRoundV2 under INT and MXFP4 quantization? Please compare V2+H with MR-GPTQ at MXFP4, and explain the small gain over V2 in Table 5.

We have added three data points to Table 11 to demonstrate the improvement of V2+H over H. Due to several factors, running these experiments is very time-consuming, so we do not expect to provide additional results during the rebuttal period.

The small gain over V2: Theoretically, activation loss can be compensated for through weight tuning. This possibility is further supported by recent progress in quantization-aware distillation and on-policy distillation, as demonstrated by approaches such as NVIDIA's Qwen3.8-27B-NVFP4. To be honest, although we observe only a small gain, we hypothesize that this is because a large portion of the evaluation tasks are relatively simple. Rotation may still be beneficial for smaller models and more challenging tasks.

We will not explore more sophisticated rotation methods in combination with our method, as the two approaches are orthogonal and combining them is not expected to hurt performance.

Q4 Could you demonstrate compatibility and compare with Hessian and rotation-based methods such as QuIP and YAQA?

QuIP, QuIP# is a successor to QuIP that has demonstrated better results than the original QuIP. The original manuscript already compares against QuIP# in Table 2. Although our method performs slightly worse than QuIP#, it is 100× faster, as shown in Table 9.

YAQA, we have conducted additional experiments in Appendix A. Under the 4-bit setting, our method achieves a higher recovery ratio than YAQA-A (99.53% vs. 98.98%) while requiring no rotation and 100× fewer calibration tokens (0.2M vs. 20M). In the ultra-low 2-bit regime, our uniform quantization (W2A16G64) yields a lower recovery ratio compared to YAQA-A and YAQA-B (89.32% vs. 91.75% and 92.12%). However, this gap is expected and acceptable considering the substantial differences in quantization constraints: YAQA relies on rotation transforms, 10× to 64× more calibration data (20M–128M tokens), and codebook-based vector quantization (QTIP), which inherently offers stronger representational capacity than standard uniform quantization at 2-bit precision.

Reviewer_XfZ4OpenReview ↗

Response to Authors’ Rebuttal

I thank the authors for the detailed response and the additional experiments. Comparing abs_dot and LLM-MQ, which differ in where the gradient is evaluated, abs_dot performs better on INT. I consider this a contribution. My questions on Hessian-based metrics are largely resolved by Appendix K. I also thank the authors for the YAQA comparison in Appendix A.

Justification for the L1 aggregation. The L1 aggregation, which distinguishes DeltaLoss from LLM-MQ, is not well supported by the results. On INT, abs_dot outperforms DeltaLoss, while on MXFP the advantage of DeltaLoss is within about one point. The sign-cancellation argument motivating the L1 form is not supported in these results. In addition, the accuracy of the first-order approximation is validated with the signed term gᵀδ, not with Eq. 7. The reason the preferred variant differs between data types is not explained, so the recommendation to choose the metric by data type remains empirical guidance. It would be helpful to provide a clearer rationale for keeping Eq. 7 as the default.

Limited evaluation on generative tasks. GSM8K is the only generative task in the main evaluation. However, in Table 7, DeltaLoss reaches 1.74% on GSM8K for Qwen3-14B, compared with 88.19% for BF16, while LLM-MQ and abs_dot reach 49.43% and 54.81%, respectively. This gap appears to come from DeltaLoss itself, and the failure is not visible in other metrics such as Avg.6. Therefore, the current evaluation is insufficient to assess the reliability of the method. The authors’ hypothesis that combining the method with rotation may help on harder tasks also cannot be verified with the current benchmarks. Evaluation on additional generative tasks, such as mathematical reasoning and code generation, is needed.

While I appreciate the extensive additional experiments, the concerns above remain. I therefore maintain my score.

AuthorsOpenReview ↗

Thank you!

Thanks for the quick reply, especially over the weekend. We have added a quick response here in case you have any further concerns!

1.1 Justification for L1 Aggregation

The major difference between our method and LLM-MQ lies in the gradient computation point: we compute gradients using the quantized weights rather than the full-precision weights. This is the main motivation behind using the second-order loss in prior work. I assume you would agree that, both theoretically and empirically, using quantized weights to compute gradients is more appropriate than using full-precision weights, particularly since you have not raised any further concerns regarding this point.

1.2 Regrading L1 aggregation

We agree that, as discussed in our previous response, the aggregation does not show a clear advantage across all settings. However, we still believe it has advantages for MXFP4. For MXFP4.5, since MXFP4 does not suffer from as much accuracy degradation as INT2, the potential gain is naturally smaller. As shown in Table 11, GPTQ improves the recovery ratio over RTN by only 0.4% at MXFP4, while SignRoundV2 achieves a 1.8% improvement at 4.5 bits as shown in Table.4. Therefore, achieving a 1% improvement for MXFP4 is already quite challenging.

1.3 In addition, the accuracy of the first-order approximation is validated with the signed term gᵀδ, not with Eq. 7

The results of ours in Table 7 all use Eq.7

1.4 The reason the preferred variant differs between data types is not explained

We agree that we currently lack a theoretical explanation for why the preferred variant differs across data types, and our recommendations are therefore based on empirical results. However, as discussed above, L1 aggregation is not the key component of our design.

2.1 low GSM8K on Qwen14B.

Yes, this surprised us as well. We are conducting additional experiments to better understand the reason. However, as shown in Table 3, no such issues occur when it is combined with SignRoundV2 or AWQ.

2.2 The authors’ hypothesis that cannot be verified with the current benchmarks. Evaluation on additional generative tasks, such as mathematical reasoning and code generation, is needed.

As we recently added Hadamard support on the vLLM side, the evaluation is now much faster. We have therefore added another result to Table 11: Ours+H achieves 67.75 on MMLU CoT, compared with 65.70 for Ours. We will also try to update the Qwen3-8B results in Table 11.

However, I am still a little unsure why we need to conduct this experiment. My understanding is that comparing combinations of different algorithms is not the main objective here. Shouldn't the focus be on comparing the individual algorithms under consistent settings, rather than their combinations

Reviewer_3inxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SignRoundV2, a post-training quantization (PTQ) framework designed to improve the performance of large language models under extremely low-bit quantization.

The main contribution is DeltaLoss, a sensitivity metric that combines gradient information with quantization-induced changes in weights and activations. The method uses this metric and dynamic programming to assign different bit-widths to layers under a fixed precision budget. The authors also introduce lightweight techniques, including scale initialization and loss filtering, to improve quantization accuracy.

Experiments across LLaMA and Qwen models show that SignRoundV2 achieves competitive accuracy in both weight-only and weight-activation quantization. The method also demonstrates relatively low quantization costs compared with several computationally expensive baselines.

Critical strengths

1. Strong experimental results across different models and quantization settings.

The paper provides extensive evaluations across LLaMA and Qwen models, covering multiple model sizes, precision levels, and quantization formats. The results are generally competitive, particularly in challenging low-bit settings.

The controlled comparisons using DL+AWQ in Table 3 are especially useful because they help isolate the contribution of DeltaLoss-based bit allocation from the underlying quantization method. These results provide meaningful evidence for the effectiveness of the proposed sensitivity metric.

2. Practical efficiency.

SignRoundV2 achieves competitive performance with relatively low quantization costs. For example, the reported quantization time of 2.5 GPU-hours for Llama2-70B is substantially lower than that of several computationally expensive alternatives.

This efficiency is an important practical advantage, particularly for large models where quantization time and memory requirements can limit deployment.

Critical weaknesses

1. Limited methodological novelty.

Sensitivity-based mixed-precision quantization has been widely studied in prior work, including HAWQ-V2, BMPQ, and MixLLM. First-order gradient information has also been explored for quantization error compensation.

Although DeltaLoss combines gradient information with quantization-induced perturbations, the paper does not clearly establish why this particular formulation is more effective than simpler alternatives.

For example, comparisons with gradient-only sensitivity, quantization-error-based sensitivity, and Hessian-based methods would help clarify the benefit of combining these signals.

As a result, the main methodological contribution appears incremental, despite the strong empirical results.

2. Insufficient justification of design choices and incomplete practical comparisons.

Several important design choices require further validation.

First, the loss filtering strategy removes the largest 0.1% of reconstruction losses during tuning. This threshold appears heuristic, and the paper does not sufficiently explain why it is appropriate. Although an ablation study is provided, experiments with different filtering ratios and comparisons with alternatives such as Huber loss or gradient clipping would strengthen the justification.

Second, the dynamic programming algorithm selects bit-widths from a predefined set of candidates. While this is a reasonable design choice, it limits the flexibility of the framework. The effects of different candidate precisions, quantization formats, and group sizes are not fully explored under practical memory and latency constraints.

Finally, the comparisons with efficient block-wise PTQ methods could be more comprehensive. CBQ is a particularly relevant missing baseline because it also targets low-bit LLM quantization through reconstruction-based optimization. Although OmniQuant is included in the accuracy comparisons, a more systematic evaluation of accuracy, quantization time, and peak memory usage under matched settings would help establish the practical advantages of SignRoundV2.

These limitations make it difficult to fully assess the significance and generality of the proposed improvements.

Questions for authors

1. Novelty and effectiveness of DeltaLoss.

Could the authors provide a more direct comparison between DeltaLoss and simpler sensitivity metrics, such as gradient magnitude alone, quantization reconstruction error alone, and Hessian-based sensitivity?

Such an ablation would help determine whether combining gradient information and quantization perturbations provides a clear advantage over existing approaches.

2. Sensitivity of loss filtering.

How sensitive is the performance to the 0.1% loss filtering threshold?

Could the authors evaluate different filtering ratios and compare the proposed method with robust alternatives such as Huber loss or gradient clipping? This would help clarify whether the improvement comes from the specific filtering strategy or from general robustness to high-loss elements.

3. Practical efficiency compared with block-wise PTQ methods.

Could the authors provide a more systematic comparison with efficient block-wise methods such as CBQ and OmniQuant, using matched model sizes, precision levels, group sizes, and hardware settings?

In particular, comparisons of accuracy, quantization time, and peak GPU memory usage would help establish the practical significance of SignRoundV2.

1 comment
AuthorsOpenReview ↗

Summary of Updates

Thank you for your review. Since we have not yet received your response, we have further revised our rebuttal based on the additional experiments and clarifications provided during the rebuttal period. We hope these updates address some of your concerns.

L1. Prior work on sensitivity-based mixed-precision quantization has been widely studied.

First- and second-order contributions. We analyze the magnitudes of the first- and second-order contributions in Appendix K, showing that the first-order loss contribution is 1.4–21× larger than the second-order contribution. This suggests that first-order information cannot be safely ignored in practice. We also add an ablation study in Table 38 to demonstrate the advantages of using first-order information over second-order information.

First-order gradient evaluation. We add new experiments in Tables 7 and 35 to compare our method with LLM-MQ and revise Lines 214–222 to better clarify the key difference between the two methods: the point at which gradients are evaluated.

LLM-MQ computes gradients with respect to the full-precision weights. This raises two theoretical concerns. First, when a model is well-trained and close to a local optimum, the gradients at the full-precision weights are expected to be close to zero, making them less informative for sensitivity estimation. Second, these gradients are shared across different bit-widths and therefore cannot adequately capture the loss associated with different quantization bit-widths.

In contrast, our method evaluates gradients at the quantized weights, providing more informative sensitivity estimates that account for the effects of quantization.

L2.1. The paper does not sufficiently explain why loss filtering is appropriate.

We agree that the original manuscript lacked sufficient theoretical justification for this design choice. We have clarified that loss filtering is motivated by balancing easy and hard samples during sensitivity estimation.

We have also added an ablation study on loss filtering in Table 4. For both Llama and Qwen3, loss filtering consistently outperforms the corresponding baselines without filtering across thresholds ranging from 1e-2 to 1e-5. Our choice of 1e-3 provides a good trade-off among these settings.

L2.3. Comparison with CBQ and OmniQuant

In the original manuscript, we compared our method with OmniQuant in Table 2 across five models ranging from 8B to 70B parameters. We also included comparisons with AQLM, QuIP#, and EQAT. Although we did not directly compare with CBQ for the reasons explained in our previous response, we have cited the relevant paper.

Additionally, we compared our method with YAQA, which reports better results than CBQ in Table 5 of the YAQA paper. We have also conducted additional experiments in Appendix A. Under the 4-bit setting, our method achieves a higher recovery ratio than YAQA-A (99.53% vs. 98.98%), while requiring no rotation and 100× fewer calibration tokens (0.2M vs. 20M). In the ultra-low 2-bit regime, our uniform quantization method (W2A16G64) achieves a lower recovery ratio than YAQA-A and YAQA-B (89.32% vs. 91.75% and 92.12%, respectively). However, this gap is expected given the substantial differences in quantization constraints. YAQA relies on rotation transforms, 10×–64× more calibration data (20M–128M tokens), and codebook-based vector quantization (QTIP), which provides greater representational capacity than standard uniform quantization at 2-bit precision.

We sincerely thank you for your time and thoughtful review. We hope that the additional experiments and revisions address your concerns and make the contributions of our work clearer. We greatly appreciate your constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.