acceptodds
Under review as a conference paper at ICLR 2027

RoQAT: A Robust Quantization-aware Training Method for LLMs Extremely Low-bit Downstream Reasoning

Abstract

Quantization-aware training (QAT) can largely recover the capabilities of low-bit large language models (LLMs). However, we discover that although achieving nearly lossless performance on general benchmarks, there still exists a tremendous gap on downstream reasoning capabilities at extremely low-bit, especially at 2-bit. We argue that this degradation is closely related to two challenges in extremely low-bit QAT: 1) directly perform low-bit training will cause severe initial degradation. Although progressive quantization can partially stabilize training, existing methods typically ignore sensitivity differences during progression; 2) the weights may converge to the sharp regions of the loss landscape, which will limit the robustness and generalization of the quantized model. Motivated by these observations, we propose RoQAT, a robust QAT framework for preserving downstream capabilities of extremely low-bit LLMs. RoQAT first introduces sensitivity-adaptive progressive quantization (SAPQ), which considers distinct quantization sensitivity across groups. Specifically, RoQAT dynamically assigns each group a different progression rate to the target bit-width according to the block quantization sensitivity along the actual progressive training direction, which allows more sensitive groups to approach the target bit-width more conservatively. Moreover, RoQAT incorporates quantization-aware sharpness minimization (QSAM) to establish flatter loss landscapes around low-precision weights, which further enhances downstream generalization. It performs sharpness-aware optimization in the weight space during block-wise training and the scale space during end-to-end refinement. Extensive experiments on Qwen3 instruction-tuned and Qwen3.5 hybrid-attention LLMs demonstrate that RoQAT consistently outperforms strong QAT baselines on downstream benchmarks while remaining stable general performance at extremely low-bit.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model compression, Quantization-aware training, Downstream reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bMWBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the degradation of downstream capabilities when instruction-tuned language models undergo extremely low-bit quantization. Building on EfficientQAT, RoQAT introduces sensitivity-adaptive progressive quantization (SAPQ), which adjusts each weight group’s transition toward the target bit-width. It also applies quantization-aware sharpness minimization (QSAM) to weights during block-wise training and to quantization scales during end-to-end refinement. Experiments cover three Qwen-family models at 2 and 3 bits, reporting improved average performance on mathematical reasoning, coding, and instruction following. Validation includes comparisons with progressive methods, component ablations, and matched-data experiments.

Critical strengths
  1. The paper addresses the practically relevant gap between recovering general benchmark performance and preserving downstream capabilities. Evaluating mathematical reasoning, coding, and instruction following alongside general benchmarks provides a useful assessment of quantized models.
  2. The paper provides controlled evidence for SAPQ. Table 2 uses the same training data and block-wise setting while excluding QSAM, and reports improvements over other progressive methods. The matched-data comparison in Table 5 also favors RoQAT over EfficientQAT on both models, supporting a benefit from the optimization strategy.
Critical weaknesses
  1. The attribution of the headline improvements is insufficiently explained. On Qwen3-4B, the downstream-average gain over EfficientQAT decreases from 16.15 percentage points in Table 1 to 1.47 points under matched training data in Table 5. The latter remains a positive result, but the configuration differences between these experiments are unclear, making it difficult to attribute the larger headline improvement to the proposed algorithm.

  2. Evidence for the complementary benefits of the two QSAM spaces remains limited. The component ablation in Table 3 covers one model and two tasks, and adding scale-space QSAM reduces HumanEval performance. Although the complete method performs best, the corresponding weight-space-only configuration and variability across independent runs are not reported. The necessity of using both spaces and the stability of their additional gains therefore remain uncertain.

Questions for authors
  1. What changes between Tables 1 and 5 in training data, data mixtures, and training budgets for each method? Please explain the substantial change in EfficientQAT’s performance to help distinguish the contributions of training configurations and the proposed algorithm.

  2. Under the same settings as Table 3, how does SAPQ combined with only weight-space QSAM perform? Can the authors provide repeated-run results or uncertainty estimates for the key configurations to assess the additional benefit and stability of the two-space design?

  3. The Qwen3-4B downstream average is 37.33 in Table 2 but 34.01 under the default settings in Tables 6–7. Do these results use the same data and training configurations? Please clarify any differences or explain the source of the discrepancy.

  4. How are the interpolated weights used during progressive training converted into the final target-bit representation? Please specify when and how this conversion occurs and confirm that all reported evaluations use the final target-bit weights.

Reviewer_RJNcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RoQAT, a quantization-aware training framework designed to preserve downstream reasoning capabilities of extremely low-bit LLMs. It introduces sensitivity-adaptive progressive quantization (SAPQ), which dynamically adjusts group-wise quantization progression based on directional sensitivity, and quantization-aware sharpness minimization (QSAM), which applies sharpness-aware optimization to weights during block-wise training and quantization scales during end-to-end refinement. Experiments on Qwen3 and Qwen3.5 models under 2-bit and 3-bit quantization demonstrate improved performance on mathematical reasoning, instruction following, and code generation compared with existing QAT baselines, while maintaining comparable general benchmark performance.

Critical strengths
  1. Strong empirical improvements on downstream reasoning tasks. RoQAT demonstrates substantial performance gains over existing QAT baselines, particularly under challenging 2-bit quantization. The improvements are consistently observed across Qwen3 and Qwen3.5 models on mathematical reasoning, instruction following, and code generation tasks, while maintaining comparable performance on general benchmarks. These results highlight the practical value of preserving downstream capabilities during extremely low-bit QAT.

  2. Experimental evaluation and ablation studies. The paper evaluates several model sizes and bit-width settings, compares against several recent QAT and progressive quantization methods, and provides ablation studies examining the contributions of SAPQ and QSAM. These experiments offer useful empirical evidence for the effectiveness of the proposed framework, although they do not fully establish the novelty of its individual components.

Critical weaknesses

1. Limited Methodological Novelty. The two main components of RoQAT appear to be incremental adaptations of existing techniques rather than fundamentally new methodological contributions.

For SAPQ, sensitivity-aware quantization and progressive training have been extensively explored in prior work, including INQ [1], HAWQ [2], and sensitivity-guided mixed-precision quantization with multi-stage fine-tuning [3]. More directly, HESTIA [4] already employs Hessian-guided adaptive quantization progression for extremely low-bit LLMs.

For QSAM, sharpness-aware optimization is well established by SAM [5] and ASAM [6]. More importantly, SAQ [7] and SQuAT [8] have already investigated sharpness-aware optimization for low-bit quantization.

Although RoQAT introduces group-wise online directional sensitivity and applies sharpness-aware optimization in both weight and scale spaces, these modifications are integrated into the existing two-stage EfficientQAT framework [9]. The paper does not sufficiently demonstrate that these modifications provide substantial benefits beyond straightforward adaptations and combinations of prior techniques. Consequently, the methodological novelty remains limited and insufficiently substantiated.

2. Insufficient methodological justification of SAPQ. The proposed first-order sensitivity metric (Eq. 5) may not reliably capture the actual loss increase caused by quantization, particularly near stationary points where second-order effects can dominate. More importantly, the sensitivity-dependent progression rate and EMA updates (Eqs. 7–8) do not guarantee that all quantization groups eventually reach the target bit-width ((r_g=1)). The paper does not adequately clarify how the final transition to fully quantized weights is performed or whether this transition introduces additional performance degradation. These issues raise concerns about the reliability of the sensitivity estimation and the consistency between progressive training and final low-bit deployment.

3. Insufficient Experimental Diversity and Limited Evaluation at More Aggressive Bit-widths.

The experimental evaluation is restricted to the Qwen model family (Qwen3-4B/8B and Qwen3.5-4B). Although Qwen3.5 introduces architectural differences, the paper would benefit from experiments on other widely adopted LLM families, such as LLaMA and Mistral, to demonstrate the generalizability of the proposed SAPQ and QSAM methods across different architectures.

Furthermore, the experiments only consider 2-bit and 3-bit quantization, despite the paper's emphasis on extremely low-bit LLM quantization. Recent studies have already explored 1-bit QAT and even sub-1-bit LLM compression, including NanoQuant [10], BTC-LLM [11], and LittleBit-2 [12]. While these approaches employ different quantization and compression mechanisms and are not necessarily directly comparable, evaluating RoQAT under more aggressive settings, particularly 1-bit quantization, would provide stronger evidence of its robustness and applicability to extreme compression. At minimum, the authors should discuss whether SAPQ and QSAM can be extended to these settings and clarify their limitations.

References

  1. Zhou et al. (2017). Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights. ICLR 2017.

  2. Dong et al. (2019). HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision. ICCV 2019.

  3. Liu, B., Wang, H., and Qian, Y. (2024). Towards Lightweight Speaker Verification via Adaptive Neural Network Quantization. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32, 3771–3784.

  4. Wang et al. (2026). HESTIA: A Hessian-Guided Differentiable Quantization-Aware Training Framework for Extremely Low-Bit LLMs. arXiv:2601.20745.

  5. Foret et al. (2021). Sharpness-Aware Minimization for Efficiently Improving Generalization. ICLR 2021.

  6. Kwon et al. (2021). ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks. ICML 2021.

  7. Liu, J., Cai, J., and Zhuang, B. (2021). Sharpness-aware Quantization for Deep Neural Networks. arXiv:2111.12273.

  8. Wang et al. (2022). SQuAT: Sharpness- and Quantization-Aware Training for BERT. arXiv:2210.07171.

  9. Chen et al. (2025). EfficientQAT: Efficient Quantization-Aware Training for Large Language Models. ACL 2025.

  10. Chong et al. (2026). NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models. ICML 2026.

  11. Gu et al. (2026). BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebook. ACL 2026.

  12. Lee and Kim (2026). LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment. ICML 2026.

Questions for authors

1. Methodological Novelty. What are the fundamental methodological differences between SAPQ and existing sensitivity-guided progressive quantization methods [1-3], and between QSAM and previous sharpness-aware quantization methods? Could the authors provide more comparisons to [1-3] and discussion in related works section?

2. Convergence to the Target Bit-width. According to Eqs. (7)–(8), the sensitivity-adaptive progression rate rgr_g is updated using an EMA, but these equations do not guarantee that rgr_g eventually reaches 1 for every quantization group. How do the authors ensure that all groups reach the target 2-bit or 3-bit precision at the end of training?

3. Reliability of the Sensitivity Metric. SAPQ estimates group-wise quantization sensitivity using the first-order approximation in Eq. (5). However, this approximation may become unreliable near stationary points, where first-order gradients approach zero while second-order effects remain significant. How do the authors justify using this metric rather than Hessian-based sensitivity measures, as explored in HAWQ [2] and HESTIA [4]?

4. Generalizability and Quantization Scope. The experimental evaluation is limited to the Qwen model family. What evidence supports the generalizability of SAPQ and QSAM to other LLM architectures, such as LLaMA and Mistral? Moreover, given the paper's emphasis on extremely low-bit quantization and recent developments in 1-bit and sub-1-bit LLM compression [10–12], can the proposed framework be extended to 1-bit quantization?

Reviewer_AgQ5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Low-bit quantization-aware training can recover general benchmark scores while still losing reasoning and coding abilities. RoQAT addresses this through sensitivity-adaptive progressive quantization and sharpness-aware training in both weight and quantization-scale spaces. Sensitive weight groups move toward the target precision more slowly, while sharpness-aware updates aim to improve robustness. The authors evaluate Qwen3-4B/8B and Qwen3.5-4B at 2 and 3 bits on mathematics, coding, instruction following, and general benchmarks. RoQAT achieves higher average downstream scores than the tested baselines, particularly at 2 bits, while maintaining competitive general performance.

Critical strengths
  1. The paper studies an important gap between general benchmark recovery and downstream ability after quantization. The evaluation includes several types of downstream tasks, and some improvements are substantial.
  2. The paper includes comparisons with recent progressive QAT methods, matched-data experiments, and component ablations. These provide useful evidence that the training changes help.
Critical weaknesses
  1. The matched-data results suggest that the main table overstates the improvement due to the method alone. On Qwen3-4B at 2 bits, EfficientQAT improves from 24.15 in Table 1 to 38.83 in Table 5, while RoQAT remains at 40.30. The gap to the strongest baseline therefore shrinks from 7.76 to 1.47 points. The paper should explain exactly which settings differ between these tables and give more weight to the controlled comparison.
  2. The baseline training procedures need clearer documentation. In particular, it is unclear how much of each method’s original pipeline is retained when using the shared setup. The progressive-method comparison also places HESTIA in a block-wise 2-bit setting, which differs from its original ternary training setup. This is useful as a controlled adaptation, but does not establish superiority over the full original method.
  3. The ablation starting point is unclear. With all proposed components disabled, Table 3 reports 55.42 on GSM8K, while the EfficientQAT baseline in Table 1 reaches only 36.92 under the same model and bit width. This difference needs explanation.
Questions for authors

See weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.