acceptodds
Under review as a conference paper at ICLR 2027

Data-Adaptive Transforms Can Improve the Accuracy of Quantization-Aware Training

Abstract

Performing large model training in lower precision, a process known as quantization-aware training (QAT), has become a standard approach for reducing costs, both for inference (since it produces a smaller model) and training time (since the matrix multiplications can be performed faster). However, training models accurately in lower precision is still a challenge. In this work, we propose a new, more accurate approach for QAT, based on the idea of adaptively learning a set of “smoothing” matrices for each layer's matrix multiplication, whose goal is to specifically reduce its quantization error. We show for the first time that this adaptation can be performed efficiently and stably during training, leading to significant reductions in the accuracy gap for QAT for pre-training, fine-tuning, and distillation of quantized large language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, QAT, quantization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oWeTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TWIST, which adapts WUSH transforms to quantization-aware training using periodic updates, late freezing, and randomized backward transforms. It reports improved accuracy in pretraining, distillation, and fine-tuning, especially at 3 bits.

Critical strengths
  1. Useful methodological contribution: TWIST addresses concrete obstacles to adaptive transforms during training: changing statistics, unstable quantization bases, and biased backward estimates.
  2. Broad empirical evidence: Improvements span several training settings. At 3B parameters, TWIST closes approximately 37% and 44% of the Hadamard-to-full-precision loss gap at 4 and 3 bits, respectively.
  3. Convincing diagnostics: The basis-shift experiments explain why reducing reconstruction error can worsen training loss. Freezing ablations support the proposed remedy.
Critical weaknesses
  1. Practical efficiency remains incompletely demonstrated: Transform refreshes add roughly 5–12% iteration overhead before freezing. Kernel-level inference benchmarks do not establish an end-to-end training advantage under equal wall-clock budgets.
  2. Statistical support for smaller gains is limited: Several downstream improvements are modest, particularly at 4 bits. Repeated training runs and confidence intervals would clarify whether the reported differences are reliable.
Questions for authors
  1. Under equal total training time, does TWIST still outperform Hadamard after accounting for calibration, statistics collection, and transform updates?
  2. Do the 2.0–2.2 percentage-point MATH-500 gains persist across training seeds and repeated decoding, with paired confidence intervals?
1 comment

Thank you for your time and support! We address concerns and questions below:

W1 & Q1: Efficiency at equal training time. Since a similar question was raised by Reviewer 9ybh, we address whether TWIST still outperforms Hadamard under equal total training time in the general response. TWIST retains its advantage under this comparison: with an update interval of τ=100\tau=100, the rebuild overhead drops to 0.7–1.9%, and TWIST outperforms Hadamard in all six settings (200M–800M, INT4 and INT3), e.g. 2.559 against 2.585 at 800M INT3. We would also like to clarify the role of calibration. Unlike in QAD and fine-tuning, TWIST has no separate calibration pass in pretraining: the statistics for each rebuild are collected online, from the training batch of the preceding step, so their cost is already included in the iteration times in Table 5 and in the equal-time comparison.

W2 & Q2: Statistical support.

The MATH-500 gains persist: we have rerun the 4B INT3 setting with 3 training runs and 5 decoding samples per checkpoint; please see Tables B and C in the general response. TWIST reaches 55.4 ± 0.6 against 53.2 ± 0.7 for Hadamard, a gain of +2.2 points (95% confidence interval [+0.8, +3.6]).

We agree that the gains at 4 bits are modest. In QAD, we do not claim INT4 downstream gains: Section 5 states that the INT4 differences between Hadamard and TWIST are within evaluation noise. We will state the same for INT4 fine-tuning and base our downstream claims on INT3.

Additionally, we show that the QAD KL improvements are statistically significant in all quantization settings considered for Qwen3-4B-Thinking-2507. For each method and setting, we train 3 independent runs and evaluate each final checkpoint on the full held-out split of 13,588 sequences, rather than the 128 used in the paper.

Table A. KL to the FP teacher (mean ± SD over 3 runs).

SettingIdentityHadamardTWIST
W-only INT40.0276 ± 0.00010.0233 ± 0.00020.0195 ± 0.0001
W-only INT30.1018 ± 0.00020.0920 ± 0.00020.0804 ± 0.0001
W+A INT40.1052 ± 0.00010.0557 ± 0.00040.0430 ± 0.0000
W+A INT30.3615 ± 0.00100.2291 ± 0.00060.1965 ± 0.0002

Table B. Relative KL change of TWIST versus Hadamard (negative means closer to the teacher). The 95% confidence intervals come from a paired bootstrap that resamples both training runs and evaluation sequences.

SettingTWIST vs Hadamard95% CI
W-only INT4−16.5%[−17.2, −15.8]
W-only INT3−12.6%[−12.9, −12.4]
W+A INT4−22.7%[−23.3, −22.2]
W+A INT3−14.2%[−14.5, −13.8]
Reviewer_iasgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the author propose TWIST, a new framework that incorporate WUSH transformation method in PTQ to QAT. It has 3 varients: 1. Frozen-TWIST, that just compute transformation matrix WUSH once then freezed during training, 2. Online-TWIST, that update WUSH matrices periodically during training, and 3. Stochastic-Twist, that combine WUSH matrix with Hadamard matrix for backward computation. Experiments show improvements over fixed Hadamard transformation matrix for training, distillation, fine-tuning.

Critical strengths
  1. Good empirical results, the result show consistent improvement across model size and settings
  2. The method provide insights why recomputing WUSH matrix can sometimes hurt QAT optimization, and has theoretical analysis.
Critical weaknesses
  1. Limited technical novelty. The method mainly reuse WUSH matrices, adding periodic updates on top and a backward varient. And the idea of adaptive transformation for QAT is not new (e.g. RoSTE), even if there are a few differences, make it feel a bit incremental.

  2. The insight may not be too strong. As I know, WUSH is designed to minimize only local MSE error per layer, it is not surprising why update WUSH hurt global objective. How can we know if the explanation of author about mismatch in quantization basis isn't just mismatch between local objective and global objective?

  3. Lack of comparison and baseline The experiments mostly compare with using similar setting but using Identity and hadamard matrices, (.e.g. the paper mention RoSTE but does not compare with it), or we can bring many transformation matrices designed for PTQ to QAT as baseline. Is the method WUSH-specific

Questions for authors

Please answer concern in Weakness

4 comments

Thank you for your time and feedback! We address your concerns below.

W1: Novelty.

We agree that TWIST builds on WUSH. Our contribution is what it takes to use an adaptive transform during training, where applying WUSH directly fails in two ways:

1. Forward pass. A WUSH transform recomputed during pretraining keeps a lower layer error than Hadamard throughout (Figure 6), yet the run falls behind Hadamard near the end of training (Figure 8). We trace this to basis shift (Table 6, Figure 9) and fix it by freezing the transform, which keeps the lead to the end.

2. Backward pass. A deterministic WUSH transform biases the gradient estimate and degrades as the update interval grows (Figure 3a). STOCHASTIC-TWIST keeps the data-dependent factor and redraws the Hadamard signs at every step, so the estimator stays unbiased.

We also give a QAT analysis that links the WUSH objective to the asymptotic QAT loss (Theorems E.8 and E.10), and a distributed rebuild (Section 4.1).

RoSTE applies rotations during quantization-aware fine-tuning and selects each one among random Walsh–Hadamard matrices; the rotations are orthogonal and not fitted to the layer statistics. TWIST fits learned non-orthogonal transforms to the weight and activation statistics, and covers pretraining from scratch and the quantized backward pass in addition to fine-tuning.

W2: Is the basis shift more than the mismatch between a local and a global objective?

We agree that minimizing a layer-local error need not reduce the training loss. The purpose of Section 4 is to identify the mechanism behind this in QAT, that is, how and why a recomputed transform can raise the loss although it lowers the local error. Section 4 attributes this to the basis shift, and supports it with the flip rate of the quantized weights, the distance between consecutive transforms and the freezing ablations. Following the reviewer's question, we complement this argument with a direct measurement of each quantity it involves (new Appendix G).

A recomputation of the transform has two consequences for a linear layer. First, it slightly reduces the norm of the layer's output error (Table 15). Second, it changes the quantization grid: consecutive WUSH transforms are almost unrelated (Section 4.1), and as a result the layer's output errors before and after a recomputation are nearly orthogonal at every stage of training (Table 15). The compensation by later layers discussed in Section 4 can then be quantified as the change of the loss when this error is modified.

This gives an exact decomposition of the loss change caused by a recomputation into a contribution of the norm and a contribution of the direction of the error (Table 15). The direct loss change is carried by the direction, and it grows over training, from 0.3 mNat at 8% of training to 13 mNat at 71% when all transforms are replaced (from −0.01 to 0.47 mNat for the direction term of a single layer), and the accompanying reduction of the error fails to compensate for it. This growth reflects a general property of a converging network, which becomes more sensitive to equally large perturbations of its current layer outputs. This holds for networks trained with TWIST and with Hadamard (Figures 14 and 15) and is not specific to quantization: in a full-precision model, the penalty for a random perturbation doubles over training (Figure 15). Recomputation thus becomes increasingly costly, which is why TWIST freezes the transform. Once it is frozen, the network keeps adjusting to it: replacing the transforms at the end of training would cost 52 mNat.

In summary, the local objective and the training loss diverge in QAT through a specific and measurable quantity, the direction of the layer output error. We would be glad to add further controls if the reviewer has a particular alternative explanation in mind.

W3: Choice of baselines and comparison with RoSTE.

We agree that more baselines make the comparison more informative. However, we chose Hadamard as the main baseline because it is a strong and widely adopted choice: it is the transform used in current QAT recipes, such as QuEST [1] and BitNet v2 [2], and in randomized form in Quartet II [3] and the NVFP4 recipe of NVIDIA [4]. For fine-tuning, we also compare against full-precision fine-tuning followed by WUSH PTQ (Table 10).

Following the reviewer's suggestion, we have also added RoSTE as a baseline in its original setting, QAT fine-tuning; please see Tables B and C in the general response. In the 4B INT3 setting, with the same data, quantization configuration and training budget, RoSTE reaches 52.8 ± 0.7 on MATH-500, on par with Hadamard (53.2 ± 0.7), while TWIST reaches 55.4 ± 0.6, a gain of +2.6 points over RoSTE (95% confidence interval [+1.1, +4.2]).

W3: Is TWIST specific to WUSH?

The periodic update scheme itself is not tied to WUSH, and other data-dependent transforms could be recomputed in the same way. We choose WUSH because it is the closed-form minimizer of the blockwise quantization error (Theorem E.3), and this is the quantity that matters for training: the quantization term of the QAT objective is exactly the WUSH objective (Section E.5), and it bounds the asymptotic loss floor of SGD with a fixed transform (Theorem E.8). The advantage of TWIST also grows with the block size, while Hadamard stays flat (Figure 2c), which indicates that the gains come from adapting to the data.

Other transforms can be plugged in as well, but this is not always straightforward. For example, the quantized backward pass requires randomization to keep the gradient estimates unbiased (Figure 3a). A generic data-adaptive transform has no natural place for it, whereas the WUSH transform factorizes into a data-dependent factor and a Hadamard matrix. This lets us reuse the solution of Quartet II, the state-of-the-art recipe for low-precision backward passes, and randomize the Hadamard factor with fresh random signs at every step (Section 4.3).

Although the RoSTE experiments select the rotation once, its Algorithm 1 also allows a periodic re-selection. We therefore checked how RoSTE behaves within our periodic update framework: we ran it in our pretraining setup at 200M W3A3, with the same sweep over the update interval and the freeze point as in Figure 1. Following RoSTE, each block (or each layer) chooses between the identity and the Hadamard transform, whichever gives the lower quantization error. We present our results in Figure 13 (Appendix F.3) and summarize them below:

MethodVal. loss
Hadamard2.918
RoSTE, per block, best configuration2.917
RoSTE, per layer, best configuration2.921
TWIST2.908

RoSTE does not improve over Hadamard: even its best configuration only matches Hadamard, and its results are unstable with respect to the update interval and the freeze point. TWIST improves over Hadamard by 0.010 in the same setting.

If the reviewer has specific PTQ transforms in mind that they would like to see within the periodic update framework, we would be happy to run those comparisons.

[1] Panferov et al. QuEST: Stable Training of LLMs with 1-Bit Weights and Activations. ICML 2025. [2] Wang et al. BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs. https://arxiv.org/abs/2504.18415 [3] Panferov et al. Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation. ICML 2026. [4] NVIDIA. Pretraining Large Language Models with NVFP4. https://arxiv.org/abs/2509.25149

Thank you again for your feedback. As the discussion period is ending soon, we wanted to check whether our response addressed your concerns.

In the response and revision, we discuss the novel techniques we introduced to adapt transforms to quantized training (fixing the basis shift in the forward pass by freezing the transform, and keeping the backward pass unbiased), supplement the evidence for basis shift in Appendix C2 with a new Appendix G, and add RoSTE as a baseline, which we find performs on par with always using Hadamard transforms.

If this resolves your reservations, we'd be grateful if you would consider updating your score. If anything remains unclear, we're happy to clarify.

Thank you for the rebuttal and additional experiments. I'm still evaluating whether the new results sufficiently address my concerns and will finalize my assessment during reviewer-ac discussion later

Reviewer_9ybhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TWIST, which adapts pre-quantization representations based on the statistical properties of weights and activations to reduce quantization error, while addressing the challenges posed by this adaptation mechanism in terms of training stability and computational overhead. Experiments across multiple training settings show that TWIST can balance quantization error and training stability, narrowing the performance gap between low-precision and full-precision models, particularly under 3-bit weight-and-activation quantization.

Critical strengths
  1. Pretraining improvements are consistent across model scales. In Table 1, all eight settings spanning 200M–3B parameters and INT4/INT3 outperform Hadamard. At 3B, the W4A4 loss decreases from 2.275 to 2.264, while the W3A3 loss decreases from 2.350 to 2.304. The table reports that the quantization loss gaps relative to full precision are reduced by approximately 37% and 44%, respectively. Note that these percentages represent reductions in the loss gap, not accuracy improvements.
  2. The observation regarding basis shifts is valuable. The authors find that although recomputing the transforms can reduce local quantization error, it may disrupt convergence in later stages of training, making it necessary to freeze the transforms. This reveals that lower reconstruction error does not necessarily lead to lower training loss. The authors support this explanation with ablations on refresh intervals, freezing schedules, and changes in the quantization grid. This contribution goes beyond directly applying a PTQ method to QAT.
  3. The INT3 joint quantization setting has strong downstream evidence. On the nine-task average score in QAD, the 4B model improves from 41.5 to 48.4, while the 32B model improves from 67.4 to 70.8. KL divergence and token agreement also improve consistently. These results are more convincing than the relatively small fluctuations observed under INT4 quantization.
Critical weaknesses
  1. The main INT3 comparison simultaneously changes the transform and the quantization grouping strategy. Under the INT3 quantization configuration, the Hadamard configuration with group size 64 is unstable, so the authors use its better-performing per-row scaling configuration, whereas TWIST uses group size 64. As a result, the effects of the transform itself and the scale granularity cannot be fully disentangled. The metadata overhead also differs between the two configurations. Ideally, the authors should provide complete comparisons between Hadamard and TWIST under both per-row and group-64 scaling, including failed configurations, and evaluate them under the same effective storage budget.
  2. Although overhead is measured, the actual time advantage of low-bit training has not been demonstrated.In Appendix C, Table 4, reports an iteration-time overhead of 4.7%–12.5% during the refresh stage. However, the section describes forward and backward passes using BF16 mixed precision and does not report throughput for the complete training pipeline using actual low-bit kernels, total GPU-hours, or the time required to reach the same loss. The 4.9–5.7× speedup reported in Appendix B is based on a single-layer MXFP4 inference microbenchmark and cannot directly establish training acceleration for the INT3 configuration. The current evidence provides reasonably strong support for better accuracy under the same token budget, but is insufficient to demonstrate better performance under the same compute-time budget.
  3. Some downstream gains are small and lack sufficient statistical support. On MATH-500, the most prominent SFT gains are 2.0 and 2.2 percentage points, corresponding to net differences of 10 and 11 problems, respectively. The evaluation uses one sample per problem with a temperature of 0.6. Without repeated decoding, repeated training runs, or paired confidence intervals, it is difficult to determine whether these improvements are robust.
Questions for authors

The QAD/SFT experiments use a WUSH transform that is fixed after a single calibration. Could the authors provide comparisons with RoSTE or other approaches that combine data-driven transforms with QAT, using the same quantization configurations and training budgets, to further clarify the incremental benefits of Frozen-TWIST over existing methods?

1 comment

Thank you for your time and detailed review!

We address your concerns and questions below:

W1: INT3 comparison changes both the transform and the scale granularity.

We agree that the INT3 pretraining results in Table 1 change two things at once: the transform and the scale granularity. We would like to note, however, that this confound is limited to INT3 forward-pass pretraining, where Hadamard with group 64 was unstable. All other setups, i.e. INT4 pretraining, the quantized backward pass, and QAD and fine-tuning, compare the transforms at the same group size.

For INT3 pretraining, we now add the full grid, including the unstable Hadamard group-64 runs, with bits per element counting the scales.

TransformScalesBits/element200M430M800M
Hadamardper row≈3.012.918 ± 0.0022.714 ± 0.0032.585 ± 0.005
Hadamardgroup 643.253.277 ± 0.3023.385 ± 0.4682.658 ± 0.061
TWISTper row≈3.012.908 ± 0.0022.710 ± 0.0022.567 ± 0.001
TWISTgroup 643.252.901 ± 0.0002.700 ± 0.0012.558 ± 0.002

At equal storage, with per-row scales, TWIST still outperforms Hadamard at every size. Moving to group 64 improves TWIST further but makes Hadamard unstable, so finer scales are an option that TWIST provides and Hadamard does not. Group 64 is also not an unusually fine choice: standard low-precision formats use even smaller groups, 32 in MXFP4 and 16 in NVFP4.

W2: Training-time advantage of low-bit training.

Since a similar concern was raised by Reviewer oWeT, we address the performance of TWIST under the same compute-time budget in the general response. TWIST retains its advantage under this comparison: with an update interval of τ=100\tau=100, the rebuild overhead drops to 0.7–1.9%, and TWIST outperforms Hadamard in all six settings (200M–800M, INT4 and INT3), e.g. 2.559 against 2.585 at 800M INT3.

Regarding INT3 specifically, a training speedup cannot be measured, since current hardware does not support INT3 matrix multiplications. Our pretraining runs therefore simulate quantization with BF16 mixed precision (Appendix C.1). This is the standard protocol for accuracy studies of low-bit training, used for example by QuEST [1], on which our setup is built, as well as by [2, 3].

W3: Statistical support for the MATH-500 gains.

We agree, and we will report all settings using 3 training runs and 5 decoding samples per checkpoint. So far, we have completed the rerun of the 4B INT3 joint weight and activation setting. please see Tables B and C in the general response. TWIST reaches 55.4 ± 0.6 against 53.2 ± 0.7 for Hadamard, a gain of +2.2 points (95% confidence interval [+0.8, +3.6]).

Q1: Comparison with RoSTE.

RoSTE adaptively selects which rotation to apply, choosing between Walsh–Hadamard matrices and the identity. The selection is data-driven, but the rotations themselves are fixed orthogonal matrices, whereas Frozen-TWIST fits a non-orthogonal transform to the second moments of each block's weights and activations.

We have added RoSTE with the same quantization configuration and training budget; please see the general response. RoSTE reaches 52.8 ± 0.7, on par with Hadamard, and TWIST improves over it by +2.6 points (95% confidence interval [+1.1, +4.2]).

[1] Panferov et al. QuEST: Stable Training of LLMs with 1-Bit Weights and Activations. ICML 2025.

[2] Kumar et al. Scaling Laws for Precision. ICLR 2025.

[3] Dremov et al. Compute-Optimal Quantization-Aware Training. ICLR 2026.

Comments on the paper

3 comments

We thank all reviewers for their time, careful reading, and support! We have uploaded a new revision of our submission, and all changes are highlighted in blue. We also reorganised Appendix D while adding the statistical significance discussion for downstream results: the subsections are reordered, and we removed one figure that duplicated Tables 9 and 10.

Comparison under equal total training time

Reviewers 9ybh and oWeT both asked whether TWIST still outperforms Hadamard under the same total training time. To address this, we ran the following experiments.

We keep Hadamard at its original number of steps NHN_H and shorten TWIST so that both runs take the same total time. Let tt be the iteration time without rebuilds, rr the extra time per rebuild, and ρ=0.75\rho = 0.75 the fraction of training after which the transform is frozen. Rebuilds happen every τ\tau steps and only before the freeze, so TWIST is trained for NT=NH⋅tt+ρ r/τN_T = N_H \cdot \frac{t}{t + \rho\, r/\tau} steps, with tt and rr taken from Table 5.

Table A. Iso-time comparison between TWIST and Hadamard.

SizeMethodStepsINT4 val. lossINT3 val. loss
200MFP25,4312.8122.812
Hadamard (Table 1)25,4312.8512.918
TWIST iso-token, τ=20\tau=20 (Table 1)25,4312.8442.901
TWIST iso-time, τ=20\tau=2023,2512.8512.913
TWIST iso-time, τ=100\tau=10024,9622.8442.902
430MFP41,0072.6222.622
Hadamard (Table 1)41,0072.6572.714
TWIST iso-token, τ=20\tau=20 (Table 1)41,0072.6502.700
TWIST iso-time, τ=20\tau=2038,3872.6592.708
TWIST iso-time, τ=100\tau=10040,4542.6512.702
800MFP50,8622.4852.485
Hadamard (Table 1)50,8622.5242.585
TWIST iso-token, τ=20\tau=20 (Table 1)50,8622.5112.558
TWIST iso-time, τ=20\tau=2049,1302.5142.562
TWIST iso-time, τ=100\tau=10050,5052.5112.559

Even at τ=20\tau=20, iso-time TWIST outperforms Hadamard in most settings. However, TWIST is not tied to τ=20\tau=20: the update interval is one of its parameters. Going to τ=100\tau=100 cuts the rebuild overhead by 5×, to 0.7–1.9% of total training time, at almost no cost in quality: iso-time TWIST with τ=100\tau=100 is within 0.002 of iso-token TWIST with τ=20\tau=20, and outperforms Hadamard in all six settings.

Moreover, the rebuild overhead exists only during training: at inference the transform is fixed, and the fused WUSH kernel runs within 2% of the speed of a fused block Hadamard transform (Appendix B). Also, the relative overhead shrinks with model size, as the transform-related cost is a decreasing fraction of the layer's matrix multiplication (Section 4.1); in Table 5 it falls from 12.5% at 200M to 4.7% at 800M.

QAT-SFT MATH-500 evaluation: multiple seeds and RoSTE

Two requests related to the QAT SFT experiments came up in multiple reviews: stronger statistical support for the MATH-500 gains (Reviewers 9ybh and oWeT) and a comparison with RoSTE (Reviewers 9ybh and iasg). We address both here with new experiments.

Setup. We use the 4B model in the INT3 joint weight and activation setting. For each method we train 3 independent runs, and evaluate each final checkpoint on MATH-500 with 5 decoding samples. These are fresh runs: we no longer have access to the original checkpoints.

TWIST refers to Frozen-TWIST. For RoSTE, we keep our transform layout and use RoSTE's adaptive selection rule, which chooses between the identity and a Hadamard rotation at every layer.

Table B. MATH-500 accuracy (%) for every evaluation.

MethodTraining runSample 1Sample 2Sample 3Sample 4Sample 5Run mean
Hadamard153.055.456.252.253.254.0
Hadamard253.253.252.453.851.452.8
Hadamard353.253.052.453.252.652.9
RoSTE151.852.651.252.252.052.0
RoSTE253.253.253.253.651.853.0
RoSTE354.653.452.252.853.853.4
TWIST155.854.655.656.056.255.6
TWIST257.254.056.857.054.255.8
TWIST354.453.455.054.057.054.8

Table C. Summary of Table B. Confidence intervals are Welch t-intervals over the run means.

MethodAccuracy (mean ± SD over 3 runs)Gain of TWIST95% confidence interval
Hadamard53.2 ± 0.7+2.2[+0.8, +3.6]
RoSTE52.8 ± 0.7+2.6[+1.1, +4.2]
TWIST55.4 ± 0.6––

In the camera-ready, we will report MATH-500 as the mean over 5 decoding samples and 3 training runs for all settings discussed. We note that our claim of improved downstream performance is made primarily for the INT3 settings at this model scale.

Thank you for the follow-up; we are glad W1 is resolved.

W2. We agree that 3-bit weight-and-activation formats do not yet have broad hardware support. We study them because we believe QAT research should stay ahead of hardware and cover the full low-precision range. Two developments also make this range relevant. First, hardware support for 3-bit weights is emerging: NVIDIA’s Rubin GPUs add a native 3-bit weight format [1], and Taalas’s HC1, a hardwired ASIC running Llama 3.1 8B, combines 3-bit and 6-bit weights [2]. Second, 3-bit models are attractive in accuracy per size: for weight-only quantization, ParetoQ [3] finds that they are on par with ternary and 2-bit models in the size-accuracy trade-off and generally exceed 4-bit.

W3. We apologize for the unclear wording: the results in Tables B and C are final. For the 4B INT3 joint weight and activation setting, all 3 training runs × 5 decoding samples are complete for TWIST, Hadamard and RoSTE. Since all methods are evaluated on the same 500 problems, we can also give a paired analysis, using a bootstrap that resamples problems:

  • TWIST vs Hadamard: +2.2 points, paired bootstrap 95% CI [+1.1, +3.3]
  • TWIST vs RoSTE: +2.6 points, paired bootstrap 95% CI [+1.6, +3.7]

In addition, each of the three TWIST runs has a higher mean accuracy than every Hadamard and every RoSTE run.

[1] NVIDIA, “Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI”, 2026. https://developer.nvidia.com/blog/inside-nvidia-rubin-gpu-architecture-powering-the-era-of-agentic-ai/

[2] Ljubisa Bajic. “The path to ubiquitous AI”. Taalas, February 2026. https://taalas.com/the-path-to-ubiquitous-ai/

[3] Liu et al. ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization. NeurIPS 2025.

AuthorsOpenReview ↗

Summary of the discussion

Dear Area Chair,

Below we summarize what we added in the revision.

  1. Equal-time comparison (9ybh W2, oWeT W1/Q1). We compare TWIST and Hadamard at the same total training time, including rebuilds. With an update interval of τ=100\tau=100, TWIST beats Hadamard in all six settings (Appendix F.1).
  2. Multiple seeds (9ybh W3, oWeT W2/Q2). We repeated the MATH-500 and QAD experiments on the 4B models with three training runs each. The gains hold, and the confidence intervals exclude zero (Table 2).
  3. RoSTE (9ybh Q1, iasg W1/W3). We added RoSTE as a baseline in fine-tuning and in pretraining. It performs on par with Hadamard, and TWIST beats both (Table 2, Appendix F.3).
  4. Group size at INT3 (9ybh W1). We report Hadamard and TWIST under both scale granularities. At equal storage, TWIST still wins at every model size (Appendix F.2).
  5. Basis shift (iasg W2). We measure why a recomputed transform raises the loss even though it lowers the layer error. The recomputation changes the direction of the error, and the network grows more sensitive to that change as it converges (Appendix G).

Reviewer 9ybh raised two follow-up points on BF16 simulation and the MATH-500 reruns; we answer both in our reply to them. Thank you again for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.