- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TWIST, which adapts WUSH transforms to quantization-aware training using periodic updates, late freezing, and randomized backward transforms. It reports improved accuracy in pretraining, distillation, and fine-tuning, especially at 3 bits.
Critical strengths
- Useful methodological contribution: TWIST addresses concrete obstacles to adaptive transforms during training: changing statistics, unstable quantization bases, and biased backward estimates.
- Broad empirical evidence: Improvements span several training settings. At 3B parameters, TWIST closes approximately 37% and 44% of the Hadamard-to-full-precision loss gap at 4 and 3 bits, respectively.
- Convincing diagnostics: The basis-shift experiments explain why reducing reconstruction error can worsen training loss. Freezing ablations support the proposed remedy.
Critical weaknesses
- Practical efficiency remains incompletely demonstrated: Transform refreshes add roughly 5–12% iteration overhead before freezing. Kernel-level inference benchmarks do not establish an end-to-end training advantage under equal wall-clock budgets.
- Statistical support for smaller gains is limited: Several downstream improvements are modest, particularly at 4 bits. Repeated training runs and confidence intervals would clarify whether the reported differences are reliable.
Questions for authors
- Under equal total training time, does TWIST still outperform Hadamard after accounting for calibration, statistics collection, and transform updates?
- Do the 2.0–2.2 percentage-point MATH-500 gains persist across training seeds and repeated decoding, with paired confidence intervals?
Thank you for your time and support! We address concerns and questions below:
W1 & Q1: Efficiency at equal training time. Since a similar question was raised by Reviewer 9ybh, we address whether TWIST still outperforms Hadamard under equal total training time in the general response. TWIST retains its advantage under this comparison: with an update interval of τ=100, the rebuild overhead drops to 0.7–1.9%, and TWIST outperforms Hadamard in all six settings (200M–800M, INT4 and INT3), e.g. 2.559 against 2.585 at 800M INT3. We would also like to clarify the role of calibration. Unlike in QAD and fine-tuning, TWIST has no separate calibration pass in pretraining: the statistics for each rebuild are collected online, from the training batch of the preceding step, so their cost is already included in the iteration times in Table 5 and in the equal-time comparison.
W2 & Q2: Statistical support.
The MATH-500 gains persist: we have rerun the 4B INT3 setting with 3 training runs and 5 decoding samples per checkpoint; please see Tables B and C in the general response. TWIST reaches 55.4 ± 0.6 against 53.2 ± 0.7 for Hadamard, a gain of +2.2 points (95% confidence interval [+0.8, +3.6]).
We agree that the gains at 4 bits are modest. In QAD, we do not claim INT4 downstream gains: Section 5 states that the INT4 differences between Hadamard and TWIST are within evaluation noise. We will state the same for INT4 fine-tuning and base our downstream claims on INT3.
Additionally, we show that the QAD KL improvements are statistically significant in all quantization settings considered for Qwen3-4B-Thinking-2507. For each method and setting, we train 3 independent runs and evaluate each final checkpoint on the full held-out split of 13,588 sequences, rather than the 128 used in the paper.
Table A. KL to the FP teacher (mean ± SD over 3 runs).
Table B. Relative KL change of TWIST versus Hadamard (negative means closer to the teacher). The 95% confidence intervals come from a paired bootstrap that resamples both training runs and evaluation sequences.