- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper aims to mitigate quantization error in network binarization within QAT. Specifically, the theoretical findings reveal that binarization error in activation gradients stems from both the STE and the chain-rule bias, and the authors propose a curvature correction using the empirical Fisher as a remedy for the latter. By integrating curvature correction into existing binarization approaches like QuEST and CAGE, the paper shows reduced perplexity (up to 3.2%) across the Llama family at different scales (30M to 1.6B) on the C4 validation benchmark.
Critical strengths
- The presentation is clear and easy to follow.
- The paper is technically sound and provides sufficient theoretical support for the methodology.
Critical weaknesses
- The impact on training wall-clock time and memory footprint was missing. Because curvature correction introduces additional optimizer states (i.e., a moving average of the mean-squared downstream gradient for activations) that update dynamically during training, the authors should discuss in detail how the proposed method affects training time and memory footprint and highlight its practicality.
Questions for authors
Based on the weaknesses, the author should respond to the following questions to address my major concerns.
- Could you provide the statistics on training wall-clock time and memory usage for the proposed method and compared baselines? Will curvature correction catastrophically increase training cost in exchange for a % perplexity reduction?
2 comments
I appreciate the detailed responses, which address my concerns about the practical efficiency of the proposed method. As a result, I will maintain a positive score and finalize my recommendation as a 'Weak Acceptance'.
Response to Reviewer hASH
We thank the reviewer for their endorsement of our work and their constructive feedback. We are happy that they found the presentation clear and easy to follow, and the method technically sound and supported by sufficient theoretical analysis. We answer the raised weakness and question below.
Weakness 1 and Question 1 — Training wall-clock time and memory footprint
We thank the reviewer for the question and agree that the computational and memory overhead should be quantified. With an efficient implementation, the overhead is just a few percent in runtime and negligible additional memory.
In more detail, the moving average $\kappa$ requires one FP32 scalar per corrected activation quantizer, which is negligible (e.g., 320 bytes per GPU for the 800M model). Updating it requires a mean-squared-gradient reduction and a scalar update. Additional computation arises from applying the backward correction and handling activation intermediates, including the quantization residual. These intermediates need not be stored separately; in practice we use a fused quantized-linear layer that retains the prequantized input $z$ as its saved activation and recomputes $Q(z)$ and the correction residual during backward.
We measured the correction overhead against a fused QuEST baseline, using 8 H100 GPUs. We used sequence length 512 and the correction-training batch configurations: per-GPU batches of 64/32/16 with gradient accumulation of 1/2/4 for the 100M/430M/800M models, respectively, giving global batch 512 as in the original training. After 50 warmup optimizer updates, we timed 200 complete optimizer updates. Memory is the maximum peak allocated GPU memory across ranks, reported in decimal GB.
Runtime values are mean ± standard deviation over three repetitions. The runtime overhead, calculated from the ratio of means, decreases with model size: 9.10%, 7.45%, and 6.79% for 100M, 430M, and 800M, respectively. Peak allocated memory is effectively unchanged, increasing by approximately 0.4 MB per GPU. Since correction is applied only during the final 10% of training updates, its estimated contribution to total training time is approximately 0.68–0.91%.