- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Rotated Fourier Surrogate to address the limitations of STE in neural network quantization. The main idea in the paper is to rotate the rounding function by 45 degrees into a continuous triangular wave and then use damped Fourier approximation to derive surrogate gradients in non-differentiable rounding based neural network quantization. The authors show improvements across different architectures for different tasks at varied quantization of weights and activations.
Critical strengths
- The authors propose an interesting geometric interpretation of the rounding function used in neural network quantization.
- The formulation is quite simple and easy to use. It only modifies the backward pass and it is also a generalized form that can yield STE at A=0.
- Unlike the other approximation, RFS provides explicit amplitude param to control the gradient magnitude that helps in numerical stability.
- The method works across different quantization schemes and bitwidths.
- The evaluations are exhaustive and the proposed method has been shown to provide improvements across various architectures and tasks.
Critical weaknesses
- Although the geometric derivation is interesting, the resulting method is fundamentally approximating the non-differentiable rounding operation. The paper emphasize the distinction between RFS and smooth rounding but its not very clear if that distinction really result in better optimization.
- It's not clear how the method works with adaptive optimizers which use first order or second order estimates, like Adam or AdamW.
- It seems that the main advantage of using RFS comes from being bounded quantization error dependent gradient scaling function which has also been shown in other works. Its not clear whether Fourier derived functional form has any meaningful benefit?
- The proposed surrogate suppresses the gradient near quantization centroid based on their relative stability which may yield issue of oscillation and quantized weight may not be optimal.
- The authors choose a fixed amplitude but its not clear if the same amplitude work across different quantization bitwidth and different architectures. It may not be optimal, so does the performance significantly change if you tune it for different scenarios.
Questions for authors
Please address the above weaknesses. Can the authors comment on computational complexity with cosine and division operations on large models compared to simple STE based framework?
4 comments
Continued response to Reviewer uucK
Q3. The proposed surrogate suppresses the gradient near quantization centroid based on their relative stability which may yield issue of oscillation and quantized weight may not be optimal.
A3. Thanks for raising this point. We would like to gently clarify that gradient suppression near quantization centroids does not necessarily induce oscillation or prevent convergence to favorable quantized solutions.
First, RFS does not vanish gradients near quantization centroids. It remains strictly positive, with a minimum value of under the well-conditioned amplitude range. Consequently, gradients remain informative, while their magnitudes are smoothly modulated according to the quantization geometry.
Second, convergence to optimal quantization is shown to depend more on sensitivity to stochastic gradient noise. DSQ may exhibit high gradient variance; EWGS can introduce noise-dependent systematic drift. Both effects contributes to undesirable quantization-bin transitions. In contrast, RFS does not have these limitations (summarized as Table 1). The controlled experiments also provide supporting evidence. RFS achieves higher quantized-solution recovery rates and fewer false-positive bin switches under increasing gradient noise than EWGS.
| Method | Recovery (%) ↑ | Quantized Loss ↓ | Bin-Switch Rate (%) ↓ | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Noise σ | 0.2 | 1 | 3 | 0.2 | 1 | 3 | 0.2 | 1 | 3 |
| STE | 100.00 | 99.16 | 94.59 | 0.0 | 0.012 | 0.071 | 0.00 | 1.47 | 6.22 |
| EWGS (δ=0.001) | 100.00 | 99.16 | 94.60 | 0.0 | 0.012 | 0.071 | 0.00 | 1.48 | 6.20 |
| EWGS (δ=0.01) | 100.00 | 99.24 | 94.72 | 0.0 | 0.011 | 0.069 | 0.00 | 1.34 | 6.03 |
| EWGS (δ=0.1) | 100.00 | 99.74 | 95.78 | 0.0 | 0.004 | 0.056 | 0.00 | 0.46 | 4.70 |
| RFS (A=0.05) | 100.00 | 99.52 | 96.62 | 0.0 | 0.007 | 0.044 | 0.00 | 0.85 | 3.81 |
| RFS (A=0.15) | 100.00 | 99.95 | 98.92 | 0.0 | 0.001 | 0.015 | 0.00 | 0.10 | 1.16 |
| RFS (A=0.184) | 100.00 | 99.99 | 99.36 | 0.0 | 0.0 | 0.009 | 0.00 | 0.02 | 0.68 |
| RFS (A=0.21) | 100.00 | 100.00 | 99.68 | 0.0 | 0.0 | 0.005 | 0.00 | 0.01 | 0.34 |
Q4. The authors choose a fixed amplitude but its not clear if the same amplitude work across different quantization bitwidth and different architectures. It may not be optimal, so does the performance significantly change if you tune it for different scenarios.
A4. This is a great question. The work focuses on constant amplitude paradigm. Due to the well-condition nature and empirical studies, average and lower-bound performance of RFS is not that sensitive to different amplitude values. RFS is also less likely to produce catastrophic performance collapse (observed in other surrogates). Nevertheless, the optimal constant amplitude can vary across architectures and tasks.
In experiments, we start from the default value with condition number as 10, then did a grid search if not exceeding the baselines. In the revision, the explicitly amplitude has been marked if not equals to 0.184.
Furthermore, dynamic amplitude scheduling represents a promising extension. Our preliminary investigations suggest additional improvements, but a systematic study of scheduling strategies deserve more dedicated studies, leaving for future research.
Q5. Can the authors comment on computational complexity with cosine and division operations on large models compared to simple STE based framework?
A5. Thank you for the great question. We extend the efficiency benchmarks across more architectures of varying sizes 1B-13B. As shown below, the conclusion holds that RFS with optimized kernel incurs neligible computational overhead to STE. Other alternatives like DSQ and EWGS requires signficant additional compute due to the multi-step nature and costly second-order approximation.
| Model | Method | H100 | Forward time (ms) | Forward memory (MiB) | Backward time (ms) | Backward memory (MiB) |
|---|---|---|---|---|---|---|
| Llama-3.2-1B | STE | 1 | 44.90 | 2,511 | 66.33 | 5,966 |
| DSQ | 1 | 70.11 | 2,520 | 85.17 | 5,967 | |
| RFS | 1 | 43.31 | 2,520 | 70.06 | 5,967 | |
| EWGS-Hessian | 1 | 45.28 | 2,520 | 124.41 | 18,116 | |
| Llama-3.2-3B | STE | 1 | 48.84 | 6,291 | 74.35 | 14,537 |
| DSQ | 1 | 79.87 | 6,291 | 88.21 | 14,543 | |
| RFS | 1 | 49.01 | 6,291 | 74.10 | 14,536 | |
| EWGS-Hessian | 1 | 50.69 | 6,291 | 4,085.23 | 38,580 | |
| Llama-3.1-8B | STE | 1 | 54.95 | 15,507 | 85.42 | 33,305 |
| DSQ | 1 | 114.17 | 15,507 | 121.10 | 33,501 | |
| RFS | 1 | 56.39 | 15,507 | 81.50 | 33,305 | |
| EWGS-Hessian | 1 | 66.27 | 15,507 | 6,336.71 | 85,508 | |
| Llama-2-13B | STE | 1 | 66.05 | 25,165 | 111.48 | 54,466 |
| DSQ | 1 | 181.65 | 25,165 | 223.09 | 54,728 | |
| RFS | 1 | 65.76 | 25,165 | 124.63 | 54,466 | |
| EWGS-Hessian | 2 | 70.18 | 25,377 | 1,533.49 | 144,331 |
Look forward to further discussions!
Best,
Authors
Thank you
Dear Reviewer uucK,
We sincerely thank you again for the constructive feedback and suggestions. We hope that our responses, additional experiments, and revisions have adequately addressed the insightful questions regarding comparison with other surrogate methods, amplitude selection, efficiency benchmarks, applicability to Adam, etc.
If any further questions or suggestions, we would be happy to discuss and address them.
Best wishes,
Authors
We thank the reviewer for the time and effort spent evaluating our work. We have posted a detailed, point-by-point response with new experiments and results addressing each of the concerns raised. As the discussion period is nearing its end, we would greatly appreciate hearing whether our response has resolved the concerns, and we would be glad to clarify or provide any additional evidence that may be helpful. Thank you again for the constructive feedback.
Response to Reviewer uucK
We sincerely thank Reviewer uucK for the constructive feedback and insightful questions. We particularly appreciate the reviewer's recognition of the geometric interpretation of RFS, its simple and generalized formulation that recovers STE at (A=0), the explicit amplitude control, and its broad applicability. Following the reviewer's suggestions, we have further strengthened and clarified the optimization behavior, and expanded the experimental evaluations. We address each question below.
Q1. Not clear the better optimization between RFS and soft-rounding; RFS and gradient-scaling.
A1. Thank you for this important question. We have added a comprehensive comparison of representative surrogate-gradient methods in Table 1 of general Q/A and the revision. In particular, the optimization advantages of RFS arise from different properties when compared with soft-rounding and gradient-scaling approaches.
Comparison with soft-rounding methods (DSQ (tanh like) and FDA-BNN). RFS offers a well-conditioned surrogate gradient and reduced sensitivity to hyperparameters. In contrast, both DSQ and FDA exhibit increasingly large gradient variance and becomes ill-conditioned through different manners. RFS instead achieves a favorable balance between smoothness, gradient conditioning, and optimization effectiveness.
Comparison with gradient-scaling methods (EWGS). RFS avoids the gradient-noise-dependent drift introduced by EWGS and offers smoothness. EWGS introduces a systematic correction whose magnitude increases with the gradient noise. In the controlled experiments, EWGS behaves noticeably behind than RFS under high-noise setup (shown in the below table). EWGS is also nonsmooth at upstream gradient as zero. The gain of smoothness is validated in the Section 3.3 that higher Fourier order (less smooth) yields performance degradation.
Table. Controlled experiment between EWGS and RFS under varying gradient noise over a quadratic problem.
Table. QAT for Image classification on ImageNet.
Table. QAT for LLaMA-3.2-1B. Avg of zero-shot common-sense reasoning.
The above results show that RFS and EWGS perform competitively under low-noise and stable setup, while RFS surpasses significantly under high-noisy optimization environments.
Analogy to activation function in DNN. We note that this comparison might shares an intuition with the evolution of activation functions in deep neural networks, from saturating functions such as tanh, to piecewise-linear yet unsmooth functions such as ReLU and LeakyReLU, subsequently to smooth alternatives such as SiLU. While the optimization settings differ, this analogy highlights the importance of balancing gradient conditioning, smoothness, and computational efficiency.
Q2. It's not clear how the method works with adaptive optimizers which use first order or second order estimates, like Adam or AdamW.
A2. . Thank you for raising this question. The formulation of RFS does not depend on additional gradient statistics. Therefore, it is compatible with different optimizers such as Adam and AdamW. In our experiments, we retain the optimizers and training configurations of the original quantization baselines. The majority of them employed Adam or AdamW, where we observed consistent improvements, demonstrating that RFS integrates effectively with these adaptive optimizers .