acceptodds
Under review as a conference paper at ICLR 2027

Quantization Everywhere with Rotated Fourier Surrogate

Abstract

Quantization is essential for efficient neural network deployment, whereas its optimization overwhelmingly relies on the Straight-Through Estimator (STE). However, STE discards the structure of the quantization operator by replacing its derivative with an identity mapping, resulting in a mismatch between the forward and backward passes that can fundamentally limit the achievable quality of quantized models. To address this limitation, we introduce Rotated Fourier Surrogate (RFS), a unified framework that extends STE with a principled gradient estimator. RFS analytically reflects the structure of quantization while remaining stable and computationally efficient. We further provide rigorous theoretical guarantees for RFS, demonstrating well-conditioned gradients and favorable optimization properties. Moreover, RFS is plug-and-play and incurs negligible computational overhead with our optimized kernel. We extensively evaluate RFS across diverse quantization paradigms, spanning integer and floating-point quantization, weight and activation quantization, as well as quantization-aware training (QAT), quantization-aware distillation (QAD), and post-training quantization (PTQ). Our experiments cover a broad spectrum of architectures, including large language models, diffusion language models, vision transformers, and convolutional neural networks, as well as diverse applications including generative recommendation, image classification, pre-training, and reasoning post-training. Across these diverse settings, RFS consistently improves upon strong state-of-the-art quantization baselines, demonstrating its broad applicability across quantization paradigms, model architectures, and applications.

Primary area:
optimization
Keywords:
quantization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uucKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Rotated Fourier Surrogate to address the limitations of STE in neural network quantization. The main idea in the paper is to rotate the rounding function by 45 degrees into a continuous triangular wave and then use damped Fourier approximation to derive surrogate gradients in non-differentiable rounding based neural network quantization. The authors show improvements across different architectures for different tasks at varied quantization of weights and activations.

Critical strengths
  • The authors propose an interesting geometric interpretation of the rounding function used in neural network quantization.
  • The formulation is quite simple and easy to use. It only modifies the backward pass and it is also a generalized form that can yield STE at A=0.
  • Unlike the other approximation, RFS provides explicit amplitude param to control the gradient magnitude that helps in numerical stability.
  • The method works across different quantization schemes and bitwidths.
  • The evaluations are exhaustive and the proposed method has been shown to provide improvements across various architectures and tasks.
Critical weaknesses
  • Although the geometric derivation is interesting, the resulting method is fundamentally approximating the non-differentiable rounding operation. The paper emphasize the distinction between RFS and smooth rounding but its not very clear if that distinction really result in better optimization.
  • It's not clear how the method works with adaptive optimizers which use first order or second order estimates, like Adam or AdamW.
  • It seems that the main advantage of using RFS comes from being bounded quantization error dependent gradient scaling function which has also been shown in other works. Its not clear whether Fourier derived functional form has any meaningful benefit?
  • The proposed surrogate suppresses the gradient near quantization centroid based on their relative stability which may yield issue of oscillation and quantized weight may not be optimal.
  • The authors choose a fixed amplitude but its not clear if the same amplitude work across different quantization bitwidth and different architectures. It may not be optimal, so does the performance significantly change if you tune it for different scenarios.
Questions for authors

Please address the above weaknesses. Can the authors comment on computational complexity with cosine and division operations on large models compared to simple STE based framework?

4 comments
AuthorsOpenReview ↗

Response to Reviewer uucK

We sincerely thank Reviewer uucK for the constructive feedback and insightful questions. We particularly appreciate the reviewer's recognition of the geometric interpretation of RFS, its simple and generalized formulation that recovers STE at (A=0), the explicit amplitude control, and its broad applicability. Following the reviewer's suggestions, we have further strengthened and clarified the optimization behavior, and expanded the experimental evaluations. We address each question below.

Q1. Not clear the better optimization between RFS and soft-rounding; RFS and gradient-scaling.

A1. Thank you for this important question. We have added a comprehensive comparison of representative surrogate-gradient methods in Table 1 of general Q/A and the revision. In particular, the optimization advantages of RFS arise from different properties when compared with soft-rounding and gradient-scaling approaches.

Comparison with soft-rounding methods (DSQ (tanh like) and FDA-BNN). RFS offers a well-conditioned surrogate gradient and reduced sensitivity to hyperparameters. In contrast, both DSQ and FDA exhibit increasingly large gradient variance and becomes ill-conditioned through different manners. RFS instead achieves a favorable balance between smoothness, gradient conditioning, and optimization effectiveness.

Comparison with gradient-scaling methods (EWGS). RFS avoids the gradient-noise-dependent drift introduced by EWGS and offers smoothness. EWGS introduces a systematic correction whose magnitude increases with the gradient noise. In the controlled experiments, EWGS behaves noticeably behind than RFS under high-noise setup (shown in the below table). EWGS is also nonsmooth at upstream gradient as zero. The gain of smoothness is validated in the Section 3.3 that higher Fourier order (less smooth) yields performance degradation.

Table. Controlled experiment between EWGS and RFS under varying gradient noise over a quadratic problem.

MethodRecovery (%) ↑Quantized Loss ↓Bin-Switch Rate (%) ↓
Noise σ0.2130.2130.213
STE100.0099.1694.590.00.0120.0710.001.476.22
EWGS (δ=0.001)100.0099.1694.600.00.0120.0710.001.486.20
EWGS (δ=0.01)100.0099.2494.720.00.0110.0690.001.346.03
EWGS (δ=0.1)100.0099.7495.780.00.0040.0560.000.464.70
RFS (A=0.05)100.0099.5296.620.00.0070.0440.000.853.81
RFS (A=0.15)100.0099.9598.920.00.0010.0150.000.101.16
RFS (A=0.184)100.0099.9999.360.00.00.0090.000.020.68**
RFS (A=0.21)100.00100.0099.680.00.00.0050.000.010.34

Table. QAT for Image classification on ImageNet.

SettingMethodW4A4W3A3W2A2
ResNet18EWGS70.669.767.0
DSQ69.668.765.2
Weight-RFS (A=0.05A=0.05) + Act-STE70.669.666.9
Weight-RFS (A=0.05A=0.05) + Act-RFS (A=0.01A=0.01)70.769.767.0
ResNet34EWGS73.973.371.4
DSQ72.872.570.0
Weight-RFS (A=0.05A=0.05) + Act-STE73.973.371.4
Weight-RFS (A=0.05A=0.05) + Act-RFS (A=0.01A=0.01)74.073.371.4

Table. QAT for LLaMA-3.2-1B. Avg of zero-shot common-sense reasoning.

SettingsMethodW4A16W3A16W2A16
Llama-3.2-1BParetoQ60.9659.5757.29
DSQ61.0757.4256.39
EWGS (δ=0.001\delta=0.001)61.0559.6857.03
Weight-RFS (A=0.21)61.2460.1258.08

The above results show that RFS and EWGS perform competitively under low-noise and stable setup, while RFS surpasses significantly under high-noisy optimization environments.

Analogy to activation function in DNN. We note that this comparison might shares an intuition with the evolution of activation functions in deep neural networks, from saturating functions such as tanh, to piecewise-linear yet unsmooth functions such as ReLU and LeakyReLU, subsequently to smooth alternatives such as SiLU. While the optimization settings differ, this analogy highlights the importance of balancing gradient conditioning, smoothness, and computational efficiency.

Q2. It's not clear how the method works with adaptive optimizers which use first order or second order estimates, like Adam or AdamW.

A2. . Thank you for raising this question. The formulation of RFS does not depend on additional gradient statistics. Therefore, it is compatible with different optimizers such as Adam and AdamW. In our experiments, we retain the optimizers and training configurations of the original quantization baselines. The majority of them employed Adam or AdamW, where we observed consistent improvements, demonstrating that RFS integrates effectively with these adaptive optimizers .

AuthorsOpenReview ↗

Continued response to Reviewer uucK

Q3. The proposed surrogate suppresses the gradient near quantization centroid based on their relative stability which may yield issue of oscillation and quantized weight may not be optimal.

A3. Thanks for raising this point. We would like to gently clarify that gradient suppression near quantization centroids does not necessarily induce oscillation or prevent convergence to favorable quantized solutions.

First, RFS does not vanish gradients near quantization centroids. It remains strictly positive, with a minimum value of 1−2πA1+2πA\frac{1-\sqrt{2}\pi A}{1+\sqrt{2}\pi A} under the well-conditioned amplitude range. Consequently, gradients remain informative, while their magnitudes are smoothly modulated according to the quantization geometry.

Second, convergence to optimal quantization is shown to depend more on sensitivity to stochastic gradient noise. DSQ may exhibit high gradient variance; EWGS can introduce noise-dependent systematic drift. Both effects contributes to undesirable quantization-bin transitions. In contrast, RFS does not have these limitations (summarized as Table 1). The controlled experiments also provide supporting evidence. RFS achieves higher quantized-solution recovery rates and fewer false-positive bin switches under increasing gradient noise than EWGS.

MethodRecovery (%) ↑Quantized Loss ↓Bin-Switch Rate (%) ↓
Noise σ0.2130.2130.213
STE100.0099.1694.590.00.0120.0710.001.476.22
EWGS (δ=0.001)100.0099.1694.600.00.0120.0710.001.486.20
EWGS (δ=0.01)100.0099.2494.720.00.0110.0690.001.346.03
EWGS (δ=0.1)100.0099.7495.780.00.0040.0560.000.464.70
RFS (A=0.05)100.0099.5296.620.00.0070.0440.000.853.81
RFS (A=0.15)100.0099.9598.920.00.0010.0150.000.101.16
RFS (A=0.184)100.0099.9999.360.00.00.0090.000.020.68
RFS (A=0.21)100.00100.0099.680.00.00.0050.000.010.34

Q4. The authors choose a fixed amplitude but its not clear if the same amplitude work across different quantization bitwidth and different architectures. It may not be optimal, so does the performance significantly change if you tune it for different scenarios.

A4. This is a great question. The work focuses on constant amplitude paradigm. Due to the well-condition nature and empirical studies, average and lower-bound performance of RFS is not that sensitive to different amplitude values. RFS is also less likely to produce catastrophic performance collapse (observed in other surrogates). Nevertheless, the optimal constant amplitude can vary across architectures and tasks.

In experiments, we start from the default value A=0.184A=0.184 with condition number as 10, then did a grid search if not exceeding the baselines. In the revision, the explicitly amplitude has been marked if not equals to 0.184.

Furthermore, dynamic amplitude scheduling represents a promising extension. Our preliminary investigations suggest additional improvements, but a systematic study of scheduling strategies deserve more dedicated studies, leaving for future research.

Q5. Can the authors comment on computational complexity with cosine and division operations on large models compared to simple STE based framework?

A5. Thank you for the great question. We extend the efficiency benchmarks across more architectures of varying sizes 1B-13B. As shown below, the conclusion holds that RFS with optimized kernel incurs neligible computational overhead to STE. Other alternatives like DSQ and EWGS requires signficant additional compute due to the multi-step nature and costly second-order approximation.

ModelMethodH100Forward time (ms)Forward memory (MiB)Backward time (ms)Backward memory (MiB)
Llama-3.2-1BSTE144.902,51166.335,966
DSQ170.112,52085.175,967
RFS143.312,52070.065,967
EWGS-Hessian145.282,520124.4118,116
Llama-3.2-3BSTE148.846,29174.3514,537
DSQ179.876,29188.2114,543
RFS149.016,29174.1014,536
EWGS-Hessian150.696,2914,085.2338,580
Llama-3.1-8BSTE154.9515,50785.4233,305
DSQ1114.1715,507121.1033,501
RFS156.3915,50781.5033,305
EWGS-Hessian166.2715,5076,336.7185,508
Llama-2-13BSTE166.0525,165111.4854,466
DSQ1181.6525,165223.0954,728
RFS165.7625,165124.6354,466
EWGS-Hessian270.1825,3771,533.49144,331

Look forward to further discussions!

Best,

Authors

Dear Reviewer uucK,

We sincerely thank you again for the constructive feedback and suggestions. We hope that our responses, additional experiments, and revisions have adequately addressed the insightful questions regarding comparison with other surrogate methods, amplitude selection, efficiency benchmarks, applicability to Adam, etc.

If any further questions or suggestions, we would be happy to discuss and address them.

Best wishes,

Authors

We thank the reviewer for the time and effort spent evaluating our work. We have posted a detailed, point-by-point response with new experiments and results addressing each of the concerns raised. As the discussion period is nearing its end, we would greatly appreciate hearing whether our response has resolved the concerns, and we would be glad to clarify or provide any additional evidence that may be helpful. Thank you again for the constructive feedback.

Reviewer_8YrEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Rotated Fourier Surrogate (RFS), a gradient surrogate that extends the Straight-Through Estimator (STE) to address its forward–backward mismatch. The key observation is that a 45° coordinate rotation turns the staircase rounding function into a continuous periodic triangle wave with an analytic Fourier representation. Truncating the series, treating the Fourier amplitude AA as a damping parameter, and mapping back to the original coordinates yields the surrogate gradient gRFS(x,xq,A)g_{\text{RFS}}(x, x_q, A), which recovers STE at A=0A=0. The authors use the first-order approximation (M=0M=0) with A≈0.184A \approx 0.184 chosen from a conditioning analysis (κ=10\kappa=10). RFS covers INT and FP quantization via local normalization, applies element-wise modulation for weights, and uses a normalized variant for activation quantization. Theoretically, the paper claims better alignment with a Gaussian-smoothed reference gradient than STE and bounded gradient variance in contrast to DSQ. Empirically, RFS replaces STE in ParetoQ (LLaMA-3.2-1B/3B), Diffusion-GR2 (MXFP8/MXFP4), ReQAT (NVFP4, DeepSeek-R1-14B), and NWQ (CNN PTQ on ImageNet).

Critical strengths
  1. Plug-and-play with negligible overhead and broad applicability. RFS only replaces the backward scalar of STE while keeping the exact discrete forward pass, and nearly matches STE in latency and peak memory, unlike DSQ and EWGS. It is evaluated across INT/FP, weight/activation, and QAT/QAD/PTQ settings. This is consequential because a drop-in STE replacement can be adopted in existing pipelines at no cost.

  2. Improved training stability at low bit-widths. In multi-seed QAT on LLaMA-3.2, RFS reduces run-to-run variance and mitigates the training collapse observed with DSQ. Since stability is the main bottleneck of sub-4-bit QAT, this is the most consequential empirical result.

Critical weaknesses
  1. RFS uses a hard forward pass and evaluates Eq. 7 at xq=round(x)x_q=\mathrm{round}(x) (Eq. 35), whereas Theorem 4.1 analyzes ∇L(Q~M)\nabla\mathcal{L}(\tilde{Q}_M) with a smooth forward pass. Moreover, since QQ is discontinuous, no continuous Q~M\tilde{Q}_M can achieve a smaller ϵM\epsilon_M than STE (i.e., the identity), so the strict advantage over STE claimed in Remark I does not follow.

  2. As Eq. 35 shows, RFS reduces to rescaling the STE gradient by a fixed function of the rounding residual, which belongs to the same family as element-wise gradient scaling (EWGS). However, EWGS is excluded from all accuracy comparisons due to Hessian-induced OOM, although a fixed-δ\delta variant requires no Hessian. Together with Weakness 1, neither the theory nor the experiments isolate the benefit of the Fourier-derived form, the paper's central claim, over simpler residual-based rescaling.

Questions for authors
  1. Only training curves are shown for M∈{0,1,2}M\in\{0,1,2\} (Fig. 6a/b). Please report the final downstream accuracy (mean ± std) for each MM, with κ\kappa matched rather than AA fixed. With fixed AA, larger MM drives gRFSg_{\text{RFS}} toward a constant (i.e., a scaled STE), and at M=1M=1 its minimum shifts from r=0r=0 to r≈±0.25r\approx\pm0.25. It is therefore unclear whether the gains come from the Fourier structure.

  2. EWGS is excluded due to Hessian-induced OOM. Please specify the hardware (GPU type and memory) and the training configuration (batch size, sequence length) under which OOM occurred, since Fig. 4 reports only 18.1 GB of peak backward memory for EWGS at batch size 4 and sequence length 128. Moreover, a fixed-δ\delta variant requires no Hessian and has a cost similar to RFS; does RFS still outperform it under the ParetoQ setting?

  3. PTQ is evaluated only on CNNs with NWQ, and only Act-RFS is applied. Reconstruction-based PTQ methods back-propagate through the quantizer with STE: for example, activation quantizers in LS-ViT [1] and FIMA-Q [2] for ViTs, and learnable rotations/affine transformations with clipping in SpinQuant [3] and FlatQuant [4] for LLMs. Please report results of RFS on top of these methods (e.g., DeiT/Swin at W4A4/W3A3, LLaMA at W4A4). Please also clarify whether weight-RFS is applicable when weight rounding is learned via AdaRound-style variables. Given the title "Quantization Everywhere," this directly bears on the generality claim.


Reference [1] Hwang, Hyunha, Xuan Truong Nguyen, and Hyuk-Jae Lee. "LS-ViT: Least-Squares Hessian Based Block Reconstruction for Low-Bit Post-Training Quantization of Vision Transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026. [2] Wu, Zhuguanyu, et al. "FIMA-Q: Post-training quantization for vision transformers by fisher information matrix approximation." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025. [3] Liu, Zechun, et al. "Spinquant: Llm quantization with learned rotations." International Conference on Learning Representations. Vol. 2025. 2025. [4] Sun, Yuxuan, et al. "Flatquant: Flatness matters for llm quantization." arXiv preprint arXiv:2410.09426 (2024).

5 comments
AuthorsOpenReview ↗

Response to Reviewer 8YrE

We sincerely thank Reviewer 8YrE for the constructive feedback and insightful suggestions. We appreciate the recognition of RFS's plug-and-play design, negligible computational overhead, broad applicability, and improved training stability. Following the reviewer's suggestions, we have strengthened the manuscript with revised theoretical analysis, controlled experiment, and empirical evaluations.

Q1. EWGS is excluded due to Hessian-induced OOM. Please specify the hardware (GPU type and memory) and the training configuration (batch size, sequence length) under which OOM occurred, since Fig. 4 reports only 18.1 GB of peak backward memory for EWGS at batch size 4 and sequence length 128. Moreover, a fixed-variant requires no Hessian and has a cost similar to RFS; does RFS still outperform it under the ParetoQ setting?

A1. Thank you for this important question. We have extended our comparison with EWGS through analytical comparison, controlled experiments, and practical evaluations on CNNs and LLMs, referring to the general response for details. Our conclusion is that under low-noise and stable training setups such as ResNet for image classification, RFS and EWGS perform competitively, where RFS slightly outperforms on W4A4 setup. For high-noisy training environment, RFS exhibits strong performance and robustness. This is driven by the below insights.

Theoretical insights. Unlike RFS, EWGS introduces gradient-dependent corrections that can exhibit noise-dependent drift and nonsmooth behavior. Since gradient statistics vary with data, architectures, optimizers, and training objectives, this dependence may affect EWGS's robustness across training settings. In contrast, RFS incorporates rounding information through a smooth, bounded surrogate without noise-induced drift, making it more reliable and higher performance especially in high-noisy setups.

Efficiency clarification. The OOM observation for the Hessian-based EWGS in LLM QAT happens under sequence length as 4096, which is significantly longer than 128 used in the efficiency benchmark (Section 4.3). It leads to OOM even under batch-size as 1. The GPUs that we used are NVIDIA H100s. We agree that the fixed-scalar EWGS variant avoids the Hessian computation. Accordingly, we have included this variant into the experiments for QAT LLM and observe RFS's outperformance.

Table. QAT for LLaMA-3.2-1B. Avg of zero-shot common-sense reasoning.

SettingsMethodW4A16W3A16W2A16
Llama-3-1BParetoQ60.9659.5757.29
DSQ61.0757.4256.39
EWGS (δ=0.001\delta=0.001)61.0559.6857.03
Weight-RFS61.2460.1258.08

Q2. Study the Fourier Order under controlled condition number.

A2. Thank you for this valuable suggestion. We have added controlled experiments and further analysis in Section 3.3 to investigate the effect of Fourier order on optimization performance.

Analytical insights. Increasing the Fourier order improves the approximation fidelity to the rounding function. But as the order increases, the approximation approaches the hard-rounding structure, reduces the smoothness, inherit ill-conditioned, and potentially compromises stability and performance.

Controlled experiments. In Section 3.3, we provided a controlled experiment. We observed that better structure capture fidelity sometimes leads to better performance such as second-order M=1M=1. However, the first order M=0M=0 has preserved most of the quality. As MM increases, RFS becomes less smooth, where the unsmoothness has led to performance degradation.

Table.Recovery rate (%) mean ± std under σ = 3,κ = 10

Fourier order (M)d = 100d = 1,000d = 10,000
M=098.8500 ± 0.208798.6150 ± 0.107798.5915 ± 0.0335
M=199.2000 ± 0.171798.1850 ± 0.119398.2805 ± 0.0373
M=298.4000 ± 0.302697.7350 ± 0.109697.7690 ± 0.0365
M=497.8500 ± 0.350097.0450 ± 0.133996.9255 ± 0.0339
M=896.9500 ± 0.444196.2550 ± 0.173496.3180 ± 0.0361
M=1697.0500 ± 0.294596.1000 ± 0.156896.1295 ± 0.0348
M=3296.3500 ± 0.448896.2200 ± 0.158496.1730 ± 0.0363
M=6496.7000 ± 0.398096.2050 ± 0.149696.1825 ± 0.0451
STE82.6000 ± 1.024480.4400 ± 0.18260.0000 ± 0.0000

Practical validation. We additionally reported the statistics of Fourier order on LLaMA-3.2-1B W3A16 QAT.

Table. LLaMA-3.2-1B W3A16, common-sense reasoning mean ± std

Fourier orderMean ±\pm Std
M=059.93±0.1459.93 \pm 0.14
M=160.05±0.2060.05 \pm 0.20
M=259.84±0.3959.84 \pm 0.39

Although M=1 achieves a marginally higher mean accuracy, M=0 exhibits the lowest variance, simplicity and high performance.

Conclusion. These findings support our choice of M=0 as the default, providing a favorable balance among approximation fidelity, optimization stability, smoothness, and computational efficiency.

AuthorsOpenReview ↗

Continued response to Reviewer 8YrE

Q3. PTQ on LS-ViT and FIMA-Q for ViTs, SpinQuant and FlatQuant for LLMs. Please also clarify whether weight-RFS is applicable when weight rounding is learned via AdaRound-style variables.

A3. Thank you for the valuable suggestions. We have extended experiments by applying Act-RFS into LS-ViT and FIMA-Q for ViTs, and SpinQuant and FlatQuant for LLMs. The results demonstrate that RFS can improve these strong PTQ baselines.

ModelMethodWikiText-2 PPLARC-EARC-CHellaSwagPIQAOBQASciQWinoGrandeAvg.Δ
LLaMA-2-7BBF165.470874.0745.5675.9978.8444.2091.4069.4668.50--
LLaMA-2-7BSpinQuant5.932370.1642.5873.6077.1042.6091.3066.6166.28--
LLaMA-2-7BSpinQuant + Act-RFS (A=0.07)5.689570.3743.4373.5376.4441.4091.8067.2566.320.04↑
LLaMA-2-7BFlatQuant + STE (RTN)5.760170.1642.6674.4577.9742.0090.8069.0666.73--
LLaMA-2-7BFlatQuant + Act-RFS (RTN)5.756072.5243.4374.2477.9143.2091.7068.0467.290.56↑
LLaMA-2-7BFlatQuant (GPTQ)5.750071.0943.0973.8077.2642.2090.5068.4366.62--
LLaMA-2-7BFlatQuant + Act-RFS (GPTQ)5.739071.3843.2674.2278.2443.8091.7068.2767.270.65↑
LLaMA-3-8BBF166.137977.8253.3379.1080.7945.0093.9072.9371.84--
LLaMA-3-8BSpinQuant7.322275.7248.2975.6878.1842.2092.8070.0968.99--
LLaMA-3-8BSpinQuant + Act-RFS (A=0.05)7.072377.2748.8976.8878.4043.2093.1069.2269.570.58↑
LLaMA-3-8BFlatQuant + STE (RTN)6.907775.0048.7276.8479.1143.2093.2070.8069.55--
LLaMA-3-8BFlatQuant + Act-RFS (RTN)6.897575.1749.4977.1579.1642.0093.2072.5369.810.26↑
LLaMA-3-8BFlatQuant (GPTQ)6.834473.4448.2976.9579.8243.8093.5071.1969.57--
LLaMA-3-8BFlatQuant + Act-RFS (GPTQ)6.823277.1550.1776.8680.0343.2094.1071.1970.390.82↑
MethodW/ADeiT-TDeiT-SDeiT-BSwin-S
Full precision32/3272.7179.8581.8083.23
FIMA-Q4/466.8476.8780.3381.82
FIMA-Q + Act-RFS4/466.9876.8880.3581.82
FIMA-Q3/355.5569.1376.5477.26
FIMA-Q + Act-RFS3/3(A=0.1) 55.6269.85(A=0.1) 76.5977.79
LS-ViT4/467.0576.8980.4181.82
LS-ViT + Act-RFS4/467.10(A=0.15) 76.98(A=0.17) 80.43(A=0.21) 81.86
LS-ViT3/355.7269.4176.5777.39
LS-ViT + Act-RFS3/3(A=0.1) 55.9570.10(A=0.15) 76.7478.15

These results show consistent improvements across the evaluated ViT architectures.

Applicability to weights learned by AdaRound in PTQ. This is a great point, where we did some studies. AdaRound differs from STE-based quantization by optimizing variables through a rectified-sigmoid relaxation. Although the RFS paradigm can be adapted to AdaRound by replacing the relaxed forward pass with hard rounding and using RFS for backward gradient estimation, our preliminary experiments suggest competitive performance at the moment. Sometimes RFS surpasses, sometimes not. Therefore, we did not include it due to an absence of a converged concluding point. A consistent outperformance requires more dedicated studies.

Generality clarification. We have added Table 1 in the revision to clarify that RFS's generality primarily refers to the coverage of STE-based gradient estimation.

Q4. RFS uses a hard forward pass mismatches Theorem 4.1, which analyzes with a smooth forward pass.

A4. We sincerely thank the reviewer for identifying this important issue. We acknowledge that the original proof contains a gap. Although it can be fixable with additional assumption, the revised theorem would include a couple of extra terms making the concluding point not as native as before. Therefore, we decided to pivoting the original Theorem 4.1 to Theorem C.1 instead. Theorem C.1 directly characterizes the Fourier approximation error of the rotated rounding function, providing approximation fidelity for RFS. We sincerely appreciate the reviewer's careful observation, which has helped improve the theoretical rigor of our manuscript.

Look forward to further discussions!

Best

Authors

Dear Reviewer 8YrE,

We sincerely thank you again for the constructive feedback and suggestions. We hope that our Fourier order studies, experiments on more PTQ baselines, comparison with EWGS, and theorem revision have adequately addressed the insightful questions.

If any further questions or suggestions, we would be happy to discuss and address them.

Best wishes,

Authors

Reviewer_8p2fOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Rotated Fourier Surrogate (RFS), a replacement for STE that uses coordinate rotation and a truncated Fourier expansion of the rounding staircase to construct quantization-aware surrogate gradients, recovering STE at zero amplitude. It provides theoretical analysis and evaluates RFS across quantization formats, model architectures, and tasks, reporting improvements in performance and optimization stability over STE with runtime and GPU memory costs close to STE under the tested configurations.

Critical strengths
  1. RFS has an elegant and natural mathematical design. Rotation and Fourier expansion provide a structural interpretation of the surrogate gradient, with STE included as the special case of zero amplitude. Comparisons with DSQ show that RFS achieves better optimization stability in some low-bit settings, demonstrating its practical value.

  2. Experiments spanning multiple quantization scenarios demonstrate optimization benefits and improvements over existing quantization baselines on several tasks. Efficiency experiments indicate that, under the tested configurations, RFS incurs runtime and GPU memory costs close to those of STE, facilitating its integration into existing quantization methods.

Critical weaknesses
  1. The paper needs more comparisons with similar gradient strategies: DSQ is tested only in limited settings, EWGS lacks a feasible task-quality comparison, and direct Fourier surrogates are not compared. The relationship to FDA-BNN and InvISP is also unexplained.

  2. Theorem 4.1 analyzes a continuous-surrogate forward pass, while Eq. (8) uses hard quantization; the proof does not explain why its result also applies to the implemented update. Appendix C.3 and Remark I also invoke vanishing uniform approximation error, which is incompatible with approximating discontinuous rounding by continuous functions across its jumps.

[1] Network Quantization with Element-wise Gradient Scaling.

Questions for authors
  1. Does RFS outperform similar gradient strategies under matched settings? Please add representative DSQ/EWGS and direct Fourier-surrogate comparisons, using a smaller task where EWGS avoids the reported OOM, and clarify the relation to FDA-BNN and InvISP.

  2. How does Theorem 4.1 apply to Eq. (8)'s hard-forward update, and in what norm does the approximation error vanish? Please explain the connection between these updates and revise the convergence assumptions.

[2] FDA-BNN: Learning Frequency Domain Approximation for Binary Neural Networks.

[3] Invertible Image Signal Processing.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8p2f

We sincerely thank Reviewer 8p2f for the constructive feedback and insightful questions. We appreciate the reviewer's recognition of the mathematical formulation, simplicity, practicality, and extensive empirical evaluation of RFS. Following the reviewer's suggestions, we have conducted additional experiments, expanded the comparisons with related methods, and revised the theoretical analysis. We address the questions in detail below.

Q1: Does RFS outperform similar gradient strategies under matched settings? Please add representative DSQ/EWGS and direct Fourier-surrogate comparisons, using a smaller task where EWGS avoids the reported OOM.

A1. Thanks for this great suggestion. Their characteristic comparison is provided in Table 1 in the revision and general response. Numerically, we conducted additional experiments to compare RFS with DSQ, EWGS, and FDA-BNN across CNN and LLM QAT.

Table. QAT for BNN on CIFAR10.

SettingMethodW1A1
ResNet20DSQ84.11
FDA-BNN85.86
Weight-RFS (A=0.05) + Act-STE85.22
Weight-STE + Act-RFS (A=0.05)85.51
Weight-RFS (A=0.05) + Act-RFS (A=0.05)86.14

Table. QAT for ResNet on ImageNet.

SettingMethodW4A4W3A3W2A2
ResNet18EWGS70.669.767.0
DSQ69.668.765.2
Weight-RFS (A=0.05) + Act-STE70.669.666.9
Weight-RFS (A=0.05) + Act-RFS (A=0.01)70.769.767.0
ResNet34EWGS73.973.371.4
DSQ72.872.570.0
Weight-RFS (A=0.05) + Act-STE73.973.371.4
Weight-RFS (A=0.05) +Act-RFS (A=0.01)74.073.371.4

Table. QAT for LLMs. Avg of common-sense reasoning.

SettingsMethodW4A16W3A16W2A16
Llama-3-1BParetoQ60.9659.5757.29
DSQ61.0757.4256.39
EWGS (δ=0.001\delta=0.001)61.0559.6857.03
Weight-RFS (A=0.21)61.2460.1258.08

Upon the above, first, RFS consistently outperforms DSQ across experiments and surpasses FDA-BNN in the binary setup. Second, RFS and EWGS achieve comparable performance on ResNets, with RFS matching or slightly exceeding EWGS at 4bit. Third, RFS consistently outperforms EWGS in the LLM QAT experiments.

To isolate the effects of surrogate design from model architecture, data distributions, and optimizer configurations, we further constructed a controlled quadratic quantization problem with a known optimal quantized solution (detailed in General Response, Q/A and Section 4.2 in revision). The results show that RFS and EWGS can perform comparably under low gradient noise. However, RFS exhibits stronger robustness as noise increases. This observation is consistent with our analysis.

Another advantage of RFS is its smoothness, whereas EWGS is not. This effect is indirectly verified by when the Fourier order M→∞M\to\infty, RFS of high-order approaches the unsmooth rounding, leading to performance degradation compared to the smooth lower-order RFS (Section 3.3 in revision).

Q2: Clarify the relation to FDA-BNN and InvISP.

A2. This is a great question. FDA-BNN and InvISP employ direct Fourier approximations onto sign and round in the original coordinate system, targeting binary networks and image processing, respectively. Such direct Fourier approximations may exhibit sharp gradient peaks (exploding) and near-flat regions (vanishing), affecting optimization performance, upon Fig. 1(b) in FDA-BNN. To mitigate, the efficacy of FDA-BNN relies on an additional noise adaption modules (NAM), making it more complex and limiting the use onto broader quantization setting.

Unlike direct Fourier approximations, RFS constructs its surrogate in a 45° rotated coordinate system, yielding a smooth, bounded and well-conditioned STE surrogate. It avoids high-order Fourier expansions, requires only one hyperparameter, and recovers STE at A=0A=0. These properties provide favorable gradient conditioning, computational efficiency, and general applicability. We provided Table 1 in the revision to highlight detailed characteristic difference between RFS’s rotated Fourier with damped amplitude and the direct Fourier formulation.

Q3: How does Theorem 4.1 apply to Eq. (8)'s hard-forward update, and in what norm does the approximation error vanish?

A3. We sincerely thank the reviewer for identifying this theoretical gap. We acknowledge this mismatch and proof error. Although fixing it is doable with extra assumptions, the revised theorem would introduce additional terms making the concluding point not natural as before. Therefore, after careful considerations, we removed the original Theorem 4.1. Instead, Theorem C.1 characterizes the approximation of hard rounding through the rotated Fourier construction. We have revised the theoretical discussion accordingly and appreciate this valuable correction.

Look forward to further discussions!

Best,

Authors

Dear Reviewer 8p2f,

We sincerely thank you again for the constructive feedback and suggestions. We hope that our revision including more comparison with EWGS from multiple perspectives, discussion with FDA-BNN and InvISP, and theorem correction, have adequately addressed the insightful questions.

If any further questions or suggestions, we would be happy to discuss and address them.

Best wishes,

Authors

We thank the reviewer for the positive assessment. We have posted a detailed, point-by-point response with new clarifications, experiments and results addressing each of the concerns raised. As the discussion period is nearing its end, we would greatly appreciate hearing whether our response has resolved the concerns, and we would be glad to clarify or provide any additional evidence that may be helpful. Thank you again for the constructive feedback.

Thank you for your response. I remain positive about the manuscript and will maintain my current rating.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers and ACs for their valuable time and constructive feedback, which have significantly improved the quality of our manuscript. We appreciate the reviewers' recognition of the following aspects of our work:

  • A simple and plug-and-play STE alternative. RFS has a clean mathematical formulation, modifies only the backward pass, and recovers STE at zero amplitude (Reviewers 8p2f, 8YrE, and uucK).
  • Theoretical foundations and optimization properties. RFS provides a smooth, well-conditioned surrogate-gradient formulation with explicitly bounded derivatives (Reviewers 8p2f and 8YrE).
  • Broad applicability and practical efficiency. RFS demonstrates improvements across diverse architectures, tasks, and quantization formats, while maintaining runtime and memory costs close to STE (Reviewers 8p2f, 8YrE, and uucK).

Following the reviewers' constructive suggestions, we have significantly strengthened the manuscript through more methodological clarifications, revised theoretical analyses, and extensive numerical experiments. All changes in the revised manuscript are highlighted in blue for ease of reference.

  • Discussion:
    • Overview. We added Table 1 to systematically compare representative surrogate-gradient methods. This comparison highlights RFS's distinctive combination of mathematical simplicity, smoothness, bounded and well-conditioned gradients, computational efficiency, and broad empirical effectiveness.
    • Fourier order study. We provided Section 3.3 with dedicated studies of Fourier order. The results show that low-order approximations (M=0,1) offer a favorable balance between optimization performance, smoothness, and computational efficiency. Higher order starts to inherit the unsmoothness even ill-condition natures from the approximated rounding which degrades the performance instead.
    • Constant Amplitude. We clarified that this work focuses on constant-amplitude RFS. Dynamic amplitude scheduling may offer additional benefits but deserves more dedicated investigations. All non-default amplitude settings (A≠0.184) are now explicitly reported for reproduction.
  • Theoretical revision:
    • Approximation of RFS to Rounding. Theorem C.1 fixed the mismatch drought by Theorem 4.1's smoothen referenced objective to directly analyze the approximation to the hard rounding function in a rotated coordinate system.
    • Noise-Dependent Drift Comparison. We introduced Theorem 4.2 to characterize the noise-dependent gradient drift of EWGS, making it sensitive to the variance of gradient, where RFS does not have this limitation. This distinction provides theoretical insight into their different optimization behaviors especially under noisy training environments.
  • Additional Numerical Experiments: We conducted additional numerical experiments including
    • QAT on Binary Neural Network to compare with FDA-BNN and DSQ (suggested by reviewer 8p2f)
    • QAT on ResNet18/34 on ImageNet to compare with EWGS and DSQ (suggested by reviewer 8p2f, 8YrE, uucK)
    • QAT on LLMs with EWGS with fixed scalar (suggested by reviewer 8YrE)
    • PTQ on VITs such as DeiT and Swin to improve quantization baseline FIMA and LS-ViT (suggested by reviewer 8YrE)
    • PTQ on LLMs to improve quantization baseline SpinQuant and FlatQuant (suggested by reviewer 8YrE)
    • Efficiency benchmark over larger models 1B-13B (suggested by reviewer uucK)
    • Controlled experiments with known ground-truth quantized solutions to investigate Fourier-order selection and compare RFS with EWGS under varying gradient noise levels. (address the questions of reviewer 8YrE and uucK)

These additional experiments demonstrate that RFS consistently improves existing STE-based quantization baselines with meaningful gains, outperforming alternative gradient scaling methods such as DSQ and EWGS across diverse settings, while providing further analytical insights into its optimization advantages.

We will address one common question shared across reviewers and also provide answers specific to each reviewer in the individual responses.

We thank the reviewers and ACs again for the valuable reviews and comments. Look forward to further discussions!

Best,

Authors

AuthorsOpenReview ↗

General response to shared question: more comparison with EWGS and other gradient surrogates

Q. More comparison with other STE surrogate, such as EWGS, DSQ, FDA in quantization quality.

A. We thank the reviewers for this important question. We further compare RFS with representative surrogate-gradient methods through analytical properties, controlled experiments, and practical CNN/LLM evaluations.

Analytical Comparison.

Table 1: Overview of representative surrogate-gradient methods† for quantization.

CharacteristicSTERFSDSQEWGSFDA-BNN
Formulation111−2πAcos⁡(π(x+xq))1+2πAcos⁡(π(x+xq))\frac{1-\sqrt{2}\pi A\cos(\pi(x+x_q))}{1+\sqrt{2}\pi A\cos(\pi(x+x_q))}C2(1−α)Sech⁡2(Cx−xqs)\frac{C}{2(1-\alpha)}\operatorname{Sech}^2\left(C\frac{x-x_q}{s}\right)1+δsign⁡(∂L∂xq)(x−xq)1+\delta\operatorname{sign}\left(\frac{\partial L}{\partial x_q}\right)(x-x_q)4ωπ∑k=0Kcos⁡((2k+1)ωx)\frac{4\omega}{\pi}\sum_{k=0}^{K}\cos((2k+1)\omega x)
Formulation TakeawayIdentityRotated damped FourierSoft quantization (tanh⁡\tanh)Gradient-dependent scalingTruncated direct Fourier
Forward MechanismExact roundingExact roundingSoft roundingExact roundingBinary sign
SmoothnessSmoothSmoothSmoothNon-smooth near ∂L∂xq=0\frac{\partial L}{\partial x_q}=0Smooth for finite truncation
Hyperparameters–Amplitude AASharpness α\alphaScaling factor δ\deltaFrequency ω\omega, truncation KK, NAM
Implementation Changes–Backward onlyForward and backwardBackward onlyBackward and NAM
Noise-Induced Gradient BiasNoneNoneNonePresent with gradient noisePresent with NAM
Gradient BoundnessConstantBoundedUnbounded as α→0\alpha\to0Bounded upon finite δ\deltaUnbounded as K→∞K\to\infty
Gradient VariancePreservedBoundedCan grow with α\alphaNoise and error dependentTruncation and NAM dependent
Computational Overhead–Near STEModerateLow (fixed) / High (Hessian)High (multi-order Fourier and NAM)
Empirical PerformanceBaselineStrong in tested settingsSetting-dependentStrong in low-noise settingsStrong in BNNs
Optimization StabilityBaselineStable in tested settingsSharpness-sensitiveNoise-sensitiveSetting and NAM dependent
Generality‡Multi-bit, INT/FP, W/AThe same as STEPrimarily CNNsPrimarily CNNsPrimarily binary networks
Recovers STE–Yes (A=0A=0)NoYes (δ=0\delta=0)No

Better features are bolded. Technically, all surrogate-gradients could be applied across settings. Generality accounts for the settings tested in the original papers or performs the strongest during extended validation.

Table 1 summarizes the key characteristics of representative surrogate-gradient methods, including their mathematical formulations, optimization properties, computational overhead, and empirical applicability.

Summary. RFS combines several desirable properties within a simple and unified formulation: smoothness, bounded and well-conditioned, preservation of the exact quantization forward pass, computational efficiency, and broad applicability.

Compared with DSQ, the quality gain of RFS comes from its bounded gradient variance and well-conditioned behavior. This distinction is consistent with our empirical observations, where DSQ exhibits larger training variability and more frequent collapse in low-bit LLM QAT sacrificing the performance in both CNNs and LLMs experiments.

Compared with EWGS, RFS provides two important optimization advantages: smoothness and avoiding noise-induced gradient drift under unbiased upstream gradient noise. In particular, the fixed-scalar EWGS formulation is

$ (\partial L/\partial x)_{EWGS}:=\partial L/\partial x_q+\delta |\partial L/\partial x_q|(x_q-x) $

where δ\delta controls the gradient correction. RFS employs

$ ({\partial L}/{\partial x})_{RFS}:=\frac{1-\sqrt{2}\pi Acos(\pi(x+x_q))}{1+\sqrt{2}\pi Acos(\pi(x+x_q))}\frac{\partial L}{\partial x_q}. $

Consider ∂L/∂xq∼N(0,σ2)\partial L/\partial x_q\sim\mathcal N(0,\sigma^2), then expected gradients satisfy

$ \mathbb{E}[(\partial L/\partial x)_{EWGS}]=\delta (x_q-x)\sigma\sqrt{\frac{2}{\pi}} =O(\sigma), $

whereas E[(∂L/∂x)RFS]=0.\mathbb E[({\partial L}/{\partial x})_{RFS}]=0.

This reveals an important difference. EWGS introduces a systematic gradient drift whose magnitude proportions to σ\sigma. Consequently, under noisy optimization, EWGS may introduce extra noise-dependent updates that alter quantization-bin transitions and compromise performance. In contrast, RFS avoids this additional source of bias, offering a more predictable optimization behavior across different architectures, applications, and training configurations.

AuthorsOpenReview ↗

Continued response to the shared question: numerical validation.

Controlled Experiment.

To isolate the impact of gradient noise on optimization, we construct a coupled quadratic quantization problem with a known optimal quantized solution (Section 4.2 in the revision).

Starting near the optimal quantization bins, we inject Gaussian noise N(0,σ2)N(0,\sigma^2) into the upstream gradients during optimization. We evaluate three metrics: (i) recovery rate, successful recovery of the optimal quantized solution; (ii) quantized loss; and (iii) false-positive bin-switch rate, measuring unintended transitions away from the optimal bins.

MethodRecovery (%) ↑Quantized Loss ↓Bin-Switch Rate (%) ↓
Noise σ0.2130.2130.213
STE100.0099.1694.590.00.0120.0710.001.476.22
EWGS (δ=0.001)100.0099.1694.600.00.0120.0710.001.486.20
EWGS (δ=0.01)100.0099.2494.720.00.0110.0690.001.346.03
EWGS (δ=0.1)100.0099.7495.780.00.0040.0560.000.464.70
RFS (A=0.05)100.0099.5296.620.00.0070.0440.000.853.81
RFS (A=0.15)100.0099.9598.920.00.0010.0150.000.101.16
RFS (A=0.184)100.0099.9999.360.00.00.0090.000.020.68
RFS (A=0.21)100.00100.0099.680.00.00.0050.000.010.34

The results match the analysis that as the gradient noise level increases, EWGS under varying δ\delta incurs more false-positive bin-switch leading to lower recovery rate and quantized loss compared to RFS. RFS performs reliably across varying noise level and amplitudes.

ImageNet CNN QAT. We further compare RFS with EWGS and DSQ on ImageNet classification using ResNet18 and ResNet34 across W4A4, W3A3, and W2A2 quantization settings. As shown below, RFS and EWGS achieve comparable performance across all evaluated bitwidths, consistently outperforming DSQ. RFS achieves slightly higher accuracy on W4A4 while matching EWGS in most other settings. These results suggest that both methods can achieve competitive quantization quality under well-established and tuned CNN training recipes.

Table. QAT for Image classification on ImageNet.

SettingMethodW4A4W3A3W2A2
ResNet18EWGS70.669.767.0
DSQ69.668.765.2
Weight-RFS (A=0.05A=0.05) + Act-STE70.669.666.9
Weight-RFS (A=0.05A=0.05) + Act-RFS (A=0.01A=0.01)70.769.767.0
ResNet34EWGS73.973.371.4
DSQ72.872.570.0
Weight-RFS (A=0.05A=0.05) + Act-STE73.973.371.4
Weight-RFS (A=0.05A=0.05) + Act-RFS (A=0.01A=0.01)74.073.371.4

Low-bit LLM QAT. We evaluate RFS against EWGS and DSQ on low-bit LLM QAT, where large-scale stochastic optimization presents a more challenging training environment. We adopt the fixed-scalar EWGS variant (δ=0.001)(\delta=0.001), as its Hessian-based variant incurs out-of-memory errors. As shown below, RFS consistently outperforms EWGS, DSQ, and ParetoQ across W4A16, W3A16, and W2A16 quantization settings. Although Hessian-based EWGS may provide additional optimization benefits, its significant computational and memory overhead limits its practicality for large models.

Table. QAT for LLaMA-3.2-1B. Avg of zero-shot common-sense reasoning.

SettingsMethodW4A16W3A16W2A16
Llama-3.2-1BParetoQ60.9659.5757.29
DSQ61.0757.4256.39
EWGS (δ=0.001\delta=0.001)61.0559.6857.03
Weight-RFS (A=0.21)61.2460.1258.08

Conclusion. Our analysis and empirical results demonstrate that RFS offers a favorable balance between quantization quality, optimization robustness, and computational efficiency. Although EWGS and RFS achieve competitive performance under well-established CNN training settings, their optimization behaviors diverge under increasing gradient noise. EWGS introduces a noise-dependent systematic gradient drift, whereas RFS does not have this limitation. Consequently, RFS exhibits stronger robustness and performance in high-noise experiments, while maintaining computational and memory efficiency close to STE.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.