acceptodds
Under review as a conference paper at ICLR 2027

Curvature-Corrected Activation Gradients for Binarized Neural Networks

Abstract

Binarized Neural Network (BNNs) training typically relies on the straight-through estimator (STE) as a practical approximation of the derivative of the quantization layer applied to the model weights and activations. We identify that the activation gradients in standard STE are biased and that this bias can be decomposed into two components: one related directly to the STE and another related to applying the classical chain rule through a discontinuous quantizer, which propagates the tangent of the downstream loss at the selected quantization level. In contrast, the distributional derivative of the population loss is governed by the finite loss jump between adjacent levels. A second-order expansion of this jump yields a gradient-correction term involving the quantization residual and the diagonal curvature of the downstream loss. We turn this result into a practical backward rule for activation quantizers by approximating the curvature component with an empirical-Fisher proxy and apply the proposed correction during the final 10% of training. We evaluate our method on binarized Llama-style language models trained with recent state-of-the-art methods, including QuEST and CAGE, and observe a 1.0–3.1% perplexity reduction across model sizes up to 1.6B parameters.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
quantization, aware, training, language, model, curvature, correction, gradient, activation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hASHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to mitigate quantization error in network binarization within QAT. Specifically, the theoretical findings reveal that binarization error in activation gradients stems from both the STE and the chain-rule bias, and the authors propose a curvature correction using the empirical Fisher as a remedy for the latter. By integrating curvature correction into existing binarization approaches like QuEST and CAGE, the paper shows reduced perplexity (up to 3.2%) across the Llama family at different scales (30M to 1.6B) on the C4 validation benchmark.

Critical strengths
  1. The presentation is clear and easy to follow.
  2. The paper is technically sound and provides sufficient theoretical support for the methodology.
Critical weaknesses
  1. The impact on training wall-clock time and memory footprint was missing. Because curvature correction introduces additional optimizer states (i.e., a moving average of the mean-squared downstream gradient for activations) that update dynamically during training, the authors should discuss in detail how the proposed method affects training time and memory footprint and highlight its practicality.
Questions for authors

Based on the weaknesses, the author should respond to the following questions to address my major concerns.

  1. Could you provide the statistics on training wall-clock time and memory usage for the proposed method and compared baselines? Will curvature correction catastrophically increase training cost in exchange for a ∼3\sim 3% perplexity reduction?
2 comments
AuthorsOpenReview ↗

Response to Reviewer hASH

We thank the reviewer for their endorsement of our work and their constructive feedback. We are happy that they found the presentation clear and easy to follow, and the method technically sound and supported by sufficient theoretical analysis. We answer the raised weakness and question below.

Weakness 1 and Question 1 — Training wall-clock time and memory footprint

We thank the reviewer for the question and agree that the computational and memory overhead should be quantified. With an efficient implementation, the overhead is just a few percent in runtime and negligible additional memory.

In more detail, the moving average $\kappa$ requires one FP32 scalar per corrected activation quantizer, which is negligible (e.g., 320 bytes per GPU for the 800M model). Updating it requires a mean-squared-gradient reduction and a scalar update. Additional computation arises from applying the backward correction and handling activation intermediates, including the quantization residual. These intermediates need not be stored separately; in practice we use a fused quantized-linear layer that retains the prequantized input $z$ as its saved activation and recomputes $Q(z)$ and the correction residual during backward.

We measured the correction overhead against a fused QuEST baseline, using 8 H100 GPUs. We used sequence length 512 and the correction-training batch configurations: per-GPU batches of 64/32/16 with gradient accumulation of 1/2/4 for the 100M/430M/800M models, respectively, giving global batch 512 as in the original training. After 50 warmup optimizer updates, we timed 200 complete optimizer updates. Memory is the maximum peak allocated GPU memory across ranks, reported in decimal GB.

Method100M runtime (s/update)100M memory (GB)430M runtime (s/update)430M memory (GB)800M runtime (s/update)800M memory (GB)
Baseline (QuEST)0.09445 ± 0.0014619.296430.31253 ± 0.0011232.937540.54446 ± 0.0010838.34960
Baseline + ours0.10305 ± 0.0001619.296450.33581 ± 0.0012032.937940.58143 ± 0.0007338.34964

Runtime values are mean ± standard deviation over three repetitions. The runtime overhead, calculated from the ratio of means, decreases with model size: 9.10%, 7.45%, and 6.79% for 100M, 430M, and 800M, respectively. Peak allocated memory is effectively unchanged, increasing by approximately 0.4 MB per GPU. Since correction is applied only during the final 10% of training updates, its estimated contribution to total training time is approximately 0.68–0.91%.

I appreciate the detailed responses, which address my concerns about the practical efficiency of the proposed method. As a result, I will maintain a positive score and finalize my recommendation as a 'Weak Acceptance'.

Reviewer_7x1xOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The authors propose a curvature-based correction to STE in binary neural network training.

Critical strengths
  • Good intuitive explanation why curvature is relevant to consider in Fig. 1 and 2
Critical weaknesses
  • Clarity and comparability to related work

    • The comparison to related work is superficial and needs to be expanded. Is the key novelty of this paper that it applies well-known curvature-aware corrections to activations (when they are usually applied to weights)? Or is it a new approximation of curvature-awareness? If the latter, can it also be applied to weights (which might be of wider general interest)? -- Since there is literature on curvature-awareness for weights, how does this related work transport to activations, and how does it differ from the proposed method?
    • There is no description of the method in code or algorithmic form (no supplemental code is supplied either). This makes it difficult to compare in detail to related methods and to disambiguate mathematical embellishment from the practical method.
  • Evaluation is limited

    • While the theory is concerned with correcting a biased estimate, the bias (and its correction) is not measured.
    • Results are limited to LLM and binary settings.
Questions for authors
  • What exactly is the computation you perform at backprop?

  • How does this paper relate to existing curvature corrections? (see weaknesses)

  • Does your method make the gradient estimate unbiased? It is currently not clear to me whether the story about bias is just framing or a substantial part of the method. Can you measure this?

  • Can you clarify your use of AI? The claim that AI was not used to "develop the proposed method, theoretical models, or conceptual frameworks; formulate mathematical claims; propose or refine hypotheses; derive the theoretical proofs or provide critical ingredients for them" seems to contradict the initial part of that section (AI assistance for "mathematical derivations; implementing training").

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7x1x (1/3)

We thank the reviewer for their constructive feedback. We are happy that they found Figures 1 and 2 helpful in providing an intuitive explanation of why curvature is relevant. We answer each raised weakness and question below.

Weakness 1 and Question 2 — Relationship to existing curvature corrections and novelty

We thank the reviewer for their comments and useful suggestions. We view the main contribution of the paper to be a curvature correction of the activation gradients derived from the perspective of reducing the activation gradient bias in standard QAT. We do not see this as an application of an existing curvature correction approach for weights (e.g. CAGE [X1] , LOTION [X2]) on the activations.

Note that CAGE’s [X1] practical correction for weights is applied after the optimizer update and does not directly define a correction for activations, where the correction needs to be done during backpropagation. LOTION [X2] is used for weight-only quantization and uses a full-precision forward pass with a curvature-aware regularizer during training. We see EWGS [X3] as the most related method to ours, using a different formulation of curvature correction for both weights and activations, yet evaluated only on convolutional models. Specifically, EWGS weights the residual correction by the current elementwise gradient magnitude, whereas our method uses a shared moving-average statistic per quantizer. PEGE [X4] also uses a residual correction, but with a scheduled coefficient and progressive replacement of full-precision values by quantized values.

We added a more detailed comparison in Appendix F and would integrate it in the body of the paper in the revision.

We agree with the reviewer that an evaluation of our methods on weights would be beneficial. We provide below results of our curvature correction applied on both weights and activations, and compare with our implementation of EWGS.

Results: We report final C4 validation perplexity and relative improvement compared to QuEST. For both our method and EWGS, we apply the correction only during the last 10% of training, which gives the best results for both methods. We also provide validation loss curves in Figure 8. At 100M and 200M, the two methods perform similarly; however, at 430–800M, our correction outperforms EWGS. Note that across all model sizes, applying our correction to both weights and activations performs better than using CAGE for weight correction, as in CAGE+ours in Table 1 of the paper, indicating that our method is useful for correcting weight gradients as well as activation gradients.

TransformMethod100M200M430M800M
No HTQuEST (baseline)37.2031.2825.8722.51
No HTEWGS (A&W)34.89 (-6.21%)29.39 (-6.05%)24.73 (-4.42%)21.73 (-3.48%)
No HTOurs (A&W)34.84 (-6.35%)29.31 (-6.31%)24.26 (-6.23%)21.11 (-6.22%)
HTQuEST (baseline)36.3730.4424.9221.40
HTEWGS (A&W)32.85 (-9.68%)27.77 (-8.77%)23.35 (-6.30%)20.45 (-4.4%)
HTOurs (A&W)32.91 (-9.50%)27.80 (-8.68%)22.95 (-7.90%)19.89 (-7.1%)

[X1] Tabesh et al. CAGE: Curvature-Aware Gradient Estimation for Accurate Quantization-Aware Training. 2025.

[X2] Kwun et al. LOTION: Smoothing the Optimization Landscape for Quantized Training. 2025.

[X3] Lee, Kim, and Ham. Network Quantization with Element-wise Gradient Scaling. CVPR 2021.

[X4] Zhao, Progressive Element-wise Gradient Estimation for Neural Network Quantization, 2025

AuthorsOpenReview ↗

Response to Reviewer 7x1x (2/3)

Weakness 1 and Question 1 — Practical backward computation and algorithmic description

We provide a standalone implementation of the practical method. For incoming gradient $g$, the backward pass returns $M \cdot (g + \alpha \kappa r)$, where $M$ is the baseline STE mask and r=clip⁡(x−Q(x),−Δ/2,Δ/2)r=\operatorname{clip}(x-Q(x),-\Delta/2,\Delta/2). Each quantizer maintains one FP32 scalar $\kappa$, initialized to zero. After computing the corrected gradient, it updates κ←0.9κ+0.1mean⁡(g2)\kappa\leftarrow0.9\kappa+0.1\operatorname{mean}(g^2) using the incoming, uncorrected gradient. No Hessian computation or additional backward pass is required.

We provide below a working simplified (unoptimized) implementation of our correction, based on the baseline implementation of CAGE we use in our experiments, which is now added in Appendix G. We will make the full implementation codebase public upon acceptance of the paper.

class _CurvatureSTE(torch.autograd.Function): 
  @staticmethod 
  def forward(ctx, x, xq, mask, residual, quantizer):
    ctx.save_for_backward(mask, residual) 
    ctx.quantizer = quantizer 
    return xq

  @staticmethod
  def backward(ctx, grad_output):
    mask, residual = ctx.saved_tensors
    q = ctx.quantizer
    kappa = 0.0 if q.fisher is None else q.fisher
    correction = q.alpha * kappa * residual
    grad_x = mask * (grad_output + correction)

    second_moment = grad_output.detach().float().square().mean()
    if q.fisher is None:
      q.fisher = torch.zeros_like(second_moment)
    q.fisher.mul_(q.beta).add_(second_moment, alpha=1 - q.beta)
    return grad_x.to(grad_output.dtype), None, None, None, None

class MSEQuantizerCurvatureCorrection(nn.Module): 

  def __init__(self, bits=1, centered=True, clip_scale=1.1,alpha=30000.0, beta=0.9):
    super().__init__()
      self.bits = bits
      self.n_levels = 2**bits
      self.centered = centered
      self.clip_scale = clip_scale
      self.alpha = alpha
      self.beta = beta
      self.fisher = None

  def forward(self, x):
    with torch.no_grad():
      scale = (OPTIMAL_GAUSSIAN_SCALES[self.bits]* torch.sqrt(torch.mean(x**2, dim=-1, keepdim=True)) + 1e-8)
      step = 2 * scale / (self.n_levels - 1)
      x_clip = torch.clamp(x, -scale, scale)
      xq = torch.round(x_clip / step + 0.5) * step - step / 2
      mask = (x.abs() <= scale * self.clip_scale).float()
      residual = torch.clamp(x - xq, -step / 2, step / 2)
      return _CurvatureSTE.apply(x, xq, mask, residual, self)

Weakness 2 and Question 3 — Gradient bias and its correction

We thank the reviewer for the opportunity to clarify this point. Our method does not generally make the gradient estimator completely unbiased, but rather less biased. The theoretical correction targets the bias arising from applying the classical chain rule through a discontinuous quantizer in standard QAT. Even when this component is canceled exactly, as in our quadratic toy model, bias from approximating the quantizer’s distributional derivative remains (Eqs. (7) and (12)). The practical implementation introduces additional approximations and generally leaves some of the chain-rule bias as well.

Quantifying gradient bias requires a reference gradient of the true population loss. We do not have an independently computable reference population gradient for our language-model experiments, e.g. since it requires the true data distribution (see Equation 7 of the paper), which is unavailable in machine learning.

Instead, we expanded in Appendix E the example in Figure 2, where the population gradient is available analytically. This allows us to compute the exact bias of all three methods and compare their magnitudes at matched parameter values. The new analysis shows bias reduction for both “theoretical” and “practical” corrections in the region explored during training, while also identifying their remaining bias.

We additionally trained the methods for 1,000 updates with minibatches of 1,024 samples and compared every training gradient with the exact population gradient evaluated at the same parameter. The results are:

MethodMean absolute gradient error
STE0.19624
Theoretical correction0.01528
Practical correction0.14639

The practical correction reduces this error by 25.4% relative to STE in this run. This training measurement includes both bias and sampling noise; the appendix separately compares the exact biases. Together, these results provide direct evidence for the proposed bias-correction mechanism in a controlled setting where the reference gradient is known.

AuthorsOpenReview ↗

Response to Reviewer 7x1x (3/3)

Question 4 — Clarification of AI use

The intention in “editing the manuscript, including textual content and mathematical derivations” is that we used AI tools for editing tasks such as fixing typos and grammatical errors in all parts of the paper, including notational errors in our mathematical derivations (e.g. missing subscripts, inconsistent notations, etc.). The intention in “implementing training scripts” is that we used AI tools for generating scripts used to execute the experiments, not the core implementation of the proposed method. In both cases we have reviewed and approved all outputs of AI tools. We do not see this usage to be contained in any of the mentioned usage categories.

The rebuttal addresses most of my concerns and I have raised my score.

Reviewer_p6pFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a curvature inspired correction to the STE for binarized NN (specifically for LLMs). The theory derives a second order update from Taylor expansion and a practical approximation of the second order term is provided. The gain is evaluated empirically on a toy example and on a real world example.

Critical strengths

The method provides a modest but very consistent gain. The practical approximation is computationally cheap and it's inclusion seems to have no downsides.

Critical weaknesses
  • The gain is modest and evaluation is on a single seed. Intra seed variability could invalidate the results.
  • There's a substantial gap between the theory and the practical implementation (both in form and in the results of Fig.2).
  • Similar ideas can be found in the literature, making the originality of the contribution slightly overstated.
  • Some relevant related work is cited but not included in the comparison.
Questions for authors
  • How large is the intra seed variance? Is the improvement in performance statistically significant. I would like to see multiple seeds, at least for the smallest configurations if compute is a concern.
  • Why is the method only applied to the final part of the training? Does the gain improve/degrade if the correction is used for the entire training?
  • Why isn't the method being compared to other gradient corrections? For instance, I would like to see EWGS included in the comparisons.
5 comments
AuthorsOpenReview ↗

Response to Reviewer p6pF (1/3)

We thank the reviewer for their constructive feedback. We are happy that they found the method’s gains consistent, the practical approximation computationally cheap, and the presentation clear and easy to follow. We answer each raised weakness and question below.

Weakness 1 and Question 1 — Variability across seeds and statistical significance

We have repeated the experiments of Table 1 of the paper for the 30M, 50M, and 100M models over three independent training seeds, both with and without the Hadamard transform (HT). For each seed, we train QuEST from scratch and apply the three correction methods (QuEST+ours, CAGE, CAGE+ours) during the final 10% of training.

We report for each method and model size the final validation perplexity (mean $\pm$ standard deviation). To isolate the variability of relative improvement from the variability of the baseline results across the seeds, we report separately the relative improvement percentage of each method on the same seed (QuEST+ours against QuEST, CAGE+ours against CAGE; mean $\pm$ standard deviation).

The results in the table below show consistent improvement of our method over QuEST and CAGE across the tested seeds.

TransformMethod30M PPL ↓30M Improvement (%) ↑50M PPL ↓50M Improvement (%) ↑100M PPL ↓100M Improvement (%) ↑
No HTQuEST51.62 ± 0.29—45.15 ± 0.29—37.26 ± 0.05—
No HTQuEST + ours50.79 ± 0.181.61 ± 0.2244.52 ± 0.361.40 ± 0.3036.41 ± 0.082.26 ± 0.32
No HTCAGE49.86 ± 0.31—43.28 ± 0.42—35.76 ± 0.02—
No HTCAGE + ours49.05 ± 0.231.64 ± 0.1942.61 ± 0.371.55 ± 0.0934.94 ± 0.082.27 ± 0.18
HTQuEST50.70 ± 0.21—43.86 ± 0.08—36.26 ± 0.19—
HTQuEST + ours49.42 ± 0.302.53 ± 0.5042.81 ± 0.162.41 ± 0.3435.20 ± 0.152.94 ± 0.14
HTCAGE48.75 ± 0.32—42.13 ± 0.13—34.64 ± 0.18—
HTCAGE + ours47.34 ± 0.222.89 ± 0.1940.75 ± 0.103.27 ± 0.1233.61 ± 0.142.98 ± 0.12

Regarding larger model sizes, we added in Figure 7 the appendix unsmoothed validation loss curves during training of QuEST, QuEST+ours, CAGE, CAGE+ours. The gaps between the methods can be seen visually larger than the variability of validation loss between iterations in the same seed.

Weakness 2 — Gap between theory and practical implementation

We thank the reviewer for highlighting this gap. The practical method replaces two components of the theoretical correction: (1) the Hessian diagonal with an empirical-Fisher-style curvature proxy for computational efficiency, and (2) the fixed offset given by the negative quantized activation with a clipped quantization residual, we find to perform better empirically. These approximations introduce additional error, and we agree that clearer validation of their effect would be useful.

Appendix B.4 (Figure 6) empirically evaluates the effect of these practical design choices on the LLM performance, comparing the curvature proxy versus a constant coefficient, and the quantization residual versus the theoretical offset. After sweeping $\alpha$ separately for each combination, the moving-average proxy combined with the residual achieves the lowest final validation loss at all three tested model sizes: 30M, 50M, and 100M. These controlled comparisons support both practical design choices.

We additionally expanded in Appendix E the example in Figure 2 of the paper with an analysis of the exact gradient bias of each method and an additional minibatch-training experiment. While there is a remaining gap between the practical and theoretical corrections, the analysis shows the practical correction reduces gradient bias compared with regular STE, even when the curvature proxy does not match the exact curvature.

In the empirical measurement on minibatch training, the practical method reduces the mean absolute training-gradient error by 25.4% relative to STE and improves the final population loss, although not to the same extent as the theoretical correction.

AuthorsOpenReview ↗

Response to Reviewer p6pF (2/3)

Weakness 3 — Originality relative to existing work

We thank the reviewer for the opportunity to clarify the distinction of our work.. We view the main contribution of the paper to be a curvature correction of the activation gradients derived from the perspective of reducing activation-gradient bias in standard QAT. Our analysis separates the quantizer-surrogate error from the error in approximating the finite loss jump by a local tangent. Holding the surrogate fixed, we derive a correction for the latter under our scalar assumptions.

We do not see this as an application of an existing curvature correction approach for weights (e.g. CAGE [P1] , LOTION [P2]) on the activations. CAGE’s [P1] practical correction for weights is applied after the optimizer update and does not directly define a correction for activations, where the correction needs to be done during backpropagation. LOTION [P2] is used for weight-only quantization and uses a full-precision forward pass with a curvature-aware regularizer during training. We see EWGS [P3] as the most related method to ours, using a different formulation of curvature correction for both weights and activations, yet evaluated only on convolutional models. Specifically, EWGS weights the residual correction by the current elementwise gradient magnitude, whereas our method uses a shared moving-average statistic per quantizer. PEGE [P4] also uses a residual correction, but with a scheduled coefficient and progressive replacement of full-precision values by quantized values.

We added a more detailed comparison in Appendix F and would integrate it in the body of the paper in the revision.

We provide additional comparisons with the Fixed-EWGS version of [P3] in our response to Weakness 4 and Question 3, applying both methods to weights and activations. At 100M and 200M the two methods perform similarly, while at 430M in both transform settings and at 800M without HT our correction achieves lower perplexity. Using our correction on both weights and activations also performs better than CAGE+ours at all available matching settings.

[P1] Tabesh et al. CAGE: Curvature-Aware Gradient Estimation for Accurate Quantization-Aware Training. 2025.
[P2] Kwun et al. LOTION: Smoothing the Optimization Landscape for Quantized Training. 2025.
[P3] Lee et al. Network Quantization with Element-wise Gradient Scaling. CVPR 2021.
[P4] Zhao. Progressive Element-wise Gradient Estimation for Neural Network Quantization. 2025.

Weakness 4 and Question 3 — Comparisons with other gradient corrections

We thank the reviewer for raising this suggestion and agree such a comparison would strengthen the paper. We note that EWGS is a method from 2021 and was evaluated only on smaller convolutional models, and its main variation requires computing second derivatives for measuring the Trace of the Hessian, which is impractical in our case.

To address the request of the reviewer, we compare our method against the "Fixed-EWGS" version of [P3], which replaces the Hessian trace with a fixed hyperparameter. We tune this hyperparameter separately for the weights and activations, similar to the tuning of alpha in our method. For a fair comparison, we similarly apply our correction on both weights and activations and tune alphas separately. Both methods are tuned on the 100M model without the Hadamard transform.

Results: We report final C4 validation perplexity and relative improvement compared to QuEST. For both our method and EWGS, we apply the correction only during the last 10% of training, which gives the best results for both methods. We also provide validation loss curves in Figure 8. At 100M and 200M, the two methods perform similarly; however, at 430–800M, our correction outperforms EWGS. Note that across all model sizes, applying our correction to both weights and activations performs better than using CAGE for weight correction, as in CAGE+ours in Table 1 of the paper, indicating that our method is useful for correcting weight gradients as well as activation gradients.

TransformMethod100M200M430M800M
No HTQuEST (baseline)37.2031.2825.8722.51
No HTEWGS (A&W)34.89 (-6.21%)29.39 (-6.05%)24.73 (-4.42%)21.73 (-3.48%)
No HTOurs (A&W)34.84 (-6.35%)29.31 (-6.31%)24.26 (-6.23%)21.11 (-6.22%)
HTQuEST (baseline)36.3730.4424.9221.40
HTEWGS (A&W)32.85 (-9.68%)27.77 (-8.77%)23.35 (-6.30%)20.45 (-4.4%)
HTOurs (A&W)32.91 (-9.50%)27.80 (-8.68%)22.95 (-7.90%)19.89 (-7.1%)
AuthorsOpenReview ↗

Response to Reviewer p6pF (3/3)

Question 2 — Applying the correction only during the final part of training

We thank the reviewer for raising this question. We chose to apply the curvature correction during the final 10% of training to align with CAGE, which applies its auxiliary weight-quantization correction over the same interval. We added a duration ablation on the 50M A1W1 QuEST model without HT. Correction remains helpful across all tested windows, from the final 5% to the final 30%, provided that $\alpha$ is adjusted accordingly. Longer windows generally favor smaller coefficients (Appendix D.1, Figure 9).

Applying curvature correction from scratch, however, provides no appreciable benefit: large $\alpha$ degrades performance, while tuned $\alpha$ roughly matches the uncorrected baseline (Table 4).

These two ablations provide empirical support for late application of the curvature correction. One possible explanation is that the curvature proxy becomes more accurate later in training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.