acceptodds
Under review as a conference paper at ICLR 2027

Learned Straight-Through Estimators for Binarized Neural Networks

Abstract

Binarized neural networks (BNNs) training requires an approximation of the gradient of the sign function, whose derivative is zero almost everywhere. Current methods replace the sign gradient with some hand-crafted surrogate, such as the straight-through estimator (STE) and its variants. The STE shape is decided in advance, whether held constant or varied by a progressive schedule; this is despite the distributions it acts change over layers and epochs. To address this, we propose the Learned STE (LSTE), which lets the training objective tune the shape of the STE in every quantizer. Specifically, we give each weight quantizer and each activation quantizer one learnable parameter that controls the shape of their STE. These parameters act only in the backward pass, so the current mini-batch gives them no gradient; we train them instead on the loss of the next mini-batch, which depends on the weight update they shape. The forward pass is unchanged, so LSTE combines with different forward quantizers and adds no cost at inference. The same mechanism also selects between published estimators by learning the ratio that mixes them, and shows that the preferred one depends on the quantizer. Experiments on CIFAR-10, ImageNet, and 1-bit language models of the Llama family trained on C4, show that LSTE improves on state-of-the-art methods.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
binarized neural networks, quantization-aware training, gradient estimation, meta gradient learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sNDHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LSTE introduces one learnable parameter per quantizer that controls the shape of its gradient estimator, trained on the next mini-batch's loss via a one-step hypergradient through the kept weight update, while adding no inference cost. The same construction learns a mixing ratio between two published estimators, revealing that the preferred estimator depends on the forward quantizer and can differ between weight and activation quantizers within one network. LSTE achieves state-of-the-art results on CIFAR-10, ImageNet (62.4% Top-1 for binarized ResNet-18), and 1-bit Llama models on C4, lowering validation loss at every model size. Ablations confirm that learning both weight and activation parameters together yields the largest gain, with their effects overlapping rather than adding.

Critical strengths
  • LSTE delivers consistent, state-of-the-art accuracy gains across three distinct benchmarks with zero inference overhead. Crucially, because the learned parameters act only in the backward pass, they add no parameters or operations at inference. This is consequential because it demonstrates that the proposed backward-pass-only adaptation is not a marginal trick confined to a single setting, but a general and practically deployable improvement.
  • The learned mixing-ratio experiment yields a substantive scientific finding: the preferred gradient estimator is quantizer- and modality-dependent. By mixing the clipped STE with the piecewise linear estimator and learning only the ratio, the authors show that vision models prefer the piecewise linear coefficient in both quantizer types, whereas language models prefer the piecewise linear coefficient for weights but the clipped STE for activations (Figure 3c).
Critical weaknesses
  • The motivation is not novel, and the paper omits directly relevant prior work on learnable clipped-gradient parameters. The paper's central motivation, that the distributions of real-valued weights and activations differ across layers and change during training, so no single fixed gradient estimator is optimal, is essentially common sense and has been widely discussed in prior work on learned binary quantizers. More seriously, the idea of a learnable parameter controlling a clipped gradient was already proposed in SiBNN (Sparsity-Inducing Binarized Neural Networks, AAAI 2020), which is not cited or discussed. This is consequential because it undermines the paper's claim to novelty: the core mechanism of attaching a learnable shape parameter to the backward coefficient is not new, and the paper fails to position LSTE against the closest existing method, making it impossible to assess what is genuinely original.
  • The one-step hypergradient design is unjustified, and its training cost is neither analyzed nor empirically vindicated. The paper trains the LSTE parameters on the next mini-batch's loss by keeping the computational graph of the weight update, which introduces extra computation and memory that are never quantified in the main body. Yet it seems straightforward to place Equation (6) inside the computational graph before the sign function and use the unclipped STE as the backward for the sign, letting the quantization parameter receive gradients from the current mini-batch. The paper provides no empirical evidence that the more complicated next-step formulation outperforms this simpler alternative, and no training speed or memory footprint comparison against baselines.
Questions for authors
  1. "the peak at zero, the largest gradient it passes.", this is hard to parse in the introduction. Why should the peak at zero have the largest gradient? And what does "largest" mean here: the largest gradient the quantizer receives, or the one it proposes? Some clarification would help.

  2. I personally think it's common sense that pre-quantized weights and activations have different distributions across layers, and this has been widely discussed in previous works on learned binary quantizers. So I'm not sure this motivation alone earns the paper much credit for a learned STE.

  3. I find these sentences a bit misleading: "We train these parameters on the loss of the next mini-batch: they shape the weight update of the current mini-batch, and that update decides the weights at which we evaluate the next one, so we keep the computational graph of the update and differentiate the next loss through it." The part "they shape the weight update of the current mini-batch, and that update decides the weights at which we evaluate the next one" doesn't really explain why the authors use the loss of the next mini-batch, especially since this uncommon setting adds computing cost, because the graph of the current mini-batch has to stay in memory.

  4. Could the authors explain why the distributions in Figure 1(a) look clipped rather than roughly Gaussian or Laplacian?

  5. The notation could be more unified. For instance, the learned quantization parameter in c(u;θ)c(u;\theta) is written differently for weights and activations, which makes it hard to follow.

  6. As far as I know, a learnable θ\theta for the clipped gradient was first proposed in SiBNN [1] (Sparsity-Inducing Binarized Neural Networks, AAAI 2020), but it isn't discussed here.

  7. Figure 2(b) could be polished. As it stands, it's hard to see why the gradient of Lt+1\mathcal{L}^{t+1} only goes to the quantization parameter and not to all parameters.

  8. Why use the next-step loss to tune the quantization parameters, when that adds extra compute and memory (which the main body doesn't discuss)? It seems straightforward to put Equation (6) in the graph before Sign(⋅)Sign(\cdot) and use the plain STE without a clip threshold as the backward for Sign(⋅)Sign(\cdot), so the quantization parameter gets a gradient from the current mini-batch. As the manuscript stands, I can't find empirical evidence that supports the more complicated setting.

  9. Given the extra training cost from the frozen graph, I personally find the accuracy gains marginal. I'd also like to see a comparison of training speed and memory footprint against the baselines.

Reviewer_YWrGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper deals with the choice of surrogate gradients for the sign function in BNNs. The author's claim is that prior work estimators are designed by hand and shared across all quantizers and layers. On the other hand, LSTE attached a scalar value in every weight and activation quantizer to balance peak height and support width. Since θ\theta only acts in the backward pass, the authors train it on the t+1t+1 mini-batch loss by keeping the graph of the weight update tt. Moreover, they also propose a method to learn a mixing ratio between two gradient estimators, i.e., the clipped STE and the triangular coefficient.

Critical strengths
  1. The paper is clear and the proposed method is sound, and I found the mathematical derivations correct and well-grounded
  2. The method is drop-in since the forward pass is untouched and it composes with different forward quantizers at no cost during inference
  3. The Appendix is complete and detailed, enabling full reproducibility once code is released
Critical weaknesses
  1. Ablations doesn't clearly show the effect of per-quantizer learning as opposed to just using a θ≠1\theta \ne 1, since every ablation compares a learned θ\theta with θ=1\theta = 1 and LSTE parameter trajectories shown in Appendix A.1 ends to values different than 1 (aka_k between 2.8 and 5.4, bkb_k between 1.6 and 2.1).

  2. The results on C4 dataset with the language models seem to not use LSTE parameters. Following Appendix C.3, ak=bk=1a_k = b_k = 1 are fixed, the activations use the QuEST clipped STE with τ=1.3\tau = 1.3, and the weights use the triangular coefficient with a hand-set κ=1.5\kappa = 1.5, so the gains reported in Table 3 come from a fixed, hand-designed estimator. The mixing ratio signal is also weak since on the activation side about half of the ratios end above 12\frac{1}{2}, while on the weight side they stay near their initial value, and no Ablation is shown as done for vision models in Appendix A.5. Finally, the 100M and 200M baselines are read from a published figure, although Appendix C.1 declines to quote those same in-training evaluations.

  3. All the reported results are single runs without standard deviations computed on different seeds. Moreover, the binomial SE on CIFAR-10 at ~88% is approximately 0.33%, while for Imagenet is approximately 0.22%, meaning that most of the reported improvements over SOTA are within evaluation noise, before even accounting for seed variance, which is typically larger in BNN training.

  4. Each step of the proposed method performs 2 backward passes, with the second differentiating through the graph of the previous weight gradient, which requires holding the backward graph of step tt in memory through step t+1t+1. Training cost evalutations are needed, both in terms of computational and memory requirements (e.g., wall-clock time and memory usage), ideally with a comparison against baselines given the same compute budget.

  5. The claim in related literature that every prior estimator is fixed before training is overstated. Spiking NN literature learns surrogate shapes directly (e.g., Dspike (Li et al., NeurIPS 2021), LSG (Lian et al., IJCAI 2023)), so the paper should either compare against these methods or argue why they are not comparable. Moreover, the claim that BNN training requires approximating the sign gradient holds only for surrogate-gradient training, while is not true for gradient-free rules that update binary weights directly (e.g., BEP (Colombo et al., ICLR 2026)), so the paper should scope its claim and position itself accordingly.

Questions for authors
  1. Which accuracy do vision models reach with aka_k and bkb_k fixed from step 0 at their final learned values or at a per-type mean?
  2. Are the gains over SOTA statistically significant if considering STD over multiple seeds?
  3. What are per-iteration wall-clock time and peak memory compared to fixed-coefficient baselines? What are the results under same compute budget?
Reviewer_YsmoOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  • Training BNNs requires approximations of the sign function since the sign function is non-differentiable. Over the past decade, this challenge has been extensively explored, giving rise to a suite of algorithms. This work carries out a thorough investigation into this problem and then introduces a novel function, the Learned STE (LSTE), which allows the training objective to adapt the shape of the STE within each quantizer.

  • The authors present extensive discussions and experimental validations of this function in both the main text and the appendix. The proofs are comprehensive and straightforward for readers to follow.

  • In terms of writing, the paper is well structured. Both the theoretical analysis and experimental sections are clearly laid out. The experimental results demonstrate that the proposed method works well.

Critical strengths
  • The main strength of this paper lies in its crystal-clear proofs and analysis for the LSTE method. The theoretical derivations establish the soundness of the proposed approach.
  • The authors run experiments on both conventional CNNs and Llama models to verify the generalizability of their method.
  • The formulas, figures and algorithm pipelines in this manuscript are well presented and easy to follow.
Critical weaknesses
  • The primary weakness of this paper lies in its experimental evaluation.
  1. In the main text, the authors mostly compare results on CNN architectures and skip standard Transformer baselines before moving directly to Llama. Experiments on Transformer networks are missing. While prior work in this area typically benchmarks on CNNs, evaluations on Transformer models have become necessary in the Transformer era.
  2. I notice that in Table 1 and Table 2, the proposed LSTE method achieves exactly a 0.2 accuracy gain over the second-best approach. I recommend the authors double-check these results for potential issues. If the improvement no longer appears as a neat 0.2 after using more decimal places, please report values with higher precision or add relevant explanations.
  3. I suggest running repeated experimental trials and reporting mean values together with standard deviations in the tables.

The theoretical part of this paper is reasonably sound, but I recommend expanding the experimental validation.

Questions for authors

To see above weaknesses. The primary weakness of this paper lies in its experimental evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.