- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LSTE introduces one learnable parameter per quantizer that controls the shape of its gradient estimator, trained on the next mini-batch's loss via a one-step hypergradient through the kept weight update, while adding no inference cost. The same construction learns a mixing ratio between two published estimators, revealing that the preferred estimator depends on the forward quantizer and can differ between weight and activation quantizers within one network. LSTE achieves state-of-the-art results on CIFAR-10, ImageNet (62.4% Top-1 for binarized ResNet-18), and 1-bit Llama models on C4, lowering validation loss at every model size. Ablations confirm that learning both weight and activation parameters together yields the largest gain, with their effects overlapping rather than adding.
Critical strengths
- LSTE delivers consistent, state-of-the-art accuracy gains across three distinct benchmarks with zero inference overhead. Crucially, because the learned parameters act only in the backward pass, they add no parameters or operations at inference. This is consequential because it demonstrates that the proposed backward-pass-only adaptation is not a marginal trick confined to a single setting, but a general and practically deployable improvement.
- The learned mixing-ratio experiment yields a substantive scientific finding: the preferred gradient estimator is quantizer- and modality-dependent. By mixing the clipped STE with the piecewise linear estimator and learning only the ratio, the authors show that vision models prefer the piecewise linear coefficient in both quantizer types, whereas language models prefer the piecewise linear coefficient for weights but the clipped STE for activations (Figure 3c).
Critical weaknesses
- The motivation is not novel, and the paper omits directly relevant prior work on learnable clipped-gradient parameters. The paper's central motivation, that the distributions of real-valued weights and activations differ across layers and change during training, so no single fixed gradient estimator is optimal, is essentially common sense and has been widely discussed in prior work on learned binary quantizers. More seriously, the idea of a learnable parameter controlling a clipped gradient was already proposed in SiBNN (Sparsity-Inducing Binarized Neural Networks, AAAI 2020), which is not cited or discussed. This is consequential because it undermines the paper's claim to novelty: the core mechanism of attaching a learnable shape parameter to the backward coefficient is not new, and the paper fails to position LSTE against the closest existing method, making it impossible to assess what is genuinely original.
- The one-step hypergradient design is unjustified, and its training cost is neither analyzed nor empirically vindicated. The paper trains the LSTE parameters on the next mini-batch's loss by keeping the computational graph of the weight update, which introduces extra computation and memory that are never quantified in the main body. Yet it seems straightforward to place Equation (6) inside the computational graph before the sign function and use the unclipped STE as the backward for the sign, letting the quantization parameter receive gradients from the current mini-batch. The paper provides no empirical evidence that the more complicated next-step formulation outperforms this simpler alternative, and no training speed or memory footprint comparison against baselines.
Questions for authors
-
"the peak at zero, the largest gradient it passes.", this is hard to parse in the introduction. Why should the peak at zero have the largest gradient? And what does "largest" mean here: the largest gradient the quantizer receives, or the one it proposes? Some clarification would help.
-
I personally think it's common sense that pre-quantized weights and activations have different distributions across layers, and this has been widely discussed in previous works on learned binary quantizers. So I'm not sure this motivation alone earns the paper much credit for a learned STE.
-
I find these sentences a bit misleading: "We train these parameters on the loss of the next mini-batch: they shape the weight update of the current mini-batch, and that update decides the weights at which we evaluate the next one, so we keep the computational graph of the update and differentiate the next loss through it." The part "they shape the weight update of the current mini-batch, and that update decides the weights at which we evaluate the next one" doesn't really explain why the authors use the loss of the next mini-batch, especially since this uncommon setting adds computing cost, because the graph of the current mini-batch has to stay in memory.
-
Could the authors explain why the distributions in Figure 1(a) look clipped rather than roughly Gaussian or Laplacian?
-
The notation could be more unified. For instance, the learned quantization parameter in is written differently for weights and activations, which makes it hard to follow.
-
As far as I know, a learnable for the clipped gradient was first proposed in SiBNN [1] (Sparsity-Inducing Binarized Neural Networks, AAAI 2020), but it isn't discussed here.
-
Figure 2(b) could be polished. As it stands, it's hard to see why the gradient of only goes to the quantization parameter and not to all parameters.
-
Why use the next-step loss to tune the quantization parameters, when that adds extra compute and memory (which the main body doesn't discuss)? It seems straightforward to put Equation (6) in the graph before and use the plain STE without a clip threshold as the backward for , so the quantization parameter gets a gradient from the current mini-batch. As the manuscript stands, I can't find empirical evidence that supports the more complicated setting.
-
Given the extra training cost from the frozen graph, I personally find the accuracy gains marginal. I'd also like to see a comparison of training speed and memory footprint against the baselines.