acceptodds
Under review as a conference paper at ICLR 2027

STE Is Not a Heuristic: A Variational Justification in Quantization-Aware Training

Abstract

The straight-through estimator (STE) is widely used in quantization-aware training (QAT) and other discrete optimization problems in deep learning, but it is considered to be a convenient heuristic with limited theoretical justification. We show that STE admits a principled variational derivation: starting from a variational relaxation of the combinatorial quantization objective under stochastic rounding, STE emerges as the first-order Taylor approximation of the multivariate expansion around the MAP configuration, placing it on rigorous theoretical footing. This variational perspective naturally extends beyond uniform grids: replacing stochastic rounding with a Laplace categorical distribution that is annealed toward the hard solution concentrates gradient signal on uncertain assignments and enables joint learning of non-uniform codebooks whose grid points migrate to minimise reconstruction error, an extension we call non-uniform latent QAT (NUL-QAT). Evaluating on the Qwen3 family (0.6B–32B) at 3-, 2-, and 1.58-bit precision with block-wise QAT, we find that NUL-QAT outperforms STE-QAT on WikiText perplexity and on MMLU, GSM8K, WinoGrande, HellaSwag, and ARC-C accuracy, with the largest gains at extreme bit-widths, confirming that the variational framework captures weight-distribution structure that uniform grids cannot. A custom Triton kernel evaluates the Laplace expectation without materialising the K-fold probability tensor, and the learned non-uniform grids allow efficient inference with lookup table-based kernels such as FLUTE.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model quantization, straight-through estimation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_9nPVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a coherent framework connecting STE to a first-order approximation of a variational objective and uses this perspective to motivate a differentiable relaxation for learning non-uniform quantization codebooks.

Critical strengths

NUL-QAT combines a learnable non-uniform codebook with a Laplace-based categorical relaxation and a fused Triton implementation. Its evaluation across multiple Qwen3 model sizes and 1.58-, 2-, and 3-bit settings demonstrates the practical relevance of the approach.

Critical weaknesses

The theoretical analysis provides useful intuition but does not yet fully substantiate the broad claim that STE has a variational justification. The theoretical analysis provides useful intuition but does not yet fully substantiate the broad claim that STE has a variational justification.

Questions for authors

1、Under what precise assumptions do the first-order Taylor approximation and stochastic-rounding derivation establish the claimed connection between NUL-QAT and STE? Can the authors clarify the scope of this equivalence 2、The paper highlights annealing the Laplace temperature (\sigma) as a key component of the proposed method, yet its contribution is not explicitly isolated in the ablation study.

1 comment

Thank you for the positive review! Please find our replies below:

As to the assumptions and scope, the equivalence requires only that the assignment distribution be stochastic rounding and the first-order expansion around the hard assignments. It holds for any differentiable loss and any sorted grid, uniform or otherwise. Indeed, you can generate many different differentiable proxies for quantization just by changing the assignment distribution (Laplace categorical, softmax, or anything else), and STE falls out as a special case for stochastic rounding.

On the matter of \sigma, you are correct, we did not conduct a formal ablation study on its values and annealing schedule. Based on our preliminary experiments, we did find that the final performance is sensitive to the ending value of \sigma (setting it too small tended to hurt performance), but we did not explore this part of the hyperparameter space in depth.

Reviewer_bWc6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper shows that gradients estimated with the Straight-Through Estimator (STE) can be interpreted as the gradient of the first-order Taylor expansion of a variational relaxation of the original objective under stochastic rounding. While the original objective is to minimize the loss by learning a single quantized network (i.e. a single set of grid assignments), the variational relaxation aims to minimize the expected loss under a learned categorical distribution of quantized networks. The probability of this learned distribution is parameterized by a set of learned weights mu. The paper shows when at most two categories/grid points of this learned distribution can have non-zero probability (i.e. stochastic rounding), and each probability is proportional to the distance between the corresponding grid point and mu, then the STE-estimated gradient is exactly the gradient of the first-order expansion (around the mode of the distribution) of the variational relaxation with respect to mu. The paper then proposes NUL-QAT, which allows all categories to have non-zero probability (Laplace), and these probabilities are the softmax of the negative distance between the corresponding grid points and mu. The softmax temperature is a hyperparameter sigma. The paper evaluates the performance of NUL-QAT, showing it outperforms the STE in LLM finetuning.

Critical strengths

NUL-QAT as a gradient estimator for non-uniform quantization with learned grid points is a novel contribution. The empirical results are convincing. The ideas are clearly presented and the paper is generally entertaining to read.

Critical weaknesses

There are no critical weaknesses.

Questions for authors

I think the paper is above the ICLR acceptance threshold without any modifications. Here are some suggestions. Addressing my suggestions is optional.

  1. Consider discussing activation quantization. Does the theory in Section 2 work when L is non-differentiable (e.g. when the network has activation quantizers)? Does NUL-QAT work for activation quantization?
  2. On line 257, the authors said they will show (in the Appendix) that NUL-QAT produces useful gradients with respect to grid points. I cannot find this information anywhere in the appendix. If I am wrong, then ignore this. Otherwise, consider adding the information to deliver that promise.
  3. Consider adding an experiment that shows the tradeoffs of the hyperparameter sigma. What are the downsides of setting sigma to an extremely large value? What are the downsides of setting sigma to a very small value? A grid search would be very helpful here. Consider elaborating why sigma is annealed (line 747) in practice. Is a large sigma more helpful at the beginning of training?
  4. The notations in Section 2 are extremely confusing. In section 2.1, the authors introduce W as the raw/unquantized/pre-quantized weights, the function W() which returns the de-quantized weights. In Equation 2, W, which is not a function but a variable, now refers to the de-quantized weights. On line 135, it is unclear what it means to sum over W (maybe it is to sum over all assignments q?)
  5. In Section 2.3, consider discussing when the first-order expansion is not accurate. For example, what happens if the expectation of W is far from its mode? Would the STE gradient be less accurate?
2 comments

Thank you for the constructive comments! Please find our responses below:

Q1: We think that, in principle, this could be applied when the network has activation quantizers in it. It would be straightforward to, for example, treat the quantized activations as latent random variables and to use the unquantized activation values as the parameters of their distributions. Under stochastic rounding, you would recover the STE as usual. We think this is an interesting future direction, but we haven't undertaken any experiments here.

Q2: We plan to expand upon the grid gradient derivation and we will add it to the body of the manuscript as well. In brief: under stochastic rounding, the shift in grid value and the shift in assignment probability cancel exactly, and STE gives no learning signal to the grid. In particular, the expected grid value satisfies gˉij=∑kgkpij,kSR=μij\bar{g}_{ij} = \sum_k g_k p^{\mathrm{SR}}_{ij,k} = \mu_{ij} for any grid, so ∂gˉij/∂gk=0\partial \bar g_{ij}/\partial g_k = 0. Under the Laplace categorical they do not cancel: ∂gˉij∂gk=pij,k[1−1σsgn⁡(gk−μij)(gk−gˉij)],\frac{\partial \bar g_{ij}}{\partial g_k} = p_{ij,k}\Bigl[1 - \tfrac{1}{\sigma}\operatorname{sgn}(g_k-\mu_{ij})(g_k-\bar g_{ij})\Bigr], which gives a non-zero grid gradient. The derivative of the surrogate loss w.r.t. the grid values depends on the gradient of the expectation gˉ\bar g by chain rule, which is why it is zero for STE but not Laplace.

Q3: On the matter of σ\sigma, you are correct, we did not conduct a formal ablation study on its values and annealing schedule. Based on our preliminary experiments, we did find that the final performance is sensitive to the ending value of σ\sigma (setting it too small tended to hurt performance), but we did not explore this part of the hyperparameter space in depth.

Q4: We apologize for the notational difficulties. There is too much overloading of the symbols, which we will address for the camera-ready. For example, we will reserve W for the dequantized weights throughout, writing W(q;s,z,g)\mathbf W(q; s, z, g) wherever the dependence on the assignments matters, make explicit that the sum on line 135 is over assignments, etc.

Q5: We agree that this is an interesting question, though we did not perform a theoretical analysis here. However, we did run some small-scale quantization experiments on sub-1B parameter LMs (e.g., the SmolLM family) where we computed the second-order term of the Taylor expansion (12E[(W−W^)⊤H(W−W^)]\frac{1}{2}\mathbb E\big[(W-\hat W)^\top H(W-\hat W)\big]) approximately with Hutchinson's estimator and backed out the gradients with Hessian vector products, but we did not find that it improved performance (as measured on Wikitext PPL). Scaling it up to larger models was therefore not something we pursued.

Thank you for your response. Please take my suggestions above into consideration for post-rebuttal edits, especially Q2 and Q4. Adressing Q1, Q3, and Q5 can further strengthen the paper.

I intend to keep my score until further discussions with other reviewers during the reviewer-AC discussion phase.

Reviewer_8XrFOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a mean-field distribution over discrete quantization assignments, constructs a variational objective from the expected loss, and takes a first-order expansion around the hard MAP weights. Within a local region with a fixed codebook, no saturation, and unchanged assignments, the stochastic-rounding (SR) expectation equals the latent value, recovering the usual STE gradient. The paper then combines a Laplace categorical distribution with learnable non-uniform codebooks to form NUL-QAT and evaluates block-wise training across multiple Qwen3 model sizes and extreme low-bit settings.

Critical strengths
  1. The SR expectation identity and the local first-order derivation have explanatory value. Equations 6–9 on pp. 4–5 place a commonly used gradient rule within an explicit surrogate objective. I do not consider that local identity itself to be incorrect.
  2. Figures 3–4 on pp. 7–8 show substantial empirical improvements at extreme low precision across model sizes from 0.6B to 32B, indicating that learnable non-uniform levels are worth studying. Fused computation that avoids explicitly materializing a weight-by-codepoint probability tensor also has practical implementation value.
  3. Combining a theoretical interpretation, codebook learning, and a deployable representation is a promising research direction. The main issue is that the current claims, algorithm description, and evidence do not yet align.
Critical weaknesses

W1 [Major] The local gradient identity does not support the current strength of the theoretical framing. Locations: pp. 1–2, abstract/contributions; pp. 3–5, Eqs. 2–9; pp. 13–14, Appendix B.2. The exact expectation in Eq. 2 does upper-bound the discrete minimum, but its first-order truncation J^(1) generally does not retain that upper-bound property, nor does its gradient necessarily provide a descent direction for the original expectation. A directly reproducible example is a grid {0,1}, s=1, z=0, mu=0.4, and L(w)=(w−0.2)^2. The MAP value is 0 and the discrete minimum loss is 0.04. The exact expectation is E[L]=0.28, with derivative 0.6 with respect to mu. The first-order surrogate at the MAP value is 0.04−0.4mu=−0.12, with derivative −0.4. Thus, the surrogate can fall below the discrete minimum and its gradient can point in the opposite direction to the gradient of the exact expectation. This example does not refute the identity that STE is the gradient of this first-order surrogate. It limits the stronger interpretation that this identity establishes general optimization validity or eliminates the heuristic aspect. I suggest explicitly restricting the claim to a local surrogate interpretation, or supplying error/descent conditions and evidence that they apply at extreme low precision. Another point requiring clarification is the gradient with respect to quantization parameters. The main text says that gradients through the expected weights are retained, whereas Appendix B.2 recommends STE pass-through based on the MAP term (a) for s, z, and g. MAP weights change with learnable codebook values and scales. Omitting Hessian-related terms therefore requires explicit stop-gradient definitions and conditions. Please provide automatic-differentiation pseudocode matching the actual implementation rather than using distribution peakedness as a blanket justification for all parameters.

W2 [Major] The claim that STE cannot learn non-uniform codebooks is too broad, and the ablations do not isolate the key factors. Locations: p. 5, §2.5; p. 7, §3.5; pp. 8–9, §5.1/Table 1. With fixed hard assignments and a quantized value s(g_k−z), the selected g_k still has a direct gradient equal to s times the sum of the relevant upstream gradients. Cancellation of a derivative with respect to g under a particular parameterization of the SR expectation does not imply that every training scheme using STE for latent-weight updates is unable to learn non-uniform codebooks. The standard baseline fixes g to integer values and therefore cannot itself test the capability of “STE plus a learnable codebook.” An initialization ablation conducted only at 3-bit precision is insufficient to explain the main gains at 2-bit and ternary precision. Suggested remedy: In the principal extreme low-bit settings, compare “Laplace plus a fixed non-uniform codebook,” “Laplace plus a learnable codebook,” and “hard nearest-neighbor assignments plus STE latent updates and direct codebook learning.” Control initialization, training budget, and actual storage. This would separate the effects of the assignment distribution, codebook capacity, and initialization.

W3 [Major] The complexity comparison contains a definite error. Location: p. 9, §5.2. A factorized general softmax has K logits per weight, so its parameter storage/explicit probability tensor is O(NK), not K^N. The latter is the size of an enumeration of all joint assignments. The paper’s first-order expected-weight calculation likewise does not require enumerating joint assignments. Using one latent value in the Laplace construction does reduce the number of learnable parameters, and a fused kernel can avoid writing the probability tensor to global memory. These are different advantages from the claim that a general softmax must enumerate K^N configurations. For the implementation described, which iterates over K terms, the computation also cannot simply be called linear in bit-width b when K=2^b. Please correct the distinctions among parameter storage, temporary memory, operation count, and the complexity of enumerating the exact expectation, and revise the efficiency comparison accordingly.

W4 [Major] The actual training protocol is not uniquely specified, limiting reproducibility and the connection between theory and experiments. Locations: p. 7, §3.5; pp. 13–14, Appendices B.2/C and Table 3. Table 3 specifies sigma0=sigmamin=1.0 and a cosine schedule, whereas the immediately following text says that sigma decreases linearly from 2.0 to 0.9. This is not an inconsequential typo: both distribution sharpening and the justification for omitting gradient terms depend on temperature. The description of a “Laplace forward” pass in the main text also needs to be reconciled with the earlier hard-MAP forward description through the actual implementation. Please provide the temperature configuration actually used, stop-gradient rules, constraints on the grid values, and learning rates, together with the necessary low-cost gradient checks. Where the appendix states that correction-term experiments did not help, the corresponding results and settings should be reported.

W5 [Major] The compression and deployment conclusions lack accounting for the actual representation. Locations: pp. 6–7, §§3.1/3.4; p. 12, related work; pp. 14–15, Appendix D/Table 4. The labels “1.58/2/3-bit” primarily describe codebook cardinality. The storage precision and sharing scheme of the learnable per-group values are not sufficiently specified. For example, with group size 64 and K=4, storing the codepoints independently for each group in FP16 would add 16×4/64=1 bpw for the codebook alone. This is an illustration of why byte-level accounting is necessary, not an assertion that the implementation necessarily uses that representation. Similarly, K=3 and log2(3) alone do not establish that fixed-width storage achieves 1.58 bpw. Table 4 explicitly reproduces Llama throughput from another paper. It is background evidence, rather than a measurement of an exported Qwen model using the non-uniform codebooks learned here. It cannot by itself establish that the present format preserves the same throughput in FLUTE. Please report the actual serialized size, codebook/scale/zero-point overhead, and ternary packing scheme, and demonstrate export correctness and throughput for at least one checkpoint from this paper. This does not require developing an entirely new inference framework.

Positioning relative to prior work (supporting context): The assertion that STE has lacked a principled explanation needs to be distinguished from existing work. Shekhovtsov and Yanush’s “Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networks” and Ajanthan et al.’s “Mirror Descent View for Neural Network Quantization” already explain STE from other modeling or optimization perspectives. They do not automatically cover this paper’s multilevel MAP/SR derivation, but the novelty should be framed as this particular interpretation rather than as the first theoretical justification of STE in general. https://openreview.net/pdf?id=F8lXvXpZdrL https://proceedings.mlr.press/v130/ajanthan21a.html This is a suggestion about novelty positioning. The main grounds for rejection are the inconsistencies between the claims and the implementation/experimental evidence discussed above, rather than the absence of a single citation.

Questions for authors

Q1. Which objective and stop-gradient operations are actually used in backpropagation for the latent variables, s, z, and g? How do these correspond to Appendix B.2? Q2. Is the actual temperature fixed at 1.0, governed by a cosine schedule, or linearly annealed from 2.0 to 0.9? Q3. How much benefit does the Laplace distribution retain when the STE baseline is allowed to learn a non-uniform codebook directly? Q4. What is the actual bpw after accounting for per-group codebooks and ternary packing? What throughput is measured for the exported Qwen checkpoint in FLUTE?

1 comment

We thank the reviewer for their detailed remarks. Please find our replies below.

W1: We are unsure what the counterexample demonstrates. In the counterexample, the MAP weight is 0, and STE gives d L / d mu = L'(0) = -0.4. Our first-order surrogate gives the same derivative, as expected. The disagreement with the exact expectation's gradient (which is 0.6 in the example) is not a problem in the derivation, it simply is the definition of the STE, which our derivation recovers exactly.

To be explicit: We showed that STE is the gradient of a particular first-order surrogate. We do not claim that this surrogate inherits the upper-bound property of the exact expectation, nor that its gradient is a descent direction for that expectation. STE itself does not have these properties and no derivation of STE could confer them. Our result is valuable because it identifies the objective that STE does actually optimize, which allows useful extensions to non-uniform grids when we choose a different assignment distribution. In this sense, the STE is not merely a convenient heuristic, but something that follows naturally from a first-order approximation to a variational objective. We will sharpen this language in the paper.

W2: Thank you for suggesting this additional ablation study. In Table 1, we tested initialization at 3 bits because the performance gap was smallest there, so a benefit from a non-uniform grid for STE could most easily have changed the comparison. We examined the 2-bit case as well, per your request.

2-bit WikiText PPL (lower is better)

ModelSTE-QAT, RTN initSTE-QAT, Non-uniform initNUL-QAT
Qwen3-0.6B1191.81318.7165.1
Qwen3-1.7B102.692.852.3
Qwen3-4B35.234.928.6
Qwen3-8B24.625.320.1

The results show that even after using a non-uniform initializer and allowing STE to update the grid values, the effect is mixed and small in the 2-bit case versus the RTN initialization. We shall update Table 1 in the camera-ready with our additional findings. There are more details which we have omitted for brevity in the rebuttal, but we will make the details of the grid gradient (and how it compares to the exact gradient and its second-order terms) more clear in the camera-ready.

W3: The K^n statement in the ablation section is a typo; the factorized softmax distribution requires O(Kn) memory and computation, not O(K^n). The training kernel section (Sec. 2.7) already correctly describes the N x K probability tensor and the fused implementation that avoids materialising it. We will correct the ablation text. The actual advantages (a single scalar latent per weight versus K independent logits and a fused kernel) are unaffected.

W4: Table 3 states the outdated configuration from an earlier iteration of our experiments: it lists sigma_0 = sigma_min = 1.0 and a cosine schedule. We actually used sigma_0 = 2.0 annealed linearly to sigma_min = 0.9, as stated in the text below the table. We will correct the table to follow the latter configuration.

W5: The reviewer's illustration assumes a per-group codebook of K fp16 values, yielding an overhead of 1 bpw. However, this is not what NUL-QAT does, and there is only one grid per weight tensor. The grid g = (g_1, ..., g_K) is shared across all groups, and each group applies its own per-group scale s and zero point z to produce distinct dequantized codebook values as (g_k - z) * s. In some sense, both NUL-QAT and STE-QAT store exactly K grid values per weight tensor; in STE-QAT these are the fixed uniform integers (0, 1, ..., K-1), while in NUL-QAT they are learned. The parameter count is identical for both methods.

Also, both methods store identical amounts of per-group metadata: a scale and zero point per group of 64 weights (0.5 bpw in FP16). The comparison in the paper is therefore at matched storage. We will state this more clearly in the manuscript and will add explicit bpw accounting.

The confusion arose because our earliest experiments had a group size equal to the whole tensor, and so the wording in the manuscript was ambiguous in places. We will update the manuscript to distinguish where the group size was 64 (for the scale factors and zero points) and where it was the entire weight tensor (for the grid).

Regarding throughput: since the grid is shared per tensor, a NUL-QAT layer maps onto FLUTE's format of one lookup table per matrix plus per-group scales, and there are no per-group tables.

Related work: Thank you for pointing out Shekhovtsov & Yanush (2021) and Ajanthan et al. (2021) as related work. They interpret STE from the stochastic binary network and mirror descent perspectives. Our contribution is the MAP of a Taylor expansion construction and its extension to non-uniform codebooks via the Laplace categorical distribution. We will cite these works in the manuscript.

Reviewer_H71GOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper argues that the straight-through estimator (STE) can be derived rather than assumed: linearizing the expected loss under a categorical distribution over grid assignments around the hard-rounded weights recovers the STE gradient when that distribution is stochastic rounding. Building on this, NUL-QAT replaces stochastic rounding with a "Laplace categorical" distribution (a temperature-controlled softmax over distance to each grid point). It uses hard weights in the forward pass and expected weights in the backward pass, learns a non-uniform grid per weight group, and uses a fused Triton kernel to keep memory low. Experiments apply block-wise QAT (EfficientQAT) to Qwen3 0.6B–32B at 3, 2, and 1.58 bits, comparing only against uniform-grid STE QAT on WikiText perplexity and a five-task average, plus two small 3-bit ablations. Gains are large at 2 and 1.58 bits and small at 3 bits; inference efficiency is argued via FLUTE using throughput numbers from prior work.

Critical strengths
  • Simple and cheap. The method keeps one latent value per weight, unlike softmax-style relaxations that need one logit per grid point. With the fused kernel, a 32B model can be quantized in about 10 hours on 4×A100. It is a small change to standard STE-based QAT code, so it would be easy to adopt.
  • Broad evaluation. The paper covers six model sizes and three bit-widths, with both perplexity and downstream accuracy. At 2 and 1.58 bits, NUL-QAT improves over the uniform-STE baseline at every scale.
Critical weaknesses
  • (Most important) The comparison is not at matched memory, and the baselines do not isolate the contribution. NUL-QAT stores a separate grid of K values for every group of 64 weights, while the uniform baseline stores only a scale and zero point. If the grids are kept in FP16 (the paper does not say), this adds about 2, 1 and 0.75 bits per weight at 3, 2 and 1.58 bits, so "2-bit" NUL-QAT costs about the same as 3-bit uniform and "1.58-bit" about the same as 2-bit uniform. At those budgets, Figure 3 favors uniform STE at every model size (at 8B, roughly 71% vs. 61% and 56% vs. 44%). Since effective bits per weight are never reported, the low-bit gains may come largely from extra storage rather than a better estimator.
  • The theory is overstated and does not match the method as run. Under stochastic rounding, the exact gradient of the bound is a finite difference between neighboring grid points. Replacing it with the derivative at the rounded weights is the STE approximation itself, so the derivation restates STE rather than justifying it. No bound is given on the dropped second-order term, and earlier first-order analyses of straight-through (such as ReinMax [1]) are not discussed.
  • Some typos and presentation issues:
    • Line 265 (Sec. 2.6): "interative expectation-maximisation" should be "iterative".
    • Line 255: "Lap" is used as an abbreviation for Laplace without being defined.

[1] Bridging Discrete and Backpropagation: Straight-Through and Beyond, NeurIPS 2023

Questions for authors
  • What are the effective bits per weight and the model size, including the per-group gg, ss, zz and their storage precision? Please compare at matched bits per weight (NUL-QAT 2-bit vs. uniform 3-bit; NUL-QAT 1.58-bit vs. uniform 2-bit).
  • How exactly is ∂L/∂g\partial L/\partial g computed: through wˉ\bar w, through the STE pass-through on w^\hat w, or both? Please add the missing appendix derivations.
  • Does FLUTE support a distinct lookup table for every group of 64 weights at full throughput? Please give measured throughput and memory for a NUL-QAT model.
1 comment

We thank the reviewer for the thoughtful comments. We address the questions below.

W1: The reviewer's arithmetic assumes a per-group codebook of K fp16 values, yielding overheads of 1-2 bpw. However, this is not what NUL-QAT does, and there is only one grid per weight tensor. The grid g=(g1,...,gK)g = (g_1, ..., g_K) is shared across all groups, and each group applies its own per-group scale s and zero point z to produce distinct dequantized codebook values as (gk−z)s(g_k - z) s. Both NUL-QAT and STE-QAT use exactly K grid values per weight tensor; in STE-QAT these are the fixed uniform integers (0,1,...,K−1)(0, 1, ..., K-1), while in NUL-QAT they are learned.

Also, both methods store identical amounts of per-group metadata: a scale and zero point per group of 64 weights (0.5 bpw in fp16). The comparison in the paper is therefore at matched storage. We will state this more clearly in the manuscript and will add explicit bpw accounting.

The confusion arose because our earliest experiments had a group size equal to the whole tensor, and so the wording in the manuscript was ambiguous in places as to what 'group size' meant. We regret the error, and we will update the manuscript to distinguish where the group size was 64 (for the scale factors and zero points) and where it was the entire weight tensor (for the grid).

W2: We understand that "justification" may overstate our contribution and will adopt "variational derivation" or "variational interpretation" in the camera-ready. The main insight is identifying which objective STE actually optimises, namely the first-order surrogate under stochastic rounding. This new characterisation of STE in terms of a specific objective is what makes the extension to other distributions possible. Without this identification, we would not have thought to try another distribution like the Laplace categorical instead of stochastic rounding, which enabled a learnable quantization grid.

We will cite ReinMax (Liu et al., NeurIPS 2023) as you have suggested. Our construction addresses a different question (i.e., extending the assignment distribution to enable non-uniform codebook learning) but the first-order analysis is relevant to our work. Thank you!

Regarding the specific questions:

Effective bpw: Addressed above.

Grid gradient computation: Both. We agree with the reviewer that we should elaborate on how the grid gradient is computed. We will add the following explanation to the manuscript and correct the appendix:

For each weight, the training loop implements the forward pass with the usual stop-gradient rule:

a = p + (h - p).detach()
W_dequant = s * ((a * g).sum() - z)

pp is the soft assignment, hh is the hard assignment, ss is the group scale, and zz is the zero point.

In the forward pass, the value of aa equals hh. In the backward pass, the derivative of aa equals the derivative of pp. Differentiating the product aikgka_{ik}g_k therefore gives a direct contribution from each weight that selected gkg_k, plus a contribution from how moving gkg_k changes the assignment probabilities. Let ii index the weights in a matrix, sis_i be the group scale for weight ii, GiG_i be the loss gradient with respect to its dequantized weight at the hard forward pass, and gˉi=∑jpijgj\bar g_i=\sum_j p_{ij}g_j. Summing over the weights, the grid update uses

$ {\frac{\partial L}{\partial g_k}} = \sum_i s_i G_i \left[h_{ik} + \frac{p_{ik}}{\sigma}(g_k-\bar g_i)\operatorname{sign}(\mu_i-g_k)\right].$

The first term has coefficient hikh_{ik} because the forward pass uses the hard assignments before multiplying by the grid. The second term follows by differentiating the Laplace softmax with logits −∣μi−gj∣/σ-|\mu_i-g_j|/\sigma, where μi\mu_i is the latent in grid units and σ\sigma is the temperature.

There are more details which we have omitted for brevity in the rebuttal, but we will make the details of the grid gradient (and how it compares to the exact gradient and its second-order terms) more clear in the camera-ready.

FLUTE: As explained above, since the grid is shared across all groups within a weight matrix in NUL-QAT, we do not need a distinct lookup table per group.

Typos: "Interative" -> "iterative" and "Lap" undefined: we will correct this, thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.