acceptodds
Under review as a conference paper at ICLR 2027

What Does Key Quantization Actually Change in a Delta Rule? The Step Size.

Abstract

Delta-rule linear recurrent neural networks such as DeltaNet, Gated DeltaNet and DeltaProduct update a matrix state with one least-mean-squares step per key. This step shrinks the residual it corrects only if the effective step size lies in , where is the step size and is the squared norm of the key. The models keep with a key normalizer, but a quantizer placed after the normalizer breaks this: with 4-bit integer (INT4) keys, deviates from one by up to . In models with step sizes below , every update then has a relative step-size error of . In models with step sizes up to , of the steps have , where the update flips and enlarges the residual. If and are independent, the share of such steps depends only on the tail of near and the distribution of , and measuring the two separately predicts the observed shares within for INT4 keys. NormFix divides the step size by the squared norm of the key that the kernel receives, . This restores the trained residual factor for every and makes the propagation of state errors nonexpansive for any quantizer. With 4-bit weights and INT4 keys, NormFix recovers of the long-context state-tracking loss of a Gated DeltaNet with negative eigenvalues and of the loss of a standard one, at less than extra kernel time. On the negative-eigenvalue model it is better than clipping, skipping, key renormalization and 8-bit keys on every state task and on retrieval. Its gain holds across six models and three scale axes, and where the key is normalized after the quantizer, so that , it changes no score by more than points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r94KOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that quantizing the key after its normalizer in delta-rule linear RNNs changes the effective step size of each update, because the key’s norm no longer equals one. In models whose step size stays below 1, this causes a small step-size error at every update. In models whose step size can approach 2, some updates overshoot and enlarge the residual instead of shrinking it. The authors propose NormFix, a training-free fix that divides the step size by the squared norm of the received key, and show that it recovers a substantial part of the quantization loss at little extra cost.

Critical strengths
  • Clear diagnosis backed by theory and measurement. The paper traces a substantial part of the accuracy loss to the norm of the received key, proves that NormFix makes the error transition nonexpansive for any quantizer, and predicts the overshoot rate within 7% of the measured one for INT4 keys.
  • Simple, training-free fix. NormFix adds only a squared-norm reduction and one division per head and step, with no retraining, no extra memory traffic, and less than 4% extra kernel time.
  • Broad evaluation. The experiments cover several delta-rule models, quantization recipes, and scale axes with one-factor ablations, and confirm that NormFix leaves the model unchanged where the key is already unit-norm.
Critical weaknesses
  • Unclear practical relevance. The large gains appear mainly in class II models, whose step size can approach 2, and these are mostly research architectures. It is unclear how many deployed hybrid models use class II step sizes, or how often serving stacks actually quantize the key after the normalizer.
  • Limited scale. All models have 340M parameters, while deployed hybrid models such as Qwen3-Next and Kimi Linear are much larger. It remains open whether the overshoot rate and the gain of NormFix carry over to that scale.
  • Gap between theory and results for renormalization. NormFix and key renormalization share the same erase term and differ only slightly in the write term, and the paper notes that the analysis does not rank them. Yet NormFix is clearly better empirically, and this gap remains unexplained.
  • Only partial recovery. NormFix recovers roughly half of the loss, and the remainder, attributed to the direction error of the quantized key, is left unaddressed.
Questions for authors
  • Comparison with P2 and P3. On P2 and P3 the key keeps unit norm, so the problem does not arise. How does the unrepaired model perform on these placements compared with P1 + NormFix? Also, P3 appears equivalent to the "Renormalize key" baseline. Is this the case?
  • Share of the loss from the key. What is the accuracy with W4 weights and unquantized keys? This would show what share of the loss NormFix can address in the first place.
Reviewer_2MhCOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how key quantization changes delta-rule updates when quantization follows key normalization. The resulting norm error changes the effective step size and can cause individual updates to enlarge their residuals. The proposed correction, NormFix, intentionally rescales the update step size to correct this. The paper analyzes the correction and reports improvements in state tracking and retrieval with a small runtime cost.

Critical strengths
  • The paper identifies that quantization after normalization can invalidate the unit-norm assumption used by the update.
  • The experiments examine several model variants, quantization settings, and context lengths, with comparisons against clipping, skipping, and key renormalization.
Critical weaknesses
  • The methodological novelty appears limited. The paper explicitly defines NormFix as re-normalization applied to the received key. Its contribution is therefore primarily the characterization and measurement of this deployment issue, but the framing and narrative give the correction more prominence than this distinction warrants.

  • The relationship to key renormalization further limits the methodological distinction. Proposition 5 shows that both repairs share the same state-transition matrix and differ only in their write terms. This relationship should be central to the positioning of the contribution.

  • Proposition 5 in Section 4.3 show that key renormalization can be implemented using scalar factors. However, the timing baseline in Appendix H explicitly rescales and writes back the key vector. A fused implementation that incorporates these factors directly into the update would strengthen the runtime comparison.

  • The distinction between the delta-rule update step size and the quantizer’s grid spacing is not clear enough early in the paper. This makes the title and abstract harder to interpret for readers approaching the work from quantization.

  • The residual formulation in Section 3.2 (i.e., rtr_t) appears before sufficient explanation of its connection to the state update.

  • The presentation and discussion of Theorem 3 is unnecessarily difficult to follow.

Questions for authors
  • How common is the identified issue relative to implementations that normalize after quantization?
  • Can Theorem 3 and its proof be simplified? It seems correct but it is difficult to follow. Moreover, the claim is unclear and makes the insights difficult to parse. The proof moves through the logical deductions too quickly. There seem to be more variables than necessary, e.g., why is Gˉ\bar{G} needed?
  • Beyond the reported empirical advantage, what motivates the write scaling used by NormFix over key renormalization? The current analysis establishes their shared transition behavior but does not establish which write term should give lower error.
Reviewer_TRmJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper studies quantization in DeltaNet models, specifically key quantization under aggressive quantization (INT4)
  • They consider the interaction between quantizing the key, and the effective step size of the DeltaNet update
  • In particular, they show that quantizing the key after the DN normalization step leads to possibly overshooting the step size (as the new key in no longer unit norm), and this leads to poor performance
  • They study the rate of such overshoots, and show that particularly for DN models with negative eigenvalues (those which are used for state tracking), this happens at a small, not non-negligible rate, and when \beta_t is close to 2, then it leads to expansive state updates (eigenvalues outside (-1, 1)), leading to worsened performance
  • They give a simple fix, NormFix, which basically rescales \beta_t by the norm of the quantized key.
  • This stops the expansive issue, and also improve performance, while adding minimal overhead
Critical strengths
  • Clear problem formulation and theoretical analysis. The issue addressed by the paper is well scoped and clearly motivated. In particular, the authors provide a useful theoretical characterization of how key quantization affects the DeltaNet recurrence, both through its effect on the residual magnitude and through the eigenvalues of the recurrent transition matrix. These complementary perspectives help explain why seemingly small quantization errors can have significant effects on model behavior.

  • Simple and effective solution. The proposed NormFix method is simple and well motivated by the preceding theoretical analysis. It requires no retraining, incurs minimal computational overhead, and demonstrates substantial empirical improvements across multiple DeltaNet variants and evaluation tasks.

Critical weaknesses
  • Incomplete quantization and implementation details. The paper provides limited information about the precision of other components of the DeltaNet recurrence beyond the keys and weights. In particular, what precision is used for the queries, values, recurrent state, step sizes, and intermediate computations within the recurrent kernel? For chunkwise DeltaNet implementations, intermediate states can also consume substantial memory. It is therefore unclear how much practical benefit INT4-key quantization provides when much of the recurrence may still operate at higher precision. An end-to-end comparison of inference memory and throughput against W4 weights with BF16 keys, for both prefill and decoding, would help establish the practical relevance of the proposed correction.

  • Insufficient analysis of the failure mechanism. The paper identifies both residual amplification, occurring when βtst>2\beta_t s_t>2, and potentially expansive recurrent transitions, occurring when αt∣1−βtst∣>1\alpha_t|1-\beta_t s_t|>1. However, the experiments do not distinguish their contributions to the observed degradation. In particular, it would be informative to apply NormFix only to updates for which ∥At∥2>1\|A_t\|_2>1, and compare this against the existing ablation that repairs all updates with βtst>2\beta_t s_t>2. Reporting the prevalence of each condition would help establish whether the degradation is primarily due to expansive state-error propagation or incorrect local memory updates.

  • Limited explanation of the advantage over key renormalization. The authors demonstrate that NormFix consistently outperforms renormalizing the quantized keys, despite both methods having identical recurrent transition matrices and differing only in the scaling of the write term. Although Section 4.3 characterizes this difference mathematically, it does not explain why NormFix should perform better. Further analysis of the resulting write magnitudes, state errors relative to the full-precision model, and downstream performance would help clarify this result.

  • Writing and presentation could be improved I think that the writing reads as very much like a constant list of statements and facts rather than forming a proper narrative with flow. This made understanding the motivation and the techniques difficult. For example, the theorems would benefit from plain English writing explaining what the theorems establish, and why we should care, rather than diving straight into the math. Section 5.2 (main results) for example, reads like a constant list of numbers, which is unnecessary since we already have the table. It would benefit from more analysis rather than repeating numbers.

  • Unclear additional insight from Theorem 3. The theorem characterizes the overshoot probability using the marginal distributions of betatbeta_t and sts_t, under an independence assumption. However, the overshoot rate can also be measured directly from these quantities at little additional cost. While the decomposition and first-order approximation provide some analytical insight, it is unclear how much the theorem contributes beyond predicting a readily measurable statistic. In particular, connecting the overshoot probability more directly to recurrent error propagation or downstream accuracy degradation would strengthen the theoretical contribution.

Questions for authors

See weaknesses

Comments on the paper

3 comments
Area_Chair_CsJwOpenReview ↗

Reminder to reviwers to read author's responses

Dear Reviewers,

Thank you for your efforts in providing thorough reviews.

As the discussion period is underway, I would like to kindly remind you to read the authors' responses to your reviews and provide follow-up comments where appropriate or leave a comment. In particular, please consider whether the responses address your questions and concerns, and update your assessment if the clarifications or additional information affect your initial evaluation.

Your engagement in this discussion is important for ensuring a fair and thorough review process.

Thank you again for your time and contributions!

Best regards, Area Chair

Thank you for the thorough response. In particular, the simplified version of Theorem 3 is much easier to follow, and the timing with fused renormalization provides a sufficiently strong baseline. However, while I appreciate the authors' efforts in addressing the concerns, the novelty remains limited. As such, I remain unconvinced that the diagnosis and resulting fix constitute a sufficiently substantial contribution to ICLR. As such, I will maintain my original score.

AuthorsOpenReview ↗

General Response 2: What is new in this paper, restated after the discussion

Dear AC and reviewers,

Thank you for the discussion. Your questions led to the analysis of why NormFix beats key renormalization (C1), the fused renormalization baseline, the simplified Theorem 3, the split between expansive and overshooting steps (C3), and the scale study up to an 80B hybrid model (C4). As the discussion closes, we restate in one place what is new in the paper.

In one sentence. Delta-rule kernels assume that every key they receive has unit norm; the paper shows that low-bit inference breaks this assumption once the key is quantized after the normalizer, explains the damage with theory, and identifies which repair works and why.

The ICLR 2027 Reviewer Guidelines place a paper's value in "new, relevant, impactful knowledge (incl., empirical, theoretical, for practitioners, etc)". The paper adds knowledge of each kind.

1. Empirical: a failure that standard quantization tools leave in place. With a quantizer after the normalizer, the kernel takes the step βs\beta s instead of β\beta, where ss is the squared norm of the received key. Every update gets a step-size error, and in class II, where β\beta can approach 2, 3.40% of the steps overshoot: they flip the residual and enlarge it. State tracking at 64k tokens drops by 31.0 points (W4 weights, INT4 keys; Table 1). Rotations, more bits and retraining do not remove the overshoots: QuaRot keeps 3.48% of the steps overshooting, SpinQuant 3.34%, quantization-aware training 3.34%, and INT8 keys 0.59%.

2. Theoretical: what sets the failure, and what bounds it. Theorem 3 writes the overshoot rate as a part the model sets, the density of β\beta just below 2, times a part the quantizer sets, the mean positive part of s−1s - 1. Measured separately, the two predict 3.17% against the measured 3.40%, and Table R1-2 tests the prediction on five key formats. The theorem also explains the list above: rotations and unbiased rounding leave the positive part of s−1s - 1 in place, and more bits shrink it but never to zero. Theorem 4 shows that rescaling the step size makes the propagation of state errors nonexpansive for any quantizer. Proposition 5 places all scalar repairs in one family with the same transition matrix; they differ only in the write term.

3. Algorithmic: which repair, and why. NormFix rescales the step size, η=β/s\eta = \beta / s, and leaves the key as the kernel receives it. The natural repairs, renormalizing the key or normalizing after quantizing, keep the write scale at its trained value. All three reviews asked why NormFix does better. C1 gives the reason: the per-head output RMSNorm cancels any constant write scale, so only the per-step factor 1/st1/\sqrt{s_t} of NormFix can matter, and it lowers the scale-free output error by 3.4% at layer 1 and by 19.4% at layer 32 (Table G1). The NLMS rule alone does not rank the repairs; the ranking comes from the RMSNorm after the kernel. NormFix beats key renormalization by 3.91 points in class II and 2.15 in class I, with both paired 95% intervals above zero, even when renormalization is fused into scalars, and at lower kernel cost, +3.6% against +4.0% (Table R2-1). It also beats normalizing after quantizing, by 3.69 to 3.91 points in class II and 1.99 to 2.15 in class I (C2).

4. For practitioners: where it occurs, and what the repair buys. The key is quantized after the normalizer when it becomes the operand of a low-bit matrix product in chunkwise prefill, or the input of a low-bit kernel (C5). NormFix needs no training and adds one reduction and one division per head and step. In class I, the class of deployed hybrids such as Qwen3-Next, INT4 keys with NormFix match or slightly beat unrepaired INT8 and FP8 keys on 64k state tracking, 92.30 against 92.08 and 91.27, with half the key bits (Table 1). On Qwen3-Next-80B-A3B-Instruct, NormFix recovers 80.7% of the RULER loss caused by INT4 keys, at 0.8% extra decode latency (C4).

What the reviews credit, and where novelty is questioned. Reviewer r94K calls the diagnosis clear and "backed by theory and measurement", Reviewer TRmJ finds "a useful theoretical characterization of how key quantization affects the DeltaNet recurrence", and Reviewer 2MhC credits the paper with identifying "that quantization after normalization can invalidate the unit-norm assumption used by the update". All three note the small runtime cost. Where the discussion questions novelty, it questions the rule. The rule is classical NLMS, and Section 1 says so. What is new is the knowledge above: the failure and its size, the theory that predicts it, the choice of repair with its mechanism, and where it matters in practice.

We thank the AC and the three reviewers again for their time and careful reading.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.