What Does Key Quantization Actually Change in a Delta Rule? The Step Size.

Delta-rule linear recurrent neural networks such as DeltaNet, Gated DeltaNet and DeltaProduct update a matrix state with one least-mean-squares step per key. This step shrinks the residual it corrects only if the effective step size $\beta s$ lies in $(0,2)$, where $\beta$ is the step size and $s$ is the squared norm of the key. The models keep $s=1$ with a key normalizer, but a quantizer placed after the normalizer breaks this: with 4-bit integer (INT4) keys, $s$ deviates from one by up to $9.4\%$. In models with step sizes below $1$, every update then has a relative step-size error of $|s-1|$. In models with step sizes up to $2$, $3.4\%$ of the steps have $\beta s>2$, where the update flips and enlarges the residual. If $\beta$ and $s$ are independent, the share of such steps depends only on the tail of $\beta$ near $2$ and the distribution of $s$, and measuring the two separately predicts the observed shares within $7\%$ for INT4 keys. NormFix divides the step size by the squared norm of the key that the kernel receives, $\eta =\beta /s$. This restores the trained residual factor $1-\beta$ for every $s$ and makes the propagation of state errors nonexpansive for any quantizer. With 4-bit weights and INT4 keys, NormFix recovers $57\%$ of the long-context state-tracking loss of a Gated DeltaNet with negative eigenvalues and $51\%$ of the loss of a standard one, at less than $4\%$ extra kernel time. On the negative-eigenvalue model it is better than clipping, skipping, key renormalization and 8-bit keys on every state task and on retrieval. Its gain holds across six models and three scale axes, and where the key is normalized after the quantizer, so that $s=1$, it changes no score by more than $0.01$ points.