- Rating
- 2
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper shows that quantizing the key after its normalizer in delta-rule linear RNNs changes the effective step size of each update, because the key’s norm no longer equals one. In models whose step size stays below 1, this causes a small step-size error at every update. In models whose step size can approach 2, some updates overshoot and enlarge the residual instead of shrinking it. The authors propose NormFix, a training-free fix that divides the step size by the squared norm of the received key, and show that it recovers a substantial part of the quantization loss at little extra cost.
Critical strengths
- Clear diagnosis backed by theory and measurement. The paper traces a substantial part of the accuracy loss to the norm of the received key, proves that NormFix makes the error transition nonexpansive for any quantizer, and predicts the overshoot rate within 7% of the measured one for INT4 keys.
- Simple, training-free fix. NormFix adds only a squared-norm reduction and one division per head and step, with no retraining, no extra memory traffic, and less than 4% extra kernel time.
- Broad evaluation. The experiments cover several delta-rule models, quantization recipes, and scale axes with one-factor ablations, and confirm that NormFix leaves the model unchanged where the key is already unit-norm.
Critical weaknesses
- Unclear practical relevance. The large gains appear mainly in class II models, whose step size can approach 2, and these are mostly research architectures. It is unclear how many deployed hybrid models use class II step sizes, or how often serving stacks actually quantize the key after the normalizer.
- Limited scale. All models have 340M parameters, while deployed hybrid models such as Qwen3-Next and Kimi Linear are much larger. It remains open whether the overshoot rate and the gain of NormFix carry over to that scale.
- Gap between theory and results for renormalization. NormFix and key renormalization share the same erase term and differ only slightly in the write term, and the paper notes that the analysis does not rank them. Yet NormFix is clearly better empirically, and this gap remains unexplained.
- Only partial recovery. NormFix recovers roughly half of the loss, and the remainder, attributed to the direction error of the quantized key, is left unaddressed.
Questions for authors
- Comparison with P2 and P3. On P2 and P3 the key keeps unit norm, so the problem does not arise. How does the unrepaired model perform on these placements compared with P1 + NormFix? Also, P3 appears equivalent to the "Renormalize key" baseline. Is this the case?
- Share of the loss from the key. What is the accuracy with W4 weights and unquantized keys? This would show what share of the loss NormFix can address in the first place.
Reminder to reviwers to read author's responses
Dear Reviewers,
Thank you for your efforts in providing thorough reviews.
As the discussion period is underway, I would like to kindly remind you to read the authors' responses to your reviews and provide follow-up comments where appropriate or leave a comment. In particular, please consider whether the responses address your questions and concerns, and update your assessment if the clarifications or additional information affect your initial evaluation.
Your engagement in this discussion is important for ensuring a fair and thorough review process.
Thank you again for your time and contributions!
Best regards, Area Chair