- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies post-training quantization of the recurrent state in linear-attention and SSM layers, where the fixed-size state stored for each request becomes a memory bottleneck at inference. It builds on DAMP, a very recent heuristic that keeps the most “persistent” (the ones with the slowest decay) state channels in high precision and quantizes the rest, and tries to replace that heuristic with quantitative laws for how long and how accurately a low-bit recurrent state can remember. The paper is mainly a theory paper. The authors model a linear recurrence in which quantization acts as additive white noise, and they separate two memory effects or tasks: “propagation”, where the state must still carry a trace of something written many steps earlier, and “retrieval”, where a specific stored entry must be read back.
The main claim is that the two tasks depend on different resources. Precision extends how far information propagates, while state dimension governs retrieval accuracy, and past a certain bit width extra precision is said to bring nothing for retrieval. Noise in the gates shortens memory. The authors also argue that DAMP measures channel persistence with the wrong statistic, a geometric mean of the decay, which underestimates quantization risk when gates occasionally reset, and they propose a second moment measure instead. They then use these results to explain DAMP design choices. The experiments are mostly synthetic and small scale, plus one small pretrained Mamba model, on which the proposed correction has no measurable effect; the authors attribute this to the model nearly deterministic gates.
In my opinion, the main text is very compressed and reads mostly as a list of results with pointers to the appendices, where the derivations, tables and scope qualifications are located, so I could not follow several parts of the argument from the main text alone.
Critical strengths
- The paper addresses a relevant problem. Recurrent-state quantization is a real deployment cost for hybrid models, and the existing approach it builds on (DAMP, a recent preprint) is a heuristic without theoretical grounding. The distinction between propagation and retrieval is a useful way to ask which resource (precision or state dimension) limits memory under quantization.
Critical weaknesses
-
The paper does not meet the presentation standard expected at ICLR. The main text does not stand on its own: derivations, result tables and scope qualifications are deferred to the appendices, and the body mostly states a result, gives a number and points elsewhere. The theorems are often supported by statements that a simulation agreed rather than by an argument (e.g., Theorem 7, “Verified: greedy equals exhaustive search”). Notation is used in the abstract and introduction before being defined, the model of Eq. (1) is not related to any existing architecture, and the assumptions (A1) to (A4) are listed without discussion, even though the white-noise assumption is the one the authors themselves say fails at low bit widths. The text also contains visible inconsistencies, such as two nearly identical sentences at the end of Section 2 and two different definitions of the calibration constant C. This is consequential because a paper whose contribution is a set of “laws” has to let the reader follow and verify the derivations, and here I could not do so from the main text.
-
Several central claims are contradicted by the paper own results or do not follow from its formulas. Theorem 1 predicts that each additional bit doubles propagation reach, yet Fig. 1 shows fp32 and 8-bit states coinciding at every delay (maybe, I miss-understood something here, and I welcome explanation), and the text attributes this saturation to Theorem 1, which contains no saturation term. The gating experiment cited for Theorem 5 is outside the theorem own validity condition, and its measured effect does not match the prediction. Finally, the persistence correction has no measurable end-to-end effect. This is consequential because once these points are removed, little of the claimed theory is both correct and supported by evidence.
-
The experiments do not reach the setting that motivates the paper. The motivation is degraded reasoning under INT8 state quantization in large hybrid models such as Gated DeltaNet and Kimi Delta Attention, but no pretrained model of either family is evaluated, and no reasoning benchmark is used. The evidence consists of small synthetic models and one 130M Mamba-1 checkpoint evaluated by perplexity on text outside its training language, and on that model INT8 costs only 0.12% perplexity, so the motivating degradation never appears. The paper also says its scripts are released, but I found no anonymized link or supplementary code. This is consequential because the claimed relevance to deployment cannot be checked, either from the reported experiments or by reproducing them.
Questions for authors
I do not have questions whose answers would materially change my assessment. Out of curiosity, however, I would like to understand how the authors see the assumptions of Section 2 being satisfied, or at least checked, in practice. The closed forms rely on i.i.d. zero-mean inputs, white quantization noise uncorrelated with the state, no clipping, and stationarity. In a trained language model, the inputs to the recurrence are input-dependent and temporally correlated, deterministic rounding is by construction correlated with the state (the dead-zone behavior reported in the appendix seems to be one consequence), and the gates vary from token to token. How would a practitioner verify that a given layer of a deployed model operates in the regime where these laws apply, and what should they expect when it does not?