acceptodds
Under review as a conference paper at ICLR 2027

THE LOW-BIT MEMORY LAW: PRECISION BUYS PROPAGATION, DIMENSION BUYS RETRIEVAL, AND GATING MUST BE STABLE

Abstract

Recurrent-state architectures (linear RNNs, SSMs, Gated DeltaNet, Kimi Delta Attention) replace the KV cache of most layers with a fixed-size state, making state quantization a primary deployment bottleneck: recent work reports uniform INT8 already degrades complex reasoning and INT4 collapses to near zero. No quantitative law, however, says how long a b-bit recurrent state can remember—expressivity analyses ask whether such states can compute a function, and deployment heuristics give no account of why persistence sets risk or when the heuristic breaks. We supply the law. Under finite precision, memory is governed by three resources with closed-form bounds: (i) precision buys propagation—reach is k_max = (2√3 / eC) * 2^b steps, each bit doubling it, at a unique self-consistent forgetting rate α_*(b); (ii) dimension buys retrieval—read accuracy saturates beyond b_sat(α) = 1/2 log2( C² / (3(1−α²)) ) bits, above which 8 bits is indistinguishable from fp32 at every load while 4 bits suffices only up to M ≈ d_k/4 (the retrieval bit requirement grows with load); (iii) gating noise is exponentially amplified—one standard deviation of gate-logit noise costs 0.72 state bits. A fourth result repairs current practice: the correct per-channel persistence measure is P = 1/(1 − E[a_t²]), and the geometric-mean version used by DAMP (Zhang et al., 2026) probably underestimates risk (Jensen), by 2.7–5.7× under the occasional-reset gates that selective architectures learn; correcting it removes 99.7 ± 0.6% of the excess risk of Top-K protection across five independently trained models at an identical bit budget. That gain is a statement about gate variability, not a universal one: where the gates are nearly deterministic the correction is inert. The resulting bit-requirement log-law b_req(P) = 1/2 log2(C² P /3) explains why protected channels must be FP16, why INT4 variants cannot be repaired by layout changes, and why DAMP’s Hadamard transform works (it lowers C by up to Δb ≈ 1.24 bits on heavy-tailed blocks). Beyond closed-form verification we measure a real pretrained Mamba-1 (130M, 1536 channels): its channel statistics transfer (23× energy concentration above uniform; material damage from 4–5 bits, INT8 costing 0.12% perplexity) while the ranking correction does not, because that model’s gates are nearly deterministic—mask-to-mask variation (0.44 PPL s.d.) exceeds the correction (0.19 PPL)—so its scope is gates with non-negligible Var(ln a). All results are reproducible from released scripts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
low-bit memory law, state quantization, SSM, Mamba, gating noise, finite precision, quantization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tVi9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies post-training quantization of the recurrent state in linear-attention and SSM layers, where the fixed-size state stored for each request becomes a memory bottleneck at inference. It builds on DAMP, a very recent heuristic that keeps the most “persistent” (the ones with the slowest decay) state channels in high precision and quantizes the rest, and tries to replace that heuristic with quantitative laws for how long and how accurately a low-bit recurrent state can remember. The paper is mainly a theory paper. The authors model a linear recurrence in which quantization acts as additive white noise, and they separate two memory effects or tasks: “propagation”, where the state must still carry a trace of something written many steps earlier, and “retrieval”, where a specific stored entry must be read back.

The main claim is that the two tasks depend on different resources. Precision extends how far information propagates, while state dimension governs retrieval accuracy, and past a certain bit width extra precision is said to bring nothing for retrieval. Noise in the gates shortens memory. The authors also argue that DAMP measures channel persistence with the wrong statistic, a geometric mean of the decay, which underestimates quantization risk when gates occasionally reset, and they propose a second moment measure instead. They then use these results to explain DAMP design choices. The experiments are mostly synthetic and small scale, plus one small pretrained Mamba model, on which the proposed correction has no measurable effect; the authors attribute this to the model nearly deterministic gates.

In my opinion, the main text is very compressed and reads mostly as a list of results with pointers to the appendices, where the derivations, tables and scope qualifications are located, so I could not follow several parts of the argument from the main text alone.

Critical strengths
  1. The paper addresses a relevant problem. Recurrent-state quantization is a real deployment cost for hybrid models, and the existing approach it builds on (DAMP, a recent preprint) is a heuristic without theoretical grounding. The distinction between propagation and retrieval is a useful way to ask which resource (precision or state dimension) limits memory under quantization.
Critical weaknesses
  1. The paper does not meet the presentation standard expected at ICLR. The main text does not stand on its own: derivations, result tables and scope qualifications are deferred to the appendices, and the body mostly states a result, gives a number and points elsewhere. The theorems are often supported by statements that a simulation agreed rather than by an argument (e.g., Theorem 7, “Verified: greedy equals exhaustive search”). Notation is used in the abstract and introduction before being defined, the model of Eq. (1) is not related to any existing architecture, and the assumptions (A1) to (A4) are listed without discussion, even though the white-noise assumption is the one the authors themselves say fails at low bit widths. The text also contains visible inconsistencies, such as two nearly identical sentences at the end of Section 2 and two different definitions of the calibration constant C. This is consequential because a paper whose contribution is a set of “laws” has to let the reader follow and verify the derivations, and here I could not do so from the main text.

  2. Several central claims are contradicted by the paper own results or do not follow from its formulas. Theorem 1 predicts that each additional bit doubles propagation reach, yet Fig. 1 shows fp32 and 8-bit states coinciding at every delay (maybe, I miss-understood something here, and I welcome explanation), and the text attributes this saturation to Theorem 1, which contains no saturation term. The gating experiment cited for Theorem 5 is outside the theorem own validity condition, and its measured effect does not match the prediction. Finally, the persistence correction has no measurable end-to-end effect. This is consequential because once these points are removed, little of the claimed theory is both correct and supported by evidence.

  3. The experiments do not reach the setting that motivates the paper. The motivation is degraded reasoning under INT8 state quantization in large hybrid models such as Gated DeltaNet and Kimi Delta Attention, but no pretrained model of either family is evaluated, and no reasoning benchmark is used. The evidence consists of small synthetic models and one 130M Mamba-1 checkpoint evaluated by perplexity on text outside its training language, and on that model INT8 costs only 0.12% perplexity, so the motivating degradation never appears. The paper also says its scripts are released, but I found no anonymized link or supplementary code. This is consequential because the claimed relevance to deployment cannot be checked, either from the reported experiments or by reproducing them.

Questions for authors

I do not have questions whose answers would materially change my assessment. Out of curiosity, however, I would like to understand how the authors see the assumptions of Section 2 being satisfied, or at least checked, in practice. The closed forms rely on i.i.d. zero-mean inputs, white quantization noise uncorrelated with the state, no clipping, and stationarity. In a trained language model, the inputs to the recurrence are input-dependent and temporally correlated, deterministic rounding is by construction correlated with the state (the dead-zone behavior reported in the appendix seems to be one consequence), and the gates vary from token to token. How would a practitioner verify that a given layer of a deployed model operates in the regime where these laws apply, and what should they expect when it does not?

Reviewer_cZSDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper develops a unified account of low-precision recurrent memory by separating propagation (how long a perturbation survives), retrieval (how accurately one stored entry can be read), and gating (how variability changes persistence). Under a scalar/isotropic linear recurrence with white write-back noise, it derives closed-form propagation, retrieval, gating-cost, persistence, and bit-allocation laws, and supports them with synthetic tasks, trained gates, a DAMP protocol reproduction, and one Mamba-1 checkpoint. The framework is clear and potentially useful, but I recommend rejection because the central persistence correction is not established for input/state-dependent gates in the target GDN/KDA setting, while its practical end-to-end benefit is demonstrated only on a small synthetic probe and is unresolvable on the real checkpoint.

Critical strengths
  1. The propagation/retrieval distinction is a useful organizing contribution. The paper makes the resource being measured explicit and connects it to concrete expressions such as the adaptive-calibration reach law, the retrieval saturation point, and the persistence-dependent bit requirement. The assumptions and the fixed-range versus adaptive-calibration branches are stated rather than hidden.
  2. The empirical design catches important protocol confounds. The overwrite experiment shows why a frozen low-bit state can look good under a single-write retention curve, and the paired retrieval comparisons correctly test differences on shared draws. The closed-form-to-simulation checks, seed-level reporting, and model-level replication of the trained-gate probe make the reported toy results unusually auditable.
  3. The paper is unusually candid about validity boundaries and reproducibility. It releases scripts, reports fixed seeds and paired uncertainty, measures both selective and control gates, and includes a real Mamba-1 checkpoint. In particular, explicitly reporting that the persistence correction is nearly inert and that the ranking effect is not resolvable on real text prevents the main synthetic result from being presented as universal.
Critical weaknesses
  1. Theorem 6 does not match the gate dependency in the target architectures. Its exact recursion for Ptrue=1/(1−E[at2])P^{\mathrm{true}}=1/(1-\mathbb{E}[a_t^2]) uses i.i.d. ata_t independent of the quantization error (Appendix A.5), whereas GDN/KDA/Mamba gates are input-dependent and can be correlated with the recurrence and its error history. In that setting the relevant term is generally E[at2et−12]\mathbb{E}[a_t^2 e_{t-1}^2], so the factorization used by the theorem and the resulting DAMP correction need not hold. The trained probe is input-gated, and the real checkpoint is in the nearly deterministic regime; neither supplies evidence for the theorem in a nontrivial endogenous-gate model.
  2. The practical “repair” of DAMP is not established at the deployment scale or on its target architectures. The 99.7% reduction is an excess-risk metric in a d=32d=32 synthetic CLEAR-token probe; the end-to-end protocol reproduction gives only +0.46±0.52+0.46\pm0.52 SNR at INT4 and no gain at INT8. On Mamba-1, the correction is a near no-op and corrected-versus-DAMP masks are not distinguishable under mask resampling. There is no in-distribution pretrained GDN/KDA evaluation comparing the two rankings at a fixed storage budget on reasoning, code, or language-model metrics.
  3. Two headline quantitative laws are presented more broadly than the evidence supports. The trained-model sweep in Appendix A.9 never exhibits the predicted interior α∗(b)\alpha^*(b) turnover for the clean b≥6b\ge6 rows, and recovering a turnover requires adding a task/gate floor Λ0\Lambda_0 that is not derived in the paper. The capacity fit f(β)≈2.14β−1.28f(\beta)\approx2.14\beta^{-1.28} is based on only three resolved β\beta points at dk=64d_k=64, while the free fit gives 1.96β−1.441.96\beta^{-1.44} and cannot distinguish the exponents. These results are useful conditional and asymptotic observations, but the current wording does not cleanly separate proven statements from numerically fitted or protocol-dependent laws.

Justification of Recommendation

I recommend clear rejection because the central persistence correction is not established for the dependent gates of the target architectures, and the reported experiments do not demonstrate the claimed practical repair at deployment scale. The useful propagation/retrieval decomposition and auditable synthetic experiments do not resolve those issues, and the limited support for the fitted quantitative laws further restrains the recommendation. I would rate the paper higher with theory or targeted evidence covering dependent gates, fixed-storage comparisons on pretrained GDN/KDA models and relevant task metrics, and stronger validation or appropriately restricted statements of the quantitative laws.

Questions for authors
  1. Can the authors derive the persistence recursion for input/state-dependent gates, or at least report the measured joint factorization error E[at2et−12]−E[at2]E[et−12]\mathbb{E}[a_t^2e_{t-1}^2]-\mathbb{E}[a_t^2]\mathbb{E}[e_{t-1}^2] on GDN/KDA-like gates? Under what precise adaptedness or mixing assumptions would the corrected score remain valid?
  2. Can the authors evaluate DAMP and the corrected score on an in-distribution pretrained GDN or KDA checkpoint, using the same fixed storage budget and multiple calibration/mask draws, and report paired end-to-end language/reasoning metrics? This would test whether the synthetic ranking improvement survives when channel identity actually affects the task.
  3. Which parts of Theorem 1 and Proposition 4 are intended as exact results versus asymptotic or empirical fits? Please provide broader α,b,dk,β\alpha,b,d_k,\beta sweeps (or narrow the claims), state the threshold/calibration regime for the α∗\alpha^* formula, and quantify uncertainty for the fitted capacity exponent.
Reviewer_gsmVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the quantization of the recurrent state in linear-attention models. Prior work (DAMP) shows that quantizing the recurrent state even to INT8 hurts the model, and INT4 collapses it. The paper then derives three "laws": precision buys propagation, dimension buys retrieval, and gating must be stable. It then proposes a fix to DAMP's persistence measure and explains DAMP's INT4 failure. The only real pretrained model used here is Mamba-1 130M.

Critical strengths
  1. The derived closed-form formulas are correct and checkable under the stated assumptions.
Critical weaknesses
  1. The laws have limited usefulness, and some predictions fail: there is no optimal forgetting rate, as shown in Table 4, and 4 bits falls significantly below full precision as memory fills, although the law says about 2.5 bits should suffice (Table 13).

  2. The only real pretrained model tested is Mamba-1 130M, which is not a GDN or KDA model; the GDN results come from a small toy model. In comparison, DAMP is more complete, with experiments on much larger Qwen3.6 and Kimi models. As a proposed fix or improvement to DAMP, the paper does not provide enough evidence.

  3. The writing and presentation are very hard to follow, with multiple ultra-long single paragraphs: for example, Section 4.3 and the Section 6 discussion are each a single paragraph, and Table 6 is cut off at the edge of the page.

Questions for authors

Could you use the derived laws to make predictions on real-world models and validate your claims?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.