acceptodds
Under review as a conference paper at ICLR 2027

Preserving Updates in Low-Bit Recurrent State Quantization

Abstract

Linear attention and recurrent models compress past tokens into a fixed-size state that is updated at every step. However, storing these states in high precision incurs substantial memory and bandwidth costs at large batch sizes, motivating low-bit state quantization. Unlike weights and key–value caches, a recurrent state is requantized after every update, so its quantization errors accumulate across steps. Our analysis shows that these errors are highly coherent across steps, so reducing per-step errors alone is insufficient. This coherence arises mainly from small updates that are lost in rounding. To preserve these updates, we intervene both before and at rounding. Before rounding, we propose Low-Rank Bypass (LRB), which accumulates small updates in a compact low-rank buffer and periodically merges its sufficiently large elements into the state. At rounding, we observe that a state element can even move opposite to its update, and such reversals prove particularly harmful. Therefore, we propose Bounded Directional Rounding (BDR), which rounds reversed elements toward their updates without additional storage. Although BDR enlarges the rounding error, it improves the model, suggesting that the direction of an update matters more than its rounding error at a single step. Across Gated DeltaNet, GLA, Mamba2, and RetNet, our training-free method substantially outperforms INT4, while reducing state memory by up to 5.3 and speeding up decoding by up to 1.9 relative to FP32 states.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-Training Quantization, Linear Attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RwJPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies low-bit quantization of recurrent states in linear attention and state-space models. These states are updated at each time step and repeatedly quantized for storage, so quantization errors can accumulate over long sequences. The authors show that errors from different time steps tend to align rather than cancel out. They link this effect mainly to small state updates that are lost during rounding.

To address this issue, the authors propose two techniques. LRB accumulates state updates in a compact low-rank buffer. It periodically merges the accumulated updates into the quantized state and retains a low-rank approximation of the remaining small updates. BDR adjusts the rounding result in selected cases where quantization changes a state element in the opposite direction to its intended update.

The authors evaluate their training-free method on four 1.3B recurrent models: Gated DeltaNet, GLA, Mamba2, and RetNet. The method substantially improves language modeling and recall performance compared with standard INT4 quantization. It also reduces recurrent-state memory by 4.7–5.3 times and increases decoding throughput by up to 1.9 times compared with FP32 states.

Critical strengths
  1. The paper addresses an important problem in recurrent-state quantization and provides a clear analysis of how quantization errors accumulate over time. Unlike conventional quantization approaches that mainly focus on reducing the error at each step, the authors examine how errors from different steps interact. Their results show strong correlations across time steps, especially for errors associated with lost updates. This analysis offers a useful explanation for the performance degradation caused by low-bit recurrent-state quantization. It also provides a clear motivation for the proposed method.

  2. The proposed method substantially improves performance over standard INT4 quantization across four recurrent architectures. For example, on WikiText-103 with an 8K context length, RetNet's perplexity decreases from 1193.21 under INT4 to 28.80 with the proposed method, compared with 24.30 for FP32 states.

Critical weaknesses
  1. The ablation study does not clearly isolate the contribution of low-rank residual retention. My main concern is whether low-rank residual retention provides a substantial benefit beyond delayed state quantization. In Table 8, the results with k=0k=0, where the method does not retain low-rank residuals across merges, are already close to those with k=1k=1 for GDN, GLA, and Mamba2. At an 8K context length, the perplexity differences are only 0.42, 0.77, and 0.56, respectively. However, RetNet shows a much larger improvement, from 77.86 to 28.80. Table 8 keeps k+mk+m fixed rather than the merge interval mm. As a result, the merge interval changes along with the retained rank kk. This makes it difficult to determine how much of the improvement comes from delayed quantization and how much comes from retaining low-rank residuals. Since residual retention is an important part of LRB, I believe the paper needs a more controlled ablation. In particular, the authors should compare different values of kk, including k=0k=0, while keeping mm fixed. This would help clarify the independent contribution of low-rank residual retention.

If the authors can convincingly address these concerns during the rebuttal, I would be willing to reconsider my rating.

Questions for authors
  1. In Table 8, the authors keep k+mk+m fixed, so the merge interval m changes when kk changes. Could the authors provide an ablation that varies kk, including k=0k=0, while keeping m fixed? This would help separate the benefit of delayed quantization from that of low-rank residual retention. Could the authors also explain why k=1k=1 provides only modest improvements over k=0k=0 for GDN, GLA, and Mamba2, but gives a much larger improvement for RetNet at 8K context length?

  2. The proposed method uses more memory per state element than the INT4 baselines. Could the authors compare their method with simpler alternatives, such as periodic quantization without residual retention across merges and higher-bit-width quantization, under similar memory budgets? In particular, how does the proposed method compare with these alternatives when they use similar average and peak state memory? This would help clarify whether the proposed design offers benefits beyond those of delayed quantization or additional storage.

  3. In Table 1, the update term BtB_t for Gated DeltaNet depends on the previous state. However, Equation (2) assumes that the full-precision and quantized recurrences use the same updates BtB_t. Does Equation (2) describe an analysis with fixed updates, or does it represent the actual error between independently executed FP32 and quantized models? If the authors intend the latter, could they clarify how the analysis accounts for changes in BtB_t caused by state quantization?

6 comments
AuthorsOpenReview ↗

Response to Reviewer RwJP (1/4): W1 + Q1 — Fixed- ablation of residual retention

Dear Reviewer RwJP,

Thank you for the careful and constructive review. We have run the fixed-mm ablation, together with the memory-matched comparisons and the Eq. 2 analysis you asked for. All new results use the protocol of Tables 7–9 (1.3B models, WikiText-103) unless noted otherwise.


W1 + Q1: Isolating low-rank residual retention with mm fixed, and why RetNet benefits most.

Re: Thank you for this suggestion. We have run the controlled ablation: mm is fixed and only kk varies.

TL;DR.

  • Retention helps on every model, and its gain grows steadily with context. With mm fixed, the gain is larger than Table 8 suggested.
  • Periodic merging gives most of the gain on GDN, GLA, and Mamba2; retention is decisive on RetNet (80.77 → 28.80 at 8K).
  • RetNet benefits most because its updates are the smallest and its state decays the slowest, as Sec. 2.3 predicts.

For clarity, LRB has two parts, and both follow from Sec. 2.3 (an update is lost when it is smaller than half a grid step):

  • Periodic merging (k=0k=0): updates accumulate in the bypass and are rounded only every mm steps, so they are larger when rounded.
  • Residual retention (k≥1k \ge 1): at each merge, the part that is still too small is kept as a rank-kk term for later merges, instead of being rounded away.

Table A. Fixed-mm ablation (m=15m=15; 5 for Mamba2; BDR on; WikiText-103 PPL ↓).

ModelkkAvg. bits1K2K4K8K
GDN05.1631.4827.3726.6627.59
1 (default)5.2531.3827.2326.2726.85
25.3531.3827.2126.2026.77
FP323230.8426.3824.8524.19
GLA05.2225.4522.4422.3826.00
1 (default)5.3125.4722.3622.1924.86
25.4125.4522.3422.2124.94
FP323225.0621.6620.6921.57
Mamba205.2630.5928.0728.3729.54
1 (default)5.6430.3027.6027.6628.78
26.0230.3027.5627.6428.65
FP323229.4326.2025.3225.19
RetNet05.1627.0324.5428.0880.77
1 (default)5.2526.6823.7624.9728.80
25.3526.7224.0025.2428.80
FP323225.6622.3822.5124.30

What retention contributes (k=0k=0 → k=1k=1).

  • The gain grows steadily with context on all four models. Retention improves every model from 2K onward; at 1K, GLA is tied (25.45 vs. 25.47). It costs only 0.09 bits per element (0.38 for Mamba2).
  • The gain is larger than Table 8 suggested. At 8K it is 0.74 / 1.14 / 0.76 on GDN / GLA / Mamba2, versus 0.42 / 0.77 / 0.56 in Table 8, where the k=0k=0 row had a longer merge interval (m=16m=16).
  • It does not depend on BDR. With BDR off, the 8K gains are 0.81 / 1.06 / 0.81 / 45.89.

How the gain splits. On GDN, GLA, and Mamba2, periodic merging already makes most updates large enough to survive rounding, so it gives most of the gain over INT4; retention then closes a further 17–26% of the remaining gap to FP32 at 8K. On RetNet, periodic merging alone fails at long context, and retention closes 92% of the remaining gap.

Why RetNet benefits most. Retention matters when (i) many updates are still too small after accumulating for mm steps, and (ii) the state decays slowly, so each loss persists. RetNet is the extreme case on both:

GDNGLAMamba2RetNet
Single update / Δ\Delta, median (Table 5)0.210.070.190.02
Accumulated over mm steps / Δ\Delta, median0.400.240.540.18
Still rounded away at a merge (k=0k=0)54%66%47%73%
Decay (Fig. 1)slowslowfastestslowest
Remaining gap closed by retention (8K)22%26%17%92%
  • Its updates are the smallest. Even after 15 steps, the median accumulated update is only 0.18Δ0.18\Delta, far below the Δ/2\Delta/2 needed to survive rounding.
  • It decays the slowest, so these losses keep adding up: without retention, its PPL rises from 27.03 at 1K to 80.77 at 8K, while with retention it stays at 28.80.
  • Mamba2 is the opposite case. Its accumulated updates are the largest and it forgets fastest, so retention matters least.

The benefit of retention thus follows the order predicted by Sec. 2: the smaller the updates and the slower the decay, the more it helps. We will add Table A in the appendix and state this split explicitly in the paper.

AuthorsOpenReview ↗

Response to Reviewer RwJP (2/4): Q2 — Comparison under similar average and peak memory (GDN, GLA, RetNet)

Q2: Comparison with simpler alternatives under similar average and peak memory.

Re: Thank you for this question. It asks whether our gain comes from the design or simply from the extra storage. We compare with higher-bit-width per-step quantization and with periodic quantization without retention (k=0k=0).

TL;DR.

  • The gain does not come from extra bits. With fewer average and peak bits than INT4, our method is still better on GDN, GLA, and RetNet at every length.
  • More bits alone do not close the gap. Our method beats INT5 and matches or beats INT6 while using fewer average and peak bits.
  • On our own Mamba2 checkpoint, the margin is the smallest, as our analysis predicts for a fast-decaying model. On the public Mamba2 checkpoint [1], our method is far ahead: at 2K, 19.53 vs. 44.95 for INT6, with fewer average bits.

We write our configurations as Ours (INT$b$, kk, mm): bb-bit quantized state, retained rank kk, merge interval mm. The default (Table 6) is INT4 with kk=1 and mm=15 (mm=5 for Mamba2).

GDN, GLA, RetNet. Each of our rows uses fewer average and fewer peak bits than the per-step baseline above it.

PPL ↓Avg. / peak bits1K2K4K8K1K2K4K8K1K2K4K8K
GDNGLARetNet
INT4 (per step)4.50 / 4.5036.1036.4948.1579.7630.0232.5341.6866.5559.2275.61179.61193
Ours (INT3, kk=1, mm=7)3.88 / 4.2535.2133.5036.7144.4629.1630.0737.4458.2634.6736.3749.0980.71
INT5 (per step)5.50 / 5.5032.4828.8129.0332.4226.3725.2629.1139.8832.4531.8044.89159.5
Ours (INT4, kk=1, mm=7)4.88 / 5.2531.7327.9927.4629.1825.8023.1624.1529.2327.5225.6928.1144.86
INT6 (per step)6.50 / 6.5031.5427.3826.4026.7725.4322.5824.0631.1127.5325.1029.2483.98
Ours (INT4, kk=0, mm=15)5.16 / 5.9131.4827.3726.6627.5925.4522.4422.3826.0027.0324.5428.0880.77
Ours (INT4, kk=1, mm=15), default5.25 / 6.0031.3827.2326.2726.8525.4722.3622.1924.8626.6823.7624.9728.80
FP323230.8426.3824.8524.1925.0621.6620.6921.5725.6622.3822.5124.30
  • vs. INT4: better with fewer bits. With an INT3 base, our method is better on all three models at every length, while using 0.62 fewer average and 0.25 fewer peak bits. The gain therefore cannot come from extra storage.
  • vs. INT5: better with fewer bits. With mm=7, our method is better on all three models at every length.
  • vs. INT6: better or on par with 1.25 fewer average bits. The default is better in most cases; INT6 is ahead only on GDN at 8K and GLA at 1K, by at most 0.08.
  • vs. LRB without retention (kk=0): retention adds only 0.09 bits and improves long contexts on every model, most of all RetNet at 8K (80.77 → 28.80). It also wins at 8K at identical peak storage (Table 8; see W1).
  • More bits alone do not fix the problem. On RetNet at 8K, per-step quantization reaches 1193 → 159.5 → 83.98 → 67.47 with INT4 → INT5 → INT6 → INT8, versus 28.80 for ours at 5.25 bits. As Sec. 2.3 shows, a finer grid still loses RetNet's tiny updates; preserving them is what matters.
AuthorsOpenReview ↗

Response to Reviewer RwJP (3/4): Q2 — Mamba2 and other evaluation results

Mamba2.

PPL ↓Avg. / peak bits1K2K4K8K
INT4 (per step)4.50 / 4.5032.9631.6533.8937.43
Ours (INT3, kk=1, mm=3)4.26 / 5.0232.9431.8133.7237.69
Ours (INT3, kk=1, mm=5)4.64 / 5.7832.0330.6432.2035.00
INT5 (per step)5.50 / 5.5030.9628.6829.1430.55
Ours (INT4, kk=1, mm=3)5.26 / 6.0230.5027.9228.1529.65
Ours (INT4, kk=1, mm=5), default5.64 / 6.7830.3027.6027.6628.78
INT6 (per step)6.50 / 6.5030.1327.4827.4128.14
FP323229.4326.2025.3225.19
  • Our method beats INT5 with fewer average bits (mm=3).

  • INT6 is slightly better than our default (by 0.12–0.64), with 0.86 more average bits. Mamba2 is where Sec. 2 predicts the smallest benefit: it forgets fastest and has the lowest coherent gain (about 4, versus 18–86 for the other models; Fig. 1), and its small per-head state makes each bypass column about four times as costly.

  • On the public Mamba2 checkpoint, our method is better at every length, with fewer average bits than INT6. We also evaluate the public mamba2-1.3b checkpoint [1], applying our default Mamba2 configuration without any tuning. It is trained on different data from ours (the Pile, 300B tokens vs. FineWeb-Edu, 100B tokens).

Public Mamba2-1.3B [1], PPL ↓Avg. / peak bits1K2K4K8K
INT5 (per step)5.50 / 5.5042.62103.111759431
INT6 (per step)6.50 / 6.5027.4744.95208.81097
Ours (default)5.64 / 6.7821.7619.5320.7424.73
FP323220.3317.4518.4723.20
  • Per-step quantization degrades sharply with context. From 1K to 8K, INT6 rises from 27.47 to 1097, and INT5 from 42.62 to 9431.
  • Our method stays within 2.3 PPL of FP32 at every length, using 0.86 fewer average bits than INT6.
  • This matches Sec. 2.3. Under INT4, the public checkpoint's updates are about half as large relative to the grid step (median 0.09 vs. 0.19 at 4K), and more of them are lost in rounding (82% vs. 72%). The more updates rounding loses, the more per-step quantization suffers and the more preserving them helps.

Beyond WikiText-103. We repeat the comparison on PG-19 (8K PPL ↓) and on the recall-intensive tasks of Table 4 (average accuracy ↑). Mamba2 here is our own checkpoint, as in Tables 2–4 of the paper. Ours (INT3) uses the default kk and mm with an INT3 base.

GDNGLAMamba2RetNet
BitsPG-19RecallBitsPG-19RecallBitsPG-19RecallBitsPG-19Recall
INT4 (per step)4.50220.430.84.50158.827.74.5087.9629.04.50317021.4
Ours (INT3)4.2588.0136.24.3199.2237.84.6483.3928.54.25122.530.1
INT5 (per step)5.5088.1936.25.50112.536.95.5073.5033.05.50556.634.5
Ours (default)5.2572.2838.55.3162.2439.75.6469.7233.15.2575.9636.0
FP323266.8439.43255.8540.73263.9937.93260.0240.1

Bits: average bits per state element.

  • vs. INT4: our INT3 variant is better on PG-19 for all four models, and improves recall by 5.4–10.1 points on GDN, GLA, and RetNet, where it also uses fewer average bits.
  • vs. INT5: our default is better on PG-19 for all four models, and improves recall by 1.5–2.8 points on GDN, GLA, and RetNet, where it also uses fewer average bits.
  • On our Mamba2 checkpoint, the margin is again the smallest: recall is on par with the per-step baselines. This matches the WikiText-103 results above and the prediction of Sec. 2: Mamba2 forgets fastest, so lost updates have the least time to accumulate, and preserving them helps the least.

In short, the gain comes from preserving updates rather than from extra storage: it remains when our method uses fewer bits, and per-step quantization does not match it by adding bits.

Full results will be added to Tables 2–4 in the paper.

[1] Tri Dao and Albert Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML, 2024. Checkpoints: https://huggingface.co/state-spaces

Reviewer_UEcHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies repeatedly quantized recurrent states for INT4. It proposes (1) Low Rank Bypass, which stores update factors in FP16 and accumulates them before merging them into INT4 state, (2) Bounded Directional Rounding, which corrects a subset of updates whose direction is reversed by quantization. Experiments on different state models show improvements over ordinary INT4 state quantization. The paper also reports memory saving and decoding speedup than FP32 state storage.

Critical strengths
  1. The paper is well motivated and the state quantization problem is important. Repeatedly quantizing a recurrent state is different from quantizing a tensor once. The paper gives a useful account of how small errors can accumulate across steps (e.g. on GDN 8K context length, perplexity rises from 24 with FP32 states to 79 with original INT4).

  2. LRB exploits outer-product structure of state updates. Storing low-rank factors can preserve recent updates without maintaing a full-size residual state. BDR is also easy to follow.

Critical weaknesses
  1. My most concern is about method comparison. The comparison with existing state-quantization methods is incomplete. The main accuracy baselines are FP32, ordinary INT4, and the gains over them are convincing. However, Section 5 discusses methods that are closer to the problem addressed here (Q-Mamba for Mamba2 and DAMP for GDN) without evaluating or difference discussion. Even a comparison on the overlapping architectures would help establish the practical contribution beyond the baselines already included.

  2. The efficiency results leave the practical choice of state precision unresolved. The paper reports around 5× reduction in state memory vs. FP32. But the paper does not show whether FP16/BF16 state storage would already preserve sufficient accuracy.

  3. BDR has not been compared with stochastic rounding under the same LRB configuration. Stochastic rounding is particularly relevant to paper's motivation because it offers a similar way to reduce the systematic loss of small updates. The paper discusses related work using it but without comparison.

Questions for authors
  1. How much of LRB’s gain comes from periodic requantization alone? Since with k = 0 on Table 8, LRB simply keeps the last m updates in FP16 and merges all of them every m steps. This variant already reaches 27/ 25/ 29 at 8K on GDN / GLA / Mamba2.

  2. Could you please provide more comparison or discussion with existing state quantization methods as weakness1?

7 comments
AuthorsOpenReview ↗

Response to Reviewer UEcH (1/5): W1 + Q2 — Comparison with Q-Mamba and Quamba2 on Mamba2

Dear Reviewer UEcH,

Thank you for the constructive review and for recognizing the motivation of our analysis. We address each point below; all new results use the protocol of Tables 7–9 (1.3B models, WikiText-103) unless noted otherwise. We write Ours (INT$b$) for our method with the quantized state stored in bb bits; the default is INT4 (Table 6).


W1 + Q2: Comparison with existing state-quantization methods (Q-Mamba, DAMP).

Re: Thank you for this suggestion. We compare with Q-Mamba [1] on Mamba2 and with DAMP [4] on GDN. For completeness, we also include Quamba2 [2] and the public mamba2-1.3b and mamba2-2.7b checkpoints [3], to which we apply our default Mamba2 configuration without any tuning.

TL;DR.

  • vs. Q-Mamba, at matched average bits (≈4.7): our INT3 variant is clearly better on both public Mamba2 checkpoints, and on par on our checkpoint.
  • vs. Quamba2, at its native 8 bits: our default is better on all three checkpoints at every length, with 2.4 fewer average bits.
  • vs. DAMP: it is near-lossless at its native ≈9.9 bits, but breaks down at ≈5 bits, where ours stays close to FP32.
  • Why: these methods reduce the error of each requantization, but still requantize at every step, so their errors keep accumulating (Sec. 2). Our advantage therefore grows with context.

Setup. Besides our Mamba2 checkpoint, we evaluate the public mamba2-1.3b and mamba2-2.7b checkpoints [3]. Ours (INT3) uses the same kk and mm with an INT3 base. Baselines cover state quantization only: Q-Mamba is implemented from the paper (its training-free state quantizer), Quamba2 follows the official code (8-bit is its native setting; 4-bit uses the same calibration), and DAMP is implemented from the paper.

Public Mamba2-1.3B (WikiText-103 PPL ↓)

MethodAvg. / peak bits1K2K4K8K
Quamba2, INT44.00 / 4.00136.5242.0596.41519
INT4 (per step)4.50 / 4.50124461972709160568
Ours (INT3)4.64 / 5.7827.8931.3242.4661.92
Q-Mamba, 4-bit4.75 / 4.7530.6540.3977.51213.5
Ours (default)5.64 / 6.7821.7619.5320.7424.73
Quamba2, INT88.00 / 8.0024.4525.3634.2256.41
FP323220.3317.4518.4723.20

Public Mamba2-2.7B

MethodAvg. / peak bits1K2K4K8K
Quamba2, INT44.00 / 4.0064.3194.59250.31172
INT4 (per step)4.50 / 4.50112.1339.1920.5860.4
Ours (INT3)4.64 / 5.7819.6018.6927.0078.52
Q-Mamba, 4-bit4.75 / 4.7521.1426.0065.32193.9
Ours (default)5.64 / 6.7817.7915.8717.1731.73
Quamba2, INT88.00 / 8.0019.2620.0335.77149.1
FP323217.1114.8613.9725.89

Our Mamba2-1.3B checkpoint

MethodAvg. / peak bits1K2K4K8K
Quamba2, INT44.00 / 4.0064.1773.8799.73156.3
INT4 (per step)4.50 / 4.5032.9631.6533.8937.43
Ours (INT3)4.64 / 5.7832.0330.6432.2035.00
Q-Mamba, 4-bit4.75 / 4.7532.7230.9932.0333.79
Ours (default)5.64 / 6.7830.3027.6027.6628.78
Quamba2, INT88.00 / 8.0030.4028.0028.5231.07
FP323229.4326.2025.3225.19
  • vs. Q-Mamba at matched average bits. On the public checkpoints, our INT3 variant is better at every length, and the gap widens with context (1.3B: 27.89 vs. 30.65 at 1K, 61.92 vs. 213.5 at 8K). On our checkpoint the two are close, with Q-Mamba ahead at 8K (33.79 vs. 35.00).
  • vs. Quamba2 at its native 8 bits. Our default is better on all three checkpoints at every length, with 2.4 fewer average bits. On public 1.3B, it stays within 2.3 PPL of FP32, whereas Quamba2 reaches 56.41 at 8K.
  • Why the public checkpoints differ. They are trained differently from ours (the Pile, 300B tokens vs. FineWeb-Edu, 100B tokens), so their states behave differently under quantization. Under INT4, the updates of public 1.3B are about half as large relative to the grid step as those of our checkpoint (median 0.09 vs. 0.19 at 4K), and more of them are lost in rounding (82% vs. 72%). Consistent with the prediction from Sec. 2.3, preserving updates then matters more.
AuthorsOpenReview ↗

Response to Reviewer UEcH (2/5): W1 + Q2 (continued) — Comparison with DAMP on GDN

GDN: comparison with DAMP (WikiText-103 PPL ↓)

MethodAvg. / peak bits2K8K
INT4 (per step)4.50 / 4.5036.4979.76
DAMP at ≈5 bits (FP16 rows + INT4)5.21 / 5.2130.85diverges
Ours (default)5.25 / 6.0027.2326.85
DAMP (native)9.88 / 9.8826.4324.31
FP323226.3824.19
  • DAMP is near-lossless at 9.88 bits, but it does not scale down to ≈5 bits. We extend it to this budget with the same row selection and an INT4 tier. It is already worse than ours at 2K, and diverges at 8K.

How our method differs. Q-Mamba, Quamba2, and DAMP all reduce the error of each requantization, through finer scales, static calibration, or keeping risky channels in higher precision. They still requantize the state at every step. As Sec. 2 shows, these per-step errors accumulate coherently because small updates are lost in rounding, so a smaller per-step error delays the degradation but does not stop it. Our method instead preserves the lost updates. The results follow this prediction: on public 1.3B, our lead over Quamba2 (INT8) widens from 2.7 PPL at 1K to 31.7 at 8K.

We will add these comparisons and this discussion to the paper.

[1] Chen Tianqi et al. Q-Mamba: Towards More Efficient Mamba Models via Post-Training Quantization. Findings of ACL, 2025.

[2] Hung-Yueh Chiang et al. Quamba2: A Robust and Scalable Post-Training Quantization Framework for Selective State Space Models. ICML, 2025.

[3] Tri Dao and Albert Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML, 2024. Checkpoints: https://huggingface.co/state-spaces

[4] Tao Zhang et al. DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization. arXiv:2608.27513, 2026.

AuthorsOpenReview ↗

Response to Reviewer UEcH (3/5): W2 — FP16/BF16 state storage

W2: Would FP16/BF16 state storage already preserve sufficient accuracy?

Re: Thank you for raising this practical question. We added FP16 and BF16 state storage, and also evaluated our method with an INT8 base.

TL;DR.

  • BF16 does preserve accuracy; FP16 does not, as it overflows on GLA and RetNet.
  • Below 16 bits, per-step quantization is no longer reliable, but our method is. Our method on top of an INT8 state (≈9.3 bits) matches BF16, whereas plain per-step INT8 (8.5 bits) collapses on RetNet (24.29 vs. 67.47 at 8K).
  • Our default (≈5.3 bits) uses 2.4–2.7× less state memory than BF16 and decodes faster on three of the four models, at a cost of 0.7–1.4 PPL at 2K and 2.7–4.5 at 8K.

Quality (WikiText-103 PPL ↓).

GDNGLAMamba2RetNet
MethodBits2K8K2K8K2K8K2K8K
FP323226.3824.1921.6621.5726.2025.1922.3824.30
BF16 (per step)1626.3824.2721.6521.6126.1225.3422.4324.99
FP16 (per step)1626.3824.20divergesdiverges26.1625.06divergesdiverges
INT8 (per step)8.526.5624.7221.8021.8626.6126.4222.6867.47
Ours (INT8)≈ 9.326.4024.2621.6621.5826.2925.4922.3924.29
Ours (default, INT4)≈ 5.327.2326.8522.3624.8627.6028.7823.7628.80

Bits of our rows are for GDN/RetNet; GLA is ≈0.06 higher and Mamba2 ≈0.4 higher.

  • BF16 is nearly lossless, but FP16 is not reliable. FP16 overflows on GLA and RetNet, whose states contain large outliers.
  • Our method on top of an INT8 state matches BF16 at only 60% of its bits. It is at most 0.2 PPL above BF16 on all four models at both lengths, and closer to FP32 on RetNet at 8K (24.29 vs. 24.99).
  • Simply using fewer bits per step does not work at long context. On RetNet, per-step INT8 is fine at 2K (22.68) but reaches 67.47 at 8K, while our INT8 variant stays at 24.29. As Sec. 2 shows, the errors of per-step quantization accumulate with context; preserving the updates prevents this.

Efficiency (RTX 3060, batch 48; state memory of all layers and end-to-end decoding throughput, as in Fig. 5).

GDNGLAMamba2RetNet
State memory (MiB) ↓
FP323024230446084608
BF161512115223042304
Ours (default)567437976865
Ours vs. BF162.7×2.6×2.4×2.7×
Throughput (tok/s) ↑
FP321342155310101021
BF161870214413361608
Ours (default)2085234612622025
Ours vs. BF161.11×1.09×0.94×1.26×
  • Memory: our default uses 2.4–2.7× less state memory than BF16 on all four models.
  • Throughput: it decodes 1.09–1.26× faster than BF16 on GDN, GLA, and RetNet. On Mamba2 it is slightly slower: its per-head state is the smallest, so the bypass takes a larger share of memory traffic and is merged more often (every 5 steps).

The practical choice. These results give two options, depending on the memory budget:

State storageBitsQualityWhen to use
Ours (INT8)≈ 9.3near-losslesssame quality at about 60% of the memory
Ours (default)≈ 5.30.7–4.5 PPL above FP32memory is the main constraint, e.g., large batches

We will add these results and this discussion to the paper.

Reviewer_EfzwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the loss of small updates due to rounding and the coherent accumulation of quantization errors in low-bit recurrent states. It introduces Low-Rank Bypass (LRB), which accumulates updates in a low-rank buffer, and Bounded Directional Rounding (BDR), which corrects rounding-induced update-direction reversals. Experiments on 1.3B-parameter Gated DeltaNet, GLA, Mamba2, and RetNet models demonstrate better preservation of FP32 quality than standard INT4 quantization, alongside 4.7–5.3× recurrent-state memory compression and 1.3–1.9× higher end-to-end decoding throughput at a batch size of 48.

Critical strengths
  1. This paper investigates the loss of small updates due to rounding and the coherent accumulation of quantization errors in low-bit recurrent states. It introduces Low-Rank Bypass (LRB), which accumulates updates in a low-rank buffer, and Bounded Directional Rounding (BDR), which corrects rounding-induced update-direction reversals. Experiments on 1.3B-parameter Gated DeltaNet, GLA, Mamba2, and RetNet models demonstrate better preservation of FP32 quality than standard INT4 quantization, alongside 4.7–5.3× recurrent-state memory compression and 1.3–1.9× higher end-to-end decoding throughput at a batch size of 48.
  2. The method demonstrates substantial quality improvements across four architectures and reports system-level benefits. For example, on WikiText-103 at an 8K context length, GDN’s perplexity improves from 79.76 with INT4 to 26.85, approaching the FP32 result of 24.19. Beyond language modeling, improvements on recall-intensive tasks, together with reductions in state memory and gains in end-to-end decoding throughput, support the method’s practical value.
Critical weaknesses
  1. Missing relevant baselines and comparisons at comparable storage budgets. Stochastic rounding (SR) is a directly relevant baseline, particularly because the cited Nemotron 3 Super report discusses coherent rounding bias and adopts FP16+SR for recurrent state storage. Whether SR is effective at INT4 remains untested. Moreover, the proposed method uses 6.00–6.78 peak bits per element, compared with 4.50 bits for the INT4 baselines, without evaluating alternatives at comparable storage budgets. Comparisons against Q-Mamba and Quamba2 are also absent. These omissions limit the evidence for the method’s comparative advantages.
  2. Unclear scope of the analysis and contributions of individual components. Eq. 2 assumes identical updates across the two recurrences, leaving its relationship to the actual deviation from the FP32 model unclear, particularly for GDN, whose updates depend on the previous state. In addition, Table 8 shows that k=0, which still includes update accumulation and BDR, comes within 0.4–0.8 PPL of the default configuration on GDN, GLA, and Mamba2, although residual retention provides a substantial benefit for RetNet. Because the merge interval m also changes with k, the additional contribution of residual retention is not cleanly isolated.
Questions for authors
  1. How does the proposed method compare with INT5/INT6, smaller quantization groups, or mixed precision under comparable storage budgets? Could you provide comparisons that account for both average and peak storage?
  2. Could you provide quality comparisons against INT4+SR and LRB+SR? In particular, under the same LRB configuration, how do SR and BDR compare in terms of model performance and temporal error coherence?
  3. How is the reference state in Fig. 1 constructed? Specifically, does it reuse the updates computed along the INT4 trajectory? For GDN, do the conclusions about temporal error coherence still hold when the state-dependent delta-rule correction is included in error propagation?
  4. With the merge interval m held fixed, what is the difference between k=0, which retains no residual after merging, and k>0? Could this comparison isolate the additional benefit of low-rank residual retention beyond periodic merging?
  5. Could you compare against the state quantization methods of Q-Mamba and Quamba2 on Mamba2? Could you also report results on a public Mamba2 checkpoint to which these prior methods can be readily applied?
8 comments
AuthorsOpenReview ↗

Response to Reviewer Efzw (1/8): W1 + Q5 — Comparison with Q-Mamba and Quamba2, including public checkpoints

Dear Reviewer Efzw,

Thank you for the detailed and constructive review. We address each point below; all new results use the protocol of Tables 7–9 (1.3B models, WikiText-103) unless noted otherwise. We write our configurations as Ours (INT$b$, kk, mm): the quantized state is stored in bb bits, kk is the retained rank, and mm is the merge interval. The default (Table 6) is INT4 with kk=1 and mm=15 (mm=5 for Mamba2).


W1 + Q5: Comparison with Q-Mamba and Quamba2 on Mamba2, including a public checkpoint.

Re: Thank you for this suggestion. We compare with the state quantization of Q-Mamba [1] and Quamba2 [2] on our Mamba2 checkpoint and on two public checkpoints, mamba2-1.3b and mamba2-2.7b [3]. For completeness, we also compare with DAMP [4], a recent state quantization method for GDN.

TL;DR.

  • vs. Quamba2, at its native 8 bits: our default is better on all three checkpoints at every length, with fewer average and fewer peak bits.
  • vs. Q-Mamba, at matched average bits (≈4.7): our INT3 variant is clearly better on both public checkpoints, and on par on our checkpoint.
  • vs. DAMP (GDN): it is near-lossless at its native ≈9.9 bits, but breaks down at ≈5 bits, where ours stays close to FP32.
  • The gap widens with context, as Sec. 2 predicts: these methods reduce the error of each requantization but still requantize at every step.

Setup. On the public checkpoints, we apply our default Mamba2 configuration without any tuning. Ours (INT3) uses the same kk and mm with an INT3 base. All baselines cover state quantization only: Q-Mamba is implemented from the paper (its training-free state quantizer), Quamba2 follows the official code (8-bit is its native setting; 4-bit uses the same calibration), and DAMP is implemented from the paper.

Public Mamba2-1.3B (WikiText-103 PPL ↓)

MethodAvg. / peak bits1K2K4K8K
Quamba2, INT44.00 / 4.00136.5242.0596.41519
INT4 (per step)4.50 / 4.50124461972709160568
Ours (INT3)4.64 / 5.7827.8931.3242.4661.92
Q-Mamba, 4-bit4.75 / 4.7530.6540.3977.51213.5
Ours (default)5.64 / 6.7821.7619.5320.7424.73
Quamba2, INT88.00 / 8.0024.4525.3634.2256.41
FP323220.3317.4518.4723.20

Public Mamba2-2.7B

MethodAvg. / peak bits1K2K4K8K
Quamba2, INT44.00 / 4.0064.3194.59250.31172
INT4 (per step)4.50 / 4.50112.1339.1920.5860.4
Ours (INT3)4.64 / 5.7819.6018.6927.0078.52
Q-Mamba, 4-bit4.75 / 4.7521.1426.0065.32193.9
Ours (default)5.64 / 6.7817.7915.8717.1731.73
Quamba2, INT88.00 / 8.0019.2620.0335.77149.1
FP323217.1114.8613.9725.89

Our Mamba2-1.3B checkpoint

MethodAvg. / peak bits1K2K4K8K
Quamba2, INT44.00 / 4.0064.1773.8799.73156.3
INT4 (per step)4.50 / 4.5032.9631.6533.8937.43
Ours (INT3)4.64 / 5.7832.0330.6432.2035.00
Q-Mamba, 4-bit4.75 / 4.7532.7230.9932.0333.79
Ours (default)5.64 / 6.7830.3027.6027.6628.78
Quamba2, INT88.00 / 8.0030.4028.0028.5231.07
FP323229.4326.2025.3225.19
  • vs. Quamba2 at its native 8 bits. Our default is better on all three checkpoints at every length, with 2.4 fewer average and 1.2 fewer peak bits. On public 1.3B, it stays within 2.3 PPL of FP32, whereas Quamba2 reaches 56.41 at 8K.
  • vs. Q-Mamba at matched average bits. On the public checkpoints, our INT3 variant is better at every length, and the gap widens with context (1.3B: 27.89 vs. 30.65 at 1K, 61.92 vs. 213.5 at 8K). Its peak is about 1 bit higher. On our checkpoint the two are close, with Q-Mamba ahead at 8K (33.79 vs. 35.00).
  • Why the public checkpoints differ. They are trained differently from ours (the Pile, 300B tokens vs. FineWeb-Edu, 100B tokens), so their states behave differently under quantization. Under INT4, the updates of public 1.3B are about half as large relative to the grid step as those of our checkpoint (median 0.09 vs. 0.19 at 4K), and more of them are lost in rounding (82% vs. 72%). As Sec. 2.3 predicts, preserving updates then matters more.
AuthorsOpenReview ↗

Response to Reviewer Efzw (2/8): W1 + Q5 (continued) — Comparison with DAMP on GDN

GDN: comparison with DAMP (WikiText-103 PPL ↓)

MethodAvg. / peak bits2K8K
INT4 (per step)4.50 / 4.5036.4979.76
DAMP at ≈5 bits (FP16 rows + INT4)5.21 / 5.2130.85diverges
Ours (default)5.25 / 6.0027.2326.85
DAMP (native)9.88 / 9.8826.4324.31
FP323226.3824.19
  • DAMP is near-lossless at 9.88 bits, but it does not scale down to ≈5 bits. We extend it to this budget with the same row selection and an INT4 tier. It is already worse than ours at 2K, and diverges at 8K.

How our method differs. Q-Mamba, Quamba2, and DAMP all reduce the error of each requantization, through finer scales, static calibration, or keeping risky channels in higher precision. They still requantize the state at every step. As Sec. 2 shows, these per-step errors accumulate coherently because small updates are lost in rounding, so a smaller per-step error delays the degradation but does not stop it. Our method instead preserves the lost updates. The results follow this prediction: on public 1.3B, our lead over Quamba2 (INT8) widens from 2.7 PPL at 1K to 31.7 at 8K.

We will add these comparisons and this discussion to the paper.

[1] Chen Tianqi et al. Q-Mamba: Towards More Efficient Mamba Models via Post-Training Quantization. Findings of ACL, 2025.

[2] Hung-Yueh Chiang et al. Quamba2: A Robust and Scalable Post-Training Quantization Framework for Selective State Space Models. ICML, 2025.

[3] Tri Dao and Albert Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML, 2024. Checkpoints: https://huggingface.co/state-spaces

[4] Tao Zhang et al. DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization. arXiv:2608.27513, 2026.

AuthorsOpenReview ↗

Response to Reviewer Efzw (3/8): W1 + Q1 — INT5/INT6, smaller groups, and mixed precision (GDN, GLA, RetNet)

W1 + Q1: Comparison with INT5/INT6, smaller groups, and mixed precision under comparable average and peak storage.

Re: Thank you for this question. We compare with INT5 and INT6, a smaller group size, and mixed precision, and report both average and peak bits throughout.

TL;DR.

  • At comparable storage, our method consistently outperforms these alternatives on all four models, across datasets and tasks, with the largest margins at long context.

Notation. Ours (INT$b$, mm) denotes a bb-bit quantized state with merge interval mm, and kk=1 throughout. The default (Table 6) is INT4 with mm=15 (mm=5 for Mamba2).

GDN, GLA, RetNet (WikiText-103 PPL ↓)

The rows form three groups. In each group, our row has the fewest average bits and no more peak bits than any baseline. Bits are for GDN/RetNet; GLA is ≈0.06 higher for our rows.

Avg. / peak bits1K2K4K8K1K2K4K8K1K2K4K8K
GDNGLARetNet
INT4 (per step)4.50 / 4.5036.1036.4948.1579.7630.0232.5341.6866.5559.2275.61179.61193
Ours (INT3, mm=7)3.88 / 4.2535.2133.5036.7144.4629.1630.0737.4458.2634.6736.3749.0980.71
INT5 (per step)5.50 / 5.5032.4828.8129.0332.4226.3725.2629.1139.8832.4531.8044.89159.5
Mixed precision5.25 / 5.2534.1831.8733.1036.0427.6927.4931.5342.6541.7045.9469.51111.8
Ours (INT4, mm=7)4.88 / 5.2531.7327.9927.4629.1825.8023.1624.1529.2327.5225.6928.1144.86
INT4, group size 326.00 / 6.0033.5631.6736.7654.9527.6429.2140.3073.2339.0043.7179.10537.6
INT6 (per step)6.50 / 6.5031.5427.3826.4026.7725.4322.5824.0631.1127.5325.1029.2483.98
Ours (default)5.25 / 6.0031.3827.2326.2726.8525.4722.3622.1924.8626.6823.7624.9728.80
FP323230.8426.3824.8524.1925.0621.6620.6921.5725.6622.3822.5124.30
  • vs. INT4: our INT3 variant is better on all three models at every length, with fewer average and fewer peak bits. The gain therefore does not come from extra storage.
  • vs. INT5 and mixed precision: with mm=7, ours is better on all three models at every length, with fewer average bits and no more peak bits.
  • vs. smaller groups: at the same peak bits, our default is better by a wide margin (e.g., 26.85 vs. 54.95 on GDN at 8K).
  • vs. INT6: with 1.25 fewer average and 0.5 fewer peak bits, our default is better in most cases. INT6 is ahead only on GDN at 8K and GLA at 1K, by at most 0.08.
  • More bits alone do not fix the problem. On RetNet at 8K, per-step quantization reaches 1193 → 159.5 → 83.98 with INT4 → INT5 → INT6, versus 28.80 for ours at 5.25 bits. As Sec. 2.3 shows, a finer grid still loses RetNet's tiny updates; preserving them is what matters.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.