- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies low-bit quantization of recurrent states in linear attention and state-space models. These states are updated at each time step and repeatedly quantized for storage, so quantization errors can accumulate over long sequences. The authors show that errors from different time steps tend to align rather than cancel out. They link this effect mainly to small state updates that are lost during rounding.
To address this issue, the authors propose two techniques. LRB accumulates state updates in a compact low-rank buffer. It periodically merges the accumulated updates into the quantized state and retains a low-rank approximation of the remaining small updates. BDR adjusts the rounding result in selected cases where quantization changes a state element in the opposite direction to its intended update.
The authors evaluate their training-free method on four 1.3B recurrent models: Gated DeltaNet, GLA, Mamba2, and RetNet. The method substantially improves language modeling and recall performance compared with standard INT4 quantization. It also reduces recurrent-state memory by 4.7–5.3 times and increases decoding throughput by up to 1.9 times compared with FP32 states.
Critical strengths
-
The paper addresses an important problem in recurrent-state quantization and provides a clear analysis of how quantization errors accumulate over time. Unlike conventional quantization approaches that mainly focus on reducing the error at each step, the authors examine how errors from different steps interact. Their results show strong correlations across time steps, especially for errors associated with lost updates. This analysis offers a useful explanation for the performance degradation caused by low-bit recurrent-state quantization. It also provides a clear motivation for the proposed method.
-
The proposed method substantially improves performance over standard INT4 quantization across four recurrent architectures. For example, on WikiText-103 with an 8K context length, RetNet's perplexity decreases from 1193.21 under INT4 to 28.80 with the proposed method, compared with 24.30 for FP32 states.
Critical weaknesses
- The ablation study does not clearly isolate the contribution of low-rank residual retention. My main concern is whether low-rank residual retention provides a substantial benefit beyond delayed state quantization. In Table 8, the results with , where the method does not retain low-rank residuals across merges, are already close to those with for GDN, GLA, and Mamba2. At an 8K context length, the perplexity differences are only 0.42, 0.77, and 0.56, respectively. However, RetNet shows a much larger improvement, from 77.86 to 28.80. Table 8 keeps fixed rather than the merge interval . As a result, the merge interval changes along with the retained rank . This makes it difficult to determine how much of the improvement comes from delayed quantization and how much comes from retaining low-rank residuals. Since residual retention is an important part of LRB, I believe the paper needs a more controlled ablation. In particular, the authors should compare different values of , including , while keeping fixed. This would help clarify the independent contribution of low-rank residual retention.
If the authors can convincingly address these concerns during the rebuttal, I would be willing to reconsider my rating.
Questions for authors
-
In Table 8, the authors keep fixed, so the merge interval m changes when changes. Could the authors provide an ablation that varies , including , while keeping m fixed? This would help separate the benefit of delayed quantization from that of low-rank residual retention. Could the authors also explain why provides only modest improvements over for GDN, GLA, and Mamba2, but gives a much larger improvement for RetNet at 8K context length?
-
The proposed method uses more memory per state element than the INT4 baselines. Could the authors compare their method with simpler alternatives, such as periodic quantization without residual retention across merges and higher-bit-width quantization, under similar memory budgets? In particular, how does the proposed method compare with these alternatives when they use similar average and peak state memory? This would help clarify whether the proposed design offers benefits beyond those of delayed quantization or additional storage.
-
In Table 1, the update term for Gated DeltaNet depends on the previous state. However, Equation (2) assumes that the full-precision and quantized recurrences use the same updates . Does Equation (2) describe an analysis with fixed updates, or does it represent the actual error between independently executed FP32 and quantized models? If the authors intend the latter, could they clarify how the analysis accounts for changes in caused by state quantization?
6 comments
Response to Reviewer RwJP (2/4): Q2 — Comparison under similar average and peak memory (GDN, GLA, RetNet)
Q2: Comparison with simpler alternatives under similar average and peak memory.
Re: Thank you for this question. It asks whether our gain comes from the design or simply from the extra storage. We compare with higher-bit-width per-step quantization and with periodic quantization without retention ().
TL;DR.
- The gain does not come from extra bits. With fewer average and peak bits than INT4, our method is still better on GDN, GLA, and RetNet at every length.
- More bits alone do not close the gap. Our method beats INT5 and matches or beats INT6 while using fewer average and peak bits.
- On our own Mamba2 checkpoint, the margin is the smallest, as our analysis predicts for a fast-decaying model. On the public Mamba2 checkpoint [1], our method is far ahead: at 2K, 19.53 vs. 44.95 for INT6, with fewer average bits.
We write our configurations as Ours (INT$b$, , ): -bit quantized state, retained rank , merge interval . The default (Table 6) is INT4 with =1 and =15 (=5 for Mamba2).
GDN, GLA, RetNet. Each of our rows uses fewer average and fewer peak bits than the per-step baseline above it.
| PPL ↓ | Avg. / peak bits | 1K | 2K | 4K | 8K | 1K | 2K | 4K | 8K | 1K | 2K | 4K | 8K |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GDN | GLA | RetNet | |||||||||||
| INT4 (per step) | 4.50 / 4.50 | 36.10 | 36.49 | 48.15 | 79.76 | 30.02 | 32.53 | 41.68 | 66.55 | 59.22 | 75.61 | 179.6 | 1193 |
| Ours (INT3, =1, =7) | 3.88 / 4.25 | 35.21 | 33.50 | 36.71 | 44.46 | 29.16 | 30.07 | 37.44 | 58.26 | 34.67 | 36.37 | 49.09 | 80.71 |
| INT5 (per step) | 5.50 / 5.50 | 32.48 | 28.81 | 29.03 | 32.42 | 26.37 | 25.26 | 29.11 | 39.88 | 32.45 | 31.80 | 44.89 | 159.5 |
| Ours (INT4, =1, =7) | 4.88 / 5.25 | 31.73 | 27.99 | 27.46 | 29.18 | 25.80 | 23.16 | 24.15 | 29.23 | 27.52 | 25.69 | 28.11 | 44.86 |
| INT6 (per step) | 6.50 / 6.50 | 31.54 | 27.38 | 26.40 | 26.77 | 25.43 | 22.58 | 24.06 | 31.11 | 27.53 | 25.10 | 29.24 | 83.98 |
| Ours (INT4, =0, =15) | 5.16 / 5.91 | 31.48 | 27.37 | 26.66 | 27.59 | 25.45 | 22.44 | 22.38 | 26.00 | 27.03 | 24.54 | 28.08 | 80.77 |
| Ours (INT4, =1, =15), default | 5.25 / 6.00 | 31.38 | 27.23 | 26.27 | 26.85 | 25.47 | 22.36 | 22.19 | 24.86 | 26.68 | 23.76 | 24.97 | 28.80 |
| FP32 | 32 | 30.84 | 26.38 | 24.85 | 24.19 | 25.06 | 21.66 | 20.69 | 21.57 | 25.66 | 22.38 | 22.51 | 24.30 |
- vs. INT4: better with fewer bits. With an INT3 base, our method is better on all three models at every length, while using 0.62 fewer average and 0.25 fewer peak bits. The gain therefore cannot come from extra storage.
- vs. INT5: better with fewer bits. With =7, our method is better on all three models at every length.
- vs. INT6: better or on par with 1.25 fewer average bits. The default is better in most cases; INT6 is ahead only on GDN at 8K and GLA at 1K, by at most 0.08.
- vs. LRB without retention (=0): retention adds only 0.09 bits and improves long contexts on every model, most of all RetNet at 8K (80.77 → 28.80). It also wins at 8K at identical peak storage (Table 8; see W1).
- More bits alone do not fix the problem. On RetNet at 8K, per-step quantization reaches 1193 → 159.5 → 83.98 → 67.47 with INT4 → INT5 → INT6 → INT8, versus 28.80 for ours at 5.25 bits. As Sec. 2.3 shows, a finer grid still loses RetNet's tiny updates; preserving them is what matters.
Response to Reviewer RwJP (3/4): Q2 — Mamba2 and other evaluation results
Mamba2.
| PPL ↓ | Avg. / peak bits | 1K | 2K | 4K | 8K |
|---|---|---|---|---|---|
| INT4 (per step) | 4.50 / 4.50 | 32.96 | 31.65 | 33.89 | 37.43 |
| Ours (INT3, =1, =3) | 4.26 / 5.02 | 32.94 | 31.81 | 33.72 | 37.69 |
| Ours (INT3, =1, =5) | 4.64 / 5.78 | 32.03 | 30.64 | 32.20 | 35.00 |
| INT5 (per step) | 5.50 / 5.50 | 30.96 | 28.68 | 29.14 | 30.55 |
| Ours (INT4, =1, =3) | 5.26 / 6.02 | 30.50 | 27.92 | 28.15 | 29.65 |
| Ours (INT4, =1, =5), default | 5.64 / 6.78 | 30.30 | 27.60 | 27.66 | 28.78 |
| INT6 (per step) | 6.50 / 6.50 | 30.13 | 27.48 | 27.41 | 28.14 |
| FP32 | 32 | 29.43 | 26.20 | 25.32 | 25.19 |
-
Our method beats INT5 with fewer average bits (=3).
-
INT6 is slightly better than our default (by 0.12–0.64), with 0.86 more average bits. Mamba2 is where Sec. 2 predicts the smallest benefit: it forgets fastest and has the lowest coherent gain (about 4, versus 18–86 for the other models; Fig. 1), and its small per-head state makes each bypass column about four times as costly.
-
On the public Mamba2 checkpoint, our method is better at every length, with fewer average bits than INT6. We also evaluate the public
mamba2-1.3bcheckpoint [1], applying our default Mamba2 configuration without any tuning. It is trained on different data from ours (the Pile, 300B tokens vs. FineWeb-Edu, 100B tokens).
| Public Mamba2-1.3B [1], PPL ↓ | Avg. / peak bits | 1K | 2K | 4K | 8K |
|---|---|---|---|---|---|
| INT5 (per step) | 5.50 / 5.50 | 42.62 | 103.1 | 1175 | 9431 |
| INT6 (per step) | 6.50 / 6.50 | 27.47 | 44.95 | 208.8 | 1097 |
| Ours (default) | 5.64 / 6.78 | 21.76 | 19.53 | 20.74 | 24.73 |
| FP32 | 32 | 20.33 | 17.45 | 18.47 | 23.20 |
- Per-step quantization degrades sharply with context. From 1K to 8K, INT6 rises from 27.47 to 1097, and INT5 from 42.62 to 9431.
- Our method stays within 2.3 PPL of FP32 at every length, using 0.86 fewer average bits than INT6.
- This matches Sec. 2.3. Under INT4, the public checkpoint's updates are about half as large relative to the grid step (median 0.09 vs. 0.19 at 4K), and more of them are lost in rounding (82% vs. 72%). The more updates rounding loses, the more per-step quantization suffers and the more preserving them helps.
Beyond WikiText-103. We repeat the comparison on PG-19 (8K PPL ↓) and on the recall-intensive tasks of Table 4 (average accuracy ↑). Mamba2 here is our own checkpoint, as in Tables 2–4 of the paper. Ours (INT3) uses the default and with an INT3 base.
| GDN | GLA | Mamba2 | RetNet | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Bits | PG-19 | Recall | Bits | PG-19 | Recall | Bits | PG-19 | Recall | Bits | PG-19 | Recall | |
| INT4 (per step) | 4.50 | 220.4 | 30.8 | 4.50 | 158.8 | 27.7 | 4.50 | 87.96 | 29.0 | 4.50 | 3170 | 21.4 |
| Ours (INT3) | 4.25 | 88.01 | 36.2 | 4.31 | 99.22 | 37.8 | 4.64 | 83.39 | 28.5 | 4.25 | 122.5 | 30.1 |
| INT5 (per step) | 5.50 | 88.19 | 36.2 | 5.50 | 112.5 | 36.9 | 5.50 | 73.50 | 33.0 | 5.50 | 556.6 | 34.5 |
| Ours (default) | 5.25 | 72.28 | 38.5 | 5.31 | 62.24 | 39.7 | 5.64 | 69.72 | 33.1 | 5.25 | 75.96 | 36.0 |
| FP32 | 32 | 66.84 | 39.4 | 32 | 55.85 | 40.7 | 32 | 63.99 | 37.9 | 32 | 60.02 | 40.1 |
Bits: average bits per state element.
- vs. INT4: our INT3 variant is better on PG-19 for all four models, and improves recall by 5.4–10.1 points on GDN, GLA, and RetNet, where it also uses fewer average bits.
- vs. INT5: our default is better on PG-19 for all four models, and improves recall by 1.5–2.8 points on GDN, GLA, and RetNet, where it also uses fewer average bits.
- On our Mamba2 checkpoint, the margin is again the smallest: recall is on par with the per-step baselines. This matches the WikiText-103 results above and the prediction of Sec. 2: Mamba2 forgets fastest, so lost updates have the least time to accumulate, and preserving them helps the least.
In short, the gain comes from preserving updates rather than from extra storage: it remains when our method uses fewer bits, and per-step quantization does not match it by adding bits.
Full results will be added to Tables 2–4 in the paper.
[1] Tri Dao and Albert Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML, 2024. Checkpoints: https://huggingface.co/state-spaces
Response to Reviewer RwJP (1/4): W1 + Q1 — Fixed-m ablation of residual retention
Dear Reviewer RwJP,
Thank you for the careful and constructive review. We have run the fixed-m ablation, together with the memory-matched comparisons and the Eq. 2 analysis you asked for. All new results use the protocol of Tables 7–9 (1.3B models, WikiText-103) unless noted otherwise.
Re:Thank you for this suggestion. We have run the controlled ablation: m is fixed and only k varies.TL;DR.
For clarity, LRB has two parts, and both follow from Sec. 2.3 (an update is lost when it is smaller than half a grid step):
Table A. Fixed-m ablation (m=15; 5 for Mamba2; BDR on; WikiText-103 PPL ↓).
What retention contributes (k=0 → k=1).
How the gain splits. On GDN, GLA, and Mamba2, periodic merging already makes most updates large enough to survive rounding, so it gives most of the gain over INT4; retention then closes a further 17–26% of the remaining gap to FP32 at 8K. On RetNet, periodic merging alone fails at long context, and retention closes 92% of the remaining gap.
Why RetNet benefits most. Retention matters when (i) many updates are still too small after accumulating for m steps, and (ii) the state decays slowly, so each loss persists. RetNet is the extreme case on both:
The benefit of retention thus follows the order predicted by Sec. 2: the smaller the updates and the slower the decay, the more it helps. We will add Table A in the appendix and state this split explicitly in the paper.