- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes State-Dependent Gating (SDG), which predicts element-wise memory decay from the change in recurrent state across a chunk and applies it at chunk boundaries. This introduces memory feedback while preserving the backbone’s within-chunk parallel computation. The authors provide fused implementations for four recurrent architectures, a proximal interpretation, and an idealized reconstruction-risk analysis. Experiments with 1.3B models trained on 100B tokens show consistent improvements in WikiText perplexity and average real-world retrieval, although reasoning and broader long-context results are mixed.
Critical strengths
-
A practical design for introducing memory-dependent gating. Applying feedback only at chunk boundaries preserves the existing within-chunk factorization and confines the additional dependency to the recurrent state scan.
-
Evaluation with retrieval improvements. The paired experiments cover four recurrent backbones and three hybrid architectures under matched training recipes.
Critical weaknesses
-
The framing would be better supported by one further comparison. The Ω control replaces ΔS by the chunk contribution computed from a zero incoming state, so it does not read the incoming memory, yet it lands within 0.67 points on three of four metrics (NIAH@4K 55.00 vs 55.67; real-world retrieval 31.89 vs 31.91; LongBench 11.13 vs 10.95) with no intervals reported. So part of the measured gain may come from chunk-level, content-driven decay rather than from the state feedback that motivates the title. A matched static-decay control for Mamba-3, and one sentence in the abstract naming which ingredient the evidence supports, would let readers place the contribution correctly.
-
As The training throughput of SDG decreases by about 11–15%, and on Mamba-3, it decreases by about 25–28%. These costs are relatively significant compared to the average general-purpose metric benefit of less than 1 pp. But relative to the specific retrieval benefit of about 10–14 pp, it may be worthwhile, depending on the application goals. I am concerned about the trade-off between the benefits and costs of generalizing diversed tasks. Additionally, the quality–efficiency trade-off is suggested to have further discussion. SDG improves performance while introducing additional training and inference overhead. It would strengthen the paper to establish whether these gains offer a better trade-off than alternative approaches, such as DeltaProduct [1] and Sparse Delta Memory [2], or other relevant baselines the authors consider appropriate which allocating additional computation and gaining performance improvement. Comparisons under matched training-compute budgets, together with quality versus measured inference latency and memory usage, would help demonstrate when SDG is the preferable choice.
[1] Siems et al. DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products. arXiv:2502.10297.
[2] Cabannes et al. Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity. arXiv:2607.07386.
Questions for authors
Considering total training-compute budgets, how does SDG perform in the quality–efficiency trade-off?
Could the authors include other appropriate competing methods to clarify whether SDG uses the additional computation more effectively?
4 comments
Rebuttal by Authors (2/2)
Question 1. Matched training compute
To test whether SDG's lower loss comes from spending more computation, we compare GDN and SDG at the same cumulative training FLOPs, including gate operations. Each loss averages the last 100 updates, covering 26,214,400 tokens.
| Size | Method | 1.50e18 | 1.88e18 | 2.25e18 | 2.63e18 | 3.00e18 |
|---|---|---|---|---|---|---|
| 60M | GDN | 2.900 | 2.888 | 2.842 | 2.821 | 2.794 |
| 60M | SDG | 2.897 | 2.875 | 2.839 | 2.815 | 2.789 |
| 130M | GDN | 2.903 | 2.862 | 2.835 | 2.806 | 2.787 |
| 130M | SDG | 2.896 | 2.856 | 2.829 | 2.797 | 2.778 |
| 340M | GDN | 2.905 | 2.846 | 2.795 | 2.783 | 2.759 |
| 340M | SDG | 2.901 | 2.841 | 2.788 | 2.773 | 2.750 |
SDG achieves lower training loss at every reported budget across all three model sizes. The gains persist after accounting for the cost of gating.
Question 2. Other competing methods
To compare alternative uses of training computation, we evaluate 340M GDN, GDN+SDG, Gated DeltaProduct, and SDM. GDN is trained on 15B tokens, and the other three models are trained to match its total training FLOPs, including SDG's gating cost. Their training token counts are adjusted to match this budget.
Gated DeltaProduct applies two state transformations per token, while pure SDM uses sparse reads and writes to a larger memory. All four configurations are fully recurrent, and our pure SDM configuration differs from its main hybrid configuration in their paper for fair comparisons.
We report WikiText PPL, NIAH@4K, Recall, and LongBench, using the same evaluation protocols as the other rebuttal comparisons.
| Model | WikiText PPL↓ | NIAH@4K↑ | Recall↑ | LongBench↑ |
|---|---|---|---|---|
| GDN | 26.98 | 27.27 | 20.70 | 7.53 |
| GDN+SDG | 25.99 | 35.80 | 22.34 | 8.50 |
| Gated DeltaProduct | 26.42 | 32.60 | 21.58 | 8.12 |
| Pure SDM | 27.14 | 35.67 | 21.13 | 8.04 |
At matched training FLOPs, SDG achieves lower WikiText PPL and higher NIAH@4K, Recall, and LongBench scores than GDN and Gated DeltaProduct. Its NIAH@4K score is close to pure SDM (35.80 versus 35.67), while WikiText PPL, Recall, and LongBench favor SDG. The inference measurements below show the corresponding latency and memory costs.
Inference cost. To assess inference cost, we benchmark latency and memory for the same four 340M model configurations. We use a 4,096-token prompt, batch 1, and 255 decode steps. Each model is benchmarked on a single A100.
| 340M model | TTFT (ms)↓ | TPOT (ms/token)↓ | Cache (MiB/sequence)↓ | Peak allocated (GB)↓ |
|---|---|---|---|---|
| GDN | 50.203 | 24.377 | 12.563 | 0.967 |
| SDG | 54.863 | 25.078 | 24.568 | 0.992 |
| Gated DeltaProduct | 60.568 | 28.872 | 12.938 | 1.010 |
| Pure SDM | 114.029 | 14.209 | 3072.000 | 4.159 |
Compared with Gated DeltaProduct, SDG reduces TTFT by 9.4% and TPOT by 13.1%, while its cache increases from 12.938 to 24.568 MiB per sequence. Pure SDM offers faster decoding, but has higher TTFT and requires 3 GiB of cache per sequence. SDG remains slightly slower than GDN in both TTFT and TPOT.
We want to emphasize that SDG is broadly compatible with existing linear attention methods, including Gated DeltaProduct and SDM. Its boundary gating can be combined with their state-update and memory-access mechanisms. We therefore view SDG as a complementary extension to these methods rather than a replacement.
Dear Reviewer 1ENE,
Thank you again for the time and care you have devoted to reviewing Paper #6796. Your thoughtful feedback has helped us improve the paper. We wanted to follow up on our rebuttal and ask whether you have had a chance to read it.
We have also submitted a revised manuscript incorporating our response. The text and tables we added in response to your review are marked in red.
-
State feedback, chunk content, and static decay (Weakness 1). We revised the abstract to describe state change as an effective and computationally efficient signal for adaptive boundary decay. Section 5.3 acknowledges that the original 1.3B chunk-only control Ω recovers most of the gain and scores slightly higher on LongBench. In the new matched-compute 340M comparison, ΔS exceeds Ω by 1.47 points on NIAH@4K, 0.50 on real-world retrieval, and 0.47 on LongBench, with 31.6% higher training throughput (Appendix K.2, Table 21). We also added the learned static Mamba-3 gate you suggested. Real-world retrieval is 23.14 for the baseline, 24.01 with static decay, and 25.17 with adaptive SDG. The static gate processes 33,926 training tokens/s versus 27,603 for adaptive SDG (Appendix K.2, Table 22).
-
Quality at a fixed training budget (Weakness 2 and Question 1). Section 5.4 and Appendix K.1 (Table 19) compare GDN and GDN+SDG at five cumulative training FLOPs budgets, including gate operations. SDG achieves lower training loss at all five budgets for each of the 60M, 130M, and 340M models. At 340M and the largest budget, the loss is 2.750 with SDG and 2.759 for GDN.
-
Alternative uses of compute (Question 2). Section 5.4 and Appendix K.4 (Table 25) compare four fully recurrent 340M models at the training FLOPs of a GDN trained on 15B tokens. GDN+SDG improves all four quality metrics over Gated DeltaProduct and reduces time per output token from 28.872 to 25.078 ms, while increasing cache per sequence from 12.938 to 24.568 MiB. Pure Sparse Delta Memory decodes faster (14.209 ms/token) and nearly matches SDG on NIAH@4K (35.67 versus 35.80), using 3 GiB of cache per sequence. We report quality, latency, and memory for the same four models.
We would welcome your thoughts on whether our response addresses your questions about what drives the gains and how they compare with the added cost. If any part of the rebuttal or the comparisons in the revised paper remains unclear, we would be happy to clarify it here.
Best regards,
The authors of Paper #6796
Thank you for your detailed rebuttal and the extensive additional experiments. I appreciate the substantial effort you have made to address my concerns and strengthen the paper.
While the new results have addressed several of my concerns and provided valuable supporting evidence, some questions remain only partially resolved. I will leave these points for further discussion with the AC and other reviewers during the next stage.
Thank you again for your thoughtful responses and constructive engagement throughout the discussion.
Rebuttal by Authors (1/2)
We are grateful for the reviewer's insightful feedback and constructive comments. We address each point in detail below.
Weakness 1. Ω and static control
We consider both accuracy and efficiency when comparing ΔS with Ω. Although Ω recovers much of the gain in the original ablation, forming it requires an additional matrix product, whereas ΔS is the difference of two available states. Following the helpful suggestion from Reviewer jWG3, we evaluate mean-pooled token gating as a cheaper content-only alternative. It averages normalized token representations within each 64-token chunk and uses a low-rank projection with Mamba-1-style decay to generate the gate.
The additional comparisons use 340M models, each matched to the total training FLOPs of a 340M GDN trained on 15B tokens.
Training throughput is measured on a single A100 using 4,096-token sequences and a batch size of 4.
SDG improves NIAH@4K, Recall, and LongBench over Ω by 1.47, 0.50, and 0.47 points, with 31.6% higher training throughput. Mean pooling is faster, but scores below SDG on all three metrics.
Separately, we follow the reviewer's suggestion to add a learned static gate to Mamba-3. This tests whether learned boundary decay alone is sufficient, or whether retention benefits from adapting to each input. The control preserves boundary placement and element-wise decay, but learns one input-independent retention coefficient per state element and uses it for every chunk. All three variants use the same 340M scale and training FLOPs budget described above. Training throughput is measured on a single A100 using 4,096-token sequences and a batch size of 4.
SDG achieves higher Recall than the learned static gate, supporting the benefit of adapting retention to the input.
Following the reviewer's suggestion, we will revise the abstract to reflect the ablation evidence: "Our ablations support adaptive element-wise decay at chunk boundaries, with state changes providing an effective and computationally efficient gating signal."
Weakness 2. Benefits versus overhead
To assess whether SDG's additional computation is worthwhile, we evaluate training efficiency and inference cost separately. At matched training FLOPs, SDG achieves lower training loss than GDN across three model sizes. In Question 2, we compares the four 340M models on WikiText PPL, NIAH@4K, Recall, and LongBench.
Against Gated DeltaProduct, SDG improves all four quality metrics and reduces both TTFT and TPOT, at the cost of a larger cache. Pure SDM decodes faster and has similar NIAH@4K accuracy, but SDG has better WikiText PPL, Recall, and LongBench scores and uses 24.57 MiB of cache per sequence instead of 3 GiB. These comparisons show the benefits and costs of SDG relative to each alternative.