acceptodds
Under review as a conference paper at ICLR 2027

SDG: State-Dependent Gating at Chunk Boundaries for Linear Attention

Abstract

Recent gated linear attention models use state-independent gates, computed from token representations without directly reading the recurrent memory. This design supports efficient chunkwise computation: sequences are divided into chunks of consecutive tokens, with parallel computation within each chunk. However, the same token representation produces the same gate, even when the layer stores different information. We introduce **S**tate-**D**ependent **G**ating (SDG), which computes element-wise gates from the change in memory over a chunk and applies them at the chunk boundary. This provides state feedback for forgetting while preserving parallel computation within each chunk. We develop fused kernels for Gated DeltaNet, Kimi Delta Attention, Gated DeltaNet-2, and Mamba-3. Our theoretical analysis quantifies the potential reconstruction benefits of state-dependent, element-wise gating under a signal–noise model. We also show that boundary decay is an element-wise ridge proximal step when the gate is held fixed. Empirical evaluation of 1.3B linear attention and hybrid models trained on 100B tokens shows improvements across language modeling, commonsense reasoning, synthetic and real-world retrieval, and long-context understanding. These results demonstrate that SDG is an effective approach to enhancing existing linear attention models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
linear attention, state-space model, linear transformer, subquadractic model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1ENEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes State-Dependent Gating (SDG), which predicts element-wise memory decay from the change in recurrent state across a chunk and applies it at chunk boundaries. This introduces memory feedback while preserving the backbone’s within-chunk parallel computation. The authors provide fused implementations for four recurrent architectures, a proximal interpretation, and an idealized reconstruction-risk analysis. Experiments with 1.3B models trained on 100B tokens show consistent improvements in WikiText perplexity and average real-world retrieval, although reasoning and broader long-context results are mixed.

Critical strengths
  1. A practical design for introducing memory-dependent gating. Applying feedback only at chunk boundaries preserves the existing within-chunk factorization and confines the additional dependency to the recurrent state scan.

  2. Evaluation with retrieval improvements. The paired experiments cover four recurrent backbones and three hybrid architectures under matched training recipes.

Critical weaknesses
  1. The framing would be better supported by one further comparison. The Ω control replaces ΔS by the chunk contribution computed from a zero incoming state, so it does not read the incoming memory, yet it lands within 0.67 points on three of four metrics (NIAH@4K 55.00 vs 55.67; real-world retrieval 31.89 vs 31.91; LongBench 11.13 vs 10.95) with no intervals reported. So part of the measured gain may come from chunk-level, content-driven decay rather than from the state feedback that motivates the title. A matched static-decay control for Mamba-3, and one sentence in the abstract naming which ingredient the evidence supports, would let readers place the contribution correctly.

  2. As The training throughput of SDG decreases by about 11–15%, and on Mamba-3, it decreases by about 25–28%. These costs are relatively significant compared to the average general-purpose metric benefit of less than 1 pp. But relative to the specific retrieval benefit of about 10–14 pp, it may be worthwhile, depending on the application goals. I am concerned about the trade-off between the benefits and costs of generalizing diversed tasks. Additionally, the quality–efficiency trade-off is suggested to have further discussion. SDG improves performance while introducing additional training and inference overhead. It would strengthen the paper to establish whether these gains offer a better trade-off than alternative approaches, such as DeltaProduct [1] and Sparse Delta Memory [2], or other relevant baselines the authors consider appropriate which allocating additional computation and gaining performance improvement. Comparisons under matched training-compute budgets, together with quality versus measured inference latency and memory usage, would help demonstrate when SDG is the preferable choice.

[1] Siems et al. DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products. arXiv:2502.10297.

[2] Cabannes et al. Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity. arXiv:2607.07386.

Questions for authors

Considering total training-compute budgets, how does SDG perform in the quality–efficiency trade-off?

Could the authors include other appropriate competing methods to clarify whether SDG uses the additional computation more effectively?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We are grateful for the reviewer's insightful feedback and constructive comments. We address each point in detail below.

Weakness 1. Ω and static control

We consider both accuracy and efficiency when comparing ΔS with Ω. Although Ω recovers much of the gain in the original ablation, forming it requires an additional matrix product, whereas ΔS is the difference of two available states. Following the helpful suggestion from Reviewer jWG3, we evaluate mean-pooled token gating as a cheaper content-only alternative. It averages normalized token representations within each 64-token chunk and uses a low-rank projection with Mamba-1-style decay to generate the gate.

The additional comparisons use 340M models, each matched to the total training FLOPs of a 340M GDN trained on 15B tokens.

Training throughput is measured on a single A100 using 4,096-token sequences and a batch size of 4.

Gate inputNIAH@4K↑Recall↑LongBench↑Training throughput (tokens/s)↑
Mean pooling + Mamba-1-style gate30.8720.707.7840,180
Ω34.3321.848.0328,753
ΔS35.8022.348.5037,835

SDG improves NIAH@4K, Recall, and LongBench over Ω by 1.47, 0.50, and 0.47 points, with 31.6% higher training throughput. Mean pooling is faster, but scores below SDG on all three metrics.

Separately, we follow the reviewer's suggestion to add a learned static gate to Mamba-3. This tests whether learned boundary decay alone is sufficient, or whether retention benefits from adapting to each input. The control preserves boundary placement and element-wise decay, but learns one input-independent retention coefficient per state element and uses it for every chunk. All three variants use the same 340M scale and training FLOPs budget described above. Training throughput is measured on a single A100 using 4,096-token sequences and a batch size of 4.

Mamba-3 variantWikiText PPL↓NIAH@4K↑Recall↑Training throughput (tokens/s)↑
Baseline25.1336.4723.1434,517
Learned static gate24.9737.6024.0133,926
SDG24.7939.6725.1727,603

SDG achieves higher Recall than the learned static gate, supporting the benefit of adapting retention to the input.

Following the reviewer's suggestion, we will revise the abstract to reflect the ablation evidence: "Our ablations support adaptive element-wise decay at chunk boundaries, with state changes providing an effective and computationally efficient gating signal."

Weakness 2. Benefits versus overhead

To assess whether SDG's additional computation is worthwhile, we evaluate training efficiency and inference cost separately. At matched training FLOPs, SDG achieves lower training loss than GDN across three model sizes. In Question 2, we compares the four 340M models on WikiText PPL, NIAH@4K, Recall, and LongBench.

Against Gated DeltaProduct, SDG improves all four quality metrics and reduces both TTFT and TPOT, at the cost of a larger cache. Pure SDM decodes faster and has similar NIAH@4K accuracy, but SDG has better WikiText PPL, Recall, and LongBench scores and uses 24.57 MiB of cache per sequence instead of 3 GiB. These comparisons show the benefits and costs of SDG relative to each alternative.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Question 1. Matched training compute

To test whether SDG's lower loss comes from spending more computation, we compare GDN and SDG at the same cumulative training FLOPs, including gate operations. Each loss averages the last 100 updates, covering 26,214,400 tokens.

SizeMethod1.50e181.88e182.25e182.63e183.00e18
60MGDN2.9002.8882.8422.8212.794
60MSDG2.8972.8752.8392.8152.789
130MGDN2.9032.8622.8352.8062.787
130MSDG2.8962.8562.8292.7972.778
340MGDN2.9052.8462.7952.7832.759
340MSDG2.9012.8412.7882.7732.750

SDG achieves lower training loss at every reported budget across all three model sizes. The gains persist after accounting for the cost of gating.

Question 2. Other competing methods

To compare alternative uses of training computation, we evaluate 340M GDN, GDN+SDG, Gated DeltaProduct, and SDM. GDN is trained on 15B tokens, and the other three models are trained to match its total training FLOPs, including SDG's gating cost. Their training token counts are adjusted to match this budget.

Gated DeltaProduct applies two state transformations per token, while pure SDM uses sparse reads and writes to a larger memory. All four configurations are fully recurrent, and our pure SDM configuration differs from its main hybrid configuration in their paper for fair comparisons.

We report WikiText PPL, NIAH@4K, Recall, and LongBench, using the same evaluation protocols as the other rebuttal comparisons.

ModelWikiText PPL↓NIAH@4K↑Recall↑LongBench↑
GDN26.9827.2720.707.53
GDN+SDG25.9935.8022.348.50
Gated DeltaProduct26.4232.6021.588.12
Pure SDM27.1435.6721.138.04

At matched training FLOPs, SDG achieves lower WikiText PPL and higher NIAH@4K, Recall, and LongBench scores than GDN and Gated DeltaProduct. Its NIAH@4K score is close to pure SDM (35.80 versus 35.67), while WikiText PPL, Recall, and LongBench favor SDG. The inference measurements below show the corresponding latency and memory costs.

Inference cost. To assess inference cost, we benchmark latency and memory for the same four 340M model configurations. We use a 4,096-token prompt, batch 1, and 255 decode steps. Each model is benchmarked on a single A100.

340M modelTTFT (ms)↓TPOT (ms/token)↓Cache (MiB/sequence)↓Peak allocated (GB)↓
GDN50.20324.37712.5630.967
SDG54.86325.07824.5680.992
Gated DeltaProduct60.56828.87212.9381.010
Pure SDM114.02914.2093072.0004.159

Compared with Gated DeltaProduct, SDG reduces TTFT by 9.4% and TPOT by 13.1%, while its cache increases from 12.938 to 24.568 MiB per sequence. Pure SDM offers faster decoding, but has higher TTFT and requires 3 GiB of cache per sequence. SDG remains slightly slower than GDN in both TTFT and TPOT.

We want to emphasize that SDG is broadly compatible with existing linear attention methods, including Gated DeltaProduct and SDM. Its boundary gating can be combined with their state-update and memory-access mechanisms. We therefore view SDG as a complementary extension to these methods rather than a replacement.

Dear Reviewer 1ENE,

Thank you again for the time and care you have devoted to reviewing Paper #6796. Your thoughtful feedback has helped us improve the paper. We wanted to follow up on our rebuttal and ask whether you have had a chance to read it.

We have also submitted a revised manuscript incorporating our response. The text and tables we added in response to your review are marked in red.

  1. State feedback, chunk content, and static decay (Weakness 1). We revised the abstract to describe state change as an effective and computationally efficient signal for adaptive boundary decay. Section 5.3 acknowledges that the original 1.3B chunk-only control Ω recovers most of the gain and scores slightly higher on LongBench. In the new matched-compute 340M comparison, ΔS exceeds Ω by 1.47 points on NIAH@4K, 0.50 on real-world retrieval, and 0.47 on LongBench, with 31.6% higher training throughput (Appendix K.2, Table 21). We also added the learned static Mamba-3 gate you suggested. Real-world retrieval is 23.14 for the baseline, 24.01 with static decay, and 25.17 with adaptive SDG. The static gate processes 33,926 training tokens/s versus 27,603 for adaptive SDG (Appendix K.2, Table 22).

  2. Quality at a fixed training budget (Weakness 2 and Question 1). Section 5.4 and Appendix K.1 (Table 19) compare GDN and GDN+SDG at five cumulative training FLOPs budgets, including gate operations. SDG achieves lower training loss at all five budgets for each of the 60M, 130M, and 340M models. At 340M and the largest budget, the loss is 2.750 with SDG and 2.759 for GDN.

  3. Alternative uses of compute (Question 2). Section 5.4 and Appendix K.4 (Table 25) compare four fully recurrent 340M models at the training FLOPs of a GDN trained on 15B tokens. GDN+SDG improves all four quality metrics over Gated DeltaProduct and reduces time per output token from 28.872 to 25.078 ms, while increasing cache per sequence from 12.938 to 24.568 MiB. Pure Sparse Delta Memory decodes faster (14.209 ms/token) and nearly matches SDG on NIAH@4K (35.67 versus 35.80), using 3 GiB of cache per sequence. We report quality, latency, and memory for the same four models.

We would welcome your thoughts on whether our response addresses your questions about what drives the gains and how they compare with the added cost. If any part of the rebuttal or the comparisons in the revised paper remains unclear, we would be happy to clarify it here.

Best regards,

The authors of Paper #6796

Thank you for your detailed rebuttal and the extensive additional experiments. I appreciate the substantial effort you have made to address my concerns and strengthen the paper.

While the new results have addressed several of my concerns and provided valuable supporting evidence, some questions remain only partially resolved. I will leave these points for further discussion with the AC and other reviewers during the next stage.

Thank you again for your thoughtful responses and constructive engagement throughout the discussion.

Reviewer_jWG3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes State-Dependent Gating (SDG), an add-on for chunkwise linear attention that applies an extra element-wise decay to the recurrent state only at chunk boundaries, every 64 tokens. The gate is computed from the change in state across the chunk. That change passes through a learned key-channel projection, an element-wise gain and a squareplus-based rate, so the current chunk’s outputs and each backbone’s within-chunk parallel algorithm stay untouched. The authors build fused Triton kernels for Gated DeltaNet, Kimi Delta Attention, Gated DeltaNet-2 and Mamba-3, and add two analyses. First, with the gate held fixed, boundary decay is an element-wise ridge proximal step. Second, under a signal-plus-noise model, the excess risk of fixed scalar, per-key and element-wise gates relative to an adaptive oracle splits into three nonnegative terms. With 1.3B-parameter models trained on 100B FineWeb-Edu tokens, SDG lowers WikiText perplexity by 0.10 to 0.58 and raises average real-world retrieval by 0.39 to 0.83 points on all four backbones. Commonsense, RULER and LongBench results are mixed. The retrieval gains carry over to three sliding-window hybrids. Ablations on GDN vary the gate input, sharing and projection. The efficiency section reports 85 to 89% of baseline training throughput for the delta-rule models and about 72 to 75% for Mamba-3.

Critical strengths
  1. Boundary placement is a near idea in my opinion. Per-token state feedback or fully element-wise decay would break the shared WY/UT factorization that makes GDN and KDA fast, and gating only between chunks leaves each backbone’s within-chunk algorithm alone. The authors follow through with fused kernels for four backbones (including Mamba-3, which has no delta rule), a segmentation-invariance result that keeps training, prefill and cached decoding consistent, and a zero-initialized gain so each SDG model starts out essentially at its baseline. That makes the method easy to drop into existing linear attention stacks.

  2. The empirical study is comprehensive. Paired runs share data order and initialization across four recurrent backbones and three hybrids, and WikiText perplexity and average real-world retrieval improve in all four recurrent pairs. The paper also reports the cases that go the other way (BoolQ in every recurrent pair, KDA on S-NIAH-3 at 4K, GDN-2 on LongBench) instead of hiding them. The single-boundary intervention, gate-distribution statistics and boundary-offset test are diagnostics most papers in this area skip.

Critical weaknesses
  1. The effect sizes are small, and the evidence that they exceed run-to-run variation is weak. No result reports seed variance, and the fixed boundary decay control in Table 4 (a constant 0.9999 per boundary, about 0.994 cumulative over a 4K sequence, so nearly the identity) trails the GDN baseline by 6.7 points on NIAH@4K and 2.4 on recall, more than SDG’s own gains of 6.1 and 0.63. The gate statistics add to the doubt, since in the three delta-rule models the median gate stays at its 0.9999 initialization and only about 0.85% of observed KDA gate cells fall below 0.99, yet KDA+SDG loses 12 points on S-NIAH-3 at 4K. Nothing is compute-matched either, even though SDG gives up roughly 11 to 15% of training throughput, or about a quarter for Mamba-3, where it also adds 2.4% more parameters.

  2. The idea that reading the state is what helps isn’t supported by the paper’s own ablation. The Ω input, which by construction excludes the layer’s incoming memory, lands within 0.7 points of ΔS on every metric and beats it on LongBench, while feeding the incoming state directly does worse than ΔS on all four. That points to element-wise, chunk-level decay predicted from the chunk’s own writes, with ΔS preferred mainly because it is cheaper to form than Ω. The theory doesn’t close this gap, since Remark 1 holds for any multiplicative decay (native GDN and KDA gates included) and Corollary 1 compares only against constant gates, so neither addresses state-dependent versus token-dependent gating, as the authors acknowledge.

Questions for authors
  1. A constant 0.9999 per 64-token boundary compounds to about 0.994 over 4K tokens, yet the fixed-decay control trails the GDN baseline by 6.7 points on NIAH@4K and 2.4 on recall. Is anything else different between these runs (kernel path, numerics, nondeterminism)? Can you report seed variance for the baseline and SDG, even at a smaller scale?

  2. Ω matches ΔS within noise. What evidence shows that within-layer state feedback, rather than element-wise chunk-level decay driven by chunk content, is responsible for the gains? A cheaper chunk-only control, such as a gate predicted from pooled token representations, would help settle this.

  3. What happens on RULER and real-world retrieval if you switch off every boundary update at inference in a trained SDG model? The single-boundary skip barely moves predictions, so I’d like to know whether the gate matters at test time or mainly reshapes the training trajectory.

  4. Can you compare against baselines matched on training compute or wall-clock, and against a parameter-matched Mamba-3? Separately, Table 8 shows Mamba-3+SDG decoding faster than its baseline at every length (15.94 vs 17.99 ms at 512 tokens). This conflicts with Section 5.4’s statement that Mamba-3 incurs decoding overhead. Do the two use the same decode path?

  5. The boundary interval is tied to the kernel chunk size C = 64. Have you tried decoupling them, for example by gating every second or fourth chunk? Could repeated boundary decay beyond the training length explain the S-NIAH-1 drops at 8K for GDN (84.6 to 78.6), Mamba-3 (46.6 to 39.6) and GDN-2 (99.8 to 94.8)?

I'd be happy to change my score if these concerns are comprehensively resolved.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We are grateful for the reviewer's insightful feedback and constructive comments. We address each point in detail below.

Weakness 1. Reliability and compute fairness

To assess run-to-run variation, we compares GDN and GDN+SDG across three paired seeds in Question 1. SDG improves WikiText PPL, NIAH@4K, Recall, and LongBench in every pair.

For fixed decay, Question 1 clarifies the kernel path, numerical settings, and nondeterminism. A median near initialization does not imply uniform retention: 7.34% of GDN+SDG gate cells are numerically equal to 1 in FP32, preserving the corresponding state components at that boundary. In Question 3, disabling the gates in the trained 1.3B GDN reduces NIAH@4K from 55.67 to 16.27 and Recall from 31.91 to 20.13. The learned gates therefore matter substantially to the trained model despite their near-identity median.

To account for the additional computation and parameters, Question 4 compares training loss at matched FLOPs and adds a parameter-matched Mamba-3 baseline. SDG achieves lower loss at every budget across three model sizes and outperforms the parameter-matched baseline on all four metrics.

Weakness 2. State feedback versus content

To test what state feedback adds to content-driven decay, Question 2 compares Ω and ΔS with the same gate parameterization and adds the pooled-token control suggested by the reviewer. ΔS outperforms both controls on all three quality metrics, gaining 1.47 points on NIAH@4K, 0.50 on Recall, and 0.47 on LongBench over Ω. These gains support a contribution from memory feedback beyond chunk content. ΔS also provides 31.6% higher training throughput than Ω.

Remark 1 explains the effect of multiplicative decay, and Corollary 1 compares restricted gate families. The evidence for state feedback comes from the input ablation above.

Question 1. Fixed decay and seed variance

The fixed-decay kernel path retains GDN's within-chunk computation and native token gates, adding a 0.9999 state multiplication at each boundary. For numerics, it uses the same training precision as the baseline. The decay involves no random sampling, although GPU nondeterminism may still affect training. The three-seed comparison below shows how GDN and GDN+SDG vary across runs.

In GDN+SDG, 7.34% of gate cells are numerically equal to 1 in FP32, preserving the corresponding state components at that boundary. Our paper also reports that about 4.29% of gate cells fall below 0.99. This allows the model to retain some components, potentially carrying needle information, while reducing others. A fixed 0.9999 gate decays every component equally. The median alone does not show this difference.

To assess seed variance, we compare GDN and GDN+SDG using three seeds. All models are 340M. For each seed, GDN is trained on 15B tokens, and GDN+SDG matches its total training FLOPs, including gate operations. Within each seed, the two models share backbone initialization, data order, and optimizer settings. We report each run separately below.

SeedModelWikiText PPL↓NIAH@4K↑Recall↑LongBench↑
0GDN26.9827.2720.707.53
0GDN+SDG25.9935.8022.348.50
1GDN26.6930.6720.787.43
1GDN+SDG26.1333.3321.998.13
2GDN26.7830.4720.117.94
2GDN+SDG25.7139.5322.858.37

Across the three seeds, GDN+SDG achieves higher average retrieval accuracy and lower average WikiText PPL than GDN.

Question 2. Pooled-token control

We thank the reviewer for the insightful suggestion to include a pooled-token control, which helps assess whether chunk content alone can account for the gains. We compare it with Ω and ΔS using 340M models, each trained to match the total training FLOPs of a 340M GDN trained on 15B tokens. Ω retains the SDG gate and changes only its input. The pooled control uses normalized token inputs, mean pooling over B=64, and Mamba-1-style decay:

xˉn=B−1∑t∈nxt,δn,h=softplus⁡(Wup,hWdownxˉn+bh),Λn,h,ij=exp⁡[−δn,h,iexp⁡(ah,ij)]\bar{x}_n=B^{-1}\sum_{t\in n}x_t,\qquad\delta_{n,h}=\operatorname{softplus}(W_{\mathrm{up},h}W_{\mathrm{down}}\bar{x}_n+b_h),\qquad\Lambda_{n,h,ij}=\exp[-\delta_{n,h,i}\exp(a_{h,ij})]

Training throughput is measured on a single A100 using sequence length of 4096 and a batch size of 4.

Gate inputNIAH@4K↑Recall↑LongBench↑Training throughput (tokens/s)↑
Mean pooling30.8720.707.7840,180
Ω34.3321.848.0328,753
ΔS35.8022.348.5037,835

With the same gate parameterization, ΔS improves NIAH@4K by 1.47 points, recall by 0.50 points, and LongBench by 0.47 points over Ω. It also outperforms pooled-token gating on all three metrics. These comparisons support a contribution from memory feedback beyond the current chunk's content.

SDG also has a substantial computational advantage over Ω. The training throughput has improved by 31.6%.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Question 3. Disabling boundary gates

We follow the reviewer's suggestion to disable every boundary gate during prefill and decoding, retaining native token gates. We evaluated the 1.3B models on benchmarks.

BackboneSDG gatesWikiText PPL↓NIAH@4K↑Recall↑LongBench↑
GDNOn15.9955.6731.9110.95
GDNOff17.1016.2720.136.64

The results show that disabling the gates sharply degrades both language modeling and retrieval performance. This indicates that SDG gating shapes the training trajectory and is an integral part of the trained model. Removing it creates a severe mismatch between training and inference.

Question 4. Compute matching and decoding

To account for SDG's extra computation, we include the gate operations in the training FLOPs and compare both models at the same budgets. The table reports training loss at three model sizes, averaged over the last 100 updates.

SizeMethod1.50e181.88e182.25e182.63e183.00e18
60MGDN2.9002.8882.8422.8212.794
60MSDG2.8972.8752.8392.8152.789
130MGDN2.9032.8622.8352.8062.787
130MSDG2.8962.8562.8292.7972.778
340MGDN2.9052.8462.7952.7832.759
340MSDG2.9012.8412.7882.7732.750

SDG achieves lower training loss at every reported budget across all three model sizes. The gains therefore persist after accounting for the cost of gating.

We also widen the Mamba-3 baseline's FFN to match SDG's parameter count. All three variants are 340M and match the total training FLOPs of a 340M GDN trained on 15B tokens.

Mamba-3 variantWikiText PPL↓NIAH@4K↑Recall↑LongBench↑
Baseline25.1336.4723.149.70
Parameter-matched baseline25.0839.0022.7910.13
SDG24.7939.6725.1710.96

SDG outperforms the parameter-matched baseline on all four metrics. These results indicate that the additional parameters alone do not account for the gains.

We thank the reviewer for pointing out the discrepancy between Table 8 and Section 5.4. The original benchmark inadvertently used different decoding kernels for Mamba-3 and Mamba-3+SDG, producing the apparent speedup. We repeated the measurements on a single A100 with both variants using the same decoding kernel and preprocessing.

BatchPrompt tokensMamba3 TTFT (ms)+SDG TTFT (ms)Mamba3 TPOT (ms/token)+SDG TPOT (ms/token)
151218.29222.92916.73917.255
1409690.154109.76017.02117.663
132768673.667815.35316.92617.684
451238.66448.69417.58718.030
44096306.497381.86717.84918.428
4327682532.2323137.77817.80218.426

With the kernels aligned, SDG incurs a decoding overhead in all six settings, consistent with Section 5.4. We will correct Table 8 and update the efficiency discussion with these results.

Question 5. Gate interval and extrapolation

Following the reviewer's suggestion, we vary B at C=64, computing ΔS over B tokens, and additionally evaluate C=B=32. All models are 340M. The GDN baseline (B=None) is trained on 15B tokens, and all gated variants match its total training FLOPs. B=None denotes no boundary gating.

CBWikiText PPL↓NIAH@4K↑Recall↑S-NIAH-1@8K↑
323225.8640.2024.1372.80
646425.9935.8022.3478.40
6412826.3730.7321.7082.20
6425626.5330.2721.8483.80
64None26.9827.2720.7086.60

C=B=32 gives the lowest WikiText PPL and the highest NIAH@4K and recall scores. At C=64, increasing B from 64 to 256 raises S-NIAH-1@8K from 78.40 to 83.80, while WikiText PPL rises from 25.99 to 26.53 and NIAH@4K falls from 35.80 to 30.27. B=None reaches 86.60 at 8K but has the highest PPL and lowest NIAH@4K. These results indicate that more frequent boundary gating may impair the model’s ability to extrapolate to longer sequences on some tasks such as S-NIAH-1.

Dear Reviewer jWG3,

Thank you again for the time and care you have devoted to reviewing Paper #6796. Your thoughtful feedback has helped us improve the paper. We wanted to follow up on our rebuttal and ask whether you have had a chance to read it.

We have also submitted a revised manuscript incorporating our response. The text and tables we added in response to your review are marked in red.

  1. Fixed decay and seed variation (Question 1). Appendix E.3 clarifies that the fixed 0.9999 control uses the baseline’s precision, within-chunk computation, and native token gates. The boundary multiplication introduces no sampling, although GPU training can still be nondeterministic. For learned SDG, 7.34% of GDN gate coefficients equal one in FP32 and 4.29% fall below 0.99. Appendix K.1 (Table 20) reports results for three paired seeds, comparing 340M GDN with GDN+SDG. All four metrics improve in each pair. The seed variance of the 1.3B fixed-decay control remains unmeasured.

  2. Pooled-token control and attribution (Question 2). Section 5.3 and Appendix K.2 (Table 21) add your proposed control. For 340M models at matched total training FLOPs, NIAH@4K is 30.87 for pooled tokens, 34.33 for Ω, and 35.80 for ΔS. Mean pooling trains fastest. ΔS improves all three quality metrics over Ω and processes 31.6% more training tokens per second. Section 4.2 also clarifies why the fixed-gate identity alone cannot establish an advantage for state feedback.

  3. Disabling all boundary gates (Question 3). Section 5.3 and Appendix K.3 (Table 24) report the requested intervention on the trained 1.3B GDN+SDG. Disabling every boundary gate during prefill and decoding lowers NIAH@4K from 55.67 to 16.27 and real-world retrieval from 31.91 to 20.13. This measures the trained model’s reliance on the gates. The intervention does not isolate how they affect training.

  4. Compute, parameter count, and decoding latency (Question 4). Section 5.4 and Appendix K.1 (Table 19) report lower training loss at all 15 size–budget combinations when gate FLOPs are counted. Appendix K.2 (Table 22) adds a widened, parameter-matched Mamba-3 baseline. Compared with this baseline, SDG lowers WikiText perplexity from 25.08 to 24.79 and improves NIAH@4K, real-world retrieval (22.79 to 25.17), and LongBench. Appendix D.5 (Table 10) corrects the original latency comparison, which used different decoding kernels. Using the same decoding kernel and preprocessing for both variants on one A100, SDG adds decoding overhead in all six settings. At batch size one and a 512-token prompt, time per output token is 16.739 ms for Mamba-3 and 17.255 ms with SDG.

  5. Gate interval and 8K extrapolation (Question 5). Section 5.3 and Appendix K.3 (Table 23) test intervals of one, two, and four kernel chunks in 340M GDN. With 64-token chunks, increasing the interval from 64 to 256 tokens improves S-NIAH-1@8K from 78.40 to 83.80 while lowering NIAH@4K from 35.80 to 30.27.

We would appreciate your thoughts on whether our response addresses your concerns about the controls, compute matching, and Mamba-3 timings. If any questions remain after reading the rebuttal and the corresponding changes in the paper, we would be glad to follow up here.

Best regards,

The authors of Paper #6796

Reviewer_AAu4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Gated linear attention models compute gates from token representations alone, without reading the recurrent memory, so the same token yields the same gate even when the layer holds different information. State-Dependent Gating (SDG) computes element-wise gates from the change in memory over a chunk and applies them at the chunk boundary, giving the model state feedback for forgetting while keeping parallel computation within each chunk. The authors built fused kernels for Gated DeltaNet, Kimi Delta Attention, Gated DeltaNet-2, and Mamba-3, and backed the design with a signal-noise analysis and a result showing that boundary decay acts as an element-wise ridge proximal step. In the evaluation, SDG improves language modeling, commonsense reasoning, synthetic and real-world retrieval, and long-context understanding.

Critical strengths
  1. Clear problem statement (“Same token produces same gate” -- limitation of state-independent gates).
  2. Simple and intuitive design that applies the gate only at chunk boundaries, preserving the existing within-chunk algorithm intact.
  3. Generality across 4 backbones (GDN, KDA, Mamba-3, GDN-2), with a fused kernel implementation provided.
  4. All models compared as baseline/+SDG pairs (shared data order, optimizer, and initialization) for fair comparison
  5. Broad evaluation: LM, multiple zero-shot tasks, RULER, multiple real-world retrieval, LongBench tasks, and hybrid settings.
Critical weaknesses
  1. No direct diagnostic evidence that “same token producing same gate” actually hurts performance.
  2. Ω control recovers most of the gain and even scores higher on LongBench. This weakens the claim that “state-dependency” is the key factor.
  3. Improvement margins are small (zero-shot +0 to +0.45, retrieval +0.39 to +0.83).
  4. A single setting (1.3B / 100B tokens / C=64) makes scaling trends and chunk size sensitivity unknown.
Questions for authors
  1. Could the authors provide the curve for scaling trends and chunk size sensitivity beyond the current single setting?
  2. I think the problem's granularity is "token-level" but solution's granularity is "chunk-level." Does this mismatch cause any issue?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the careful review and constructive feedback.

Weakness 1. Direct diagnostic evidence

Our motivation is that the same incoming content may be new to one memory state but redundant to another. Retention should therefore be able to respond to what is already stored. For a fixed layer input, a token-derived gate cannot make this distinction directly. SDG introduces this dependence through ΔS while preserving within-chunk parallelism.

The comparisons in Weakness 2 assess the practical value of this design. At matched training FLOPs, SDG outperforms both content-only controls on NIAH@4K, Recall, and LongBench. It also achieves higher training throughput than Ω. These results support ΔS as an effective and efficient gating signal.

Weakness 2. Ω and state-feedback attribution

We consider both accuracy and efficiency when comparing ΔS with Ω. Although Ω recovers much of the gain in the original ablation, forming Ω requires an additional matrix product, whereas ΔS is simply the difference of two available states. We therefore test mean-pooled token gating as a cheaper content-only alternative to Ω, which is suggested by Reviewer jWG3. All models are 340M and match the total training FLOPs of a 340M GDN trained on 15B tokens.

For the pooled control, we average the normalized token representations within each B=64 chunk. A low-rank projection followed by softplus predicts positive step sizes, which generate element-wise decay in the Mamba-1 form:

xˉn=B−1∑t∈nxt,δn,h=softplus⁡(Wup,hWdownxˉn+bh),Λn,h,ij=exp⁡[−δn,h,iexp⁡(ah,ij)]\bar{x}_n=B^{-1}\sum_{t\in n}x_t,\qquad\delta_{n,h}=\operatorname{softplus}(W_{\mathrm{up},h}W_{\mathrm{down}}\bar{x}_n+b_h),\qquad\Lambda_{n,h,ij}=\exp[-\delta_{n,h,i}\exp(a_{h,ij})]

Here h indexes heads, i and j index state entries, and a contains learned decay parameters. The gate reads only the pooled token representations and is applied at the same chunk boundaries as SDG.

Training throughput is measured on a single A100 using 4096 sequence length and a batch size of 4.

Gate inputNIAH@4K↑Recall↑LongBench↑Training throughput (tokens/s)↑
Mean pooling + Mamba-1-style gate30.8720.707.7840,180
Ω34.3321.848.0328,753
ΔS35.8022.348.5037,835

SDG achieves higher scores than both controls on all three metrics and 31.6% higher training throughput than Ω. Mean pooling is faster, but does not recover SDG's accuracy.

Weakness 3. Small improvement margins

Although some gains are modest, consistent improvements across backbones and configurations support SDG's effectiveness. WikiText PPL and Recall improve across all four recurrent backbones, with Recall gains also in hybrids. GDN gains 6.07 points on NIAH@4K, while Mamba-3 and GDN-2 gain 13.6 and 9.8 points on MK-NIAH-1@4K.

Weakness 4. Single-setting evaluation

Following the reviewer's suggestion, Question 1 covers 60M, 130M, and 340M models at five FLOPs budgets, with lower loss throughout. We also vary B at C=64 and evaluate C=B=32.

Question 1. Scaling and chunk sensitivity

We use common cumulative FLOPs budgets to compare how both models improve with training compute, including the added gating cost. Training loss averages the last 100 updates, covering 26,214,400 tokens.

ScaleMethod1.50e181.88e182.25e182.63e183.00e18
60MGDN2.9002.8882.8422.8212.794
60MSDG2.8972.8752.8392.8152.789
130MGDN2.9032.8622.8352.8062.787
130MSDG2.8962.8562.8292.7972.778
340MGDN2.9052.8462.7952.7832.759
340MSDG2.9012.8412.7882.7732.750

SDG achieves lower training loss at all five budgets and all three model sizes. At 340M and 3.00e18 FLOPs, loss is 2.750 versus GDN's 2.759, supporting gains beyond the original 1.3B setting.

For chunk sensitivity, we vary B at C=64 and additionally evaluate C=B=32, using 340M models. The GDN baseline (B=None) is trained on 15B tokens, and all gated variants match its total training FLOPs. B=None denotes no boundary gating.

CBWikiText PPL↓NIAH@4K↑Recall↑S-NIAH-1@8K↑
323225.8640.2024.1372.80
646425.9935.8022.3478.40
6412826.3730.7321.7082.20
6425626.5330.2721.8483.80
64None26.9827.2720.7086.60

C=B=32 gives the best WikiText PPL, NIAH@4K, and Recall. At C=64, increasing B from 64 to 256 improves S-NIAH-1@8K from 78.40 to 83.80, while PPL and NIAH@4K worsen. These results suggest that more frequent boundary gating may impair length extrapolation on tasks like S-NIAH-1.

Question 2. Token versus chunk granularity

Although computed at chunk boundaries, the gate modifies the state used by every subsequent token. It therefore affects token-level predictions and complements native token gates. We will clarify this distinction in the revision.

Dear Reviewer AAu4,

Thank you again for the time and care you have devoted to reviewing Paper #6796. Your thoughtful feedback has helped us improve the paper. We wanted to follow up on our rebuttal and ask whether you have had a chance to read it.

We have also submitted a revised manuscript incorporating our response. The text and tables we added in response to your review are marked in red.

  1. Motivation and evidence for state feedback (Weaknesses 1–2). In Section 1, we clarify why a token-only gate gives the same value for a fixed layer input even when the layer’s recurrent memory differs. Section 5.3 and Appendix K.2 (Table 21) add a 340M comparison at matched training FLOPs. NIAH@4K scores are 30.87 for mean-pooled tokens, 34.33 for the chunk-only input Ω, and 35.80 for the state change ΔS. ΔS also gives 31.6% higher training throughput than Ω. These model-level ablations measure downstream performance. They do not isolate the effect on prediction accuracy of varying only the incoming memory while holding the token representation fixed. The original 1.3B Ω control recovers most of the gain, and we have narrowed the abstract’s claim accordingly.

  2. Size and consistency of the gains (Weakness 3). Section 5.3 highlights the 6.07-point NIAH@4K gain for 1.3B GDN. Section 5.4 and Appendix K.1 (Table 20) report results for three paired seeds at 340M. WikiText perplexity, NIAH@4K, real-world retrieval, and LongBench improve in every pair, with NIAH@4K gains ranging from 2.66 to 9.06 points.

  3. Model scale and gate frequency (Weakness 4 and Question 1). Section 5.4 and Appendix K.1 (Table 19) report lower training loss for SDG at all five matched FLOPs budgets for each of the 60M, 130M, and 340M GDN models. Section 5.3 and Appendix K.3 (Table 23) distinguish kernel chunk size from gate interval. With the kernel chunk fixed at 64 tokens, increasing the interval from 64 to 256 lowers NIAH@4K from 35.80 to 30.27 but raises S-NIAH-1@8K from 78.40 to 83.80. We also evaluate the joint 32-token setting.

  4. Token versus chunk granularity (Question 2). Section 3.1 now explains the timing explicitly. The current chunk’s outputs are already computed when the boundary gate acts. Every token in the next chunk inherits the gated memory, while the native gates continue to operate at each token.

We would welcome your thoughts on whether our response addresses your questions about state feedback and gate frequency. If any part of the rebuttal or the corresponding changes in the paper would benefit from further clarification, we would be happy to discuss it here.

Best regards,

The authors of Paper #6796

Thank you for the additional experiments and careful responses. Most of my concerns are addressed, but due to the limited experimental settings regarding different values of B and C, it is difficult to justify a stronger recommendation. I therefore maintain my score of 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.