acceptodds
Under review as a conference paper at ICLR 2027

Robust Gated Linear RNNs via a Single Linear Time-Invariant Recurrent Layer

Abstract

Gated linear Recurrent Neural Networks (RNNs) incorporating input-dependent gate mechanisms have shown promising performance on sequence modeling. While input-dependent gating allows their dynamics to adapt flexibly to each input, it also makes the dynamics sensitive to task-irrelevant inputs. To characterize and address this issue, we directly analyze the gate mechanism and show that the gate has a *non-constant rate of change*: When the gate value is close to one, a decrease in the pre-activation causes a larger change in the gate than an increase of the same magnitude. This asymmetry can cause a downward shift in the gate values, making it difficult to preserve information over long sequences. Moreover, a larger learning rate can amplify this downward shift since the update of the pre-activation scales with the learning rate. This insight leads to our proposal: We replace a single input-dependent recurrent layer with a *linear time-invariant* (LTI) *recurrent layer* in a gated linear RNN while leaving the rest of the architecture unchanged, which we call the *single-LTI hybrid*. Since the state-transition matrix of the LTI layer is independent of the input, this replacement mitigates the effect of task-irrelevant inputs and large learning rates. Interestingly, replacing only a single layer is sufficient because residual connections commonly used in gated linear RNNs provide an identity path, allowing the features extracted by the LTI layer to be preserved through the subsequent layers. Our experiments show that the proposed single-LTI hybrid improves trainability on long-sequence modeling tasks, particularly with large learning rates, reduces undesirable sensitivity to inputs, and achieves competitive or better performance on language modeling.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Linear RNN, State Space Model, Mamba

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pd5cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes a training-stability issue in gated linear RNNs caused by input-dependent decay gates, and proposes a minimal architectural fix. The authors show that for a gate of the form fα(z)=exp⁡[−α softplus(−z)]f_\alpha(z)=\exp[-\alpha\,\mathrm{softplus}(-z)], the gate is monotonically increasing but concave for z>log⁡αz>\log\alpha. Consequently, when the gate is close to one, an equal-magnitude decrease in the pre-activation produces a larger drop in the gate than the corresponding increase. Over long sequences, this asymmetry accumulates multiplicatively and can cause information loss and training failure, especially at larger learning rates. The proposed remedy, called “single-LTI hybrid,” replaces one input-dependent recurrent layer (typically the middle layer) with an LTI state-space layer such as S4D, while keeping the rest of the gated linear RNN unchanged. The paper argues that residual connections provide a direct additive path for the LTI features, so a single LTI layer can be sufficient. Experiments cover a copying task across learning rates and sequence lengths, Pathfinder from LRA, language modeling on PG-19 at 4K–32K context lengths, and ablations on residual connections and LTI-layer placement.

Critical strengths

Strength 1: The theoretical analysis is focused and yields a concrete, falsifiable mechanism. The paper does not merely observe that gated linear RNNs are unstable at large learning rates; it isolates a specific asymmetric sensitivity of the gate function and connects it to multiplicative information decay over time. Theorem 1 and the subsequent unrolling argument give a clear causal story: gate values near one are fragile under downward pre-activation perturbations, and this fragility compounds over long sequences. This is consequential because it motivates a minimal intervention—replacing a single layer—rather than a broad architectural overhaul, and it makes the empirical predictions testable (e.g., larger learning rates and longer sequences should benefit more).

Strength 2: The proposed intervention is simple and empirically supported across multiple settings. Replacing one gated layer with an S4D layer is a lightweight change that does not require redesigning the entire architecture. The copying-task heatmaps show a wider trainable region for the hybrid, the Pathfinder results show that the hybrid succeeds at learning rates where the baselines fail or overfit, and the PG-19 results show consistent perplexity improvements at most context lengths. The residual-connection ablation further supports the claimed mechanism: without residual connections, the hybrid becomes sensitive to LTI-layer placement, which is consistent with the argument that residual paths are what allow a single LTI layer to suffice.

Critical weaknesses

Weakness 1: The theoretical analysis relies on simplified assumptions that are not clearly connected to the actual training regimes of the models used in experiments. The derivation considers a single SGD step and a scalar pre-activation zt=w⊤xt+bz_t=w^\top x_t+b, whereas the experiments use AdamW, deep stacks of layers, normalization, and gated linear RNNs whose gate parameterizations may differ from the specific fα(z)=exp⁡[−α softplus(−z)]f_\alpha(z)=\exp[-\alpha\,\mathrm{softplus}(-z)] form analyzed. The paper does not establish that the scalar SGD analysis quantitatively predicts the behavior of Mamba-2 or Gated DeltaNet under AdamW. This matters because the central claim is that the asymmetric gate sensitivity explains training failure at large learning rates; if the analysis is only qualitatively suggestive, the mechanism remains plausible but not decisively established. A stronger paper would either derive the effect under Adam-like updates or provide targeted measurements of gate-value distributions during training to confirm the predicted downward shift.

Weakness 2: The empirical gains are real but modest, and the experimental protocol leaves several confounds unresolved. On PG-19, the average perplexity improvement is about 0.74 for Gated DeltaNet and 0.75 for Mamba-2; at 16K, Mamba-2 single-LTI is actually worse than the baseline. The paper does not report multiple seeds or variance for the language-modeling results, so it is difficult to judge whether these differences are robust. The LTI layer is placed at the middle layer in most experiments, but the placement ablation shows that the first-layer replacement is less consistent; the explanation (raw input mixing) is plausible but post hoc. Moreover, the copying task uses randomly sampled dummy tokens rather than the standard fixed dummy token, which changes the task relative to prior work and makes direct comparison with published copying results less straightforward. These issues do not invalidate the method, but they weaken the strength of the empirical support relative to the paper’s broad claims about long-sequence trainability.

Questions for authors
  1. The theoretical analysis assumes a scalar SGD update and a specific gate parameterization. Can you provide empirical measurements of gate-value distributions during training for Mamba-2 and Gated DeltaNet under AdamW, showing that the predicted asymmetric downward shift actually occurs and correlates with the observed training failures?

  2. For the PG-19 language-modeling results, how many random seeds were used? Can you report mean and variance across seeds, and clarify whether the 16K Mamba-2 result (where single-LTI is worse than the baseline) is within run-to-run noise?

  3. The LTI layer is placed at the middle layer by default, but the placement ablation shows that the first layer is less effective. Is there a principled criterion for choosing the replacement layer, or is it purely empirical? How sensitive is the method to this choice across different backbones and sequence lengths?

  4. The copying task uses randomly sampled dummy tokens instead of the standard fixed dummy token. Can you report results on the standard copying setup as well, so that the trainability claims can be compared directly with prior work?

2 comments
AuthorsOpenReview ↗

Response to Reviewer pd5c (1/2)

We appreciate your insightful feedback and suggestions.

Q1. The theoretical analysis assumes a scalar SGD update and a specific gate parameterization.

Thank you for pointing this out. Although Equation (7) is derived for SGD, the analysis in Sections 3.2 and 3.3 depends only on Δzt\Delta z_t and is therefore optimizer-agnostic. Under AdamW, the change Δzt=(w+−w)⊤xt+(b+−b)\Delta z_t = (w^{+} - w)^\top x_t + (b^{+} - b) likewise depends on xtx_t and, for a given optimizer state, scales linearly with the learning rate η\eta. We use AdamW in all experiments. We have added this discussion to Section 3.1 of the revised version.

To examine whether the downward shift actually occurs during training, we conducted an additional experiment on the copying task with Gated DeltaNet. At each training step, we compare the gate values after the actual update with those after an update whose change is replaced by its average over the dummy positions. To isolate the effect of the concavity analyzed in Section 3, we remove the linear part of this difference and report the resulting relative decrease in the geometric-mean gate.

Figure 2 of the revised version shows the training accuracy and the downward shift of Gated DeltaNet with T=1500T=1500. With learning rates of 0.01 and 0.015, the downward shift remains small and the training accuracy increases, whereas with 0.02 and 0.04 the downward shift is larger and training fails. These results are consistent with our analysis that a larger learning rate amplifies the downward shift. We have added these results and the discussion to Section 5.1 of the revised version.

Q2. For the PG-19 language-modeling results, how many random seeds were used?

Thank you for the comment. Each PG-19 configuration was trained with a single run. This follows common practice in language-modeling comparisons of recurrent architectures. For example, Mamba-2 (Dao & Gu, 2024) and Gated DeltaNet (Yang et al., 2025) report a single result per configuration without error bars. Although we cannot provide error bars, the results are consistent across configurations. As shown in Table 1, the single-LTI hybrid achieves lower perplexity than the baseline in all configurations except Mamba-2 at 16K. As shown in Figure 8 of the revised version, all placements of the LTI layer other than the first layer improve on the Gated DeltaNet baseline at every context length.

We follow the training protocol of Karami et al. (2025), including their peak learning rate and learning-rate schedule. All models use the same optimizer settings, learning-rate schedule, batch size, and training budget. Thus, these experiments show that replacing a single input-dependent recurrent layer with an LTI layer achieves competitive or better performance on language modeling under the same training budget, rather than demonstrating the best performance after tuning each architecture separately. We have added the reference to Karami et al. (2025) and this clarification to Appendix C.

On PG-19, we evaluate training on a long-sequence modeling task with context lengths from 4K to 32K, where the single-LTI hybrid improves perplexity in seven of eight configurations (Table 1). To examine whether the single-LTI hybrid preserves general language modeling performance at a larger scale, we trained approximately 340M-parameter Mamba-2 and Gated DeltaNet models and their single-LTI hybrids. All four models follow the 340M training recipe of Tumma et al. (2026), including its global batch size, and are trained on 15.7B SlimPajama tokens with a sequence length of 2,048.

ModelWiki. ppl ↓LMB. ppl ↓Avg. acc (8 zero-shot tasks) ↑Avg. recall (SWDE, SQuAD, FDA) ↑
Mamba-228.6240.7843.2321.96
Single-LTI Mamba-228.5542.7943.5221.43
Gated DeltaNet27.0430.0944.5924.01
Single-LTI Gated DeltaNet27.0630.0044.3126.68

As shown in the table above (Table 3 in the revised version), the average zero-shot score changes from 43.23 to 43.52 for Mamba-2 and from 44.59 to 44.31 for Gated DeltaNet, while the average in-context recall score changes from 21.96 to 21.43 and from 24.01 to 26.68, respectively. General language-modeling perplexities are also broadly comparable. Combined with the results on PG-19, these results indicate that replacing a single input-dependent layer with an LTI layer improves long-context language modeling while preserving general language modeling performance. We have added these results to Section 5.5 and Appendix C.6 of the revised version.

Reference Mahdi Karami, Razvan Pascanu, and Vahab Mirrokni. Lattice: Learning to Efficiently Compress the Memory. arXiv preprint arXiv:2504.05646, 2025.

Neehal Tumma, Noel Loo, and Daniela Rus. Preconditioned DeltaNet: Curvature-Aware Sequence Modeling for Linear Recurrences. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.

AuthorsOpenReview ↗

Response to Reviewer pd5c (2/2)

Q3. The LTI layer is placed at the middle layer by default, but the placement ablation shows that the first layer is less effective.

Figure 8 in the revised version shows test perplexity when different layers of the 12-layer Gated DeltaNet are replaced. Replacing the first layer does not consistently improve test perplexity, whereas every other placement improves on the Gated DeltaNet baseline at all four context lengths. Moreover, the best placement depends on the context length, and no placement is consistently best. Given this, we use the middle layer (k = 7) as the default, as a neutral position between the first and last layers. Although this does not mean that the middle layer is the best placement, every placement other than the first improves on the baseline.

Q4. The copying task uses randomly sampled dummy tokens instead of the standard fixed dummy token.

Thank you for the valuable suggestion. To examine the effect of the variation of task-irrelevant inputs, we also conduct the experiments in Section 5.2 of the revised version using the standard copying task (Arjovsky et al., 2016) as it is, in which the same dummy token is used at all dummy positions, while keeping all model and training settings unchanged.

Figures 9 and 10 in the revised version show the training accuracy of Mamba-2 and Gated DeltaNet and their single-LTI hybrids with the fixed dummy token. The fixed dummy token reduces the failure region for both the baselines and the single-LTI hybrids compared with the randomly sampled dummy tokens in Figures 3 and 4. The reduction is larger for the baselines, and the difference between the baselines and the single-LTI hybrids becomes smaller than with the randomly sampled dummy tokens.

With the fixed dummy token, the representation xtx_t in the first layer is the same at all dummy positions, and thus the change Δzt\Delta z_t in Equation (7) takes the same value at these positions in that layer. These results are consistent with our analysis that input-dependent gates cause the downward shift when the representations vary across positions. We have added these results to Appendix C.3 of the revised version.

Reviewer_bVxEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes replacing a single gated RNN layer with a simpler linear time-invariant layer to improve stability and long-term memory. The idea is simple and practical, with promising results on synthetic tasks and language modeling, including Mamba-2 and Gated DeltaNet.

Critical strengths

Simple, practical architectural change: replacing one gated layer with S4D. Consistent improvements across Mamba-2 and Gated DeltaNet, including better robustness to learning rates and sequence lengths. Interesting theoretical motivation concerning gate sensitivity and long-term memory.

Critical weaknesses

The theoretical argument does not establish that gate asymmetry actually causes the observed performance degradation. Similar limitations of selective state-space models were already investigated by Yu & Erichson (2025); differentiation is insufficient (just 1 layer replaced) Missing controlled ablations and direct comparisons with existing remedies. Limited experimental scale and insufficient statistical evidence.

Questions for authors

Can you directly measure gate decay during training to validate the proposed mechanism?

Can you reproduce the relevant Yu & Erichson (2025) experiments, comparing your method against B₂S₆ under identical conditions (except for the substitution you propose)

2 comments
AuthorsOpenReview ↗

Response to Reviewer bVxE (1/2)

We appreciate your valuable feedback and suggestions.

Can you directly measure gate decay during training to validate the proposed mechanism?

Thank you for your valuable comment. To examine whether the downward shift actually occurs during training, we conducted an additional experiment on the copying task with Gated DeltaNet. At each training step, we compare the gate values after the actual update with those after an update whose change is replaced by its average over the dummy positions. To isolate the effect of the concavity analyzed in Section 3, we remove the linear part of this difference and report the resulting relative decrease in the geometric-mean gate.

Figure 2 of the revised version shows the training accuracy and the downward shift of Gated DeltaNet with T=1500T=1500. With learning rates of 0.01 and 0.015, the downward shift remains small and the training accuracy increases, whereas with 0.02 and 0.04 the downward shift is larger and training fails. These results are consistent with our analysis that a larger learning rate amplifies the downward shift. We have added these results and the discussion to Section 5.1 of the revised version.

Missing controlled ablations and direct comparisons with existing remedies.

This feedback has helped us further strengthen our contributions. To examine which input-dependent component causes the training failure, we conducted an additional ablation using variants of the S4D layer in the single-LTI hybrid Gated DeltaNet. In the S4D layer, the state and input matrices in Equation (11) are given by Aˉ=exp⁡(ΔA)\bar{A} = \exp(\Delta A) and Bˉ=ΔB\bar{B} = \Delta B, where AA is a diagonal matrix, BB is an input matrix, and Δ\Delta is a fixed step size. We constructed variants in which the step size in Aˉ\bar{A}, Bˉ\bar{B}, or both is replaced with an input-dependent step size.

Figure 5 of the revised version shows the training accuracy of these variants. Making the step size input-dependent in Aˉ\bar{A}, Bˉ\bar{B}, or both enlarges the region in which training fails compared with the S4D layer. This enlargement is larger when the input-dependent step size is applied only to Aˉ\bar{A} than when it is applied only to Bˉ\bar{B}. In contrast, making only BB input-dependent leaves the failure region largely unchanged. Combined with our response above, these results empirically support our analysis that the downward shift can contribute to training failure. We have added these results to Section 5.3 and Appendix C.4 of the revised version. We also unified the notation for the discretized matrices Aˉ\bar{A} and Bˉ\bar{B} in Section 4 accordingly (no change in content).

Limited experimental scale and insufficient statistical evidence.

Thank you for pointing this out. On PG-19, we evaluate training on a long-sequence modeling task with context lengths from 4K to 32K, where the single-LTI hybrid improves perplexity in seven of eight configurations (Table 1). To examine whether the single-LTI hybrid preserves general language modeling performance at a larger scale, we trained approximately 340M-parameter Mamba-2 and Gated DeltaNet models and their single-LTI hybrids. All four models follow the 340M training recipe of Tumma et al. (2026), including its global batch size, and are trained on 15.7B SlimPajama tokens with a sequence length of 2,048.

ModelWiki. ppl ↓LMB. ppl ↓Avg. acc (8 zero-shot tasks) ↑Avg. recall (SWDE, SQuAD, FDA) ↑
Mamba-228.6240.7843.2321.96
Single-LTI Mamba-228.5542.7943.5221.43
Gated DeltaNet27.0430.0944.5924.01
Single-LTI Gated DeltaNet27.0630.0044.3126.68

As shown in the table above (Table 3 in the revised version), the average zero-shot score changes from 43.23 to 43.52 for Mamba-2 and from 44.59 to 44.31 for Gated DeltaNet, while the average in-context recall score changes from 21.96 to 21.43 and from 24.01 to 26.68, respectively. General language-modeling perplexities are also broadly comparable. Combined with the results on PG-19, these results indicate that replacing a single input-dependent layer with an LTI layer improves long-context language modeling while preserving general language modeling performance. We have added these results to Section 5.5 and Appendix C.6 of the revised version.

Reference Mahdi Karami, Razvan Pascanu, and Vahab Mirrokni. Lattice: Learning to Efficiently Compress the Memory. arXiv preprint arXiv:2504.05646, 2025.

Neehal Tumma, Noel Loo, and Daniela Rus. Preconditioned DeltaNet: Curvature-Aware Sequence Modeling for Linear Recurrences. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.

AuthorsOpenReview ↗

Response to Reviewer bVxE (2/2)

Similar limitations of selective state-space models were already investigated by Yu & Erichson (2025)

Yu & Erichson (2025) showed that large inputs and long sequences can make training unstable in the case where, in the forward pass, the input dependence enters only through the write and readout operations. In contrast, we analyze the case where the decay gate itself is input-dependent, and study how a parameter update changes the gate values rather than the gradient magnitude by Theorem 1, an update whose effect on the pre-activation varies across inputs shifts the gate values downward, and a larger learning rate amplifies this shift.

Can you reproduce the relevant Yu & Erichson (2025) experiments, comparing your method against B₂S₆ under identical conditions (except for the substitution you propose)

Thank you for your suggestion. B2S6 (Yu & Erichson, 2025) extends the S6 unit of Mamba with block-wise selective dynamics and channel-specific biases, and additionally reduces the learning rate of the parameters of Δ\Delta. These architectural modifications are specific to the S6 unit. B2S6 also notes that its multihead structure was already suggested in Mamba-2 (Dao & Gu, 2024), and both Mamba-2 and Gated DeltaNet already compute the decay gate separately for each head. Our method replaces one layer without modifying the recurrent unit, so it is orthogonal to such architectural changes. We therefore compare against the training-side component of B2S6, namely the reduced learning rate for the Δ\Delta parameters, which correspond to the decay gate parameters in Gated DeltaNet.

To examine whether reducing the learning rate of only the decay gate enables Gated DeltaNet to learn the Pathfinder task, we train the Gated DeltaNet baseline with a learning rate of 1×10−31 \times 10^{-3} and reduce the learning rate of the decay gate in the fourth layer, which is the layer replaced in the single-LTI hybrid, to 4×10−44 \times 10^{-4} following B2S6. The decay gate parameters are ww and bb in Equation (3), and all other parameters use the learning rate of 1×10−31 \times 10^{-3}.

Figure 12 in the revised version shows the test accuracy on the Pathfinder task. Reducing the learning rate of the decay gate does not enable the model to learn the task, and its test accuracy remains at about 0.5. In contrast, the single-LTI hybrid Gated DeltaNet achieves near-optimal test accuracy, although the S4D layer is also trained with the learning rate of 1×10−31 \times 10^{-3}. Although a smaller learning rate reduces the change Δzt\Delta z_t in Equation (7), the decay gate still depends on the input. Combined with the results of the input-dependent variants of the S4D layer in Section 5.3, these results suggest that the input dependence of the decay gate makes training difficult. This empirically supports our proposal of replacing an input-dependent recurrent layer with an LTI layer, instead of only reducing the learning rate of its decay gate. We have added these results to Appendix C.5 of the revised version.

Reviewer_W4mwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how gated RNNs learn to use memory paths. In temporal-XOR tasks, replacing some hard examples with equally long short-dependency examples improves hard-only test accuracy, with partial transfer across token groups. The proposed explanation is that easier supervision teaches shared gates selective retention.

Critical strengths

I think the paper has simple designs, and the proposed module is simple yet effective over baseline methods.

Critical weaknesses

However, I think the proposed method has critical drawbacks:

  1. The relationship between this work and prior work is somehow weird. Compared to prior work, Gu et al. (2020) study gate learnability, and Khajehabdollahi et al. (2024) demonstrate curriculum benefits for long-memory learning in GRUs/LSTMs. This work is somehow strange and only performs experiments on small datasets, and ppl is the only evaluation metric. I don't think the proposed method is convincing and generalizable.

  2. The explanation sounds reasonable, but I don't think it is well verified. Gate concavity alone does not show that gate values decrease during training. Replacing a whole layer with S4D also changes more than the gate, so I am not sure which change actually helps.

  3. The LM experiments are limited to PG-19, with ~110M params and PPL as the metric. All models use the same peak LR. Since LR sensitivity is a main point here, do the gains remain after tuning the baselines separately? The current results are not enough to convince me that the gains generalize well.

Questions for authors

My question mostly lies in the weaknesses part.

  1. Do the PG-19 gains remain after tuning the LR separately for each model?
  2. Do you actually observe the predicted downward drift in gate values? Can a simpler fix, e.g., lowering the gate LR or making only the decay input-independent, give similar gains?
2 comments
AuthorsOpenReview ↗

Response to Reviewer W4mw (1/2)

We appreciate your constructive feedback and suggestions.

Do the PG-19 gains remain after tuning the LR separately for each model? This work is somehow strange and only performs experiments on small datasets, and ppl is the only evaluation metric. I don't think the proposed method is convincing and generalizable.

Thank you for these comments and questions.

We follow the training protocol of Karami et al. (2025), including their peak learning rate and learning-rate schedule. All models use the same optimizer settings, learning-rate schedule, batch size, and training budget. Thus, these experiments show that replacing a single input-dependent recurrent layer with an LTI layer achieves competitive or better performance on language modeling under the same training budget, rather than demonstrating the best performance after tuning each architecture separately. We have added the reference to Karami et al. (2025) and this clarification to Appendix C.

On PG-19, we evaluate training on a long-sequence modeling task with context lengths from 4K to 32K, where the single-LTI hybrid improves perplexity in seven of eight configurations (Table 1). To examine whether the single-LTI hybrid preserves general language modeling performance at a larger scale, we trained approximately 340M-parameter Mamba-2 and Gated DeltaNet models and their single-LTI hybrids. All four models follow the 340M training recipe of Tumma et al. (2026), including its global batch size, and are trained on 15.7B SlimPajama tokens with a sequence length of 2,048.

ModelWiki. ppl ↓LMB. ppl ↓Avg. acc (8 zero-shot tasks) ↑Avg. recall (SWDE, SQuAD, FDA) ↑
Mamba-228.6240.7843.2321.96
Single-LTI Mamba-228.5542.7943.5221.43
Gated DeltaNet27.0430.0944.5924.01
Single-LTI Gated DeltaNet27.0630.0044.3126.68

As shown in the table above (Table 3 in the revised version), the average zero-shot score changes from 43.23 to 43.52 for Mamba-2 and from 44.59 to 44.31 for Gated DeltaNet, while the average in-context recall score changes from 21.96 to 21.43 and from 24.01 to 26.68, respectively. General language-modeling perplexities are also broadly comparable. Combined with the results on PG-19, these results indicate that replacing a single input-dependent layer with an LTI layer improves long-context language modeling while preserving general language modeling performance. We have added these results to Section 5.5 and Appendix C.6 of the revised version.

Reference Mahdi Karami, Razvan Pascanu, and Vahab Mirrokni. Lattice: Learning to Efficiently Compress the Memory. arXiv preprint arXiv:2504.05646, 2025.

Neehal Tumma, Noel Loo, and Daniela Rus. Preconditioned DeltaNet: Curvature-Aware Sequence Modeling for Linear Recurrences. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.

Do you actually observe the predicted downward drift in gate values?

Thank you for your valuable comment.

To examine whether the downward shift actually occurs during training, we conducted an additional experiment on the copying task with Gated DeltaNet. At each training step, we compare the gate values after the actual update with those after an update whose change is replaced by its average over the dummy positions. To isolate the effect of the concavity analyzed in Section 3, we remove the linear part of this difference and report the resulting relative decrease in the geometric-mean gate.

Figure 2 of the revised version shows the training accuracy and the downward shift of Gated DeltaNet with T=1500T=1500. With learning rates of 0.01 and 0.015, the downward shift remains small and the training accuracy increases, whereas with 0.02 and 0.04 the downward shift is larger and training fails. These results are consistent with our analysis that a larger learning rate amplifies the downward shift. We have added these results and the discussion to Section 5.1 of the revised version.

AuthorsOpenReview ↗

Response to Reviewer W4mw (2/2)

Replacing a whole layer with S4D also changes more than the gate, so I am not sure which change actually helps.

This feedback has helped us further strengthen our contributions. To identify which component in the S4D layer is effective, we conducted an additional ablation using variants of the S4D layer in the single-LTI hybrid Gated DeltaNet. In the S4D layer, the state and input matrices in Equation (11) are given by Aˉ=exp⁡(ΔA)\bar{A} = \exp(\Delta A) and Bˉ=ΔB\bar{B} = \Delta B, where AA is a diagonal matrix, BB is an input matrix, and Δ\Delta is a fixed step size. We constructed variants in which the step size in Aˉ\bar{A}, Bˉ\bar{B}, or both is replaced with an input-dependent step size.

Figure 5 of the revised version shows the training accuracy of these variants. Making the step size input-dependent in Aˉ\bar{A}, Bˉ\bar{B}, or both enlarges the region in which training fails compared with the S4D layer. This enlargement is larger when the input-dependent step size is applied only to Aˉ\bar{A} than when it is applied only to Bˉ\bar{B}. In contrast, making only BB input-dependent leaves the failure region largely unchanged. Combined with our response above, these results empirically support our analysis that the downward shift can contribute to training failure. We have added these results to Section 5.3 and Appendix C.4 of the revised version. In Section 4, we also added a paragraph explaining why S4D is used as the LTI layer, and unified the notation for the discretized matrices Aˉ\bar{A} and Bˉ\bar{B} (a notational change only).

Can a simpler fix, e.g., lowering the gate LR or making only the decay input-independent, give similar gains?

To examine whether reducing the learning rate of only the decay gate enables Gated DeltaNet to learn the Pathfinder task, we train the Gated DeltaNet baseline with a learning rate of 1×10−31 \times 10^{-3} and reduce the learning rate of the decay gate in the fourth layer, which is the layer replaced in the single-LTI hybrid, to 4×10−44 \times 10^{-4} following B2S6 (Yu & Erichson, 2025). The decay gate parameters are ww and bb in Equation (3), and all other parameters use the learning rate of 1×10−31 \times 10^{-3}.

Figure 12 in the revised version shows the test accuracy on the Pathfinder task. Reducing the learning rate of the decay gate does not enable the model to learn the task, and its test accuracy remains at about 0.5. In contrast, the single-LTI hybrid Gated DeltaNet achieves near-optimal test accuracy, although the S4D layer is also trained with the learning rate of 1×10−31 \times 10^{-3}. Although a smaller learning rate reduces the change Δzt\Delta z_t in Equation (7), the decay gate still depends on the input. Combined with the results of the input-dependent variants of the S4D layer in Section 5.3, these results suggest that the input dependence of the decay gate makes training difficult. This empirically supports our proposal of replacing an input-dependent recurrent layer with an LTI layer, instead of only reducing the learning rate of its decay gate. We have added these results to Appendix C.5 of the revised version.

The relationship between this work and prior work is somehow weird. Compared to prior work, Gu et al. (2020) study gate learnability, and Khajehabdollahi et al. (2024) demonstrate curriculum benefits for long-memory learning in GRUs/LSTMs.

Thank you for pointing this out. Gu et al. (2020) pointed out that gated RNNs such as GRUs and LSTMs struggle to learn gate values close to 0 or 1, because the gradient of the gate vanishes near saturation. Their methods make it easier for gates to reach values close to 0 and 1. In contrast, we analyze long-term dependencies from the perspective of the asymmetric response of the gate, which can cause a downward shift in the gate values that a larger learning rate amplifies. Also, their methods modify the gate parameterization and initialization, whereas we replace a single input-dependent layer with an LTI layer.

Khajehabdollahi et al. (2024) showed that a curriculum that trains a model to hold multiple concurrent memories improves training speed and robustness to perturbations on long-memory tasks, and reported that the curriculum also improves the training of GRUs and LSTMs. Their mechanistic analysis focuses on leaky RNNs, which do not have input-dependent gate mechanisms, whereas we analyze the input-dependent gates of gated linear RNNs. We have added this discussion to the related work section in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.