- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes a training-stability issue in gated linear RNNs caused by input-dependent decay gates, and proposes a minimal architectural fix. The authors show that for a gate of the form , the gate is monotonically increasing but concave for . Consequently, when the gate is close to one, an equal-magnitude decrease in the pre-activation produces a larger drop in the gate than the corresponding increase. Over long sequences, this asymmetry accumulates multiplicatively and can cause information loss and training failure, especially at larger learning rates. The proposed remedy, called “single-LTI hybrid,” replaces one input-dependent recurrent layer (typically the middle layer) with an LTI state-space layer such as S4D, while keeping the rest of the gated linear RNN unchanged. The paper argues that residual connections provide a direct additive path for the LTI features, so a single LTI layer can be sufficient. Experiments cover a copying task across learning rates and sequence lengths, Pathfinder from LRA, language modeling on PG-19 at 4K–32K context lengths, and ablations on residual connections and LTI-layer placement.
Critical strengths
Strength 1: The theoretical analysis is focused and yields a concrete, falsifiable mechanism. The paper does not merely observe that gated linear RNNs are unstable at large learning rates; it isolates a specific asymmetric sensitivity of the gate function and connects it to multiplicative information decay over time. Theorem 1 and the subsequent unrolling argument give a clear causal story: gate values near one are fragile under downward pre-activation perturbations, and this fragility compounds over long sequences. This is consequential because it motivates a minimal intervention—replacing a single layer—rather than a broad architectural overhaul, and it makes the empirical predictions testable (e.g., larger learning rates and longer sequences should benefit more).
Strength 2: The proposed intervention is simple and empirically supported across multiple settings. Replacing one gated layer with an S4D layer is a lightweight change that does not require redesigning the entire architecture. The copying-task heatmaps show a wider trainable region for the hybrid, the Pathfinder results show that the hybrid succeeds at learning rates where the baselines fail or overfit, and the PG-19 results show consistent perplexity improvements at most context lengths. The residual-connection ablation further supports the claimed mechanism: without residual connections, the hybrid becomes sensitive to LTI-layer placement, which is consistent with the argument that residual paths are what allow a single LTI layer to suffice.
Critical weaknesses
Weakness 1: The theoretical analysis relies on simplified assumptions that are not clearly connected to the actual training regimes of the models used in experiments. The derivation considers a single SGD step and a scalar pre-activation , whereas the experiments use AdamW, deep stacks of layers, normalization, and gated linear RNNs whose gate parameterizations may differ from the specific form analyzed. The paper does not establish that the scalar SGD analysis quantitatively predicts the behavior of Mamba-2 or Gated DeltaNet under AdamW. This matters because the central claim is that the asymmetric gate sensitivity explains training failure at large learning rates; if the analysis is only qualitatively suggestive, the mechanism remains plausible but not decisively established. A stronger paper would either derive the effect under Adam-like updates or provide targeted measurements of gate-value distributions during training to confirm the predicted downward shift.
Weakness 2: The empirical gains are real but modest, and the experimental protocol leaves several confounds unresolved. On PG-19, the average perplexity improvement is about 0.74 for Gated DeltaNet and 0.75 for Mamba-2; at 16K, Mamba-2 single-LTI is actually worse than the baseline. The paper does not report multiple seeds or variance for the language-modeling results, so it is difficult to judge whether these differences are robust. The LTI layer is placed at the middle layer in most experiments, but the placement ablation shows that the first-layer replacement is less consistent; the explanation (raw input mixing) is plausible but post hoc. Moreover, the copying task uses randomly sampled dummy tokens rather than the standard fixed dummy token, which changes the task relative to prior work and makes direct comparison with published copying results less straightforward. These issues do not invalidate the method, but they weaken the strength of the empirical support relative to the paper’s broad claims about long-sequence trainability.
Questions for authors
-
The theoretical analysis assumes a scalar SGD update and a specific gate parameterization. Can you provide empirical measurements of gate-value distributions during training for Mamba-2 and Gated DeltaNet under AdamW, showing that the predicted asymmetric downward shift actually occurs and correlates with the observed training failures?
-
For the PG-19 language-modeling results, how many random seeds were used? Can you report mean and variance across seeds, and clarify whether the 16K Mamba-2 result (where single-LTI is worse than the baseline) is within run-to-run noise?
-
The LTI layer is placed at the middle layer by default, but the placement ablation shows that the first layer is less effective. Is there a principled criterion for choosing the replacement layer, or is it purely empirical? How sensitive is the method to this choice across different backbones and sequence lengths?
-
The copying task uses randomly sampled dummy tokens instead of the standard fixed dummy token. Can you report results on the standard copying setup as well, so that the trainability claims can be compared directly with prior work?
2 comments
Response to Reviewer pd5c (2/2)
Q3. The LTI layer is placed at the middle layer by default, but the placement ablation shows that the first layer is less effective.
Figure 8 in the revised version shows test perplexity when different layers of the 12-layer Gated DeltaNet are replaced. Replacing the first layer does not consistently improve test perplexity, whereas every other placement improves on the Gated DeltaNet baseline at all four context lengths. Moreover, the best placement depends on the context length, and no placement is consistently best. Given this, we use the middle layer (k = 7) as the default, as a neutral position between the first and last layers. Although this does not mean that the middle layer is the best placement, every placement other than the first improves on the baseline.
Q4. The copying task uses randomly sampled dummy tokens instead of the standard fixed dummy token.
Thank you for the valuable suggestion. To examine the effect of the variation of task-irrelevant inputs, we also conduct the experiments in Section 5.2 of the revised version using the standard copying task (Arjovsky et al., 2016) as it is, in which the same dummy token is used at all dummy positions, while keeping all model and training settings unchanged.
Figures 9 and 10 in the revised version show the training accuracy of Mamba-2 and Gated DeltaNet and their single-LTI hybrids with the fixed dummy token. The fixed dummy token reduces the failure region for both the baselines and the single-LTI hybrids compared with the randomly sampled dummy tokens in Figures 3 and 4. The reduction is larger for the baselines, and the difference between the baselines and the single-LTI hybrids becomes smaller than with the randomly sampled dummy tokens.
With the fixed dummy token, the representation in the first layer is the same at all dummy positions, and thus the change in Equation (7) takes the same value at these positions in that layer. These results are consistent with our analysis that input-dependent gates cause the downward shift when the representations vary across positions. We have added these results to Appendix C.3 of the revised version.
Response to Reviewer pd5c (1/2)
We appreciate your insightful feedback and suggestions.
Thank you for pointing this out. Although Equation (7) is derived for SGD, the analysis in Sections 3.2 and 3.3 depends only on Δzt and is therefore optimizer-agnostic. Under AdamW, the change Δzt=(w+−w)⊤xt+(b+−b) likewise depends on xt and, for a given optimizer state, scales linearly with the learning rate η. We use AdamW in all experiments. We have added this discussion to Section 3.1 of the revised version.
To examine whether the downward shift actually occurs during training, we conducted an additional experiment on the copying task with Gated DeltaNet. At each training step, we compare the gate values after the actual update with those after an update whose change is replaced by its average over the dummy positions. To isolate the effect of the concavity analyzed in Section 3, we remove the linear part of this difference and report the resulting relative decrease in the geometric-mean gate.
Figure 2 of the revised version shows the training accuracy and the downward shift of Gated DeltaNet with T=1500. With learning rates of 0.01 and 0.015, the downward shift remains small and the training accuracy increases, whereas with 0.02 and 0.04 the downward shift is larger and training fails. These results are consistent with our analysis that a larger learning rate amplifies the downward shift. We have added these results and the discussion to Section 5.1 of the revised version.
Thank you for the comment. Each PG-19 configuration was trained with a single run. This follows common practice in language-modeling comparisons of recurrent architectures. For example, Mamba-2 (Dao & Gu, 2024) and Gated DeltaNet (Yang et al., 2025) report a single result per configuration without error bars. Although we cannot provide error bars, the results are consistent across configurations. As shown in Table 1, the single-LTI hybrid achieves lower perplexity than the baseline in all configurations except Mamba-2 at 16K. As shown in Figure 8 of the revised version, all placements of the LTI layer other than the first layer improve on the Gated DeltaNet baseline at every context length.
We follow the training protocol of Karami et al. (2025), including their peak learning rate and learning-rate schedule. All models use the same optimizer settings, learning-rate schedule, batch size, and training budget. Thus, these experiments show that replacing a single input-dependent recurrent layer with an LTI layer achieves competitive or better performance on language modeling under the same training budget, rather than demonstrating the best performance after tuning each architecture separately. We have added the reference to Karami et al. (2025) and this clarification to Appendix C.
On PG-19, we evaluate training on a long-sequence modeling task with context lengths from 4K to 32K, where the single-LTI hybrid improves perplexity in seven of eight configurations (Table 1). To examine whether the single-LTI hybrid preserves general language modeling performance at a larger scale, we trained approximately 340M-parameter Mamba-2 and Gated DeltaNet models and their single-LTI hybrids. All four models follow the 340M training recipe of Tumma et al. (2026), including its global batch size, and are trained on 15.7B SlimPajama tokens with a sequence length of 2,048.
As shown in the table above (Table 3 in the revised version), the average zero-shot score changes from 43.23 to 43.52 for Mamba-2 and from 44.59 to 44.31 for Gated DeltaNet, while the average in-context recall score changes from 21.96 to 21.43 and from 24.01 to 26.68, respectively. General language-modeling perplexities are also broadly comparable. Combined with the results on PG-19, these results indicate that replacing a single input-dependent layer with an LTI layer improves long-context language modeling while preserving general language modeling performance. We have added these results to Section 5.5 and Appendix C.6 of the revised version.
Reference Mahdi Karami, Razvan Pascanu, and Vahab Mirrokni. Lattice: Learning to Efficiently Compress the Memory. arXiv preprint arXiv:2504.05646, 2025.
Neehal Tumma, Noel Loo, and Daniela Rus. Preconditioned DeltaNet: Curvature-Aware Sequence Modeling for Linear Recurrences. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.