- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper describes the results of an empirical study on length generalization of various sequence models. The focus is on recurrent models that compute an inner (“fast”) loop of intermediate steps between the (“slow”) steps of the task sequence. The main findings of this study are that state-dependent hidden-to-hidden transitions and feature mixing in the hiddens are important for length generalization. The paper also shows some ablations, such as on the number of intermediate steps and weight sharing between the intermediate steps.
Critical strengths
Length generalization is an important research area and it is an Achilles’ heel of existing common sequence models, including practically all frontier models based on transformers or SSMs. The paper is fairly well-written and fairly easy to follow.
Critical weaknesses
Performance of the recurrent models can be very sensitive to hyperparameter choices, such as layer sizes, initializations, etc. Although the hyperparameter choices are mentioned in the Appendix, and learning rates are evaluated using a sweep, I’m afraid the absolute performance figures could vary a lot with variations in the other choices. This makes it impossible to draw ony sound conclusions from these numbers.
The importance of feature-wise mixing for length-generalization in recurrent models is well-known both empirically and theoretically, and the results in this paper barely add anything new in this regard.
See also Questions For Authors below.
Questions for authors
Why is the OOD scenario in the Dyck-language experiments defined as by a 1-regular run and not simply kept at length generalization. This confounds the variables under investigation (longer horizons), by adding a seemingly arbitrary other dimension of generalization. I do not understand the point of this. It would be important to know what the results in the simpler setting (just length-generalization) look like.
The importance of state-dependent transitions is confusing to me. Doesn’t this simply amount to being a higher-order (that is, quadratic or generally non-linear) function of Z? State-dependent transitions are also reminiscent of the well-known need for input-dependent transitions in SSMs, and I am wondering what the relation is between the two.
5 comments
Q1. "Why is the OOD scenario in the Dyck-language experiments defined as by a 1-regular run and not simply kept at length generalization. ... It would be important to know what the results in the simpler setting (just length-generalization) look like."
The results for length generalization alone are in Appendix C.1.1 (Fig. 13a). On random Dyck-(30,5) strings up to length 2,560 (64 times the training length), LSTM, Mamba2, and FSRM stay above 90%, and the Transformer drops to about 60%.
We use 1-regular runs because a longer random string does not contain longer dependencies. At length 2,560, 99% of the openers in a random string are closed within 47 steps, which is about the training length. Length generalization on random strings therefore tests the same dependency distances as training. In a 1-regular run, the first opener must be kept for the whole stream (about 2,558 steps), so the dependency distance grows with the length. It is the simplest sequence with this property.
This second axis is not arbitrary. Long dependencies also occur in random strings, although rarely (0.06% exceed 100 steps). A long enough stream will contain them, and a model that cannot keep an opener for that long fails there. The 1-regular run tests this case directly.
To separate the dependency distance from the regular pattern, we also evaluated a second kind of string. Its bottom opener stays open for the whole length (2,560), and random Dyck content is placed on top of it. The table reports the accuracy of recalling the bottom opener at positions 1,280–2,560 (mean ± std over 3 seeds), with each model at its default configuration (Table 5).
| Depth of the content on top | 1 | ≤ 2 | ≤ 3 | ≤ 4 |
|---|---|---|---|---|
| LSTM | 0.000 ± 0.000 | 0.000 ± 0.000 | 0.000 ± 0.000 | 0.000 ± 0.000 |
| Mamba2 | 0.006 ± 0.008 | 0.003 ± 0.005 | 0.012 ± 0.013 | 0.010 ± 0.010 |
| Transformer | 0.000 ± 0.000 | 0.000 ± 0.000 | 0.000 ± 0.000 | 0.000 ± 0.000 |
| FSRM | 0.978 ± 0.022 | 0.999 ± 0.001 | 0.841 ± 0.102 | 0.021 ± 0.021 |
With content of depth up to 3, FSRM recalls the opener with 84–100% accuracy, whereas the baselines reach at most about 1%.
Q2. "The importance of state-dependent transitions is confusing to me. Doesn't this simply amount to being a higher-order (that is, quadratic or generally non-linear) function of Z? State-dependent transitions are also reminiscent of the well-known need for input-dependent transitions in SSMs, and I am wondering what the relation is between the two."
We agree that state-dependent transitions are nonlinear functions of . However, our experiments compare different nonlinear parameterizations. In particular, MLP-Mixer already uses nonlinear token and feature mixing, yet exhibits substantially weaker OOD generalization than several attention-based and gated cores (Sec. 5.3, Fig. 6). Thus, the presence of nonlinearity alone does not distinguish the successful cores. Our results support explicit state-dependent mixing or gating as a useful architectural feature among the tested cores, without establishing that it is uniquely responsible or that other nonlinear parameterizations could not perform equally well.
The connection to input-dependent transitions in selective SSMs is indeed relevant. Our adapted Mamba2 core illustrates this connection. Its selective mechanism receives , so its input-dependent coefficients depend on the evolving outer latent (Appendix A.1.2). We therefore regard selective SSMs as a related instance of this design principle, but SSMs differ in that they use an internal transition matrix. Our contribution is the empirical investigation of these structures within persistent fast–slow recurrence, showing how the choice of nonlinear core affects generalization from short training sequences to much longer horizons. We added this discussion to Appendix A.1.2.
We hope the new results address your concerns about robustness and the Dyck setup.
Regarding learning rate sweep: What is the list of learning rates tested in each scenario?
References on input-dependent, non-diagonal transitions: Fan, et al. (2023). Advancing Regular Language Reasoning in Linear Recurrent Neural Networks. arXiv. Grazzi, et al. (2024). Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues. arXiv. Merrill, et al. (2024). The Illusion of State in State-Space Models. arXiv. Sarrof, et al. (2024). The Expressive Capacity of State Space Models: A Formal Language Perspective. arXiv. Ebrahimi, et al. (2025). Revisiting Bi-Linear State Transitions in Recurrent Neural Networks. arXiv. Terzić, et al. (2025b). Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models. arXiv.
Thank you for the review. We address each point, with new results for W1, W2, and Q1.
We agree that absolute performance depends on these choices, but the comparison with the baselines does not. The paper concludes that the baselines do not reach the OOD accuracy of FSRM, and this conclusion would be wrong only if some hyperparameter choice let a baseline reach it. We varied the choices you list and found no such choice. None of the 345 baseline configurations reaches 0.95 OOD accuracy, whereas 7 of the 27 FSRM configurations do (table below).
On Dyck, we varied layer size, depth, initialization, learning rate, training length, and number of loops in 372 configurations. As in the paper, each configuration is scored by its mean OOD accuracy over 3 seeds. For each model, the table gives the number of configurations whose mean OOD accuracy reaches 0.95 and 0.99.
OOD accuracy is token accuracy on 1-regular runs at positions 1,280–2,560. Besides the library default, the initializations are a forget-gate bias of 1 and chrono initialization (Tmax=40) for the LSTM, N(0,0.022) weights for the Transformer and the Looped Transformer, and a ten times smaller step size Δ for Mamba2.
The protocol of the submission sweeps the learning rate and keeps the other choices fixed, which follows common practice in prior work on length generalization. Delétang et al. (2023) tune only the learning rate for their recurrent models (three values, ten seeds), and Grazzi et al. (2025) also tune the learning rate with three seeds. The sweep above varies more choices than these studies do. We will add this sweep to the appendix of the final version.
We agree that the importance of feature-wise mixing is known, and we do not claim it as new. Sec. 3.2 introduces both properties as established concepts and cites the prior work. What the paper adds is how the effect of feature-wise mixing (ii) depends on state-dependent transitions (i) and on repeated updates.
The effect of (ii) is large in cores that also have (i) and small in the other cores (Table 3).
The effect also depends on repeated updates. We repeated the same ablation with one update per observation (T=1). With T=5, cores with both properties exceed the mean of the other three groups by 0.204 in OOD accuracy (table above). With T=1, this gap is 0.045. The known results concern recurrent models with one update per input, and to our knowledge this dependence on repeated updates has not been reported.
If you know of prior work that shows this interaction, we would be grateful for the reference and will cite and discuss it. Appendix A.1.2 now states what is known and what the paper adds.