acceptodds
Under review as a conference paper at ICLR 2027

Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalization

Abstract

We study out of distribution generalization in streaming tasks where models are trained on short sequences but must operate over much longer, unknown horizons under bounded memory. Our focus is on a persistent fast slow recurrent formulation in which a latent state is maintained across observations rather than reset at each stream step. For each incoming observation, the model performs multiple weight-shared latent updates with a recurrent core and then carries the resulting state forward to the next observation. This allows the model to maintain and refine a compact stream-level state without reprocessing a growing context. We evaluate this formulation across symbolic sequence prediction, supervised navigation, and partially observable reinforcement learning tasks. Across these settings, persistent latent recurrence improves OOD generalization over recurrent, state-space, and Transformer baselines. Through recurrent-core ablations, we identify architectural ingredients that are consistently associated with strong OOD performance, including state-dependent transitions and feature-wise nonlinear mixing. Together, these results highlight the value of revisiting persistent recurrence as an architectural bias for more generalizable sequence prediction.

Primary area:
learning on time series and dynamical systems
Keywords:
Latent recurrent model, Long-horizon sequence modeling, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4encOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper describes the results of an empirical study on length generalization of various sequence models. The focus is on recurrent models that compute an inner (“fast”) loop of intermediate steps between the (“slow”) steps of the task sequence. The main findings of this study are that state-dependent hidden-to-hidden transitions and feature mixing in the hiddens are important for length generalization. The paper also shows some ablations, such as on the number of intermediate steps and weight sharing between the intermediate steps.

Critical strengths

Length generalization is an important research area and it is an Achilles’ heel of existing common sequence models, including practically all frontier models based on transformers or SSMs. The paper is fairly well-written and fairly easy to follow.

Critical weaknesses

Performance of the recurrent models can be very sensitive to hyperparameter choices, such as layer sizes, initializations, etc. Although the hyperparameter choices are mentioned in the Appendix, and learning rates are evaluated using a sweep, I’m afraid the absolute performance figures could vary a lot with variations in the other choices. This makes it impossible to draw ony sound conclusions from these numbers.

The importance of feature-wise mixing for length-generalization in recurrent models is well-known both empirically and theoretically, and the results in this paper barely add anything new in this regard.

See also Questions For Authors below.

Questions for authors

Why is the OOD scenario in the Dyck-language experiments defined as by a 1-regular run and not simply kept at length generalization. This confounds the variables under investigation (longer horizons), by adding a seemingly arbitrary other dimension of generalization. I do not understand the point of this. It would be important to know what the results in the simpler setting (just length-generalization) look like.

The importance of state-dependent transitions is confusing to me. Doesn’t this simply amount to being a higher-order (that is, quadratic or generally non-linear) function of Z? State-dependent transitions are also reminiscent of the well-known need for input-dependent transitions in SSMs, and I am wondering what the relation is between the two.

5 comments

Thank you for the review. We address each point, with new results for W1, W2, and Q1.

W1. "Performance of the recurrent models can be very sensitive to hyperparameter choices, such as layer sizes, initializations, etc. ... I'm afraid the absolute performance figures could vary a lot with variations in the other choices. This makes it impossible to draw any sound conclusions from these numbers."

We agree that absolute performance depends on these choices, but the comparison with the baselines does not. The paper concludes that the baselines do not reach the OOD accuracy of FSRM, and this conclusion would be wrong only if some hyperparameter choice let a baseline reach it. We varied the choices you list and found no such choice. None of the 345 baseline configurations reaches 0.95 OOD accuracy, whereas 7 of the 27 FSRM configurations do (table below).

On Dyck, we varied layer size, depth, initialization, learning rate, training length, and number of loops in 372 configurations. As in the paper, each configuration is scored by its mean OOD accuracy over 3 seeds. For each model, the table gives the number of configurations whose mean OOD accuracy reaches 0.95 and 0.99.

ModelVariedConfigsMean OOD ≥ 0.95Mean OOD ≥ 0.99
LSTMhidden {256, 512, 1024}, layers {1, 2, 4}, init {default, forget-gate bias 1, chrono (Tallec & Ollivier, 2018)}, lr 5e-4 to 3e-2, 30 or 100 epochs12900
Transformerwidth {128, 256, 512}, layers {2, 4, 6}, 2 inits, 3 learning rates5400
Looped Transformerwidth {128, 256, 512}, layers {2, 4, 6}, 2 inits, 3 learning rates, loops {5, 10}10800
Mamba2width {128, 256, 512}, layers {2, 4, 6}, 2 inits, 3 learning rates5400
FSRMwidth {16, 32, 64}, initial γ\gamma {0.05, 0.1, 0.2}, lr {1e-3, 5e-3, 1e-2}2775

OOD accuracy is token accuracy on 1-regular runs at positions 1,280–2,560. Besides the library default, the initializations are a forget-gate bias of 1 and chrono initialization (Tmax⁡=40T_{\max}=40) for the LSTM, N(0,0.022)\mathcal{N}(0, 0.02^2) weights for the Transformer and the Looped Transformer, and a ten times smaller step size Δ\Delta for Mamba2.

The protocol of the submission sweeps the learning rate and keeps the other choices fixed, which follows common practice in prior work on length generalization. Delétang et al. (2023) tune only the learning rate for their recurrent models (three values, ten seeds), and Grazzi et al. (2025) also tune the learning rate with three seeds. The sweep above varies more choices than these studies do. We will add this sweep to the appendix of the final version.

W2. "The importance of feature-wise mixing for length-generalization in recurrent models is well-known both empirically and theoretically, and the results in this paper barely add anything new in this regard."

We agree that the importance of feature-wise mixing is known, and we do not claim it as new. Sec. 3.2 introduces both properties as established concepts and cites the prior work. What the paper adds is how the effect of feature-wise mixing (ii) depends on state-dependent transitions (i) and on repeated updates.

The effect of (ii) is large in cores that also have (i) and small in the other cores (Table 3).

Mean Dyck OOD acc.without (ii)with (ii)
without (i)0.5120.534
with (i)0.5210.726

The effect also depends on repeated updates. We repeated the same ablation with one update per observation (T=1T{=}1). With T=5T{=}5, cores with both properties exceed the mean of the other three groups by 0.204 in OOD accuracy (table above). With T=1T{=}1, this gap is 0.045. The known results concern recurrent models with one update per input, and to our knowledge this dependence on repeated updates has not been reported.

If you know of prior work that shows this interaction, we would be grateful for the reference and will cite and discuss it. Appendix A.1.2 now states what is known and what the paper adds.

Q1. "Why is the OOD scenario in the Dyck-language experiments defined as by a 1-regular run and not simply kept at length generalization. ... It would be important to know what the results in the simpler setting (just length-generalization) look like."

The results for length generalization alone are in Appendix C.1.1 (Fig. 13a). On random Dyck-(30,5) strings up to length 2,560 (64 times the training length), LSTM, Mamba2, and FSRM stay above 90%, and the Transformer drops to about 60%.

We use 1-regular runs because a longer random string does not contain longer dependencies. At length 2,560, 99% of the openers in a random string are closed within 47 steps, which is about the training length. Length generalization on random strings therefore tests the same dependency distances as training. In a 1-regular run, the first opener must be kept for the whole stream (about 2,558 steps), so the dependency distance grows with the length. It is the simplest sequence with this property.

This second axis is not arbitrary. Long dependencies also occur in random strings, although rarely (0.06% exceed 100 steps). A long enough stream will contain them, and a model that cannot keep an opener for that long fails there. The 1-regular run tests this case directly.

To separate the dependency distance from the regular pattern, we also evaluated a second kind of string. Its bottom opener stays open for the whole length (2,560), and random Dyck content is placed on top of it. The table reports the accuracy of recalling the bottom opener at positions 1,280–2,560 (mean ± std over 3 seeds), with each model at its default configuration (Table 5).

Depth of the content on top1≤ 2≤ 3≤ 4
LSTM0.000 ± 0.0000.000 ± 0.0000.000 ± 0.0000.000 ± 0.000
Mamba20.006 ± 0.0080.003 ± 0.0050.012 ± 0.0130.010 ± 0.010
Transformer0.000 ± 0.0000.000 ± 0.0000.000 ± 0.0000.000 ± 0.000
FSRM0.978 ± 0.0220.999 ± 0.0010.841 ± 0.1020.021 ± 0.021

With content of depth up to 3, FSRM recalls the opener with 84–100% accuracy, whereas the baselines reach at most about 1%.

Q2. "The importance of state-dependent transitions is confusing to me. Doesn't this simply amount to being a higher-order (that is, quadratic or generally non-linear) function of Z? State-dependent transitions are also reminiscent of the well-known need for input-dependent transitions in SSMs, and I am wondering what the relation is between the two."

We agree that state-dependent transitions are nonlinear functions of ZZ. However, our experiments compare different nonlinear parameterizations. In particular, MLP-Mixer already uses nonlinear token and feature mixing, yet exhibits substantially weaker OOD generalization than several attention-based and gated cores (Sec. 5.3, Fig. 6). Thus, the presence of nonlinearity alone does not distinguish the successful cores. Our results support explicit state-dependent mixing or gating as a useful architectural feature among the tested cores, without establishing that it is uniquely responsible or that other nonlinear parameterizations could not perform equally well.

The connection to input-dependent transitions in selective SSMs is indeed relevant. Our adapted Mamba2 core illustrates this connection. Its selective mechanism receives Z+CZ+C, so its input-dependent coefficients depend on the evolving outer latent ZZ (Appendix A.1.2). We therefore regard selective SSMs as a related instance of this design principle, but SSMs differ in that they use an internal transition matrix. Our contribution is the empirical investigation of these structures within persistent fast–slow recurrence, showing how the choice of nonlinear core affects generalization from short training sequences to much longer horizons. We added this discussion to Appendix A.1.2.

We hope the new results address your concerns about robustness and the Dyck setup.

Regarding learning rate sweep: What is the list of learning rates tested in each scenario?

References on input-dependent, non-diagonal transitions: Fan, et al. (2023). Advancing Regular Language Reasoning in Linear Recurrent Neural Networks. arXiv. Grazzi, et al. (2024). Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues. arXiv. Merrill, et al. (2024). The Illusion of State in State-Space Models. arXiv. Sarrof, et al. (2024). The Expressive Capacity of State Space Models: A Formal Language Perspective. arXiv. Ebrahimi, et al. (2025). Revisiting Bi-Linear State Transitions in Recurrent Neural Networks. arXiv. Terzić, et al. (2025b). Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models. arXiv.

Reviewer_Ds6POpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

As the authors suggest, this paper considers a variation of latent recurrent reasoning models with two key properties: (i) the input vector is not fixed but changes to accommodate streaming inputs, and (ii) a persistent latent recurrent state is maintained across input steps by initializing the latent state for the next input with the final recurrent state from the previous input. The authors note that this formulation reduces to a standard RNN when T=1T=1. They experiment with two important properties of the recurrent backbone: (i) state-dependent mixing and (ii) nonlinear dimensional mixing, providing insights into the design of effective recurrent backbones. The authors demonstrate that their fast–slow formulation improves out-of-distribution (OOD) generalization in streaming tasks.

Critical strengths
  1. The paper is well-written, clearly organized, and easy to follow.
  2. The experiments are interesting and provide useful insights into the recurrent backbone properties that contribute to improved OOD generalization.
Critical weaknesses

1. Limited methodological novelty. The paper's main methodological contribution appears to be maintaining a persistent latent recurrent state across streaming inputs. While this formulation may be novel in the context of latent recurrent reasoning models, similar mechanisms have already been explored in the recurrent neural network literature. The authors note that their formulation reduces to a standard RNN when T=1T=1. However, prior work, such as Adaptive Computation Time for Recurrent Neural Networks (Graves, 2016), already considers multiple recurrent state updates for the same input (T>1T>1) while maintaining state across inputs. Could the authors clarify how their proposed formulation differs from this prior work and what constitutes its key methodological novelty?

The authors mention in Lines 67–68 that a key objective is to identify recurrent-core designs that support OOD generalization. Could the authors further clarify the main methodological contribution of this work, particularly how it extends beyond an empirical investigation of suitable recurrent architectures?

2.Could the authors provide a clearer explanation of the mechanisms underlying the observed OOD generalization improvements? Specifically, what are the relative contributions of persistent memory across observations and repeated latent updates within each observation? Since existing architectures already incorporate persistent memory (e.g., Transformer-XL) or repeated computation (e.g., looped Transformers), why are these mechanisms individually insufficient, and what makes their combination in FSRM more effective?

3. Computational cost versus OOD generalization gains. The proposed FSRM incurs substantially higher training costs than several baselines. Given the unclear relative contributions of persistent memory and repeated latent updates, are these additional costs justified by the OOD performance improvements? Specifically, if persistent memory is the primary contributor, how do the gains compare with the strongest persistent-memory baseline? Conversely, if repeated latent updates are the main contributor, are the improvements sufficient to justify the additional computation compared with the strongest loop-based baseline?

Questions for authors
  1. Motivation and role of recurrent-core properties. The paper identifies state-dependent transitions and nonlinear feature-wise mixing as important for OOD generalization. Could the authors clarify why these two properties were specifically selected and provide insight into their relationship with OOD generalization? Additionally, are these properties particularly beneficial within the proposed fast–slow formulation, or do they generally improve OOD generalization in recurrent architectures?
3 comments

Thank you for the positive assessment and the precise questions.

W1-1. "prior work, such as Adaptive Computation Time for Recurrent Neural Networks (Graves, 2016), already considers multiple recurrent state updates for the same input ... what constitutes its key methodological novelty?"

We did not intend to claim recurrence itself as the contribution. The paper states "Our contribution is not the introduction of recurrence per se" and discusses ACT.

ACT and our work answer different questions. ACT decides how many updates to apply to each input. It does not specify what is iterated and was applied to standard RNN and LSTM cells. We instead fix the number of updates and study what to iterate, asking which recurrent core lets a model trained on short sequences generalize to much longer ones. The iterated state also differs. It is a single vector in those RNN and LSTM cells, whereas every core we test updates a set of nn tokens Z∈Rn×dZ\in\mathbb{R}^{n\times d} and normalizes it after every update (Sec. 3, Eq. 4). Since ACT only decides when to stop, it can be combined with our model. We fix TT so that only the core differs between the models we compare, and we list the combination with adaptive halting as future work.

We make two contributions. First, we show that persistent fast–slow recurrence improves OOD length generalization on symbolic (Dyck), navigation (Local-Maze), and reinforcement-learning (Minigrid) tasks. Second, we identify two properties shared by the cores that generalize, namely (i) state-dependent transition and (ii) feature-wise mixing.

W1-2. "Could the authors further clarify the main methodological contribution of this work, particularly how it extends beyond an empirical investigation of suitable recurrent architectures?"

Our main methodological contribution is to provide a guideline for choosing the recurrent core. Among the cores we tested, those that generalize combine property (i) and property (ii). Beyond comparing architectures, we test each property separately and explain the role of each.

We changed one property at a time and kept the rest of the model the same. To change (ii), we added or removed the FFN. With the FFN, the LSTM core reaches 0.993 OOD accuracy on Dyck and the Mamba2 core reaches 0.861. To change (i), we compared attention-based cores with MLP-Mixer, which has (ii) but whose token mixing does not depend on the state. MLP-Mixer stays at 0.511 (Sec. 5.3).

Property (i) matters because the update is repeated. Within one observation the input is fixed, so repeating an update that is linear in the state is equivalent to one linear update (see Q1). With (i), the operator itself depends on the state, so each repetition adds computation. This predicts a smaller advantage when the update is applied only once, and a new ablation agrees. With five updates per observation, cores with both properties exceed the mean of the other three groups in Table 3 by 0.204 in OOD accuracy on Dyck, and with one update the gap is 0.045.

The benefit of (ii) is already known for recurrent models (Appendix A.1.2). We add that it appears mainly in cores that also have (i) (Table 3).

W2. "what are the relative contributions of persistent memory across observations and repeated latent updates ... why are these mechanisms individually insufficient ...?"

Both mechanisms contribute, and neither is sufficient alone. In the table below, adding repeated updates to a persistent state raises OOD accuracy from 0.743 to 0.939 on Dyck and from 0.306 to 0.612 on Local-Maze. For repeated updates only, the table gives the strongest loop-based baseline on each task.

OOD acc.Repeated updates onlyPersistent state only (T=1T{=}1)Both (T=5T{=}5)
Dyck (Table 9)0.500 (Looped Transformer)0.7430.939
Local-Maze (Table 2, Fig. 5)0.510 (TRM)0.3060.612

To see why each mechanism alone is insufficient, we trained the two architectures you mention on Dyck.

Transformer-XL has persistent memory and no repeated updates. Its memory stores the activations of a fixed number of recent tokens and is not rewritten by repeated updates. On Dyck, its OOD accuracy stays at 0.500, and it is also below FSRM on the Minigrid OOD environments (Fig. 5c–e).

The Looped Transformer has repeated updates and no persistent state. Because it carries no state across observations, it must take all past observations as input at every step. At test time this input is longer than any input seen in training. On Dyck, its OOD accuracy also stays at 0.500.

The combination applies the same learned update to a fixed-size state, so the computation per observation is the same at step 40 and at step 10410^4. We will add the Transformer-XL and Looped Transformer results on Dyck to the appendix of the final version.

W3. "are these additional costs justified by the OOD performance improvements? ... how do the gains compare with the strongest persistent-memory baseline? ... compared with the strongest loop-based baseline?"

The gain remains when the wall-clock training time is the same. In the paper, every model was trained for the same number of epochs, so FSRM used more training time than the baselines. We therefore trained each baseline for the same wall-clock time as FSRM on Dyck and Local-Maze and compared again.

DyckOOD acc. at the same wall-clock training time as FSRM
LSTM0.557
Mamba20.500
Transformer0.500
Looped Transformer0.502
FSRM0.999
Local-MazeOOD acc. at the same wall-clock training time as FSRM
LSTM0.095
Mamba20.165
Transformer0.292
Looped Transformer0.057
DeltaProduct0.290
TRM0.510
FSRM0.612

On Dyck, the baselines remain near 0.50 with the longer training, and the highest single run is one LSTM seed at 0.672. On Local-Maze, the best baseline is TRM at 0.510, below the 0.612 of FSRM.

These results also answer your two comparisons. The strongest persistent-memory baseline is LSTM on Dyck (0.557 against 0.999 for FSRM) and DeltaProduct on Local-Maze (0.290 against 0.612). The strongest loop-based baseline is the Looped Transformer on Dyck (0.502) and TRM on Local-Maze (0.510). We will add the Dyck comparison at equal training time to the appendix of the final version.

Q1. "Could the authors clarify why these two properties were specifically selected and provide insight into their relationship with OOD generalization? Additionally, are these properties particularly beneficial within the proposed fast–slow formulation, or do they generally improve OOD generalization in recurrent architectures?"

We chose these two properties because they are the two components of the block used by recent latent-reasoning models (the LRMs of Sec. 3.1). Attention is a state-dependent mixer across tokens, and the FFN is a mixer across features. All the cores we tested can be classified along these two axes (Table 1).

Property (i) matters for OOD generalization because of the repeated updates. Within one observation, the update is applied TT times to the same input. Suppose the update is linear in the state, R(Z,C)=A(C)Z+b(C)R(Z,C)=A(C)Z+b(C). Applying it TT times gives

Zs(T)=A(Cs)TZs(0)+∑k=0T−1A(Cs)k b(Cs).Z_s(T)=A(C_s)^T Z_s(0)+\textstyle\sum_{k=0}^{T-1}A(C_s)^k\,b(C_s).

This has the same form as one update, only with a different matrix and offset. A linear update repeated TT times can therefore do only what one linear update can do. With (i), the matrix is computed from the current state, so it is a different matrix at every update, and the TT updates cannot be merged into one. Repeated updates raise OOD accuracy (W2), so the property that makes them more than a single update also matters for OOD generalization.

The effect of (ii) appears mainly in cores that also have (i). On Dyck, cores with (i) have a mean OOD accuracy of 0.521 without (ii) and 0.726 with (ii). Cores without (i) have 0.512 without (ii) and 0.534 with (ii) (Table 3).

The two properties improve OOD accuracy mainly in the fast–slow formulation, where the update is repeated several times per observation. If the explanation above is right, their benefit should shrink when the update is applied only once, as in an ordinary recurrent model. We tested this by repeating the Dyck core ablation with one update per observation (T=1T{=}1). With T=5T{=}5, cores with both properties exceed the mean of the other three groups by 0.204 in OOD accuracy (Table 3). With T=1T{=}1, this gap is 0.045.

We hope these responses clarify the contribution, and we would be glad to discuss further.

Thank you again for your thoughtful review. As the discussion period is nearing its end, we would appreciate it if you could review our responses and additional results. We hope they address your concerns, and we would be grateful if you could reconsider your assessment.

Reviewer_pJ12OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the properties of out of distribution generalization of a number of models, and tries to understand which components of a fast-slow recurrent network are crucial for OOD generalization. The authors perform a number of experiments in a few different domains (such as Dyck language, Local-Maze, and Minigrid). Their results suggest, among other things, the importance of state-dependent transitions and weight-shared recurrent updates.

Critical strengths
  • The paper investigates genuinely interesting question of OOD generalization. In particular, the potential benefits of adding weight-shared LRM-style recurrence.
  • The paper is quite thorough in providing a large number of experiments and ablations.
Critical weaknesses

I very much struggled with how to evaluate this paper. On the one hand, the question it considers is interesting and potentially impactful. On the other hand, there are many small flaws and issues with experimental reporting and design that don't allow me to form a coherent picture of the contribution as a whole.

For an example of clearly strange experimental result, we can look at Figure 4. There, we can see that under the authors' experimental conditions, a reasonably capable LLM architecture (Qwen3 4B Base SFT) is struggling to learn a toy task even in distribution. This clearly suggests severe undertraining or some other issue with the training setup. The authors stress multiple times that they keep the training protocol fixed for different models, but this is not necessarily a fair decision. For example, when it comes to shared-weight vs non-shared weight comparison, the non-shared weight model is, of course, much more expressive, and might need more data/time to converge to a good solution. Around the line 1466 in the appendix, the authors provide a brief experiment showing that their model doesn't just get worse (and, in fact, gets better) with increased parameter count, but overall, I believe this matter was not given sufficient attention.

Again, on the same Figure 4, QWen shows a very strange generalization pattern with a second peak out of distribution, around 10^3 length. This makes me suspect that something strange is going on either in the data or in the experiment, but the authors don't investigate this issue. I am giving this not as a critical flaw, but as an example of the overall breadth over depth impression that the paper gives.

There are also many questions about the choice of baselines and tasks. For example, why was it valuable to evaluate algorithmically-prompted frontier LLMs in section 5.1? Such models already have completely different training pipelines and model sizes, and then when we add to that that one model is directly trained on the task while another is prompted, it begs the question of "what could we possibly conclude about the importance of fast-slow recurrence based on the outcomes of such a comparison?"

I think, unfortunately, a similar argument could be extended to many other experiments reported in the paper. For yet another example, in the recurrent core ablations/experiments, the authors, among other things, added an FFN wrapper to a large number of models, and observed that "although the value of adding a separate FFN depends on the task and on the core’s native inductive bias". In other words, that it sometimes helps and sometimes does not. I understand that adding an FFN relates to the idea of feature-mixing that authors posit as important. But the key question of the paper is about the recurrence structure, and it seems that this extra investigation muddles the focus of the paper and doesn't actually provide any additional strong insights.

Overall, I have a feeling that the paper spreads itself a little too thin and lacks a clear and coherent story. This also applies to the literature review part of the paper. While the authors list some major contribution, I did not feel that they covered the topic of generalization across sequence length in sufficient depth and hence did not fully characterize how their work fits into the broader context of existing research.

Unfortunately, because of these issues, at present, I can not recommend the paper for acceptance.

Questions for authors
  1. On lines 158-159, the authors say "holding Cs = C constant makes FSRM equivalent to continuing the static-input LRM recurrence across successive blocks of T updates". I am a little confused by this, as it seems that we can't hold Cs constant if we have a sequence of observations O_1... O_2. I.e. whether or not we can hold it constant doesn't seem to be an architectural decision, but rather something dictated by the nature of the problem being considered.

To clarify, this isn't a major flaw, I suspect it's just confusing phrasing, and I just would like to understand what they meant.

  1. Regarding language modeling experiments: "The comparison should be interpreted under the stated context policy: recurrent models have cross-chunk state, while the Transformer baseline has no cross-chunk attention or state." this seems like an extremely unfair setting for testing Transformer architectures. What was the motivation to do it like this?

  2. What were the stoppage criteria in different training setups for different models. I did not read the whole appendix, but I did try to find information on whether training was done to convergence, or whether the number of epochs was adjusted for parameter counts, and I couldn't.

3 comments

Thank you for the detailed review. Several of your points concern the coherence of the paper, so we first summarize its claim and then answer each point in order.

The paper makes one claim, namely that persistent fast–slow recurrence improves train-short, test-long generalization in streaming tasks, and that the improvement is largest for recurrent cores that combine state-dependent transitions (i) and feature-wise mixing (ii). Each experiment tests one part of this claim.

ClaimEvidence
Fast–slow recurrence outperforms one-step sequence models out of distribution (RQ1)Fig. 5
Repeated updates per observation raise OOD accuracy. At the same number of updates, a weight-shared loop matches or exceeds an unshared stack (RQ2, RQ3)Table 2, Table 10
The gain is largest for cores that combine (i) and (ii) (RQ4)Table 3, Fig. 6

W1. "(Qwen3 4B Base SFT) is struggling to learn a toy task even in distribution"

We apologize that Figure 4 gave this impression. On the training distribution, the fine-tuned Qwen reaches a token accuracy of 99.96%. We measured this on 1,000 newly sampled random Dyck-(30,5) strings of length 10–40 that are disjoint from the training set.

The shaded area of Figure 4 only marks the range of training lengths and is not an in-distribution evaluation. The curve is evaluated on OOD 1-regular runs, not on the random strings used for training (Sec. 4, Appendix B.6), and W2 explains why accuracy drops inside the shaded area as well. We added the training-distribution accuracy to Appendix B.6.2, and we will state in the caption of Figure 4 that all curves are evaluated on OOD 1-regular runs.

W2. "a very strange generalization pattern with a second peak out of distribution"

The second peak appears because Qwen's curve in Figure 4 shows per-position accuracy within a single 20,480-token input (Appendix B.6.2). Qwen is fine-tuned in a sequence-to-sequence format, in which it reads the whole input and then emits one prediction per position. For example, given the input ({[], it outputs )}]}, the closing bracket for the most recent unclosed bracket at each position. For Qwen, the position within the input and the input length are therefore different axes. Even the predictions at positions ≤ 40 are made after reading all 20,480 tokens, which is why accuracy drops inside the shaded area as well. When each input length is instead evaluated on separate inputs, Qwen's accuracy decreases monotonically.

Input length (1-regular runs)401606401,2802,5605,12010,24020,480
Qwen3-4B-Base SFT, acc. (%)99.9599.6899.2499.0998.2393.9959.4858.54

Within one long input, the curve is non-monotone because each output position must copy from the aligned input token, which lies about nn tokens back for an input of length nn. For large nn, the attention heads that perform this copy miss the correct token, and how often they miss varies with position. Such a dependence on the position of the relevant token is known for LLM retrieval from long contexts (Liu et al., 2024). On separate inputs, Qwen degrades more slowly than Figure 4 suggests, but it still degrades sharply beyond 5,120 tokens and falls below 60% at 10,240. We added this per-length evaluation and the explanation of the second peak to Appendix B.6.2. We will also state in the caption of Figure 4 that Qwen's curve shows per-position accuracy within a single input.

W3. "they keep the training protocol fixed ... not necessarily a fair decision ... the non-shared weight model ... might need more data/time to converge"

We checked this for both the baselines and the unshared stacks.

For the baselines (LSTM, Transformer, and Mamba2), we varied size, depth, initialization, learning rate, and training length on Dyck. None of the 237 configurations (711 runs) reaches a mean OOD accuracy of 0.95 over its 3 seeds (see our response to 4enc, W1), and training three times longer does not change this (see Q3).

For the unshared stacks, the learning rate is scaled with depth by the Depth-µP rule (Appendix C.2.2), because they performed worse without this scaling. Even so, with the same number of updates per observation, the weight-shared loop matches or exceeds the stack while using half the parameters.

OOD acc.Shared 1⊗51\otimes5Stack 5⊗15\otimes1
Dyck (Table 9)0.9390.595
Local-Maze (Table 2)0.6120.509
LavaCrossing S11N5 (Table 10)0.3610.340

We will add the baseline sweep and the longer-training runs to the appendix of the final version.

W4. "what could we possibly conclude about the importance of fast-slow recurrence based on the outcomes of such a comparison?"

We apologize that the paper did not make the purpose of this comparison clear. The comparison shows two things. First, strong general-purpose frontier models do not solve this task easily even when given the stack algorithm. At position 320 in Figure 4, the three LLMs reach 31–57%, in line with other studies in which frontier LLMs fail on long tasks even when given the algorithm (Shojaee et al., 2025; Sinha et al., 2025). Second, a small model with the fast–slow structure, trained only on lengths ≤ 40, generalizes beyond that range, as FSRM stays above 99.7% at every position up to 320.

Whether this generalization is due to the fast–slow structure is tested separately, by the comparison with baselines trained on the same data under the same protocol (Fig. 5) and by the ablations (Sec. 5.2–5.3). Appendix B.1 therefore describes the LLM comparison as a context-setting stress test, and we expanded it to state the purpose given above.

W5. "added an FFN wrapper ... it sometimes helps and sometimes does not ... muddles the focus of the paper and doesn't actually provide any additional strong insights"

The FFN ablation is not a separate investigation. OOD accuracy depends on which core is iterated, so the choice of core is one of the paper's questions (RQ4). The FFN ablation is the controlled test of property (ii) within that question, since adding or removing the FFN changes (ii) and nothing else.

The insight from this ablation is that feature-wise mixing (ii) raises OOD accuracy mainly when it is combined with state-dependent transitions (i). For cores with (i), adding (ii) raises the mean OOD accuracy by 0.135 on Local-Maze and by 0.205 on Dyck. For cores without (i), it adds only 0.035 and 0.022 (Table 3).

Mean OOD acc.without (ii)with (ii)
Local-Maze, without (i)0.2010.236
Local-Maze, with (i)0.4510.586
Dyck, without (i)0.5120.534
Dyck, with (i)0.5210.726

At the level of single cores, adding the FFN improves OOD accuracy or leaves it about the same (Fig. 6).

W6. "lacks a clear and coherent story ... I did not feel that they covered the topic of generalization across sequence length in sufficient depth"

On the story, the summary at the top of our first response states the paper's single claim and the experiment that tests each part.

On related work, we revised the first paragraph of Sec. 2 and added an extended discussion in Appendix D. Both cover empirical limits of Transformers, positional-encoding approaches, recurrent and looped computation, and state tracking in SSMs and linear RNNs, and state how our work relates to them.

Q1. "whether or not we can hold it [Cs] constant doesn't seem to be an architectural decision"

Thank you for pointing this out. You are right. The sentence describes a special case, not a design choice, and shows that FSRM contains the LRM. We revised it to "When the observation is static (Cs=CC_s = C for all ss), FSRM after ss observations equals the static-input LRM recurrence run for sTsT steps."

Q2. "the Transformer baseline has no cross-chunk attention or state ... an extremely unfair setting"

The setting follows the condition that the paper studies, where a model carries only a fixed-size state instead of the whole history. It is not meant to handicap the Transformer. All models are trained on 1,024-token sequences. The Transformer attends directly to all 1,024 tokens, whereas FSRM splits the same sequence into four 256-token segments and carries only a latent state of 8 tokens between segments. Within 1,024 tokens, FSRM therefore has less direct context than the Transformer. Beyond 1,024 tokens, only the recurrent models carry information across chunks, and this difference is what the experiment measures. We added these details to Appendix C.5.

Q3. "What were the stoppage criteria ... whether the number of epochs was adjusted for parameter counts"

All models are trained for the same fixed budget of 30 epochs for Dyck, 300 for Local-Maze, and 10610^6 environment steps for Minigrid (Table 4).

The budget is not adjusted for parameter count. On Dyck, training the baselines three times longer leaves their OOD accuracy near 0.50, which indicates that the budget is sufficient.

Dyck OOD acc.LSTMMamba2Transformer
30 epochs0.5000.5000.508
90 epochs0.5000.5000.515

On Local-Maze, longer training does not close the gap to FSRM either (see our response to Ds6P, W3).

Parameter counts are of the same order on Local-Maze and Minigrid, and for Dyck we also report a parameter-matched comparison (Table 5, Appendix C.7).

We hope these responses address your concerns and are glad to clarify further.

Thank you again for your thoughtful review. As the discussion period is nearing its end, we would appreciate it if you could review our responses and additional results. We hope they address your concerns, and we would be grateful if you could reconsider your assessment.

Comments on the paper

1 comment

General Response

We thank all reviewers for their careful and insightful reviews.

Reviewer pJ12 found that the paper "investigates genuinely interesting question of OOD generalization", in particular "the potential benefits of adding weight-shared LRM-style recurrence", and that it is "quite thorough in providing a large number of experiments and ablations".

Reviewers Ds6P and 4enc found the paper well-written and easy to follow, and Reviewer Ds6P found the experiments "interesting" with "useful insights into the recurrent backbone properties that contribute to improved OOD generalization".

Below we first answer the main concerns shared across the reviews and then summarize what we provided in the rebuttal. Revisions to the paper are marked in blue.

Concern 1. Robustness to hyperparameters and training budget (4enc, pJ12, Ds6P)

The conclusion that the baselines do not reach the OOD accuracy of FSRM holds across these choices. We swept size, depth, initialization, training length, and number of loops, with the same six learning rates for every model (672 configurations, 2,016 runs). None of the 618 baseline configurations reaches a mean OOD accuracy of 0.95 over 3 seeds, whereas 7 of the 54 FSRM configurations do. Training the baselines three times longer leaves their Dyck OOD accuracy near 0.50. At the same wall-clock training time as FSRM, the best baseline reaches 0.557 against 0.999 for FSRM on Dyck, and 0.510 against 0.612 on Local-Maze.

Concern 2. Contribution beyond recurrence itself (Ds6P, 4enc)

Our contribution is not recurrence itself but a guideline for choosing what to iterate. The cores that generalize combine two properties. (i) State-dependent transition means that the operator applied to the latent state is computed from the state itself, as in attention or gating. (ii) Feature-wise mixing means that the update mixes the feature dimensions of each token, as an FFN does. The effect of (ii) is large mainly in cores that also have (i) (Table 3). Property (i) matters because the update is repeated on a fixed input. A transition that is linear in the state, or that depends only on the input, merges into one linear map when repeated, whereas a state-dependent transition does not. Consistent with this, the gap between cores with both properties and the other cores is 0.204 with five updates per observation and 0.045 with one. Nonlinearity alone is not sufficient, as MLP-Mixer, whose token mixing does not depend on the state, stays at 0.511.

Concern 3. Roles of persistent state and repeated updates (Ds6P)

Both are needed. Adding repeated updates to a persistent state raises OOD accuracy from 0.743 to 0.939 on Dyck and from 0.306 to 0.612 on Local-Maze. On Dyck, Transformer-XL (persistent memory only) and the Looped Transformer (repeated updates only) stay at 0.500.

Concern 4. Evaluation setup (4enc, pJ12)

The Dyck OOD set uses 1-regular runs because longer random strings do not contain longer dependencies. At length 2,560, 99% of openers close within 47 steps. Results for length generalization alone are in Appendix C.1.1. On strings whose bottom opener stays open under random content of depth up to 3, FSRM recalls the opener with 84–100% accuracy and the baselines with at most about 1%. In Figure 4, the fine-tuned Qwen reaches 99.96% on the training distribution, and its second peak appears because the curve shows per-position accuracy within one 20,480-token input. The LLM comparison is a context-setting stress test, and the effect of the fast–slow structure is tested by the baselines and ablations.

Our rebuttal actions

In the rebuttal, we added the following experiments.

  1. A sweep over size, depth, initialization, training length, number of loops, and six learning rates for every model (672 configurations, 2,016 runs)
  2. Training the baselines three times longer
  3. Training the baselines for the same wall-clock time as FSRM on Dyck and Local-Maze
  4. Transformer-XL and the Looped Transformer on Dyck
  5. The core ablation with one update per observation
  6. Dyck strings that separate the dependency distance from the regular pattern
  7. Qwen evaluated on the training distribution and for each input length

The revised PDF adds Appendix D on length generalization and updates Sec. 2, Sec. 3, and Appendices A.1.2, B.1, B.6.2, and C.5.

In the final version, we will add the Dyck results of experiments 1–6 to the appendix, clarify the caption of Figure 4, and discuss the references suggested by Reviewer 4enc in Appendix D.

We hope this summary is helpful for the final evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.