- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies length generalization in linear recurrences. It shows that sufficient hidden-state coverage and small initialization ensure arbitrary-length generalization, and proposes that slowly learned hidden directions explain why longer sequences become accurate later in training.
Critical strengths
The question of length generalization in recurrences is interesting. As far as I can assess, the authors seem to provide an involved framework to tackle the problem.
Critical weaknesses
The paper uses technical language that makes it hard to follow without the necessary mathematical background. While I understand the difficulty of striking the correct balance between taking a reader along without starting from zero. It should provide the reader with an intuition for the method, which it currently does not.
Questions for authors
The authors might want to consider making the work more digestible, f.e. by accompanying different steps with plots that convey an intuition.
3 comments
I want to thank the reviewers for the rebuttal. The overview does improve the readability of the paper.
However, I have remaining concerns about the presentation. Starting with the introduction, the opening motivates recurrence through its ability to apply learned computation across lengths. But processing variable-length inputs is also possible with Transformers, where grokking-like length generalization has been reported [1]. Although the paper cites broader related work, it should explain more clearly what is specific to recurrence and how its proposed mechanism complements existing work on grokking and length generalization.
Such positioning work would help resolve my general concern about readability and clarity of the paper at hand. I maintain my weak rejection recommendation.
[1] Zhou et al., Transformers Can Achieve Length Generalization But Not Robustly
Thank you for your follow-up and for pointing us to Zhou et al. (2024). We agree that the original manuscript did not sufficiently distinguish our work from existing studies on length generalization and grokking. We have revised the Introduction (p. 2) to clarify the following two points.
1. What is specific to recurrence?
While both Transformers and recurrent models can exhibit length generalization and grokking, they face different challenges when processing longer sequences. Standard Transformers can use attention to retrieve information directly from earlier tokens. Recurrent models, in contrast, compress past information into a fixed-dimensional hidden state and repeatedly apply a shared transition rule.
This repeated composition creates a challenge: two transition rules can produce nearly identical outputs on short sequences but behave very differently on longer ones. Errors that are invisible to the decoder during short rollouts may become observable after further recurrent steps. Moreover, because these errors have little effect on the training loss, they can be learned very slowly. Recurrence thus provides a natural setting for studying how short-horizon fitting, long-horizon error propagation, and optimization dynamics interact to produce grokking.
2. How does our mechanism complement existing work on grokking and length generalization?
Existing studies on Transformer length generalization have examined data formats and positional encodings (Zhou et al., 2024), algorithmic simplicity (Zhou et al., 2024), and whether a length-generalizing solution can be identified from finite-length training data (Huang et al., 2025; Izzo et al., 2026). For recurrent models, Buitrago and Gu (2025) highlight the importance of hidden-state coverage, while Van Rossem and Saxe (2025) study delayed length generalization through the emergence of a generalizing state representation.
As stated in our revised Introduction (p. 2), these results raise two related questions that have not yet been fully resolved: When does training on finite sequences recover a length-generalizing recurrent rule? And why can learning this rule take much longer than fitting short sequences? We establish a quantitative connection between hidden-state coverage, the learning rates of residual error modes, and the resulting grokking delay. This connection also motivates a principled augmentation strategy to accelerate length generalization.
More broadly, our findings suggest a new direction for studying length generalization in other architectures, including Transformers: identifying modes that are weakly constrained at training lengths but become important at longer lengths, and understanding how their learning dynamics affect the emergence of generalization.
We appreciate the reviewer’s thoughtful feedback on the clarity and presentation of our work. The comments have helped us communicate the main ideas of our paper more effectively. On the rebuttal:
Response to W1:
We thank the reviewer for this feedback. We agree that the original presentation was too technical and did not clearly explain the main ideas. We have revised the presentation to make the story easier to follow. In particular, we have added an overview figure (Figure 1, p. 2) to illustrate the main mechanism of our paper. It connects hidden-state augmentation, learning dynamics, and length generalization, and provides a roadmap for the theoretical results and practical implications that follow.
Our paper starts with a simple question: Can an RNN trained only on short sequences learn to handle much longer ones?
We first study when this is possible. The key idea is that the training data must be rich enough for the RNN to learn the underlying rule, rather than just perform well on short sequences. We show that, under suitable conditions, learning from short sequences can indeed lead to generalization to arbitrarily long ones.
But being able to generalize does not mean that generalization happens immediately. An RNN may already perform almost perfectly on short sequences while still making large errors on longer ones. Only after further training does its performance on long sequences improve. This is the length grokking phenomenon we aim to explain.
Why does this delay happen? Our main insight is that some hidden directions are learned much more slowly than others. Their errors may be almost invisible on short sequences, but become much more noticeable when the same process is repeated over many steps. As a result, the RNN can appear to have learned the task on short sequences while still needing more time to generalize to longer ones.
Finally, this explanation suggests a practical way to reduce the delay: provide better coverage of these hidden directions during training. We develop an augmentation method based on this idea and show experimentally that it can substantially reduce the observed delay.
Together, these results explain when length generalization is possible, why it can be delayed, and how that delay can be reduced.
Response to Q1:
As suggested by the reviewer, we have added an overview figure (Figure 1, p. 2) to make the main ideas of our paper easier to understand. The figure explains how different training strategies affect length generalization.
1. Normal training (first row). When we train an RNN on short sequences, some hidden-state directions receive little training signal. The model can learn to fit short sequences well, but it may fail to generalize to longer ones.
2. Random augmentation (second row). Adding random perturbations to the hidden states allows the model to learn previously invisible directions. However, some directions are still learned slowly. The remaining errors may be small on short sequences but become much more noticeable on longer ones. This is why the model needs additional training time to generalize.
3. Well-conditioned augmentation (third row). This observation suggests a natural way to reduce the delay. Instead of choosing augmentation directions randomly, we design them to cover the hidden-state space more evenly. The previously slow directions can then be learned faster, allowing the model to generalize earlier.
To summarize, hidden-state coverage helps explain whether length generalization is possible, while its conditioning helps explain how quickly it occurs. Our theory connects these two aspects and suggests a practical way to improve length generalization.