acceptodds
Under review as a conference paper at ICLR 2027

From Memorization to Length Generalization: Grokking in Recurrent Models

Abstract

Recurrent models can in principle process arbitrarily long sequences by reusing the same state transition, but fitting short sequences does not guarantee that the learned transition will generalize to longer ones. We study this gap in an overparameterized linear RNN trained on sequential modular addition. We show that zero training loss alone is insufficient for length generalization because decoder-invisible hidden directions can remain unconstrained. In contrast, sufficient hidden-state coverage forces every zero-loss solution to implement the correct reusable recurrence. Combined with a small-initialization convergence result, this gives a tractable setting where exact arbitrary-length generalization can be characterized. We then explain why length generalization may emerge much later than short-horizon fitting. Near a generalizing solution, the delay is controlled by slowly decaying training modes and by their visibility at different rollout horizons. For decoder-invisible directions, we derive the local curvature and link the slowest learning rate to the conditioning of hidden-state coverage. This provides a quantitative mechanism for length grokking: recurrent errors can become negligible on short rollouts while remaining significant on longer ones. Our results connect hidden-state coverage, optimization dynamics, and the emergence of reusable recurrent computation from finite-horizon data.

Primary area:
learning theory
Keywords:
length generalization, grokking, recurrent neural networks, memorization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AcMSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies length generalization in linear recurrences. It shows that sufficient hidden-state coverage and small initialization ensure arbitrary-length generalization, and proposes that slowly learned hidden directions explain why longer sequences become accurate later in training.

Critical strengths

The question of length generalization in recurrences is interesting. As far as I can assess, the authors seem to provide an involved framework to tackle the problem.

Critical weaknesses

The paper uses technical language that makes it hard to follow without the necessary mathematical background. While I understand the difficulty of striking the correct balance between taking a reader along without starting from zero. It should provide the reader with an intuition for the method, which it currently does not.

Questions for authors

The authors might want to consider making the work more digestible, f.e. by accompanying different steps with plots that convey an intuition.

3 comments

We appreciate the reviewer’s thoughtful feedback on the clarity and presentation of our work. The comments have helped us communicate the main ideas of our paper more effectively. On the rebuttal:

Response to W1:

We thank the reviewer for this feedback. We agree that the original presentation was too technical and did not clearly explain the main ideas. We have revised the presentation to make the story easier to follow. In particular, we have added an overview figure (Figure 1, p. 2) to illustrate the main mechanism of our paper. It connects hidden-state augmentation, learning dynamics, and length generalization, and provides a roadmap for the theoretical results and practical implications that follow.

Our paper starts with a simple question: Can an RNN trained only on short sequences learn to handle much longer ones?

We first study when this is possible. The key idea is that the training data must be rich enough for the RNN to learn the underlying rule, rather than just perform well on short sequences. We show that, under suitable conditions, learning from short sequences can indeed lead to generalization to arbitrarily long ones.

But being able to generalize does not mean that generalization happens immediately. An RNN may already perform almost perfectly on short sequences while still making large errors on longer ones. Only after further training does its performance on long sequences improve. This is the length grokking phenomenon we aim to explain.

Why does this delay happen? Our main insight is that some hidden directions are learned much more slowly than others. Their errors may be almost invisible on short sequences, but become much more noticeable when the same process is repeated over many steps. As a result, the RNN can appear to have learned the task on short sequences while still needing more time to generalize to longer ones.

Finally, this explanation suggests a practical way to reduce the delay: provide better coverage of these hidden directions during training. We develop an augmentation method based on this idea and show experimentally that it can substantially reduce the observed delay.

Together, these results explain when length generalization is possible, why it can be delayed, and how that delay can be reduced.

Response to Q1:

As suggested by the reviewer, we have added an overview figure (Figure 1, p. 2) to make the main ideas of our paper easier to understand. The figure explains how different training strategies affect length generalization.

1. Normal training (first row). When we train an RNN on short sequences, some hidden-state directions receive little training signal. The model can learn to fit short sequences well, but it may fail to generalize to longer ones.

2. Random augmentation (second row). Adding random perturbations to the hidden states allows the model to learn previously invisible directions. However, some directions are still learned slowly. The remaining errors may be small on short sequences but become much more noticeable on longer ones. This is why the model needs additional training time to generalize.

3. Well-conditioned augmentation (third row). This observation suggests a natural way to reduce the delay. Instead of choosing augmentation directions randomly, we design them to cover the hidden-state space more evenly. The previously slow directions can then be learned faster, allowing the model to generalize earlier.

To summarize, hidden-state coverage helps explain whether length generalization is possible, while its conditioning helps explain how quickly it occurs. Our theory connects these two aspects and suggests a practical way to improve length generalization.

I want to thank the reviewers for the rebuttal. The overview does improve the readability of the paper.

However, I have remaining concerns about the presentation. Starting with the introduction, the opening motivates recurrence through its ability to apply learned computation across lengths. But processing variable-length inputs is also possible with Transformers, where grokking-like length generalization has been reported [1]. Although the paper cites broader related work, it should explain more clearly what is specific to recurrence and how its proposed mechanism complements existing work on grokking and length generalization.

Such positioning work would help resolve my general concern about readability and clarity of the paper at hand. I maintain my weak rejection recommendation.

[1] Zhou et al., Transformers Can Achieve Length Generalization But Not Robustly

Thank you for your follow-up and for pointing us to Zhou et al. (2024). We agree that the original manuscript did not sufficiently distinguish our work from existing studies on length generalization and grokking. We have revised the Introduction (p. 2) to clarify the following two points.

1. What is specific to recurrence?

While both Transformers and recurrent models can exhibit length generalization and grokking, they face different challenges when processing longer sequences. Standard Transformers can use attention to retrieve information directly from earlier tokens. Recurrent models, in contrast, compress past information into a fixed-dimensional hidden state and repeatedly apply a shared transition rule.

This repeated composition creates a challenge: two transition rules can produce nearly identical outputs on short sequences but behave very differently on longer ones. Errors that are invisible to the decoder during short rollouts may become observable after further recurrent steps. Moreover, because these errors have little effect on the training loss, they can be learned very slowly. Recurrence thus provides a natural setting for studying how short-horizon fitting, long-horizon error propagation, and optimization dynamics interact to produce grokking.

2. How does our mechanism complement existing work on grokking and length generalization?

Existing studies on Transformer length generalization have examined data formats and positional encodings (Zhou et al., 2024), algorithmic simplicity (Zhou et al., 2024), and whether a length-generalizing solution can be identified from finite-length training data (Huang et al., 2025; Izzo et al., 2026). For recurrent models, Buitrago and Gu (2025) highlight the importance of hidden-state coverage, while Van Rossem and Saxe (2025) study delayed length generalization through the emergence of a generalizing state representation.

As stated in our revised Introduction (p. 2), these results raise two related questions that have not yet been fully resolved: When does training on finite sequences recover a length-generalizing recurrent rule? And why can learning this rule take much longer than fitting short sequences? We establish a quantitative connection between hidden-state coverage, the learning rates of residual error modes, and the resulting grokking delay. This connection also motivates a principled augmentation strategy to accelerate length generalization.

More broadly, our findings suggest a new direction for studying length generalization in other architectures, including Transformers: identifying modes that are weakly constrained at training lengths but become important at longer lengths, and understanding how their learning dynamics affect the emergence of generalization.

Reviewer_GJK6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an overparameterized linear RNN trained on sequential modular addition, focusing on length generalization and its delayed emergence during training. With a fixed encoder and decoder, the authors augment the initial hidden states with balanced Gaussian perturbations in decoder-invisible directions. They prove that sufficiently small initialization yields convergence to zero training loss, while full hidden-state coverage ensures that every zero-loss solution generalizes exactly to all finite sequence lengths.

The authors then propose a local spectral explanation of delayed length generalization: weakly constrained parameter modes decay slowly, and a delay arises when these modes affect long-horizon predictions more strongly than short-horizon predictions. At a canonical solution, they calculate the relevant curvature exactly and relate it to the smallest eigenvalue of the Gaussian augmentation covariance. The resulting explanation of grokking at actual trained endpoints remains conditional on additional assumptions, and the stated spectral-delay theorem requires stronger control of its asymptotic remainder.

Critical strengths
  1. A precise sufficient condition for arbitrary-length generalization. Theorem 3.3 proves that, with decoder-consistent sources spanning the hidden space, zero training loss is equivalent to for every action. This identity guarantees correctness for every initial task state and every finite action sequence.

  2. An exact connection between augmentation geometry and local training curvature. At the canonical solution, Theorem 5.1 derives the invisible-to-visible block’s Hessian eigenvalues, independently of training horizon. This rigorously distinguishes sufficient coverage from good conditioning: full-rank augmentation can still leave directions with arbitrarily small local learning rates.

Critical weaknesses

The paper’s explanation of grokking has three limitations:

  1. The delay formula describes a mechanism but does not establish why it should arise. A substantial delay requires a slowly learned mode that affects long sequences much more than short ones. The paper identifies slow directions at a canonical solution, but does not generally prove that training produces a mode with both properties.

  2. The asymptotic result may not explain the observed delay. Theorem 4.1 considers error thresholds approaching zero, when training occurs sufficiently late. At practical thresholds, faster modes with larger amplitudes may still dominate. Applying the theorem to measured grokking therefore requires showing that the slowest visible mode dominates before those thresholds are crossed.

  3. The proof of Theorem 4.1 needs stronger remainder control. Equation (12) makes the trajectory remainder negligible relative to the slowest active parameter mode. If that mode is evaluation-invisible, the remainder may still dominate the faster visible mode used in the proof on Proof the Theorem 4.1 when κmin<κmem\kappa_{min} < \kappa_{mem}. The proof needs to show that the evaluation-visible remainder is negligible relative to the selected visible mode.

Questions for authors
  1. What are the sufficient conditions for gj(T)≫mj(L)g_j(T) \gg m_j(L)?

  2. Could you provide more empirical evidence that Theorem 4.1 remains informative at finite thresholds?

  3. Could you fix the issue in the proof of Theorem 4.1?

2 comments

We appreciate the reviewer’s thoughtful comments, which have helped us strengthen our theoretical analysis.

Regarding our rebuttal, we find that W1 and Q1, W2 and Q2, and W3 and Q3 are closely related, respectively. We therefore address each pair together.

Part I: Response to W1 and Q1

Response to W1

We thank the reviewer for pointing out this gap. We agree that identifying a slow direction at the canonical solution does not show that actual training will produce it.

To address this, we added a new trajectory result (Theorem C.1, pp. 29–30; Appendix C, pp. 29–33). It establishes explicit conditions under which actual training produces a slowly learned component that has a much stronger effect on long sequences than on short ones. Consider a full-coverage source pool whose smallest covariance eigenvalue μ1\mu_1 is simple. When κ=μ1/d\kappa = \mu_1/d is smaller than both the slowest A-sector rate and the next B-sector rate, we prove that Gaussian initialization activates the corresponding weak B-subspace along the actual gradient-flow trajectory. This activation follows from training.

We also show that this slow component can have a much stronger effect on longer sequences. For any amplification factor r>1r > 1 and δ>0\delta > 0, every sufficiently large fixed test horizon TT satisfies

P(QTQL>r)≥1−δ,\mathbb{P}\left(\frac{Q_T}{Q_L} > r\right)\geq 1-\delta,

where QKQ_K measures the leading horizon-KK output response of the training-induced component.

Thus, under explicit conditions, actual training produces a component that is both slowly learned and strongly amplified over longer horizons. The sufficient conditions for horizon amplification are discussed further in our response to Q1.

Response to Q1

We agree that it is meaningful to provide an explicit sufficient condition for gj(T)≫mj(L)g_j(T)\gg m_j(L). The key condition is that the perturbations caused by a mode accumulate coherently over time. More precisely, after aligning the output responses with the target transitions, suppose there exist a unit vector vv and constants ℓj≠0\ell_j\neq 0 and βj\beta_j such that

E⟨v,D~j,t⟩=tℓj−βj,\mathbb{E}\langle v,\widetilde{D}_{j,t}\rangle=t\ell_j-\beta_j,

where D~j,t\widetilde{D}_{j,t} is the aligned first-order output response at step tt.

Under this condition, we prove that

gj(T)≥1T∑t=1T(tℓj−βj)2.g_j(T)\geq\frac{1}{T}\sum_{t=1}^{T}(t\ell_j-\beta_j)^2.

Thus, gj(T)g_j(T) grows at least quadratically with TT. For any factor M>1M > 1, sufficiently large TT guarantees gj(T)>Mmj(L)g_j(T) > M m_j(L).

The interpretation for this condition is that recurrent networks reuse the same parameters at every step. If the errors introduced by a mode remain aligned across steps, they do not cancel each other out. Their effect can therefore be much stronger on long sequences than on short ones.

We have added this sufficient condition and its proof to Appendix B.2 (Proposition B.1, p. 27).

Continuing our response in Part I, we address the remaining two pairs of concerns and questions: W2/Q2 and W3/Q3.

Part II: Response to W2/Q2 and W3/Q3

Response to W2

We thank the reviewer for this important observation. We agree that a slow mode does not necessarily dominate at practical thresholds, since faster modes may still have larger amplitudes.

We now identify sufficient conditions under which this concern can be ruled out. In particular, our new Theorem C.2 (p. 30) shows that when the error threshold is sufficiently small relative to the initialization scale, faster modes become negligible before the threshold is crossed. Theorem C.2 (p. 30) shows that, for

τε=εp,p>4λfλf−κ,\tau_\varepsilon=\varepsilon^p,\qquad p > \frac{4\lambda_f}{\lambda_f-\kappa},

where ε\varepsilon is the initialization scale, τε\tau_\varepsilon is the error threshold, κ\kappa is the weak-mode decay rate, and λf>κ\lambda_f > \kappa is the smallest competing decay rate, the training-induced weak component dominates the output errors near both threshold-crossing times.

We also tested whether the mechanism remains informative at finite thresholds. At τ=10−5ε4\tau=10^{-5}\varepsilon^4, the original single-mode delay formula achieves median relative prediction errors of 16.6% for h=3h=3 and 13.4% for h=5h=5, across 240 gradient-descent runs.These results support the relevance of the asymptotic mechanism to observed grokking. They do not establish slow-mode dominance at every practical threshold. We provide further finite-threshold evidence in our response to Q2.

The theoretical result is given in Appendix C (pp. 29–33) (Theorem C.2, p. 30), and the numerical results are reported in Appendix B.3 (p. 28).

Response to Q2

We conducted additional experiments to test whether Theorem 4.1 (p. 7) can predict the observed grokking delay at finite error thresholds.

First, as discussed in our response to W2, our additional experiments show that the single-mode delay formula remains informative at finite thresholds.

Second, we fixed the initialization scale at ε=0.2\varepsilon=0.2 and tested five hidden dimensions (h=3,5,7,9,11h=3,5,7,9,11) under continuous gradient flow. As we decreased the error threshold, the prediction generally became more accurate. For example, at τ=10−8\tau=10^{-8}, the median errors ranged from 5.72% to 20.87%. At τ=10−16\tau=10^{-16}, they decreased to 1.36%–14.73%.

These results show that Theorem 4.1 (p. 7) remains informative at finite thresholds, and its predictions generally improve as the threshold decreases.

The additional results are reported in Appendix B.3 (p. 28).

Response to W3/Q3

We thank the reviewer for identifying this gap. We agree that Eq. (12) (p. 7) alone does not provide sufficient control of the evaluation-visible remainder.

We have revised Theorem 4.1 (p. 7) by adding two conditions, while keeping Eq. (12) (p. 7) and the original delay formula.

First, we require R⋆R_\star to be a regular zero-loss endpoint, where the zero-loss set is locally smooth and the training Hessian has positive curvature in its normal directions.

Second, we require

κmem<2κmin⁡,κgen<2κmin⁡.\kappa_{\mathrm{mem}} < 2\kappa_{\min},\qquad \kappa_{\mathrm{gen}} < 2\kappa_{\min}.

Here κmin⁡\kappa_{\min} is the slowest active parameter decay rate, while κmem\kappa_{\mathrm{mem}} and κgen\kappa_{\mathrm{gen}} are the slowest active rates visible at the two evaluation horizons.

The intuition is simple. The nonlinear remainder decays at least at the rate 2κmin⁡2\kappa_{\min}, up to a polynomial factor. Our new conditions ensure that the selected evaluation-visible modes decay more slowly. Therefore, the remainder cannot dominate these modes, even if the globally slowest parameter mode is evaluation-invisible.

Under these conditions, Theorem 4.1 (p. 7) gives

ΔtL,T(τ)=12(1κgen−1κmem)log⁡1τ+log⁡Agen2κgen−log⁡Amem2κmem+o(1).\Delta t_{L,T}(\tau)=\frac{1}{2}\left(\frac{1}{\kappa_{\mathrm{gen}}}-\frac{1}{\kappa_{\mathrm{mem}}}\right)\log\frac{1}{\tau}+\frac{\log A_{\mathrm{gen}}}{2\kappa_{\mathrm{gen}}}-\frac{\log A_{\mathrm{mem}}}{2\kappa_{\mathrm{mem}}}+o(1).

We provide the revised theorem in Section 4.1 (p. 7) and the complete evaluation-level remainder proof in Appendix B.1 (pp. 26–27; Theorem B.1, p. 26).

Reviewer_w8G7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates conditions for length generalization of overparameterized linear RNNs on a modular addition task. The authors establish that zero training loss guarantees arbitrary-length generalization under sufficient hidden-state coverage, achieved through a decoder-consistent augmentation of the training data. Moreover, they investigate the phenomenon of length grokking, where short-horizon training errors become negligible long before the model generalizes to longer sequences. They provide a local spectral explanation for this delay, relating slowly decaying training modes and their different effects on short- and long-horizon predictions to the conditioning of hidden-state coverage.

Critical strengths

The authors provide a careful mathematical analysis of length generalization and grokking. The results are conceptually interesting and provide theoretical insight, particularly through the connection between hidden-state coverage, local training curvature, and convergence rates. The broader problem is important and of high practical relevance.

Critical weaknesses
  • The results are derived in a very restricted setting. The paper lacks concrete implications of the theory for designing or training recurrent networks with improved length generalization. Translating the theoretical results and their conditions into practical recommendations does not seem straightforward.

  • The theoretical explanation of length grokking is conditional on additional assumptions. Theorem 4.1 relies on (12), the connection of Theorem 5.1 to actual training trajectories requires Assumption 5.2. Although the authors provide supporting numerical evidence in their considered setting, the generality of the proposed mechanism remains unclear.

Questions for authors
  • Can the identified relationship between hidden-state coverage, conditioning, and convergence times be exploited in more realistic recurrent architectures?

  • In a similar spirit: Theorem 5.1 and the subsequent discussion around (16) suggest that the training speed is controlled by the augmentation vectors gig_i. The authors propose to draw these randomly from a Gaussian distribution. However, given their direct influence on hidden-state coverage and training rates, would it be more meaningful to select them deterministically, so that GGTGG^T is well-conditioned with a sufficiently large smallest eigenvalue? This would provide a design principle (and also avoid the "almost surely'" phrases from the results). Moreover, an experiment comparing Gaussian augmentation with deterministic, well-conditioned augmentation could demonstrate whether deliberately improving hidden-state coverage reduces the observed grokking delay.

  • Theorems 3.1 and 3.2 are not uniform in dd, hh, and LL. Can you provide an estimate of δ\delta, quantifying its dependence on the problem dimensions and potentially the source pool? This is would also be relevant in Corollary 3.1 and could yield a design recommendation for initializing the parameters.

2 comments

We thank the reviewer for the constructive feedback and insightful questions. Below, we address each of the weaknesses and questions in detail.

Weaknesses Part

Response to W1:

We agree that our theoretical analysis is conducted in a simplified recurrent setting. However, even in linear RNNs, length grokking is nontrivial to analyze, as it involves gradient-flow dynamics, the interaction between different hidden-state directions, and long-horizon error amplification. Our simplified setting allows us to isolate and rigorously study this mechanism in a tractable way.

Importantly, while our formal analysis is restricted, the design implications extend beyond the specific linear parameterization. For nonlinear RNNs, weakly observed hidden-state directions can be identified locally through decoder Jacobians, while recurrent Jacobians describe how errors propagate over longer horizons. This provides a concrete way to study similar mechanisms in more realistic architectures.

Our theory also leads to a practical design principle: improving the coverage of weakly constrained hidden-state directions. We developed a deterministic tight-frame augmentation and proved its optimality under a fixed energy budget in our setting. To examine its broader applicability, we further tested this idea in a nonlinear tanh RNN. Across 64 paired runs, tight-frame augmentation achieved length-50 generalization in 64/64 runs, compared with 48/64 for energy-matched Gaussian augmentation. The restricted mean grokking delay also decreased from 1043.8 to 133.4 training steps (2,500-step budget; Appendix E.6, Table 1, p. 42; Section 6, p. 10).

These results suggest that our analysis can inform practical training strategies beyond the simplified model.

Response to W2:

We agree that Theorem 4.1 (p. 7) and the connection of Theorem 5.1 (p. 8) to training trajectories rely on additional conditions.

The broader insight, however, does not require the exact spectral formulas of our model. Recurrent networks share two essential properties: parameters are reused across time, and training on short sequences does not necessarily constrain their behavior over longer horizons. This creates the possibility that slowly learned hidden-state directions contribute disproportionately to long-horizon errors. Our theory provides a rigorous characterization of this mechanism in one setting.

As discussed in W1, the nonlinear RNN experiment further supports the practical relevance of improving hidden-state coverage beyond our theoretical architecture. While this does not establish that the identical spectral mechanism governs all nonlinear RNNs, it provides evidence that this design principle is not limited to our setup.

Questions Part

Response to Q1:

Yes, the identified relationship can be exploited to improve length generalization in more realistic recurrent architectures. We conducted additional experiments with a conventional nonlinear tanh RNN to confirm that the relationship between hidden-state coverage, conditioning, and convergence time extends beyond our linear setting. We have the following observations.

First, we observed length grokking in the nonlinear RNN. The model learned the short training sequences before generalizing to longer sequences.

Second, hidden-state coverage strongly affects convergence time. We varied the source covariance condition number across 1, 3, 10, and 30, while keeping the total augmentation energy fixed. The restricted mean length-50 convergence time increased from 206.7 to 307.5, 807.5, and 1305.0 steps, respectively.

Finally, improving coverage substantially reduced the grokking delay. Across 64 paired runs, tight-frame augmentation reduced the restricted mean delay from 1043.8 to 133.4 steps compared with energy-matched Gaussian augmentation. The length-50 generalization success rate also increased from 48/64 to 64/64.

These results show that our design principle can also improve length generalization in nonlinear RNNs, even though our formal guarantees are limited to the linear setting (Appendix E.6, pp. 40–43).

Response to Q2:

We thank the reviewer for this insightful suggestion. Yes, deliberately improving hidden-state coverage through deterministic augmentation can substantially reduce the observed grokking delay.

Motivated by this suggestion, we developed a tight-frame augmentation. Instead of sampling augmentation directions randomly, we choose them to cover the decoder-invisible subspace evenly, so that GG⊤=nIkGG^\top = nI_k. We also prove that this construction maximizes the smallest eigenvalue of the source covariance under a fixed augmentation-energy budget.

We compared tight-frame augmentation with Gaussian and energy-matched Gaussian augmentation across 64 paired runs. At τ=10−6\tau = 10^{-6}, the median grokking delay dropped from 346.5 steps for energy-matched Gaussian augmentation to just 1.46 steps for the tight frame.

These results support the reviewer’s proposed design principle. We have added the construction and theoretical justification to Section 6 (p. 10; Training Implications: Well-Conditioned Augmentation), the experimental comparison to Figure 4 (p. 10), and the full experimental setup to Appendix E.2 (pp. 36–37).

Response to Q3:

We have derived explicit initialization bounds for both Theorems 3.1 (p. 4) and 3.2 (p. 5).

For reset-only training (Theorem 3.1, p. 4), we obtain

δreset(d,L)=eLe−IL1+MLTd,LeL.\delta_{\mathrm{reset}}(d,L)=\frac{e_L e^{-I_L}}{1+M_L T_{d,L}e_L}.

For training with source augmentation (Theorem 3.2, p. 5), we obtain

δaug(d,L,Σu)=eLe−IL1+(7+5∥Σu∥op)L3Td,LeL.\delta_{\mathrm{aug}}(d,L,\Sigma_u)=\frac{e_L e^{-I_L}}{1+(7+5\lVert\Sigma_u\rVert_{\mathrm{op}})L^3T_{d,L}e_L}.

Here all constants are explicitly defined. The first bound depends on dd and LL, while the second also depends on the source covariance.

These bounds also give an initialization recommendation for Corollary 3.1 (p. 6). For Gaussian initialization with (Ra(0))ij∼N(0,σR2/h)(R_a(0))_{ij}\sim\mathcal{N}(0,\sigma_R^2/h), choosing

σR<h δaugχdh2,1−η2\sigma_R<\frac{\sqrt{h}\,\delta_{\mathrm{aug}}}{\sqrt{\chi^2_{dh^2,1-\eta}}}

guarantees convergence with probability at least 1−η1-\eta. With full source coverage, this further guarantees exact length generalization.

What could be improved is that these bounds are conservative sufficient conditions, rather than optimal initialization scales. Full proofs are provided in Appendices A.2 (pp. 13–19) and A.4 (pp. 19–24).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.