acceptodds
Under review as a conference paper at ICLR 2027

A theory of noisy linear recurrent neural network learning dynamics

Abstract

Biological neural networks are noisy: their dynamics involve difficult-to-predict fluctuations which are likely due not just to chaos, but also to a variety of low-level biophysical processes. Despite the omnipresence of intrinsic noise, its theoretical consequences are incompletely understood, especially in the context of learning. For example, does noise meaningfully shape representation learning, or is it better viewed as a nuisance factor whose presence or absence changes little about what solution is found and when? Here, we argue that intrinsic noise can significantly modulate representation learning in recurrent neural networks (RNNs), since it can strongly break solution-space degeneracies and in some cases control whether a non-degenerate solution exists at all. We show that this is true for an analytically tractable model of noisy RNN learning with a linear activation function and white noise inputs. We find that, in the so-called rich regime (where weights are initialized with small values), noise strongly modulates the directions in which the read-in, readout, and recurrent weights align. Moreover, we find that noise produces a kind of phase transition: for sufficiently high noise, a non-degenerate solution is not even optimal if parameter scales are limited (e.g., via regularization). These results extend recent analytic work on RNN representation learning and show that noise can act as a fundamental determinant of representations.

Primary area:
learning theory
Keywords:
recurrent neural network, noise, stochasticity, dynamics, theory, linear model, learning, representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mogLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a theory of noisy linear RNN learning dynamics under a simplified setting and show that noise can decisively shape representation learning. The key result is that in the regime where the weights are initialized with small values, noise strongly modulates the directions in which the read-in, readout, and recurrent weights align. Moreover, noise can produce a "phase transition": for sufficiently high noise, only a trivial solution is learned.

Critical strengths
  • Overall the paper is well written and the mathematical derivations seem to be correct.
  • The findings and detailed analysis on how noise can shape representation learning and lead to the phase transition behavior are interesting.
Critical weaknesses
  • The setting studied is relatively simple and the assumptions are overly restrictive. For instance, the target filter is assumed to be exponential, the RNN update equation is assumed to be linear in the state, input and noise, all of which are not realistic in practical settings.
  • The paper shows that noise can decisively shape representation learning in the simple linear RNN setting but does not provide empirical results to test whether similar claims also hold for nonlinear RNNs and the more realistic settings.
  • It is not clear what are the practical gains from the studies on the simple RNN setting. There are no detailed analysis of the implications of the findings on downstream tasks such as next-value prediction and classification for time series data. The paper also seems to be better suited for a physics journal than ICLR.
Questions for authors
  • Would similar findings/claims still hold for linear RNNs with different structures, e.g. the linear antisymmetric RNNs [1] plus noise?
  • What are the implications of the findings on downstream tasks such as next-value prediction for real-world time series data and chaotic dynamical system data?
  • In addition to shaping the representation learned, how would the noise and its magnitude affect trainability? How are the impacts of noise on trainability related to those on representation learning?

[1] Moe, S., Remonato, F., Grøtli, E. I., & Gravdahl, J. T. (2020, July). Linear antisymmetric recurrent neural networks. In Learning for Dynamics and Control (pp. 170-178). PMLR.

5 comments
AuthorsOpenReview ↗

Rebuttal (part 1)

Thank you for your close reading of our paper and for your helpful comments. We hope to convincingly address your concerns below, but please let us know if we fall short of doing so!

On the simplicity of the setting

The setting studied is relatively simple and the assumptions are overly restrictive. For instance, the target filter is assumed to be exponential, the RNN update equation is assumed to be linear in the state, input and noise, all of which are not realistic in practical settings.

This is absolutely a good point, and it is something we try to explicitly highlight in the last paragraph of the Discussion. We do indeed make a large number of simplifying assumptions. In our view, this is unfortunate, but also necessary given what we are trying to accomplish. Our goal is not to construct a state-of-the-art model or to qualitatively analyze RNNs through experiments. Rather, we aim to construct a specific kind of mathematically detailed explanation for learning dynamics: we aim to characterize, in a mathematically rigorous way, full learning dynamics trajectories and key qualitative features of them (e.g., whether there is a trivial or nontrivial solution given the noise level).

We are heavily inspired by Bordelon et al. (https://arxiv.org/abs/2503.18754) and Proca et al. (https://proceedings.mlr.press/v267/proca25a.html), and a large body of prior literature (e.g., the Saxe et al. characterizations of learning dynamics in linear feedforward networks; https://arxiv.org/abs/1312.6120) that tries to accomplish something similar. Because it's very hard to exactly solve the ODEs associated with neural network learning dynamics, all of these past papers had to make strong simplifying assumptions. While our assumptions are strong, we think this is typical of previous work that proceeds along similar lines.

It is not clear what are the practical gains from the studies on the simple RNN setting. ... The paper also seems to be better suited for a physics journal than ICLR.

We agree that the practical relevance of our work is not immediately clear. In our defense, the same is true for the above tradition of mathematical work on learning dynamics: given various strong assumptions, it is sometimes unclear what one learns about 'realistic' scenarios. However, in our view, the big-picture idea is to use these simple settings to build an understanding of how to model and think about learning dynamics. The hope is that concepts learned by analyzing simple, tractable networks can eventually be used to understand bigger and more complex networks. Or it may be possible to leverage these concepts to speed up training, something that has arguably proven true in work related to scaling laws and the μ\muP parameterization.

Finally, we note that fixing the scope to linear networks (and not simulating any nonlinear networks) is not unusual for this type of theoretical work. Bordelon et al. and Proca et al. (the latter of which was published in ICML) both do this. We think this is just a matter of effort: although the settings of these papers are limited, it's a lot of work to mathematically characterize them thoroughly.

AuthorsOpenReview ↗

Rebuttal (part 2)

(continued)

Numerical validation

The paper shows that noise can decisively shape representation learning in the simple linear RNN setting but does not provide empirical results to test whether similar claims also hold for nonlinear RNNs and the more realistic settings.

There are no detailed analysis of the implications of the findings on downstream tasks such as next-value prediction and classification for time series data.

What are the implications of the findings on downstream tasks such as next-value prediction for real-world time series data and chaotic dynamical system data?

We think that it is likely that certain qualitative insights survive in the nonlinear setting---in particular, networks should preferentially compute with non-noisy state space directions, and too much noise pushes networks into a 'trivial' regime---and are currently checking these claims in very simple experiments. On the other hand, we do not expect that our exact expressions for post-alignment dynamics hold anymore, since these are more closely tied to our specific setting. We will add brief comments on the nonlinear setting to the revised version.

Also, we have now made a serious effort to compare our theory with numerical BPTT experiments. We are happy to report that we have now conducted extensive experiments with moderate-sized linear RNNs (N≤128N \leq 128). Our high-level conclusion is that our theory is correct within its domain of validity. See the very end of the updated paper PDF for some illustrative (but still not 100% polished) figures.

  • 1. BPTT agrees with the gradient descent on the analytic objective in the large-batch, small-time-step limit. One reason linear RNNs are nice is that we can avoid doing BPTT on the original objective JJ (Eq. 6) and instead perform gradient descent on a more explicit equivalent objective that doesn't require BPTT (Eq. 7), i.e., which doesn't require differentiating through time steps. While this is mathematically true, is it numerically true? Yes, at least when one expects: when the batch size BB per SGD step is taken to be very large (∼128\sim 128 is generally high enough) and the time discretization Δt\Delta t is small enough. Here, we used Δt=0.1\Delta t = 0.1 and 0.050.05. These values should be understood relative to the 'overall' time scale set by the target leaky integration rate, which we set to 11 in these experiments.

  • 2. Our theory works given small weight initializations. We compared gradient descent on the analytic objective (Eq. 7) with our theory's predictions. (We can also do this with BPTT and the Eq. 6 objective, but training networks this way is much slower.) We find that SGD empirically matches our predictions provided parameter initializations are small. In particular, there is an alignment phase followed by effectively low-dimensional dynamics.

  • 3. Phase transitions occur where the theory predicts. The previous two points illustrate that our theory works when its assumptions hold. As expected, there are also phase transitions where we predict.

In the process of doing this validation, we also learned other interesting things:

  • Training with SGD and a fixed learning rate is generally extremely unstable (or takes a prohibitively long time). To make training stable, we had to use an adaptive learning rate; without this trick, we couldn't get BPTT-based training to work at all. In this specific setting, we know a lot about the mathematical structure of the objective and its gradients, so we could use our math to periodically approximate the Hessian and use it to decide whether the learning rate should be made larger or smaller. To be conservative, we set the learning rate to be proportional to 1/(largest eigenvalue).
  • Using too-large initial weights makes our theory still qualitatively work (e.g., there is an alignment phase), but the precise dynamics are a little different from what the theory predicts, probably because some interactions with the non-aligned parameters can survive and affect dynamics.
  • Near a phase transition (i.e., noise too high), training really slows down and requires many more epochs for convergence. This is empirically annoying, but it's consistent with our theoretical insight that the objective gets much flatter (see, e.g., Fig. 5c).
  • Something interesting happens if BPTT with a small-to-moderate (less than 128128, say) batch size is used for training. Noise associated with using small batches can cause updates to WW that are not aligned with the dominant rank-one component. This can actually cause the alignment between WW and uu (and vv) to steadily go down after initially becoming high during the alignment phase. The effect completely goes away if either the analytically simplified (Eq. 7), non-BPTT-requiring objective is used, or if there's a small amount of regularization on WW.
AuthorsOpenReview ↗

Rebuttal (part 3)

(continued)

On noise and trainability

In addition to shaping the representation learned, how would the noise and its magnitude affect trainability? How are the impacts of noise on trainability related to those on representation learning?

This is something we address in the original paper, although we will think about how to communicate it more clearly in the revised version. In Sec. 4 and Appendix C of the paper, we characterize the initial 'alignment phase' of dynamics. During this phase, u\mathbf{u}, v\mathbf{v}, and W\mathbf{W} align in the direction of the least-noisy eigenvector of S\mathbf{S}, the matrix that controls the RNN's intrinsic noise.

One result we derive indicates how the convergence time during this phase depends on the noise matrix S\mathbf{S}. Eq. 14 shows that, as the amount of noise gets larger, training really slows down. One way of stating this fact is that these linear RNNs get harder to train, since they require many more epochs for convergence.

For example, during our numerical experiments reproducing the phase transition discussed in Sec. 6 and Fig. 5 (discussed in part 2 above), we found that

  • high noise levels close to the phase transition required a large number of training epochs to converge (~ 30k);
  • lower noise levels, which are farther from the phase transition, required a much smaller number of training epochs to converge (~ 3k).

Hence, we find that noise has consequences for both representation learning (parameters align in the least noisy directions) and trainability (training takes much longer as the noise level gets higher). Importantly, this is true even in the limit of infinite-sized batches; these conclusions apply to the noise-averaged, simplified objective (Eq. 7).

Linear asymmetric RNNs

Would similar findings/claims still hold for linear RNNs with different structures, e.g. the linear antisymmetric RNNs [1] plus noise?

This is a great question. You've inspired us to think about two possible extensions: to symmetric matrices (Wij=WjiW_{i j} = W_{j i}), and to antisymmetric matrices (Wij=−WjiW_{i j} = - W_{j i}).

Both constraints yield slightly different gradient descent updates for W\mathbf{W} (see Eq. 8 from Sec. 2). If we use FijF_{i j} to denote the unconstrained right-hand side of dWij/dτdW_{i j}/d\tau, in the symmetric case this becomes dWijdτ=Fij+Fji\frac{d W_{i j}}{d\tau} = F_{i j} + F_{j i} and in the antisymmetric case it becomes \frac{d W_{i j}}{d\tau} = F_{i j} - F_{j i} \ In the symmetric case, the alignment-phase ODE for WijW_{i j} is nearly the same as its unconstrained counterpart. The answer turns out almost the same, which is unsurprising since W\mathbf{W} of the form bbT\mathbf{b} \mathbf{b}^T (where b\mathbf{b} denotes the alignment direction) is already symmetric. The subsequent dynamics are the same as before. Hence, choosing WW to be symmetric doesn't change much.

Meanwhile, using an antisymmetric W\mathbf{W} yields a very different answer, since bbT\mathbf{b} \mathbf{b}^T is not antisymmetric. In fact, the lowest-order contribution to alignment-phase W\mathbf{W} dynamics actually vanishes here because of the antisymmetry constraint, and we need to approximate the right-hand side of dWij/dτdW_{ij}/d\tau using more terms. We end up getting both a different alignment phase (whose result is that W\mathbf{W} has rank ≥2\geq 2 rather than rank one) and different subsequent dynamics. On the other hand, this shouldn't be that surprising, since antisymmetric matrices can't have rank one.

A more thorough examination of this phenomenon would probably be out of scope. An antisymmetric linear network can't even solve our task, since the eigenvalues of W\mathbf{W} can only be imaginary (and the task requires developing a specific real eigenvalue). In any case, we will comment on these generalizations in the revised Discussion (and possibly a new appendix if we run out of space). We will also cite the interesting paper you mentioned.

Reviewer_2aQVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the role of colored noise in training linear RNNs, and generalizes an earlier derivation by Bordelon et al, 2025, using a simple linear filtering task. The authors show that noise selects noise-cancelling directions early in the alignment phase of learning. They additionally show how noise induces phase transitions in learning leading to amplitude collapse.

Critical strengths
  • Interesting extension of the theory of learning by Bordelon et al to the noisy case.

  • The role of noise in breaking the symmetry and inducing early-phase alignment appears new and interesting per se.

Critical weaknesses
  • The wording in the abstract about the role of noise is suggestive but maybe a bit too strong for the limited toy problem the authors studied (this limitation is acknowledged by the authors in the Conclusion section).

  • There seems to be no numerical check, parameters are not given for some plots, the theory should also be validated with bptt in the presence of sampled noise.

  • The recurrent weights are not regularized explicitly. That would break some of the symmetries, so the specific role of noise vs weight regularization should be clarified further.

  • There seems to be a direct connection of the noise-averaged loss and the noise-regularization equivalence [Chris M. Bishop. Training with Noise is Equivalent to Tikhonov Regularization. Neural Computation, 7(1):108–116], which is not properly discussed.

Questions for authors
  • A check with discrete-time bptt with sampled noise is in order: do you observe the same dynamics at early times?

  • How does alignment depend on the structure of the S (or better G) matrix?

2 comments
AuthorsOpenReview ↗

Rebuttal (part 1)

Thank you for your close reading of our paper and for your helpful comments. We hope to convincingly address your concerns below, but please let us know if we fall short of doing so!

Numerical validation

There seems to be no numerical check, parameters are not given for some plots, the theory should also be validated with bptt in the presence of sampled noise.

A check with discrete-time bptt with sampled noise is in order: do you observe the same dynamics at early times?

It is indeed (unfortunately) true that the original submission did not involve extensive numerical validation. We took for granted the theoretical picture provided by Bordelon et al. and Proca et al., and found that it held in various small-scale numerical experiments as we prepared the original version; this is what led us to focus on mathematical results. However, we absolutely agree that such validation would bolster our arguments and lend credence to our theory.

We are happy to report that we have now conducted extensive numerical experiments with moderate-sized linear RNNs (N≤128N \leq 128). Our high-level conclusion is that our theory is correct within its domain of validity. See the very end of the updated paper PDF for some illustrative (but still not 100% polished) figures. As a side note, we will also add parameter details to the paper to make reproducibility easier.

  • 1. BPTT agrees with the gradient descent on the analytic objective in the large-batch, small-time-step limit. One reason linear RNNs are nice is that we can avoid doing BPTT on the original objective JJ (Eq. 6) and instead perform gradient descent on a more explicit equivalent objective that doesn't require BPTT (Eq. 7), i.e., which doesn't require differentiating through time steps. While this is mathematically true, is it numerically true? Yes, at least when one expects: when the batch size BB per SGD step is taken to be very large (∼128\sim 128 is generally high enough) and the time discretization Δt\Delta t is small enough. Here, we used Δt=0.1\Delta t = 0.1 and 0.050.05. These values should be understood relative to the 'overall' time scale set by the target leaky integration rate, which we set to 11 in these experiments.

  • 2. Our theory works given small weight initializations. We compared gradient descent on the analytic objective (Eq. 7) with our theory's predictions. (We can also do this with BPTT and the Eq. 6 objective, but training networks this way is much slower.) We find that SGD empirically matches our predictions provided parameter initializations are small. In particular, there is an alignment phase followed by effectively low-dimensional dynamics.

  • 3. Phase transitions occur where the theory predicts. The previous two points illustrate that our theory works when its assumptions hold. As expected, there are also phase transitions where we predict.

In the process of doing this validation, we also learned other interesting things:

  • Training with SGD and a fixed learning rate is generally extremely unstable (or takes a prohibitively long time). To make training stable, we had to use an adaptive learning rate; without this trick, we couldn't get BPTT-based training to work at all. In this specific setting, we know a lot about the mathematical structure of the objective and its gradients, so we could use our math to periodically approximate the Hessian and use it to decide whether the learning rate should be made larger or smaller. To be conservative, we set the learning rate to be proportional to 1/(largest eigenvalue).
  • Using too-large initial weights makes our theory still qualitatively work (e.g., there is an alignment phase), but the precise dynamics are a little different from what the theory predicts, probably because some interactions with the non-aligned parameters can survive and affect dynamics.
  • Near a phase transition (i.e., noise too high), training really slows down and requires many more epochs for convergence. This is empirically annoying, but it's consistent with our theoretical insight that the objective gets much flatter (see, e.g., Fig. 5c).
  • Something interesting happens if BPTT with a small-to-moderate (less than 128128, say) batch size is used for training. Noise associated with using small batches can cause updates to WW that are not aligned with the dominant rank-one component. This can actually cause the alignment between WW and uu (and vv) to steadily go down after initially becoming high during the alignment phase. The effect completely goes away if either the analytically simplified (Eq. 7), non-BPTT-requiring objective is used, or if there's a small amount of regularization on WW.
  • In the paper, we assumed no WW regularization since this slightly simplifies the math. In practice, when using BPTT and SGD, such regularization seems necessary since it can really improve the stability of training (without changing the answer much if the regularization coefficient is small).
AuthorsOpenReview ↗

Rebuttal (part 2)

(continued)

On regularization of the recurrent weights

The recurrent weights are not regularized explicitly. That would break some of the symmetries, so the specific role of noise vs weight regularization should be clarified further.

This is a good point. To address this concern, we will do two things. First, we will explicitly comment on the effect of weight regularization in a new appendix section. Second, we have done experiments using recurrent weight regularization (see above and updated paper PDF).

Theoretically, weight regularization does not qualitatively change our paper's story. There is still an alignment phase and the post-alignment dynamics are still described by a reduced objective. Recurrent weight regularization κW2∥W∥F2\frac{\kappa_W}{2} \Vert \mathbf{W} \Vert_F^2 contributes a term κW2(γ−c)2\frac{\kappa_W}{2} (\gamma - c)^2 to the reduced objective which pushes cc toward the leak rate γ\gamma. However, the downside of including recurrent weight regularization is that various math bits (e.g., determining where a phase transition occurs) become intractable. Hence, it doesn't qualitatively change the story, but it does prevent some analytic derivations.

Experimentally, something more interesting happens (mentioned in one of the bullet points above). Without recurrent weight regularization, when using BPTT and finite batches, batch-associated noise can accumulate in W\mathbf{W}, which can gradually reduce alignment after the alignment phase. With this regularization, this effect goes away, since increasing these components (which are orthogonal to the readout) does not improve performance, but does increase the norm of W\mathbf{W}. Hence, finite-batch simulations and our theory are actually more likely to agree when recurrent weight regularization is used.

Alignment and noise

How does alignment depend on the structure of the S (or better G) matrix?

This is something we address in the original version of the paper, although we could be more explicit if you prefer. There are two main effects (see Sec. 4 and Appendix C):

  • The system aligns in the direction of the least noisy eigenvectors of S\mathbf{S}. (That is, u\mathbf{u} and v\mathbf{v} become parallel to that subspace, and W\mathbf{W} becomes approximately rank-one.) G\mathbf{G} only matters through determining S\mathbf{S}. If there are multiple eigenvectors of S\mathbf{S} with the same (lowest) eigenvalue, the initial conditions of u\mathbf{u} and v\mathbf{v} fix the particular alignment direction.
  • Convergence time depends on the eigenvalues of S\mathbf{S} according to a specific formula (see Eq. 14). In particular, when S\mathbf{S} is noisier, convergence can really slow down.

Other concerns

There seems to be a direct connection of the noise-averaged loss and the noise-regularization equivalence [Chris M. Bishop. Training with Noise is Equivalent to Tikhonov Regularization. Neural Computation, 7(1):108–116], which is not properly discussed.

Thanks for the pointer to this paper; we have read it and will discuss it in the revised version. We do comment in a few places on the idea of 'noise as a regularizer' (in the second paragraph of the Introduction and the third paragraph of the Discussion), but agree that we could cover this material in a more thorough way. We plan to expand the Discussion to do this. Incidentally, a core component of the paper is very much in the spirit of this Bishop paper: we show that training with noise (Eq. 6) is equivalent to training with a certain deterministic objective (Eq. 7) which includes a regularization-like term.

The wording in the abstract about the role of noise is suggestive but maybe a bit too strong for the limited toy problem the authors studied (this limitation is acknowledged by the authors in the Conclusion section).

We added the qualifier "linear" to every mention of "RNN" in the abstract, which hopefully makes the scope of our claims clearer. We think the abstract is otherwise accurate---for example, although our task is far from the most general one we could consider, it's a sort of 'proof of principle' that noise can shape representations in a major way---but are happy to make additional changes if you think doing so is important.

Reviewer_DvxEOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores the effects of intrinsic noise on the learning dynamics and learned representation of an analytically tractable, noisy linear RNN. It extends Bordelon et al., 2025 to consider the case of (anisotropic) Gaussian white noise added into the RNN hidden state and shows mathematically that this leads to a number of interesting consequences: (i) in the rich regime, the input and output directions align along the least noisy direction in state-space (ii) without regularization, noise introduces asymmetry such that the read-in weights are driven to have maximal magnitude while the read-out weights are driven to have minimal magnitude (iii) with regularization, there are phase transitions associated with the magnitude of noise - in particular, beyond a critical noise level, the optimal solution collapses to a trivial one where output noise is minimized at the cost of not learning the desired filter.

Critical strengths

The paper is a detailed mathematical analysis of noisy linear RNN learning dynamics, where the main results are derived algebraically and the phase transition is shown to exist beyond an analytically derived critical noise level σcrit2\sigma_{\text{crit}}^2 rather than just observed empirically. The paper contains complete derivations of all results.

The presentation is extremely clear and the main ideas, such as that of noise breaking the "balancing" assumption, are explained very well.

Critical weaknesses

The main weakness is that this work could be seen as relatively incremental over Bordelon et al., 2025, in that some of the results follow naturally from the setup - for example, since noise is added in the hidden state, it is not too surprising that an unregularized network is driven to maximise the magnitude of uu, since that trivially increases the signal-to-noise ratio in the hidden state.

Another minor issue is that this work is presented as mostly theoretical and so it is unclear to what extent RNNs in practice follow the learning dynamics predicted by this paper. If the authors have performed simulations to verify this, a small section on the setup and results would strengthen the paper.

Questions for authors

I have only a few questions:

  1. For your main results, do you have empirical observations from simulations that show that your predicted optimal solutions are actually achieved? Does the phase transition occur at the predicted critical noise threshold in practice?
  2. Do the main results hold under different applications of noise? For example, would you expect significantly different learning dynamics if the noise was instead applied in the inputs rather than in the hidden state?
2 comments
AuthorsOpenReview ↗

Rebuttal (part 1)

Thank you for your close reading of our paper, for your helpful comments, and for your support. We hope to convincingly address your concerns below, but please let us know if we fall short of doing so!

Numerical validation

Another minor issue is that this work is presented as mostly theoretical and so it is unclear to what extent RNNs in practice follow the learning dynamics predicted by this paper. If the authors have performed simulations to verify this, a small section on the setup and results would strengthen the paper.

For your main results, do you have empirical observations from simulations that show that your predicted optimal solutions are actually achieved? Does the phase transition occur at the predicted critical noise threshold in practice?

It is indeed (unfortunately) true that the original submission did not involve extensive numerical validation. We took for granted the theoretical picture provided by Bordelon et al. and Proca et al., and found that it held in various small-scale numerical experiments as we prepared the original version; this is what led us to focus on mathematical results. However, we absolutely agree that such validation would bolster our arguments and lend credence to our theory.

We are happy to report that we have now conducted extensive numerical experiments with moderate-sized linear RNNs (N≤128N \leq 128). Our high-level conclusion is that our theory is correct within its domain of validity. See the very end of the updated paper PDF for some illustrative (but still not 100% polished) figures.

  • 1. BPTT agrees with the gradient descent on the analytic objective in the large-batch, small-time-step limit. One reason linear RNNs are nice is that we can avoid doing BPTT on the original objective JJ (Eq. 6) and instead perform gradient descent on a more explicit equivalent objective that doesn't require BPTT (Eq. 7), i.e., which doesn't require differentiating through time steps. While this is mathematically true, is it numerically true? Yes, at least when one expects: when the batch size BB per SGD step is taken to be very large (∼128\sim 128 is generally high enough) and the time discretization Δt\Delta t is small enough. Here, we used Δt=0.1\Delta t = 0.1 and 0.050.05. These values should be understood relative to the 'overall' time scale set by the target leaky integration rate, which we set to 11 in these experiments.

  • 2. Our theory works given small weight initializations. We compared gradient descent on the analytic objective (Eq. 7) with our theory's predictions. (We can also do this with BPTT and the Eq. 6 objective, but training networks this way is much slower.) We find that SGD empirically matches our predictions provided parameter initializations are small. In particular, there is an alignment phase followed by effectively low-dimensional dynamics.

  • 3. Phase transitions occur where the theory predicts. The previous two points illustrate that our theory works when its assumptions hold. As expected, there are also phase transitions where we predict. (See the end of the updated PDF.)

In the process of doing this validation, we also learned other interesting things:

  • Training with SGD and a fixed learning rate is generally extremely unstable (or takes a prohibitively long time). To make training stable, we had to use an adaptive learning rate; without this trick, we couldn't get BPTT-based training to work at all. In this specific setting, we know a lot about the mathematical structure of the objective and its gradients, so we could use our math to periodically approximate the Hessian and use it to decide whether the learning rate should be made larger or smaller. To be conservative, we set the learning rate to be proportional to 1/(largest eigenvalue).
  • Using too-large initial weights makes our theory still qualitatively work (e.g., there is an alignment phase), but the precise dynamics are a little different from what the theory predicts, probably because some interactions with the non-aligned parameters can survive and affect dynamics.
  • Near a phase transition (i.e., noise too high), training really slows down and requires many more epochs for convergence. This is empirically annoying, but it's consistent with our theoretical insight that the objective gets much flatter (see, e.g., Fig. 5c).
  • Something interesting happens if BPTT with a small-to-moderate (less than 128128, say) batch size is used for training. Noise associated with using small batches can cause updates to WW that are not aligned with the dominant rank-one component. This can actually cause the alignment between WW and uu (and vv) to steadily go down after initially becoming high during the alignment phase. The effect completely goes away if either the analytically simplified (Eq. 7), non-BPTT-requiring objective is used, or if there's a small amount of regularization on WW.
AuthorsOpenReview ↗

Rebuttal (part 2)

(continued)

Input noise

Do the main results hold under different applications of noise? For example, would you expect significantly different learning dynamics if the noise was instead applied in the inputs rather than in the hidden state?

This is a great question. Our first thought was that the distinction between input and intrinsic noise here might just be 'semantic'. After all, the input is assumed to be white noise, so in some sense it is just another noise term. Is input noise really so different from intrinsic noise?

Perhaps surprisingly, the answer is that they are substantially different. This is a really interesting conceptual point, and we plan to add it to the Discussion (and possibly to an appendix if we run out of space).

The key difference between the terms is that the input term couples to the read-in vector u\mathbf{u} (since it is of the form x(t)ux(t) \mathbf{u}), whereas the intrinsic noise term does not. By an argument analogous to the one we use to derive our simplified objective (Eq. 7), assuming that input includes an additive random component (x(t)→x(t)+σξ(t)x(t) \to x(t) + \sigma \xi(t)) yields an objective function term ∫0∞e−2γt2vTeWt(noise covariance)eWTtv dt .\int_0^{\infty} \frac{e^{- 2 \gamma t}}{2} \mathbf{v}^T e^{\mathbf{W} t} (\text{noise covariance}) e^{\mathbf{W}^T t} \mathbf{v} \ dt \ . But here the noise covariance is σ2uuT\sigma^2 \mathbf{u} \mathbf{u}^T, so this new term equals σ2∫0∞e−2γt2vTeWtuuTeWTtv dt=σ2∫0∞12f(t)2 dt .\sigma^2 \int_0^{\infty} \frac{e^{- 2 \gamma t}}{2} \mathbf{v}^T e^{\mathbf{W} t} \mathbf{u} \mathbf{u}^T e^{\mathbf{W}^T t} \mathbf{v} \ dt = \sigma^2 \int_0^{\infty} \frac{1}{2} f(t)^2 \ dt \ . In other words, the new term only depends on the parameters through f(t)f(t)! This term affects optimization quite differently from the intrinsic-noise-associated term, because it does not produce the same kind of representational pressure: what is important is the input-output function rather than how that function is computed. By a slightly more general argument, this also appears to be true if the system receives a vector of inputs rather than just a scalar input, so this is not just a consequence of having scalar inputs.

As a final comment, we think the situation is also different for non-white input noise, but this extension seems (mathematically) harder to deal with.

Other concerns

The main weakness is that this work could be seen as relatively incremental over Bordelon et al., 2025, in that some of the results follow naturally from the setup - for example, since noise is added in the hidden state, it is not too surprising that an unregularized network is driven to maximise the magnitude of uu, since that trivially increases the signal-to-noise ratio in the hidden state.

We agree that we owe a lot to Bordelon et al. for the general formulation and the two-phase approach to describing learning dynamics. We also agree that, in hindsight, the insight that the system is driven to compute with 'less noisy' state space directions is not that surprising. We think the primary virtue of our work is that we make this insight quantitatively precise---for example, by identifying the precise direction of alignment (which can also depend on initialization) and when phase transitions occur. This quantitative exploration has led to some surprises. We didn't expect, before we started, that there would be multiple distinct types of phase transitions here.

Comments on the paper

2 comments
AuthorsOpenReview ↗

A summary of some of the discussion so far

Thank you to all of the reviewers for engaging carefully with our work. We feel that your comments and suggestions have tremendously improved the paper, and we are exceedingly grateful for your time and energy.

In case it is helpful, we would like to summarize some of the discussion so far.

Perceived strengths and weaknesses

All reviewers appear to agree that the paper is clear, well-written, and mathematically correct. Furthermore, all reviewers seem to agree that our work is both novel and interesting---especially, according to their comments, the content related to phase transitions.

(We promise that we are not exaggerating here. "Clear", "well written", "correct", "novel", and "interesting" are all direct quotes!)

The main perceived weaknesses appear to be related to scope, significance, and numerical validation rather than (for example) correctness. More precisely:

  • Reviewers DvxE and 2aQV wanted some numerical validation for our mathematical results on noisy linear RNNs.
  • Reviewer mogL wonders what an analysis of linear RNNs teaches us about more 'realistic' RNNs, which have nonlinear architectures and are trained on more complex tasks. This is a scope and significance issue.
  • Reviewers 2aQV and mogL think that the setting is simple. This is also a scope and significance issue.

There are also more minor concerns, like questions about extensions of our work. DvxE asked whether input noise behaves differently than intrinsic noise (it does! see our reply to them), 2aQV asked about the consequences of regularizing the recurrent weights (we explored this both theoretically and numerically; see our reply to them), and mogL asked about antisymmetric linear RNNs (they behave quite differently; see our reply to them).

We are happy to report that, over the course of the discussion period, we have more or less addressed most of these issues.

On numerical validation

This is our biggest update relative to the original submitted version. We put a lot of work into efficient simulating many moderate-sized (N∼128N \sim 128) linear RNNs and comparing their empirical behavior during training to what our theory predicts. See our reviewer responses, and the end of the appendices in the revised PDF, for more details; in short, our theory works and agrees with these new numerical experiments. These experiments also help clarify the scope of the theory. When some of its assumptions are violated (e.g., the initial parameter values are not small enough), there is some disagreement with the theory. Some results (e.g., alignment along one direction) are more robust to violations of our assumptions than others (e.g., precisely following the predicted dynamics of the reduced objective).

We think that this convincingly addresses the concerns of reviewers DvxE and 2aQV on this front.

On extensions of our work

As mentioned above, reviewers asked various questions regarding extensions of our results. We answered questions related to input noise vs intrinsic noise, the consequences of recurrent weight regularization, how alignment depends on the structure of SS and GG, how noise affects trainability (it decreases it), and how antisymmetric linear RNNs would be different than unconstrained linear RNNs. These are all interesting questions, and we hope that the reviewers feel we answered them convincingly.

(continued)

On scope and significance

This is the biggest question mark remaining. Given the above, all reviewers agree that our work is clear, well-written, mathematically correct, and now numerically validated. But reviewer mogL wonders how much we really learn by studying linear RNNs.

They suggested that we conduct some experiments involving more complex tasks and architectures. Unfortunately, we were not able to do this within the limited rebuttal period. We hope that this is at least understandable---the required technical effort is orthogonal to what the rest of the paper required. (Also, we must acknowledge that our talents and training are much more in theory than in numerical experiments.)

We still think that our work is a worthy contribution to the literature, and we humbly ask you to consider it. We think this because:

  • Although the setting is 'simple', it could fairly be described as on the 'cutting edge' of theory. It's so difficult to do theory for RNNs that people are just beginning to do serious work on linear RNNs. For an indication of where the field is, see, e.g., Bordelon et al. 2025 and Proca et al. 2025, two papers that really inspired us.
  • Not unrelatedly, it is a lot of work to do a detailed mathematical characterization of linear RNNs, so it is rare that related work involves extensive numerical experiments. Neither Bordelon et al. 2025 nor Proca et al. 2025 do experiments involving nonlinear RNNs and 'realistic' tasks, for example.
  • In theoretical work like this, it's not always clear what the 'payoff' is. Often, it is kind of indirect: over time, people keep improving their understanding and mathematical tools, and eventually they figure out how to use those tools to explain 'realistic' networks. Work on infinite-width and lazy networks, for example, led to a better understanding of more realistic feedfoward networks, and even (arguably) an understanding of hyperparameter scaling, something actually useful. We think it's okay if our theory has no major immediate applications, as in our experience, this is rather typical.

In short, we think that there's a reasonable case that this is a solid 'theory' contribution. It is clear, correct, and interesting---and if it's a little useless, that's too bad, but it is also fairly standard for work like this. Given the sorry state of our theoretical understanding of RNNs, we think it's still valuable to take a small step forward!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.