acceptodds
Under review as a conference paper at ICLR 2027

Linear-to-Nonlinear Transitions in the Learning Dynamics of Recurrent Neural Networks

Abstract

How do recurrent neural networks (RNNs) acquire the temporal computations required by a task? We use the best causal linear (BCL) predictor—determined by input and target second-order statistics—as a task-defined reference for answering this question. First, we connect the BCL predictor to its minimal recurrent realization, providing an exact functional benchmark and task-defined poles. Second, we show that an overparameterized linear RNN realizes this predictor through a small set of residue-carrying modes while most of its spectrum remains behaviorally weak. Recurrent gain accelerates learning only up to a near-marginal regime, where predictive modes approach the BCL poles as nearby unused modes move away. Third, on a nonlinear memory task, a nonlinear RNN reaches BCL-level performance before improving beyond the optimal-linear loss and forming stable memory states around the same slow scaffold. A controlled filtered-cubic task shows that this linear-first ordering depends on the training regime. The BCL thus links task statistics, recurrent dynamics, and the emergence of nonlinear computation during learning.

Primary area:
learning theory
Keywords:
Learning Dynamics, Recurrent Neural Networks, Bifurcations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7JMVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors study the learning dynamics of nonlinear task-trained recurrent neural networks. To do this, they use the concept of the best causal linear (BCL) predictor, and show that nonlinear RNNs undergo a transition from initially emulating the BCL predictor, to eventually learning a nonlinear filter of the inputs. To facilitate comparisons betwen trained networks and BCL, they analyze the spectral structure of networks' recurrent weights over learning.

Critical strengths
  • To the best of my knowledge, the observation that nonlinear RNNs first learn the optimal linear filter approximation of a task's input-output relation before eventually developing nonlinear dynamics (e.g. multistability in the flipflop task) is not well known, and is interesting insight.
Critical weaknesses
  • Apologies for my bluntness, but the writing of this paper was quite poor. The motivation was difficult to follow; for instance, why are we discussing scaling laws in the intro, and contrasting the ideas in this paper with scaling? Seems like a non-sequitur. My general impression is that the paper currently reads as a disorganized collection of facts and observations. Overall, the incoherent organization of thoughts and excessive claudespeak made it difficult to read.

  • The vast majority of analysis is empirical in nature. This is not problematic in itself, but considering how simple the tasks studied are, and the fact that many of the experimental observations in the linear case have already been fully characterized analytically, the overall contribution of this paper is rather limited. For instance, in the work of Bordelon et al. that you cite, they already achieve a fairly sharp characterization of learning from teachers with one real pole, two complex poles, and empirics for learning from teachers with multiple poles. Further, the central finding that nonlinear RNNs first learn the BCL is just presented as an observation, and only for a single simple task.

Overall, I would suggest that the paper needs improved writing and sharper theoretical insight to match the empirics before it would be ready for acceptance.

Questions for authors
  • Is there any insight the authors can provide to shed light on the generality of this phenomenon of linear-to-nonlinear transitions in the learning dynamics of nonlinear RNNs? And does this generically occur across a variety of optimizers/architectures/nonlinearities?
1 comment
AuthorsOpenReview ↗

Thank you for your review

Thank you for your thorough and insightful review. We are pleased that you find the central observation of our paper, that nonlinear RNNs first learn the optimal linear filter approximation of a task’s function, to be an interesting insight.

Now we address your core concerns.

Readabilility issues

We appreciate your bluntness regarding the writing quality of the paper. We apologize that you had an unpleasant experience reading the text. In the revised submission, we have substantially re-written the main text, including sharpening the motivation, and improved the overall organization. We hope that the updated version is easier to follow, more pleasant to read, and more in line with your standards.

Is there any insight the authors can provide to shed light on the generality of this phenomenon of linear-to-nonlinear transitions in the learning dynamics of nonlinear RNNs? And does this generically occur across a variety of optimizers/architectures/nonlinearities?

This is a very good question. Our experiments indicate that linear-first learning depends on the training setting. In Figure 4, in the appendix, we show that increasing the curvature of the tanh nonlinearity strength at fixed initialization speeds up nonlinear component acquisition and shortens the linear phase. We agree that exploring the generality of this phenomenon across architectures and optimizers, systematically, is a very important direction.

Thank you again for your review. We believe your comments and questions have helped us sharpen the narrative and spurred further experiments about the generality of the linear-to-nonlinear learning dynamics.

Reviewer_sYVGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how the best causal linear predictor can be compared to trained RNN representations in the learning of sequence models. It illustrates a general framework on how the properties of the BCN can be compared to properties of trained RNNs, and shows how the signature of the BCN can be observed both in linear and nonlinear tasks as a potential scaffold for learning.

Critical strengths
  • The overall BCL framework is well motivated and follows a clear logical structure throughout the paper.
  • The paper does a good job linking theoretical results to concrete interpretable structures emerging during learning.
Critical weaknesses
  • Eq. 13 introduces both a decay and an oscillatory timescale. But all the tasks actually trained on in the main text have only decaying timescales. The appendix then introduces tasks like the sine-wave task in F.2. This task is never used, referenced, or trained on, except in B.3, for which I also could not find any reference in the main text (even though F.2 explicitly frames it as the dual of the flip-flop task that is trained on, and claims the two tasks "span the two extremes of the autocorrelation space considered in this paper"). This claim is simply false in this version. So there seems to have been a version of this paper that also featured the oscillatory tasks, but they were either removed or never included, without however removing their introduction in the appendix. The two tasks also don't really fit into the paper's narrative, since the sine-wave task has no inputs at all, so there is no BCL predictor for it. On the other hand, the driven resonator in B.3 has inputs and known complex BCL poles thatwould have fit the framework perfectly, so it is puzzling that it is introduced but was apparently never run. This is the central limitation to me, since it both underscores a very sloppy assembly of the paper, and a central missing half of the framework that is never addressed (why is only the decay half of the taxonomy tested?).

Smaller points:

  • The discussion is short and reads mostly like a summary. Tthere is only a very brief mention of limitations: "The filtered-cubic experiment shows that this linear-first trajectory is a training regime, not a universal law". What does this tell me about finding similar BCN structure in other nonlinear asks?
  • The Reproducibility Statement is literally the default copy paste, if added it should actually argue to what extent the results are reproducible. No code is provided, while the Appendix provides sufficient context so the main results could likely be reproduced.
  • Other parts of the Appendix also look sloppy (e.g. Equation 56 spans almost one page).
  • The first theorem and the linear results are relatively close to other work on linear RNN learning (Proca, Bordelon etc.). It is fair since these are cited, but it would help more clearly clarify what the central novel contribution of this paper is in this aspect.
Questions for authors
1 comment
AuthorsOpenReview ↗

Addressing your core concerns + driven resonator experiment results

Thank you for your insightful review. We are glad that you found the BCL framework well-motivated and that it follows a clear logical structure. We are also encouraged that you believe the paper does a good job linking theoretical results to concrete interpretable structures that emerge during learning.

Here we address your core concerns.

Results for Driven Resonator Task

We agree that the submitted experiments left the oscillatory part of the framework untested. On your suggestion, we have now trained 3 linear RNN seeds on the input-driven resonator task defined in Appendix B.2. Its analytic BCL has three complex-conjugate pole pairs, jointly specifying decay times and oscillation periods. Figure 5 in Appendix D.4, analogous to Figure 2 in the main text but for the resonator task, shows that training recovers the driven impulse responses and predictive conjugate pairs, with all selected poles landing on their corresponding targets. This complements the real-pole memory experiments by directly testing oscillatory BCL dynamics. We have also removed the unused sine-wave discussion and its associated coverage claim.

Reproducibility Statement

Thank you for pointing this out. We have revised the reproducibility statement and provided an anonymized repository containing training code, configurations, and scripts to regenerate the experimental figures from supplied numerical data.

Appendix Organization

We agree that the appendix of the original submission was poorly organized and contained formatting errors. We have substantially shortened and reorganized it, removed outdated and redundant material, and corrected the formatting issues, including the excessive spacing around Equation 56, for example.

Thank you again for your comprehensive review of our paper, we believe your suggestions have strengthened the quality of the results and we hope you consider raising your score.

Reviewer_pFsTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors apply classic results from control theory to analyze gradient-descent-trained RNNs. They define the "best causal linear" (BCL) predictor from the task's second-order statistics and compare its poles and residues to the trained network's spectrum. On an overparameterized linear RNN trained on a 3-channel flip-flop task, the eigenvalues approach the BCL poles. Changing the network gain shows a non-monotonic learning-speed tradeoff — acceleration from g = 0.3 to g = 0.9, slowdown at the near-marginal g = 0.99. At g = 0.99 analysis shows pole-matched modes moving toward the BCL poles while nearby unused bulk modes move away. On a nonlinear tanh RNN, the network reaches BCL-level loss first and later improves below it while forming nonlinear memory fixed points.

Critical strengths
  • Classical tools are applied correctly. Wiener-Hopf → Ho-Kalman → modal decomposition is an appropriate pipeline for this question. Equation 12 is explicitly flagged as standard (Ho & Kalman 1966, Kailath 1980).
  • The non-monotonic gain tradeoff (Figure 2D) is a concrete empirical finding. This is not predicted by classical control theory and is a specific prediction about SGD + overparameterization.
  • The BCL predicted linear solution preceding the nonlinear solution (Fig 3) is interesting.
Critical weaknesses
  • The methodology is almost entirely classical. Wiener-Hopf (1949), Ho-Kalman (1966), modal decomposition and residue analysis. The paper's contribution is in applying these tools to characterize a specific empirical phenomenon in neural network learning. This is a reasonable contribution but not a methodological or theoretical advance.
  • Several of the empirical findings are expected under classical control theory. Residue-based mode selection concentrating predictive power in a low-dimensional subspace of an overparameterized network is close to tautological: a rank p input-output map in a 512-dim state has to live in a p-dim subspace, and residues are the standard way to find it. Mode eigenvalues approaching BCL poles under MSE training is what Wiener filtering guarantees in the limit.
  • The paper invokes "Wiener-Hopf" repeatedly, but on the paper's task the input is i.i.d. Bernoulli across time, i.e., temporally white. For white input, the Wiener-Hopf solution degenerates to the per-lag division H_k = C_yx(k) / σ², which is the formula shown in Figure 1 and which coincides with the asymptotic OLS estimator for an FIR filter. The paper never performs a spectral factorization, the step that distinguishes Wiener-Hopf from OLS. On this task family, "Wiener-Hopf" is a dressed-up name for OLS, and the only non-trivial step in the pipeline is Ho-Kalman realization of the OLS solution to obtain poles. A convincing demonstration would use colored input, e.g., x_t = ρ x_{t−1} + η_t. On that task the BCL poles would depend jointly on the input AR coefficient ρ and the target dynamics, spectral factorization is non-trivial, and OLS on finite taps no longer coincides with the infinite-horizon Wiener-Hopf solution. Does the trained RNN's spectrum still approach the BCL poles under colored input?
  • The introduction does not make the case for why a task-defined linear reference is important. By the end of the introduction, I understood what the authors did but not why this framework matters for improving our understanding of RNN training. Providing more context for how this methodology will help drive research forward would benefit the Introduction.
Questions for authors
  • Does the linear-first regime persist for tasks with non-white input autocovariance, where the BCL filter requires nontrivial Wiener-Hopf spectral factorization?
  • The result in Figure 2E depends on correctly tracking which final-state eigenvalue corresponds to which initial-state eigenvalue. The paper uses minimum-displacement matching. This procedure can misattribute motion when mode trajectories cross during training: if two near-degenerate modes have trajectories that pass through each other at some intermediate checkpoint, minimum-displacement matching will prefer the identity-swap pairing because it has the shorter total displacement. Since the paper’s claim depends on correct identity tracking, could the authors clarify (i) how finely spaced the training checkpoints are relative to the typical mode motion between checkpoints, (ii) whether identity-swap events were detected across the three g = 0.99 seeds and how they were handled, and (iii) whether the signed-displacement result in Figure 2E is robust under an alternative tracking method such as eigenvector-tracked continuous trajectories through intermediate checkpoints?
  • Could the framework be extended to modern state-space architectures (S4, Mamba) whose explicit linear state-space structure aligns directly with Ho-Kalman realization?
2 comments
AuthorsOpenReview ↗

Addressing your core questions

Thank you for your thorough review. We are pleased that you appreciated the Wiener-Hopf -> Ho-Kalman -> modal decomposition pipeline, and that you found our empirical results interesting, including the non-monotonic gain tradeoff (Figure 2D) and the linear solution preceding the nonlinear solution (Figure 3).

Here we address your remaining core concerns:

"The methodology is almost entirely classical ... This is a reasonable contribution but not a methodological or theoretical advance"

In our paper, we indeed repurpose classical tools from control theory to study the learning dynamics of overparameterized RNNs. We believe that the specific methodology described in the paper is itself a valuable contribution that can be applied to interrogate the learning process of neural networks more broadly.

Does the linear-first regime persist for tasks with non-white input autocovariance, where the BCL filter requires nontrivial Wiener-Hopf spectral factorization?

We agree that white input simplifies the Wiener–Hopf equations; we will clarify this in the manuscript. Following your suggestion, we repeated the filtered-cubic experiment with stationary unit-variance AR(1) input at ρ=0.8\rho=0.8, alongside the white-input control. In all three seeds under each condition, the linear component emerged before the cubic component. Cubic recovery occurred substantially earlier with colored input. This establishes linear-first acquisition with non-white input autocovariance in this task.

Thank you again for your in-depth review, we hope that this clarifies your core remaining concerns and questions.

We acknowledge the additional work contributed by the authors. Given the late response to my review, I cannot substantively reply. My score was already generous and not all concerns were addressed so I will remain with my initial assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.