- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
In this paper, the authors study the learning dynamics of nonlinear task-trained recurrent neural networks. To do this, they use the concept of the best causal linear (BCL) predictor, and show that nonlinear RNNs undergo a transition from initially emulating the BCL predictor, to eventually learning a nonlinear filter of the inputs. To facilitate comparisons betwen trained networks and BCL, they analyze the spectral structure of networks' recurrent weights over learning.
Critical strengths
- To the best of my knowledge, the observation that nonlinear RNNs first learn the optimal linear filter approximation of a task's input-output relation before eventually developing nonlinear dynamics (e.g. multistability in the flipflop task) is not well known, and is interesting insight.
Critical weaknesses
-
Apologies for my bluntness, but the writing of this paper was quite poor. The motivation was difficult to follow; for instance, why are we discussing scaling laws in the intro, and contrasting the ideas in this paper with scaling? Seems like a non-sequitur. My general impression is that the paper currently reads as a disorganized collection of facts and observations. Overall, the incoherent organization of thoughts and excessive claudespeak made it difficult to read.
-
The vast majority of analysis is empirical in nature. This is not problematic in itself, but considering how simple the tasks studied are, and the fact that many of the experimental observations in the linear case have already been fully characterized analytically, the overall contribution of this paper is rather limited. For instance, in the work of Bordelon et al. that you cite, they already achieve a fairly sharp characterization of learning from teachers with one real pole, two complex poles, and empirics for learning from teachers with multiple poles. Further, the central finding that nonlinear RNNs first learn the BCL is just presented as an observation, and only for a single simple task.
Overall, I would suggest that the paper needs improved writing and sharper theoretical insight to match the empirics before it would be ready for acceptance.
Questions for authors
- Is there any insight the authors can provide to shed light on the generality of this phenomenon of linear-to-nonlinear transitions in the learning dynamics of nonlinear RNNs? And does this generically occur across a variety of optimizers/architectures/nonlinearities?
Thank you for your review
Thank you for your thorough and insightful review. We are pleased that you find the central observation of our paper, that nonlinear RNNs first learn the optimal linear filter approximation of a task’s function, to be an interesting insight.
Now we address your core concerns.
Readabilility issues
We appreciate your bluntness regarding the writing quality of the paper. We apologize that you had an unpleasant experience reading the text. In the revised submission, we have substantially re-written the main text, including sharpening the motivation, and improved the overall organization. We hope that the updated version is easier to follow, more pleasant to read, and more in line with your standards.
Is there any insight the authors can provide to shed light on the generality of this phenomenon of linear-to-nonlinear transitions in the learning dynamics of nonlinear RNNs? And does this generically occur across a variety of optimizers/architectures/nonlinearities?
This is a very good question. Our experiments indicate that linear-first learning depends on the training setting. In Figure 4, in the appendix, we show that increasing the curvature of the tanh nonlinearity strength at fixed initialization speeds up nonlinear component acquisition and shortens the linear phase. We agree that exploring the generality of this phenomenon across architectures and optimizers, systematically, is a very important direction.
Thank you again for your review. We believe your comments and questions have helped us sharpen the narrative and spurred further experiments about the generality of the linear-to-nonlinear learning dynamics.