TUNE: A Non-stationary Kernel Perspective on Linear Echo State Networks

Echo State Networks are high-dimensional randomized recurrent neural networks where only the output layer is trained. It is common to apply a non-linear transfer function to the recurrent layer while training a linear perceptron at the output layer. Recently, it has been demonstrated that Echo State Networks, which rely on a linear recurrent layer followed by a non-linear output layer, are also performant. This design choice is attractive, as the resulting networks are fully parallelizable and can perform inference faster and more efficiently. In this study, we demonstrate that the response function of a finite-width linear Echo State Network is, in expectation over its input random projection matrix, a matrix-valued Gaussian Process that, depending on the topology of the recurrent layer, the induced kernel can be stationary, locally stationary, or non-stationary. Following this perspective, we show that tunability between the different statistical properties of Gaussian Processes provides insight into the dynamics of the recurrent layer and can lead to richer representations. Furthermore, it motivates a novel architecture: Tuneable and Unconstrained Non-stationary Echo State Network (TUNE). In this architecture, random features are drawn from a parameterizable spectral measure and combined with tuneable temporal weighting, replacing the recurrent dynamics. We show that TUNE admits a lower computational complexity of the hidden state initialization than linear Echo State Networks with dense recurrent connectivity, providing competitive performance on both synthetic and real-world datasets for classification and (long-term) forecasting.