acceptodds
Under review as a conference paper at ICLR 2027

TUNE: A Non-stationary Kernel Perspective on Linear Echo State Networks

12 views

Abstract

Echo State Networks are high-dimensional randomized recurrent neural networks where only the output layer is trained. It is common to apply a non-linear transfer function to the recurrent layer while training a linear perceptron at the output layer. Recently, it has been demonstrated that Echo State Networks, which rely on a linear recurrent layer followed by a non-linear output layer, are also performant. This design choice is attractive, as the resulting networks are fully parallelizable and can perform inference faster and more efficiently. In this study, we demonstrate that the response function of a finite-width linear Echo State Network is, in expectation over its input random projection matrix, a matrix-valued Gaussian Process that, depending on the topology of the recurrent layer, the induced kernel can be stationary, locally stationary, or non-stationary. Following this perspective, we show that tunability between the different statistical properties of Gaussian Processes provides insight into the dynamics of the recurrent layer and can lead to richer representations. Furthermore, it motivates a novel architecture: Tuneable and Unconstrained Non-stationary Echo State Network (TUNE). In this architecture, random features are drawn from a parameterizable spectral measure and combined with tuneable temporal weighting, replacing the recurrent dynamics. We show that TUNE admits a lower computational complexity of the hidden state initialization than linear Echo State Networks with dense recurrent connectivity, providing competitive performance on both synthetic and real-world datasets for classification and (long-term) forecasting.

Primary area:
learning on time series and dynamical systems
Keywords:
Reservoir Computing, Echo State Networks, non-stationary Gaussian processes, Random Fourier Features, classification, long-term forecasting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jCRYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper focuses on linear ESNs and addresses the problem of selecting optimal reservoir topology in such architectures. The paper first demonstrates the parallelisms between linear ESNs and matrix-valued Gaussian Processes and it shows that the induced kernel can be stationary, locally stationary or non-stationary. Starting from this finding, the paper proposes a novel architecture named TUNE, which allows to adjust explicitly the different statistical components of the ESN’s response function. The novel proposed architecture is tested on classification and forecasting tasks and compared with existing ESN baselines.

Critical strengths
  1. The addressed problem is relevant and of scientific interest, as linear ESNs are attractive in terms of computational demands.
  2. The paper gives an interesting perspective on the response function of linear ESNs, showing that it can be decomposed in stationary, locally stationary and non-stationary terms. In Table 1 this is also used to reinterpret common reservoir topologies as different kernel classes, allowing more interpretable design choices.
Critical weaknesses
  1. The proposed methodology and its novelty are not fully clear. The connection between the theory and the architecture design is not described clearly and the reasons for each component are not well explained. Additionally, ParalESN appears to be a special case of TUNE, so the novelty seems incremental.
  2. The experimental evaluation is limited and some comparisons are not valid. The paper compares with linear and non-linear ESNs and only the shallow ParalESN, excluding other existing competitors. Additionally, the timing budget is arbitrary and appears decisive in some cases.
  3. Several sentences are too long, resulting hard to follow and interpret. For example, the contribution sentence in the abstract appears to be incorrect.
Questions for authors
  1. Can you expand and better clarify the description of the proposed architecture? Why is each component needed in TUNE?
  2. Why was a 10s budget selected for the experiments? Can you provide results with different timing budgets and include all baselines for the different datasets?
  3. Why were sMNIST and psMNIST resplit (Appendix E)? The provided results for the additional baselines (LSTM, transformer, ..) are obtained with the same split?
  4. Can you discuss more in detail the results in the tables? In particular, in Table 2 it is difficult to interpret the behaviour of TUNE with respect to the baselines at different parameter sizes, both in terms of accuracy and time.
  5. Why TUNE is compared only with shallow ParalESN, when its authors also report a deep variant? Was ParalESN tuned with its original hyperparameter ranges? Its results on sMNIST at 160K suggests instability. Can you also compare with other existing competitors already cited in the Related Work section (e.g. Piatti et al., 2026)?
3 comments

We thank the reviewer for a careful and constructive review, and for pointing out the issues of our study.

Weaknesses 1 (Novelty): We thank the reviewer for bringing up these issues. We agree that the original submission did not sufficiently clarify the connection between the theoretical analysis and TUNE as also noted by Reviewer Cinz (Weakness 3). We address this in our response to Question 1. Regarding ParalESN, we view its connection to TUNE as supporting the generality of our proposal. While ParalESN is motivated by efficient engineering principles, TUNE is motivated by the stationarity classes, enabling explicit tuning of spectral properties and temporal modulation beyond the ParalESN construction.

Weaknesses 2 (Experimental evaluation): We agree that the experimental evaluation in the original submission had limitations. We have taken steps to address these concerns, as detailed in our responses to Questions 2, 3, and 5 below.

Weaknesses 3 (Presentation): We thank the reviewer for highlighting these presentation issues. We revised the contribution sentence in the abstract by splitting it into two sentences. We also reviewed the paper to identify and shorten unnecessarily long sentences to improve the overall clarity and readability.

Question 1 (Motivation of the proposal): We agree that the original submission did not sufficiently clarify the motivation behind TUNE. Our theoretical analysis identifies that exact stationary, locally stationary, and non-stationary components in the response function, which are opaque when not relying on recurrence. This motivates TUNE, which replaces recurrent dynamics with an explicitly parametrizable response function, enabling more efficient initialization. The spectral component controls stationary part, exponential decay introduces locally stationary modulation, and the Gaussian envelope enables additional non-stationary temporal localization. We acknowledge that the original description needs to be clarified. We revised the manuscript to clarify the role of each component and its connection to the stationarity class it controls.

Question 2 (10-second cutoff): We agree that the original 10-second cutoff applied to individual fits (5 fits per configuration, up to 500 configurations per model) required clearer justification, as also noted by Reviewer XiZe. We clarified its motivation in terms of the total runtime budget and, to reduce its influence, increased the budget from 7 to 24 hours per model. Under the same hyperparameter search procedure, this corresponds to increasing the per-fit cutoff to approximately 35 seconds. We reran all models under our controlled experimental protocol across all datasets. The relative rankings remain unchanged, with no statistically significant changes in performance.

Question 3 (MNIST evaluation): We agree that our original nonstandard splits complicated comparisons with prior work, as also noted by Reviewer XiZe. We adopted the established splits and reran all affected experiments under the increased runtime budget (Question 2). The relative rankings among the models evaluated under the same protocol remain unchanged. We also revisited the experimental protocols for the imported results and noted that they do not adhere to the established splits. Following Reviewer XiZe's suggestion we now present these results separately as contextual references rather than controlled comparisons. All comparative conclusions are based exclusively on models evaluated under our controlled experimental protocol.

Question 4 (Discussion of results): We agree that Table 2 requires a clearer interpretation. We expanded the discussion and will add a figure to the Appendix comparing accuracy and runtime across parameter sizes. As suggested by Reviewer Cinz, we also clarified that stationary and non-stationary components perform comparably on some tasks (Table 11), but their benefits depend on the task, with temporal localization particularly useful for KBumps and BumpDistance.

Question 5 (Additional baselines): We compared against shallow ParalESN to match TUNE's single-layer architecture; the deep variant introduces additional nonlinear processing (Appendix C.3). Prompted by the reviewer's comment on instability, we identified that the hidden state of ParalESN occasionally induced ill-conditioned ridge regression matrices, which we resolved by replacing standard ridge regression with QR-based ridge regression. This revision was applied before the full rerun mentioned in response to Question 3 and improved ParalESN's stability and performance, although TUNE retains its advantage. To strengthen the evaluation, we also added the baselines suggested by Reviewer Cinz including a simpler random features baseline on MNIST and ETT and MiniRocket on MNIST. TUNE outperforms both baselines in this evaluation. We are also extending the evaluation to LinOSS and vanilla Transformers/PatchTST (in progress).

Reviewer_jCRYOpenReview ↗

Answer to rebuttal

Thanks for the rebuttal and for having considered the highlighted weaknesses and questions.

However, while the authors state to have revised the manuscript, the currently uploaded version appears to be the original one. Without the revised version of the paper, I cannot verify the declared clarifications and improvements and neither the additional experimental evaluations. In particular, answers to question 3 and question 5 are not completely clear to me.

Without enough time to read the revised paper, I won't change my previous assessment.

Dear Reviewer jCRY,

Thank you for following up on our responses. Following your suggestion, we have prioritized incorporating the necessary revisions into the manuscript and have just uploaded a revised version of our submission. As we noted above, we do not yet have the results for LinOSS and vanilla Transformer (indicated in Tables 2 and 3). However, we agree with your point that it is important to leave sufficient time for any additional discussion that may be needed.

Reviewer_XiZeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents a linear echo state network’s impulse response, with a fixed transition matrix and Gaussian input projection, as a finite-width matrix-valued Gaussian process. Its covariance connects transition-matrix structure to temporal covariance structure. Motivated by this view, TUNE directly samples oscillatory response functions with decay and localized envelopes, then fits a nonlinear-feature ridge readout. The paper presents experiments that compare it with reservoir baselines on classification and forecasting tasks, with additional ablations and an analysis of frequency tuning through signal-to-noise ratio.

Critical strengths
  1. Sampling the impulse response directly avoids dense transition-matrix construction and propagation. The separation of frequency, decay, and temporal localization makes the architecture easier to tune and interpret.
  2. The paper provides a wide range of empirical studies. The experiments cover several task families and rerun shallow ParalESN under a shared protocol.
Critical weaknesses
  • Theorem 3.4 omits the condition Sn(μ)>0S_n(\mu)>0 used in the appendix. The GP claim also needs clarification: fixing the random envelope center can give a Gaussian process, but averaging over that center generally does not. Corollary 3.2 additionally needs a condition ensuring the slow variation required by the paper’s local-stationarity definition.

  • The performance and efficiency comparisons are difficult to assess. MNIST is re-split, but the imported baselines are not shown to use the same protocol. The 10-second limit applies per fit, with up to 500 tuning trials, and time and energy cover different operations. No code is supplied in the supplementary material, making these experimental details harder to verify.

Questions for authors
  1. See weaknesses.
  2. Is the GP claim conditional on the shared envelope center? Marginalizing it generally gives a Gaussian mixture. Also, what additional timescale condition makes Corollary 3.2 meet Definition 3.2, beyond the normal-matrix factorization?
  3. Do the imported MNIST baselines use the same split, training sample count, and pixel permutation? If not, please separate contextual results from controlled rankings.
  4. Which operations enter the reported time, energy, and 10-second cutoff: response construction, feature extraction, ridge search, or readout fitting? Can existing records separate these costs and clarify total tuning time?
2 comments

We thank the reviewer for their careful review and comments.

Weaknesses 1 (Theorem 3.4 and the GP claim): We thank the reviewer for identifying the discrepancy between the main text and the Appendix. Indeed, the condition Sn(μ)>0S_n(\mu) > 0, used in the Appendix proof, was inadvertently omitted from the statement of Theorem 3.4. We revised the theorem to align its statement with the proof. We address the reviewer's concerns regarding the GP claim in response to Question 2.

Weaknesses 2 (MNIST evaluation and 10-second cutoff): We appreciate the reviewer's points regarding the experimental evaluation, also raised by Reviewer jCRY. We took steps to address these issues and clarify the experimental protocol. The corresponding revisions are described in our responses to Questions 3 and 4. Regarding reproducibility, we are preparing a GitHub repository to accompany the paper but we need to complete the additional code cleanup before it can be released. In the meantime, if the reviewer would find it helpful to see a particular model or experiment, we would be happy to prioritize preparing and sharing an anonymized script.

Question 2 (GP claim and Corollary 3.2): As pointed out by the reviewer, the GP claim holds conditionally on the shared envelope center. Indeed, marginalizing over this center generally yields a Gaussian mixture rather than a GP. Although the Appendix already considers the covariance conditional on the envelope center, we should not have extended this interpretation to the marginal process in the main text. We therefore revised the presentation to explicitly distinguish the conditional GP limit from the marginal mixture and will clarify the corresponding component-wise covariance analysis for the stationary, locally stationary, and non-stationary response functions. Regarding Corollary 3.2, we agree that normality of the transfer matrix alone does not guarantee the slow variation required by Definition 3.2 and additional separation of timescales is necessary for local stationarity. We therefore revised the corollary to explicitly include this condition, formulated in terms of the relation between the real and imaginary eigenvalues of the transfer matrix, and hence dependent on its particular realization. Importantly, the structural separation between midpoint and lag dependence holds for any normal transfer matrix, regardless of initialization, whereas local stationarity requires the additional condition.

Question 3 (MNIST evaluation): Initially, we did not follow the standard data splits for sMNIST, psMNIST, and ETT, which we agree complicates comparisons with previously reported results. To address this, we revised the experimental protocol to follow the established splits and rerun all affected experiments, using a total runtime budget of 24 hours per model (see our response to Question 4 below). The relative rankings among the models evaluated under the same protocol remain unchanged, and we found no statistically significant changes in their performance. We also revisited the experimental protocols for the imported results and noted that they do not adhere to the established splits. Consequently, following the reviewer's recommendation, we explicitly separated our controlled comparisons, including the independently rerun ParalESN baseline, from the contextual results reported in prior work. The two groups are now presented separately in the results tables, instead of being distinguished by italics alone. Importantly, all performance rankings and comparative conclusions are based exclusively on models evaluated under our controlled experimental protocol.

Question 4 (Time, energy, and the 10-second cutoff): We agree with the reviewer that the time and energy measurements in the original submission were not directly comparable, as they covered different operations. We re-organized the presentation of the results to clarify time and energy measurements by reporting separately i) the forward pass that involves constructing the response function and computing the hidden state and ii) training that includes ridge search and readout fitting. Regarding the 10-second cutoff, this constraint was applied to individual fits within the hyperparameter search. Each hyperparameter configuration was evaluated using 5 independent fits, with up to 500 configurations per model. To reduce the influence of this constraint (a concern raised by Reviewer jCRY), we increased the total runtime budget from 7 hours to 24 hours. Under the same hyperparameter search, this corresponds to increasing the per-fit cutoff to approximately 35 seconds. The results in the original submission did not report the individual steps but timed the complete hyperparameter search trial. To provide a clearer assessment, we are also preparing a new section to the Appendix with a detailed breakdown of runtime, energy consumption, and peak VRAM usage (suggested by Reviewer Cinz) for the MNIST experiment.

Dear Reviewer XiZe,

We would like to briefly follow up to let you know that we have incorporated the necessary revisions into the manuscript and have just uploaded a revised version of our submission.

Reviewer_CinzOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors propose a Gaussian Process and kernel-based interpretation of linear Echo State Networks. They show that, when randomness is introduced through the input projection matrix, the impulse response of a finite-width linear ESN defines a matrix-valued Gaussian Process. They characterize how different reservoir topologies influence its covariance kernel, particularly in relation to stationarity and non-stationarity.

Critical strengths

1- The authors establishe a finite-width Gaussian Process interpretation of linear ESNs and characterizes how different reservoir topologies influence the induced covariance kernel (which is interesting).

2- Motivated by the kernel interpretation, they are introducing TUNE, which replaces recurrent dynamics with parameterizable random Fourier features and temporal modulation.

3- The proposed method demonstrates competitive performance across synthetic tasks, sequential classification, long-term forecasting, and chaotic dynamical systems.

4- The paper includes detailed investigations of spectral tuning, temporal decay, non-stationary modulation, random-feature counts, and memory capacity (nice ablation study which is required for such paper).

5- The authors provide theoretical analysis of computational and memory complexity.

Critical weaknesses

1- In Appendix A.5, Lemma A.4, the empirical kernel convergence argument to me appears mathematically incorrect (could you double check?), as the expansion of the product of random Fourier features omits cross terms that vanish only in expectation. Although this does not affect the expected covariance derivation.

2- The classification of normal reservoir matrices as locally stationary seems to not fully established for me. Agree that the covariance depends on both temporal lag and midpoint, but this alone does not guarantee the separation of timescales required by your definition of local stationarity. Additional assumptions or a more exact characterization seems to be needed.

3- The theoretical analysis characterizes non-stationarity through commutator interactions in the reservoir dynamics, but your TUNE introduces non-stationarity through a heuristically chosen Gaussian. The relationship between these two mechanisms is not formally established (or I did not see enough clarification), leaving the proposed architecture only indirectly motivated by the theoretical findings.

4- Your synthetic experiments demonstrate benefits from temporal localization, but the ablation results in Table 11 show that removing the non-stationary extension sometimes gets comparable or even better performance (besides randomness, any specific reason?). This raises questions about how much of TUNE's improvement is attributable to non-stationarity rather than spectral tuning or other architectural components.

5- The optimal spectral bandwidth you are deriving in Theorem 3.4 relies on restrictive assumptions about signal and noise characteristics. Moreover, the experiments in Appendix F.4 indicate that SNR-optimal bandwidth does not consistently correspond to optimal predictive accuracy (to me at least this is limiting the theorem's ability to explain the empirical improvements).

6- Despite the extensive experiments (which is nice), most comparisons focus on ESN-based architectures, particularly linear ESNs and shallow ParalESN. Stronger comparisons against contemporary efficient sequence models and simpler random-feature alternatives under matched experimental conditions can better establish your TUNE (of course I am not pushing to do more experiments in limited rebuttal time, but there are several recent sequential and memory modeling in top-tiers beyond mamba and s4 families).

7- Beside theoretical memory analysis, please report your empirical memory usage and latency in the experiments to show it is in alignment with your theory analysis (I could not find in appendix, otherwise please point me to right place)

Questions for authors

my questions are embedded in the weaknesses above, please find them there.

Currently my evaluation is borderline (which is not listed in the grades). But I would be happy to remind depending on how you answer my queries in rebuttal.


POST REBUTTAL : I am satisfied to give accept (3) to this paper

6 comments

We thank the reviewer for their detailed reading and insightful comments.

Weakness 1 (Incorrect Lemma A.4): We thank the reviewer for identifying this error. We double-checked and revised the statement of Lemma A.4 to correct the empirical kernel convergence claim.

Weakness 2 (Definition of local stationarity): We agree that normality of the transfer matrix alone does not guarantee the slow variation required by Definition 3.2, and that an additional separation of timescales is necessary for local stationarity. This concern was also raised by Reviewer XiZe in Question 2. To address it, we revised Corollary 3.2 to explicitly include the required timescale condition, formulated in terms of the relation between the real and imaginary eigenvalues of the transfer matrix, and hence dependent on its particular realization. Importantly, the structural separation between midpoint and lag dependence holds for any normal transfer matrix, regardless of initialization, whereas local stationarity requires the additional condition.

Weakness 3 (Motivation of the proposal): We agree that the original submission did not sufficiently clarify the motivation behind TUNE. Our theoretical analysis suggests that exact stationary, locally stationary, and non-stationary components in the response function cannot be modeled explicitly without relying on recurrence. This motivates TUNE, which replaces recurrent dynamics with a response function that allows direct parametrization of these components, fulfilling our goal of efficient initialization whilst retaining some statistical characteristics. We acknowledge that the proposed Gaussian envelope is a heuristic construction rather than an exact representation of the commutator terms. We will clarify this distinction and strengthen the connection between the theoretical analysis and the proposed architecture in the revised text.

Weakness 4 (Benefits of non-stationary extension): As pointed out by the reviewer, Table 11 shows that the stationary and non-stationary response functions can achieve comparable performance, with neither consistently outperforming the other. This is expected since the benefits of the temporal localization depend on the task. For example, long-term memorization may favor stationary response functions, consistent with the memory capacity results in Appendix F.3. In contrast, the synthetic KBumps and BumpDistance tasks demonstrate clearer benefits from non-stationary extension. We will clarify that TUNE's advantage lies in its flexibility to parametrize different stationarity classes rather than in non-stationary being universally beneficial.

Weakness 5 (Theorem 3.4): We agree that Theorem 3.4 relies on restrictive assumptions and does not fully explain the empirical improvements. Rather, we view it as a theoretical justification for spectral tuning and a starting point for further analysis. Appendix F.4 demonstrates that SNR-optimal bandwidth can correspond to accuracy-optimal bandwidth on certain datasets. These results suggest that SNR can serve as a useful heuristic for predictive performance, while extending the analysis to more general noise conditions remains future work.

Weakness 6 (Additional baselines): We agree with the reviewer that comparisons beyond ESN-based architectures would strengthen the experimental evaluation, as also noted by Reviewer jCRY. Following the reviewer's suggestion, on MNIST and ETT we added a simpler random features baseline and also implemented MiniRocket, which uses efficient random convolutions, on MNIST. TUNE outperforms both baselines in this evaluation. We are also extending the evaluation to LSSM and one transformer-based architectures (LinOSS and vanilla Transformers/PatchTST). As these architectures require adaptations to our experimental pipeline, their implementation and evaluation are still in progress.

Weakness 7 (Memory usage): We thank the reviewer for this suggestion. Following concerns raised by Reviewers XiZe and jCRY, we have revised the experimental protocol to use standard data splits for sMNIST, psMNIST, and ETT and increased the runtime budget beyond the original 10-second per-fit cutoff. We have also incorporated measurements of empirical memory usage and runtime into the new experiments. We are preparing a dedicated Appendix section reporting these measurements for MNIST, complementing the theoretical analysis.

Reviewer_CinzOpenReview ↗

reply to authors for rebuttal

Thanks for the rebuttal and answers.

For W6 and W7, did you update your manuscript to involve them? I went through the rebuttal pdf, but could not find any reported results concerning W6 and W7.

Dear Reviewer Cinz,

Thank you for following up on our responses. We are currently finalizing our response to Reviewer jCRY. Once that is complete, we will incorporate all the necessary revisions into the manuscript, including those related to W6 and W7. We plan to upload the fully revised paper before the discussion period ends on November 18.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.