- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper focuses on linear ESNs and addresses the problem of selecting optimal reservoir topology in such architectures. The paper first demonstrates the parallelisms between linear ESNs and matrix-valued Gaussian Processes and it shows that the induced kernel can be stationary, locally stationary or non-stationary. Starting from this finding, the paper proposes a novel architecture named TUNE, which allows to adjust explicitly the different statistical components of the ESN’s response function. The novel proposed architecture is tested on classification and forecasting tasks and compared with existing ESN baselines.
Critical strengths
- The addressed problem is relevant and of scientific interest, as linear ESNs are attractive in terms of computational demands.
- The paper gives an interesting perspective on the response function of linear ESNs, showing that it can be decomposed in stationary, locally stationary and non-stationary terms. In Table 1 this is also used to reinterpret common reservoir topologies as different kernel classes, allowing more interpretable design choices.
Critical weaknesses
- The proposed methodology and its novelty are not fully clear. The connection between the theory and the architecture design is not described clearly and the reasons for each component are not well explained. Additionally, ParalESN appears to be a special case of TUNE, so the novelty seems incremental.
- The experimental evaluation is limited and some comparisons are not valid. The paper compares with linear and non-linear ESNs and only the shallow ParalESN, excluding other existing competitors. Additionally, the timing budget is arbitrary and appears decisive in some cases.
- Several sentences are too long, resulting hard to follow and interpret. For example, the contribution sentence in the abstract appears to be incorrect.
Questions for authors
- Can you expand and better clarify the description of the proposed architecture? Why is each component needed in TUNE?
- Why was a 10s budget selected for the experiments? Can you provide results with different timing budgets and include all baselines for the different datasets?
- Why were sMNIST and psMNIST resplit (Appendix E)? The provided results for the additional baselines (LSTM, transformer, ..) are obtained with the same split?
- Can you discuss more in detail the results in the tables? In particular, in Table 2 it is difficult to interpret the behaviour of TUNE with respect to the baselines at different parameter sizes, both in terms of accuracy and time.
- Why TUNE is compared only with shallow ParalESN, when its authors also report a deep variant? Was ParalESN tuned with its original hyperparameter ranges? Its results on sMNIST at 160K suggests instability. Can you also compare with other existing competitors already cited in the Related Work section (e.g. Piatti et al., 2026)?
3 comments
Answer to rebuttal
Thanks for the rebuttal and for having considered the highlighted weaknesses and questions.
However, while the authors state to have revised the manuscript, the currently uploaded version appears to be the original one. Without the revised version of the paper, I cannot verify the declared clarifications and improvements and neither the additional experimental evaluations. In particular, answers to question 3 and question 5 are not completely clear to me.
Without enough time to read the revised paper, I won't change my previous assessment.
Dear Reviewer jCRY,
Thank you for following up on our responses. Following your suggestion, we have prioritized incorporating the necessary revisions into the manuscript and have just uploaded a revised version of our submission. As we noted above, we do not yet have the results for LinOSS and vanilla Transformer (indicated in Tables 2 and 3). However, we agree with your point that it is important to leave sufficient time for any additional discussion that may be needed.
We thank the reviewer for a careful and constructive review, and for pointing out the issues of our study.
Weaknesses 1 (Novelty): We thank the reviewer for bringing up these issues. We agree that the original submission did not sufficiently clarify the connection between the theoretical analysis and TUNE as also noted by Reviewer Cinz (Weakness 3). We address this in our response to Question 1. Regarding ParalESN, we view its connection to TUNE as supporting the generality of our proposal. While ParalESN is motivated by efficient engineering principles, TUNE is motivated by the stationarity classes, enabling explicit tuning of spectral properties and temporal modulation beyond the ParalESN construction.
Weaknesses 2 (Experimental evaluation): We agree that the experimental evaluation in the original submission had limitations. We have taken steps to address these concerns, as detailed in our responses to Questions 2, 3, and 5 below.
Weaknesses 3 (Presentation): We thank the reviewer for highlighting these presentation issues. We revised the contribution sentence in the abstract by splitting it into two sentences. We also reviewed the paper to identify and shorten unnecessarily long sentences to improve the overall clarity and readability.
Question 1 (Motivation of the proposal): We agree that the original submission did not sufficiently clarify the motivation behind TUNE. Our theoretical analysis identifies that exact stationary, locally stationary, and non-stationary components in the response function, which are opaque when not relying on recurrence. This motivates TUNE, which replaces recurrent dynamics with an explicitly parametrizable response function, enabling more efficient initialization. The spectral component controls stationary part, exponential decay introduces locally stationary modulation, and the Gaussian envelope enables additional non-stationary temporal localization. We acknowledge that the original description needs to be clarified. We revised the manuscript to clarify the role of each component and its connection to the stationarity class it controls.
Question 2 (10-second cutoff): We agree that the original 10-second cutoff applied to individual fits (5 fits per configuration, up to 500 configurations per model) required clearer justification, as also noted by Reviewer XiZe. We clarified its motivation in terms of the total runtime budget and, to reduce its influence, increased the budget from 7 to 24 hours per model. Under the same hyperparameter search procedure, this corresponds to increasing the per-fit cutoff to approximately 35 seconds. We reran all models under our controlled experimental protocol across all datasets. The relative rankings remain unchanged, with no statistically significant changes in performance.
Question 3 (MNIST evaluation): We agree that our original nonstandard splits complicated comparisons with prior work, as also noted by Reviewer XiZe. We adopted the established splits and reran all affected experiments under the increased runtime budget (Question 2). The relative rankings among the models evaluated under the same protocol remain unchanged. We also revisited the experimental protocols for the imported results and noted that they do not adhere to the established splits. Following Reviewer XiZe's suggestion we now present these results separately as contextual references rather than controlled comparisons. All comparative conclusions are based exclusively on models evaluated under our controlled experimental protocol.
Question 4 (Discussion of results): We agree that Table 2 requires a clearer interpretation. We expanded the discussion and will add a figure to the Appendix comparing accuracy and runtime across parameter sizes. As suggested by Reviewer Cinz, we also clarified that stationary and non-stationary components perform comparably on some tasks (Table 11), but their benefits depend on the task, with temporal localization particularly useful for KBumps and BumpDistance.
Question 5 (Additional baselines): We compared against shallow ParalESN to match TUNE's single-layer architecture; the deep variant introduces additional nonlinear processing (Appendix C.3). Prompted by the reviewer's comment on instability, we identified that the hidden state of ParalESN occasionally induced ill-conditioned ridge regression matrices, which we resolved by replacing standard ridge regression with QR-based ridge regression. This revision was applied before the full rerun mentioned in response to Question 3 and improved ParalESN's stability and performance, although TUNE retains its advantage. To strengthen the evaluation, we also added the baselines suggested by Reviewer Cinz including a simpler random features baseline on MNIST and ETT and MiniRocket on MNIST. TUNE outperforms both baselines in this evaluation. We are also extending the evaluation to LinOSS and vanilla Transformers/PatchTST (in progress).