- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work stdies the statiscal information encoded in ESN readouts. It characterizes the mechanism that higher-order temporal statistics become visible through nonliner dynamics, complex scaling law, and identifies representation blind spots associated with conditional-mean dependence. Experiments on synthetic time series are conducted to validate the scientific findings.
Critical strengths
- Theoretical characterization of fitted-model representations. The paper provides a relatively sound analysis of how higher-order temporal statistics are reflected in fitted ESN readout weights. The relation between statistical order, input scaling, and representation visibility is theoretically interesting.
- Sampling-complexity analysis. The analysis provides insights into why higher-order statistical differences become increasingly difficult to detect from finite observations.
Critical weaknesses
- Limited novelty beyond general nonlinear feature regression. The main theoretical results appear to rely primarily on nonlinear feature expansions followed by ridge regression. In particular, the Theorem 3 and Theorem 4 seem potentially applicable to general fixed nonlinear feature extractors, such as MLPs with lagged inputs. The paper does not sufficiently clarify which theoretical insights fundamentally depend on recurrent reservoir dynamics.
- Insufficient experimental comparisons. The experiments validate the theoretical predictions on synthetic processes specifically constructed to exhibit higher-order statistical differences, but with limited evaluation on general-purpose time-series benchmarks. Consequently, it remains unclear whether the proposed theoretical insights are useful in practical settings. If standard benchmarks are unsuitable for evaluating the claimed contributions, the authors should clarify why. And the baseline comparisons are relatively limited, making it difficult to assess the significance of the proposed analysis relative to alternative representations and statistical methods.
- Limited applicability beyond the weakly nonlinear regime. Most of the theoretical analysis focuses on the small-input-scaling regime, where reservoir dynamics are dominated by the linear response and nonlinear effects appear as higher-order perturbations. The resulting statistical properties largely follow established principles of nonlinear feature expansions and statistical estimation, offering limited surprising insights or new mechanisms specific to ESNs. Meanwhile, the paper provides a less comprehensive characterization of the broader nonlinear regime, where reservoir dynamics may exhibit richer behavior. This limits the scope and significance of the theoretical contributions.
Questions for authors
- What theoretical insights in Theorems 3 and 4 are specific to ESNs?
- Why are the experiments limited to specially constructed synthetic datasets? Could the authors provide more comprehensive baseline comparisons?
- To what extent do the theoretical findings extend beyond the small-input-scaling regime?
3 comments
Thank you for the detailed rebuttal and additional clarifications. The revisions partially address my concerns, particularly by distinguishing general nonlinear feature regression properties from ESN-specific results.
However, my main concern regarding theoretical novelty remains. The memory-related findings, such as lag-dependent weighting and the inability of finite-memory reservoirs to capture dependencies beyond their accessible history, appear closely related to established ESN memory-capacity and fading-memory theory. The paper does not sufficiently clarify what fundamentally new insights these results provide beyond prior work.
The Krylov-based noise geometry is potentially more distinctive, but its theoretical significance is not adequately developed in the main text. Much of the analysis supporting the claimed novelty is deferred to the appendix, while the main presentation emphasizes relatively predictable perturbative scaling laws. Moreover, the broader nonlinear regime remains insufficiently characterized.
I appreciate the authors' efforts and acknowledge the improvements, but I remain unconvinced that the paper establishes sufficiently novel and significant theoretical contributions. I therefore maintain my original rating.
Thank you for following up on your concerns. We agree that geometric lag decay and finite-window dependence are consistent with established fading-memory theory, and we do not present them as main contributions. Our question is different: when does a difference between two input laws become visible in the fitted forecasting readout, and how long must a sequence be to detect it?
Table 1 illustrates why the prediction target matters. For the martingale-difference pair, the next-step readout from a reservoir with a bias inside the tanh has a test error of . Adding a squared-target readout on the same states reduces the error to . Corollary 1 explains why: both classes have zero conditional mean, so the population next-step readout is zero in both. On the 128 datasets of the UCR archive, adding the squared target also improves the accuracy of a practical model space on 94 to 101 datasets, depending on the classifier (see our reply to Reviewer HNeB).
Appendix F.12 illustrates why sampling noise matters. For the two-tap pair, three reservoirs of equal width and spectral radius yield values of , , and . Below each reservoir's crossover scale, the measured squared separation is close to . Proposition D.5 explains how signal outside the leading Krylov directions can yield a large : the linear sampling noise is small in those directions. Section 5 discusses this after Theorem 4.
Thank you for the review. We address the three questions in turn.
Q1. New Remark 5 separates the general order-counting argument from what the ESN supplies. The first possible order in Theorem 3(a), that a difference in moments of order 2k+2 enters at λ2k+1, also holds for a ridge readout on any fixed odd map with bounded derivatives of a finite window of scaled inputs, such as a tanh network without biases on lagged inputs (Appendix C.2.1). The recurrence decides the rest.
Q2. Our paper is a theory of the basic model-space representation. Methods built on fitted readouts are evaluated on standard archives in their own papers, where they report the highest average accuracy (SARC, ICML 2025, on the 128 UCR datasets against Rocket, MiniRocket, Hydra and others; SVGL, AAAI 2026, on 30 UEA datasets). What the fitted readout retains has not been analysed, and this is our question. Our claims say which class difference reaches the readout and at what cost, and they can be checked only when the class difference is known. On a benchmark, it is not, and an accuracy change cannot be attributed to a mechanism. Table 7 compares the readout with the raw sequence, twenty generic statistics and AR(20) coefficients under three classifiers, now with standard deviations and, in Table 8, with cross-validated hyperparameters. Appendices F.9 and F.11 compare it with explicit higher-order statistics. Fourth-order reversibility statistics with the right lags have 1.4 to 20 times the separation per unit length of the readout at its best scale on the two-tap and eight-tap pairs, while on a pair whose first difference is at sixth order the same statistics stay at chance, and the readout reaches error 0.02 at L=4096 and λ=2.
Q3. Several results hold at every scale: the conditional-mean restriction and the sign symmetry (Corollary 1, Remark 2), and the exact description of finite-memory reservoirs driven by finite-order moving averages of finitely-valued innovations (Proposition D.6), which gives the population readout and its noise at every scale and locates the onset of detection at order-one scales (Figure 3c). Appendix D.4 gives the saturated limit. At order-one scales, several orders contribute at once, and there we compute rather than expand. A closed-form theory at order-one scales for reservoirs with infinite memory remains open.
On novelty. The tools (Taylor expansion, central limit theorems, ridge algebra) are standard, as in most learning theory, but the results are new: the first order of the input scale at which a difference in moments of a given order can enter the population readout, with a sufficient condition for it to enter for almost every reservoir; the witness being a joint cumulant with the forecasting target, so that the target decides what is visible; the λ4kL law with a lower bound for every classifier with convex or semialgebraic decision regions of bounded complexity; and the blind spots with their remedies. To our knowledge, none of these had been established for model-space representations.