acceptodds
Under review as a conference paper at ICLR 2027

What Does a Fitted Model Represent? Graded Orders and Sampling Cost in Echo-State Model Spaces

Abstract

Representing a time series by the parameters of a model fitted to it is a common approach, from model-space learning to weight-space learning, yet little is known about what such a representation retains. We analyse echo-state networks, in which a ridge readout of a fixed random reservoir is fitted to forecast each sequence and serves as its representation. The fitted readout is a statistic of the law of the process. Its linear part depends only on the autocovariance, and higher-order structure enters order by order: for scalar input, once lower-order moments agree, a difference of order enters at order in the input scale , generically when it is a joint cumulant of the forecasting target and the current input. Detecting it from one trajectory of length is governed by , which we establish for bounded moving averages at a fixed ridge with a shared non-degenerate linear noise as a local Gaussian limit, with a lower bound on the error of every classifier of bounded complexity that population LDA attains. The same analysis reveals blind spots created by the forecasting target and by the symmetry of the reservoir, and shows how to remove them. Together, these results explain when a fitted-model representation distinguishes time series from different processes, and what to change when it does not.

Primary area:
learning theory
Keywords:
echo state networks, reservoir computing, model space learning, time series representation, learning theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_a8jPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work stdies the statiscal information encoded in ESN readouts. It characterizes the mechanism that higher-order temporal statistics become visible through nonliner dynamics, complex scaling law, and identifies representation blind spots associated with conditional-mean dependence. Experiments on synthetic time series are conducted to validate the scientific findings.

Critical strengths
  1. Theoretical characterization of fitted-model representations. The paper provides a relatively sound analysis of how higher-order temporal statistics are reflected in fitted ESN readout weights. The relation between statistical order, input scaling, and representation visibility is theoretically interesting.
  2. Sampling-complexity analysis. The analysis provides insights into why higher-order statistical differences become increasingly difficult to detect from finite observations.
Critical weaknesses
  1. Limited novelty beyond general nonlinear feature regression. The main theoretical results appear to rely primarily on nonlinear feature expansions followed by ridge regression. In particular, the Theorem 3 and Theorem 4 seem potentially applicable to general fixed nonlinear feature extractors, such as MLPs with lagged inputs. The paper does not sufficiently clarify which theoretical insights fundamentally depend on recurrent reservoir dynamics.
  2. Insufficient experimental comparisons. The experiments validate the theoretical predictions on synthetic processes specifically constructed to exhibit higher-order statistical differences, but with limited evaluation on general-purpose time-series benchmarks. Consequently, it remains unclear whether the proposed theoretical insights are useful in practical settings. If standard benchmarks are unsuitable for evaluating the claimed contributions, the authors should clarify why. And the baseline comparisons are relatively limited, making it difficult to assess the significance of the proposed analysis relative to alternative representations and statistical methods.
  3. Limited applicability beyond the weakly nonlinear regime. Most of the theoretical analysis focuses on the small-input-scaling regime, where reservoir dynamics are dominated by the linear response and nonlinear effects appear as higher-order perturbations. The resulting statistical properties largely follow established principles of nonlinear feature expansions and statistical estimation, offering limited surprising insights or new mechanisms specific to ESNs. Meanwhile, the paper provides a less comprehensive characterization of the broader nonlinear regime, where reservoir dynamics may exhibit richer behavior. This limits the scope and significance of the theoretical contributions.
Questions for authors
  1. What theoretical insights in Theorems 3 and 4 are specific to ESNs?
  2. Why are the experiments limited to specially constructed synthetic datasets? Could the authors provide more comprehensive baseline comparisons?
  3. To what extent do the theoretical findings extend beyond the small-input-scaling regime?
3 comments

Thank you for the review. We address the three questions in turn.

Q1. New Remark 5 separates the general order-counting argument from what the ESN supplies. The first possible order in Theorem 3(a), that a difference in moments of order 2k+22k+2 enters at λ2k+1\lambda^{2k+1}, also holds for a ridge readout on any fixed odd map with bounded derivatives of a finite window of scaled inputs, such as a tanh network without biases on lagged inputs (Appendix C.2.1). The recurrence decides the rest.

  • Infinite memory. The ESN state depends on the whole past. Constructing its stationary state and controlling its Taylor remainder with unbounded inputs needs the contraction argument of Lemma C.3, Proposition C.4 and Lemmas C.5 and C.6. A window map needs neither.
  • Which lags enter, with what weight. The cubic coefficient is a fixed linear image of the fourth-order lag moments, with weights bounded geometrically in the lag when ∥W∥2≤q<1\|W\|_2\leq q<1 (Proposition E.3). For a map of the last p+1p+1 inputs, as for a reservoir of memory pp, two processes with one joint law of the target and this window have one population readout at every scale and ridge (Proposition D.6(a)). On the lag-two pair, the population rule of the memoryless reservoir is therefore at chance at every scale, while memory one or two separates the classes (Figure 3b).
  • Genericity. Theorem 3(b): a non-zero witness suffices for almost every (W,Win)(W,W_{\mathrm{in}}) and for all but countably many ρ\rho along the orthogonal family.
  • The noise geometry of Theorem 4. The sampling law needs a non-degenerate linear noise V1V_1. For a reservoir, V1V_1 acts through the Krylov directions WjWinW^jW_{\mathrm{in}}, and for iid input V1≻0V_1\succ0 whenever (W,WWin)(W, WW_{\mathrm{in}}) is controllable (Lemma D.3). A window of p+1p+1 inputs gives at most p+1p+1 directions of linear noise, so V1V_1 is singular once the map has more than p+1p+1 outputs and the law in its stated form does not apply. The Krylov structure also sets the constant, which is large when the signal lies outside the leading Krylov directions (Proposition D.5). New Appendix F.12 shows this numerically: on one reservoir the constant is 1.1×1031.1\times10^3 with one input channel and about 0.20.2 with two.

Q2. Our paper is a theory of the basic model-space representation. Methods built on fitted readouts are evaluated on standard archives in their own papers, where they report the highest average accuracy (SARC, ICML 2025, on the 128 UCR datasets against Rocket, MiniRocket, Hydra and others; SVGL, AAAI 2026, on 30 UEA datasets). What the fitted readout retains has not been analysed, and this is our question. Our claims say which class difference reaches the readout and at what cost, and they can be checked only when the class difference is known. On a benchmark, it is not, and an accuracy change cannot be attributed to a mechanism. Table 7 compares the readout with the raw sequence, twenty generic statistics and AR(20) coefficients under three classifiers, now with standard deviations and, in Table 8, with cross-validated hyperparameters. Appendices F.9 and F.11 compare it with explicit higher-order statistics. Fourth-order reversibility statistics with the right lags have 1.4 to 20 times the separation per unit length of the readout at its best scale on the two-tap and eight-tap pairs, while on a pair whose first difference is at sixth order the same statistics stay at chance, and the readout reaches error 0.02 at L=4096L=4096 and λ=2\lambda=2.

Q3. Several results hold at every scale: the conditional-mean restriction and the sign symmetry (Corollary 1, Remark 2), and the exact description of finite-memory reservoirs driven by finite-order moving averages of finitely-valued innovations (Proposition D.6), which gives the population readout and its noise at every scale and locates the onset of detection at order-one scales (Figure 3c). Appendix D.4 gives the saturated limit. At order-one scales, several orders contribute at once, and there we compute rather than expand. A closed-form theory at order-one scales for reservoirs with infinite memory remains open.

On novelty. The tools (Taylor expansion, central limit theorems, ridge algebra) are standard, as in most learning theory, but the results are new: the first order of the input scale at which a difference in moments of a given order can enter the population readout, with a sufficient condition for it to enter for almost every reservoir; the witness being a joint cumulant with the forecasting target, so that the target decides what is visible; the λ4kL\lambda^{4k}L law with a lower bound for every classifier with convex or semialgebraic decision regions of bounded complexity; and the blind spots with their remedies. To our knowledge, none of these had been established for model-space representations.

Thank you for the detailed rebuttal and additional clarifications. The revisions partially address my concerns, particularly by distinguishing general nonlinear feature regression properties from ESN-specific results.

However, my main concern regarding theoretical novelty remains. The memory-related findings, such as lag-dependent weighting and the inability of finite-memory reservoirs to capture dependencies beyond their accessible history, appear closely related to established ESN memory-capacity and fading-memory theory. The paper does not sufficiently clarify what fundamentally new insights these results provide beyond prior work.

The Krylov-based noise geometry is potentially more distinctive, but its theoretical significance is not adequately developed in the main text. Much of the analysis supporting the claimed novelty is deferred to the appendix, while the main presentation emphasizes relatively predictable perturbative scaling laws. Moreover, the broader nonlinear regime remains insufficiently characterized.

I appreciate the authors' efforts and acknowledge the improvements, but I remain unconvinced that the paper establishes sufficiently novel and significant theoretical contributions. I therefore maintain my original rating.

Thank you for following up on your concerns. We agree that geometric lag decay and finite-window dependence are consistent with established fading-memory theory, and we do not present them as main contributions. Our question is different: when does a difference between two input laws become visible in the fitted forecasting readout, and how long must a sequence be to detect it?

Table 1 illustrates why the prediction target matters. For the martingale-difference pair, the next-step readout from a reservoir with a bias inside the tanh has a test error of 0.5000.500. Adding a squared-target readout on the same states reduces the error to 0.0000.000. Corollary 1 explains why: both classes have zero conditional mean, so the population next-step readout is zero in both. On the 128 datasets of the UCR archive, adding the squared target also improves the accuracy of a practical model space on 94 to 101 datasets, depending on the classifier (see our reply to Reviewer HNeB).

Appendix F.12 illustrates why sampling noise matters. For the two-tap pair, three reservoirs of equal width and spectral radius yield κ⋆\kappa_\star values of 1.1×1031.1\times10^3, 8.0×1038.0\times10^3, and 6.1×1076.1\times10^7. Below each reservoir's crossover scale, the measured squared separation is close to λ4Lκ⋆\lambda^4 L\kappa_\star. Proposition D.5 explains how signal outside the leading Krylov directions can yield a large κ⋆\kappa_\star: the linear sampling noise is small in those directions. Section 5 discusses this after Theorem 4.

Reviewer_HNeBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a mostly theoretical paper that analyzes reservoirs from the perspective of model-space representation. In the setup, time series drive a reservoir of an echo state network, and the readout (the only trained part of the architecture) is used as a representation for the time series. The paper shows that the readout essentially represents higher-order differences between the modeled processes only weakly when the reservoir uses a small input scaling, thereby detecting them might need very long sequences. The paper also shows two situations in which the readout is not able to distinguish between two processes, and proposes simple ways to overcome this (using a bias and a squared target). Experiment analysis is shown using synthetic processes.

Critical strengths
  1. To the best of my knowledge, this is the first theoretically rooted analysis of what a trained readout of an echo state network can represent about the input sequences. The theory seems novel to me, and potentially useful for the reservoir computing community (even if results are under assumptions whose relevance to practical applications is still not very clear to me).

  2. The theory is complemented by experimental analysis, which is appreciable for this kind of contribution.

Critical weaknesses
  1. There are quite a few assumptions in the analysis. In particular, the main results are given basically only for small values of the input scalings and for univariate input time series. From my understanding of the paper, moreover, the result on the sequence length assumes simple linear processes and fixed regularization for the readout. On the other hand, in practice, the selected value of the input scaling that is selected is definitely larger (Appendix F), and from what I can see in Figure2b, at small values of the input scaling, the readout is not able to separate classes with a realistic number of samples. This risks implying that the theory developed in the paper describes a regime that is different from the one that is used in practical situations.

  2. The experiments are limited to synthetic data, which reduces the practical significance of the work. The comparison in Table 6, from my understanding, shows that the proposed remedies show some relevant impact only in 2 of the analyzed cases. In addition to this, values of critical hyper-parameters of the reservoir computing networks, e.g., reservoir size, spectral radius, and regularization coefficient, are fixed. Results are reported without standard deviations or baselines.

Questions for authors
  • Please, analyze the effect of the remedies proposed in the paper on time series benchmarks for real-world applications (see e.g., Bianchi et al., 2021), identifying possible relevant baselines in the comparison (e.g., ROCKET, reservoir models with fully tuned hyper-parameters).

  • From my understanding, the result on the sequence length is tested under the assumptions made in the theoretical part. Can you please assess if this still holds in more realistic settings, including for instance reservoirs with standard random weights, in case of multivariate input signals and non-linear input processes?

  • Please, provide the std values in Table 6. It would also be good if you could check if the results change if model selection tunes the reservoir size, the spectral radius of the regularization coefficient.

  • Suggestion to the authors: it would be really good if you could make the results from the paper a bit more accessible from a practitioner's point of view, for example, by adding a short summary (like a box or a table) including the main practical takeaways from an application standpoint. LDA is also defined in the appendix (from what I understood from the paper), so it would be better defined at the first use in the manuscript.

3 comments

Thank you for the concrete suggestions.

W1. Three parts of the paper are not small-scale statements. (i) The blind spots hold at every input scale and every ridge: the population readout depends on the target only through its conditional mean (Corollary 1), and with zero initial state, no bias, and no intercept, a process and its reflection give one law of the fitted readout (Remark 2). (ii) For reservoirs of finite memory driven by finite-order moving averages of finitely-valued innovations, Proposition D.6 gives the population readout and its noise exactly at every scale. (iii) Appendix D.4 gives the limit λ→∞\lambda\to\infty. Figure 2b combines two costs. Theorem 4 concerns the length LL of one trajectory: at small scale and a fixed ridge, keeping the limiting separation fixed requires LL to grow as λ−4k\lambda^{-4k}. The number NtrN_{\mathrm{tr}} of training sequences adds the cost of learning the discriminant: with 25 sequences per class, shrinkage LDA stays near chance over a range where LDA trained on 1000 sequences already separates the classes. Held-out prediction selects order-one and larger scales (Appendix F.7, λ≥0.5\lambda\geq0.5 in 450 of 450 selections at ρ=0.9\rho=0.9). The order of a difference also sets where detection starts: computed from these exact moments (Figure 3c), the scale at which the Gaussian-limit error reaches 0.1 at L=1024L=1024 is 0.22 to 1.67 at order four and 0.94 to 4.39 at order six (Table 14).

W2. The remedies change the result in four of the five tasks. Under LDA: four families 0.500→0.1050.500\to0.105, AR against bilinear 0.319→0.0000.319\to0.000, AR against AR-GARCH 0.503→0.1430.503\to0.143, AR against threshold AR 0.177→0.0270.177\to0.027. The exception is the Kuramoto task, where the plain readout already separates the classes (0.0820.082), and there is no blind spot to remove.

Baselines, standard deviations, fixed hyper-parameters (and Q3). Table 7 includes baselines (raw sequence, 20 generic statistics, AR(20) coefficients) and now standard deviations over reservoir draws, at most 0.044. New Table 8 chooses nn, ρ\rho, the ridge and λ\lambda jointly by five-fold cross-validation. Most errors fall (four families under LDA: 0.350 / 0.262 / 0.069 for plain / bias / both, against 0.500 / 0.277 / 0.105). On every task and rule, the variant with both remedies has the lowest error or lies within 0.01 of it. On AR against AR-GARCH, the plain readout and the readout with a bias stay at chance under LDA (0.499, 0.485), with both remedies at 0.112. The role of each remedy is isolated by the controlled pair of martingale differences in Table 1, for which Corollary 1 makes both population next-step readouts zero: plain, bias only, squared target only, and both give 0.502, 0.500, 0.493, and 0.000. On AR against bilinear, the tuned plain readout reaches 0.026, and a bias lowers the error by 0.023 there.

Q1. Our paper is a theory of the basic model-space representation rather than a new classifier. Benchmark comparisons are published for methods built on fitted readouts: Bianchi et al. (2021) evaluate tuned reservoir model spaces on multivariate archives, and SARC (ICML 2025) is compared with Rocket, MiniRocket, and Hydra on the 128 UCR datasets and reports the highest average accuracy. These methods add structure to the basic readout, which is the object of our theorems. Our claims concern mechanisms: which class difference reaches the readout, at what order, at what cost in length. On a benchmark, the difference between the classes is unknown, so an accuracy change cannot be attributed to the blind spot or the order a theorem names, and the experiments therefore control the class difference. We make no claim about the accuracy gain of the remedies on real benchmarks: they matter when the class difference lies in a blind spot, and whether it does is a property of each benchmark.

Q2. New Appendix F.12 and Figure 6 (width 10, spectral radius 0.5, ranges and slopes over cells with d2≥0.2d^2\geq0.2). On dense and sparse Gaussian reservoirs, for which we did not check Assumption 2, and on the orthogonal one, d2/(λ4Lκ⋆)d^2/(\lambda^4L\kappa_\star) is 0.95 to 1.14 below a crossover scale, with κ⋆=6.1×107\kappa_\star=6.1\times10^7 on the dense one. With two input channels d2/(λ4L)d^2/(\lambda^4L) is 0.20 to 0.23 for λ\lambda from 0.1 to 0.4. On an exponential autoregression against its time reversal, it is 320 to 410 for λ≤0.03\lambda\leq0.03. On a threshold autoregression with non-zero mean, the ratio is still rising at the smallest measurable scale. Wherever two or more lengths qualify, d2d^2 grows linearly in LL (log-log slope 0.94 to 1.01). For vector input, Proposition D.2 already gives the cubic separation law, though not the lower bound.

Suggestions. Section 7 (Table 2) now gives the practical consequences, and LDA is defined at its first use in Section 3.

Thank you for the response, based on which I keep my positive evaluation of the paper.

The added information and material adds most of the issues raised in my review. The limitations section is now better clear to me and I believe it would make the scope of the paper more clear for the readers too. Also, the effects of the remedies are more clear now.

I understand the point of the authors on the real-world datasets as the possible differences among the classes might not be known. However, I still think that even one single example on real dates would certainly help the potential reader to better see when these blind spots matter in applications.

Thank you for your follow-up and suggestion. Corollary 1 also applies to the model spaces used in practice, whose reservoirs have a bias input and whose readouts have an intercept: the population next-step readout depends on the target only through its conditional mean. We therefore tested the squared target it points to on the 128 datasets of the UCR archive, with the official train/test splits and 3 seeds per dataset. Both variants use a reservoir with a bias input and a readout intercept, and both representations have 40 coordinates: a 40-unit reservoir predicting the next value, against a 20-unit reservoir predicting the next value and its square.

ClassifierMean gain in test accuracyWins / ties / losses
Ridge+4.6 percentage points94 / 18 / 16
Shrinkage LDA+4.6 percentage points101 / 11 / 15
1-NN+4.8 percentage points94 / 11 / 23

Differences below 0.5 percentage points count as ties, and the LDA results cover 127 datasets. Using the value two steps ahead as the second target instead, on the same reservoirs, raises the mean accuracy by less than one percentage point. This suggests that the choice of the second target matters, rather than simply adding another output.

Reviewer_CW4kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The underlying context of the work presented by the authors is that high-dimensional and/or complex data can often be more efficiently summarised by fitting a model to it, and using that model as a representation of the data, upon which one can use other classification methods to distinguish different classes of data. In the context of time series data, the authors focus on echo-state or reservoir networks, where fixed input to hidden and recurrent weights produce a high-dimensional, feature map with memory that is read out downstream by a linear (ridge) decoder. The interesting problem then becomes under which circumstances such a model can still distinguish sequences produced by different processes. This is what the authors address.

In particular, their results focus primarily on manipulations to the relative scale of inputs in the recurrent pool λ\lambda, the recurrent weight spectral radius (memory timescale) ρ\rho, length of sequences LL, and number of sequences NN (they also report changes to hidden layer width nn and ridge penalty γ\gamma); networks are trained to predict sequences, which gets better with LL, and an LDA classifier is trained to distinguish sequences of different processes, which gets better with NN. They provide analytical results for how these variables produce varying degrees of distinguishability between sequences from different processes and also derive sampling time/costs. Finally, they provide useful (simple) architectural changes that resolve issues with distinguishing processes, namely adding a bias to the recurrent pool and squaring the target.

Critical strengths

The approach is sound, as far as I can tell. The proofs are sensible, and the conclusions are interesting, useful, and sound. There is also clear validation of their results, which is presented fairly succinctly (though text could be clearer). The results are potentially useful/illustrative for a range of fields, including (most importantly) system identification and biological dynamical systems.

Critical weaknesses

The critical weakness of this paper is the lack of clarity in its exposition. The paper was quite hard to get through and was, at times, almost incomprehensible. For example, the phrases two-tap and eight-tap come out of nowhere, and this terminology is not familiar to me. Another example is the introduction itself, which does a decent job of convincing me that using models as a representation of data is important but cites related work in a choppy way that serves primarily to introduce jargon without explanation. Moreover, the exact motivation for studying ESNs in particular is lost on me, though I implicitly have some intuition for why they might be interesting in and of themselves. The rest of the paper, including the final few sections with the results, are particularly dense and unclear. The only reprieve is the beginning of section 2, which is decently well-written, but barring that, the paper is overly dense and verbose, which detracts from the (potentially sound) points it is trying to make. Importantly, section 6 should be moved up to be section 2 and should be made a lot clearer.

A very minor point is that I would rename the variable nn to something else, as it was confusing to read NN (number of sequences) and nn (number of hidden neurons).

Questions for authors

More of a clarifying question for myself but could be relevant to clarifying the narrative in the paper: is the point of studying ESNs to study nonlinear feature maps with memory? Or is there implicitly a claim that ESNs might be a better architecture than kernel machines or some other kind of representation? Why not fit e.g., a GRU/LSTM (or is this not analytically tractable enough)?

Was there any reason for picking LDA in particular? Have you tried other classification methods?

3 comments

Thank you for the detailed comments on the exposition. We revised the paper based on all of them.

Clarity, jargon, and the order of sections. Related work is now Section 2, directly after the introduction, as you suggested. The first paragraph of the introduction is rewritten. It explains what model-space learning does (each sequence drives one fixed random recurrent network, a linear readout is fitted to forecast the sequence, and a classifier works on the fitted readouts) before citing the methods built on it, and it no longer lists citations that introduce terms without explanation. The "two-tap" and "eight-tap" pairs are now introduced with the term: a moving average is a linear filter of iid innovations, its coefficients are its taps, and a pair compares a filter with the same taps in reverse order, which keeps the autocovariance and the marginal law and changes only higher-order moments. Section 6 opens with a roadmap, and each paragraph begins with what it tests. A new table (Section 7) summarizes the results by the way two classes can differ.

nn and NN. The number of training sequences per class is now NtrN_{\mathrm{tr}} throughout, including the figure legends, and nn denotes only the reservoir width.

Q1. Why echo-state networks, and why not a GRU or LSTM? The echo-state readout is the representation of model-space learning, the method whose theory we develop. Two properties make it the right object. First, the recurrent weights are fixed, and only a linear readout is fitted, in closed form, so each sequence has exactly one readout, and it converges to a functional of the law of the process. This makes "what does the representation retain" a well-posed question. A GRU or LSTM has no such fit. All of its weights must be trained by backpropagation through time on a non-convex loss, so the parameters fitted to a sequence depend on the initialization and the optimizer, and are identified only up to the symmetries of the network, such as a permutation of the hidden units. They are therefore not a statistic of the process in the same sense, and fitting one network per sequence takes an iterative training run instead of one linear solve. Second, the features are nonlinear, so the readout captures more than second-order statistics while staying tractable, as random-feature models do. We make no claim that ESNs beat kernel machines or other representations. Part of the analysis transfers to other fitted models (new Remark 5): the conditional-mean restriction needs only features that are functions of the past, the sign symmetry only odd features, and the counting of orders holds for a ridge readout on odd smooth maps of a finite window of scaled inputs.

Q2. Why LDA? Theorem 4(a) shows that, in the local limit of the sampling law and for two classes that share the linear noise V1V_1 (a hypothesis at fourth order, automatic at higher orders), the fitted readouts are asymptotically Gaussian with one shared covariance. In that model LDA is the Bayes rule, and Theorem 4(b) shows that LDA with the population parameters attains the lower bound that holds for every classifier with convex or semialgebraic decision regions of bounded complexity, a class that contains QDA, trees, forests, ReLU networks of fixed size and nearest-neighbour rules with a fixed training set. With few training sequences we use shrinkage LDA. We also report 1-NN and an RBF support vector machine (Tables 7 and 8), and the main effects of the remedies are also visible under these two rules.

Thank you for your response to the review. It is clear from the updated manuscript that the authors have taken more care in their exposition, and the paper is much more readable now. The explanations in the response make sense, though I think the GRU/LSTM comment is a imprecise. You could train the recurrent weights of a network and fit sequence-dependent readouts? Wouldn't that achieve the same guarantees? Could you clarify which results would not carry over from your setup if you did this instead? I would greatly appreciate the clarification. In any case, I am willing to upgrade my score to a 3.

Thank you for reconsidering your score. Our first reply addressed training a separate network for each sequence. In the setup you suggest, a shared GRU or LSTM is trained once and then held fixed, so each sequence still has a unique, closed-form ridge readout. The conditional-mean restriction (Corollary 1) also applies whenever the stationary state is a function of the inputs observed so far. The reflection blind spot (Remark 2) and the order counting in Theorem 3(a) rely on an odd state map, which standard GRUs and LSTMs do not guarantee. Theorem 3(b) concerns random reservoir weights, and the sampling law and lower bound of Theorem 4 would need a separate analysis for a trained network. We appreciate the suggestion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank the reviewers for their careful reading. We first restate what the paper is for, since two reviewers asked for benchmark comparisons, and then list the changes in the revised PDF.

What the paper is. Model-space learning represents each time series by the readout of a fixed random reservoir fitted to forecast it, and classifies the fitted readouts (Chen et al., 2014; Aswolinskiy et al., 2018; Bianchi et al., 2021). This representation is the common core of later methods that add spectral information (SARC, ICML 2025), graphs over variables and scales (SVGL, AAAI 2026), or continuous-time models. SARC reports the highest average accuracy among nine methods on the 128 UCR datasets, including Rocket, MiniRocket, Hydra, and InceptionTime, and SVGL reports the highest average accuracy among twelve methods on 30 UEA datasets. To our knowledge, this line has no theory of what the fitted readout retains. Our paper develops one for the basic representation these methods share: which differences between two processes reach the readout, at what order of the input scale, at what cost in sequence length, and which differences it cannot see at all.

We do not propose a new classifier. The works above motivate the representation and compare accuracy across benchmark collections. Our experiments instead measure what the theorems describe: whether a specified class difference reaches the readout, at which order of the input scale and at what length, and whether a bias or a squared target restores a missing signal. They use processes whose class difference is known, because only then can a measured effect be attributed to the mechanism a theorem names.

Changes in the revised PDF:

  1. Presentation (all reviewers). The introduction explains model-space learning in plain terms and why echo-state networks are the object (closed-form fit, one readout per sequence). Related work moved from Section 6 to Section 2. Taps (filter coefficients) and LDA are defined at first use. The number of training sequences per class is now NtrN_{\mathrm{tr}}, so that nn is only the reservoir width. Section 6 opens with a roadmap, and each paragraph states what it tests.
  2. Practical consequences (HNeB). New Section 7 with Table 2: for each way two classes can differ, what the plain readout sees and what to try.
  3. What depends on the reservoir (a8jP). New Remark 5: the order counting of Theorem 3(a) holds for a ridge readout on any odd map with bounded derivatives of a finite window of scaled inputs (argument in Appendix C.2.1). The recurrence decides which lags enter, the genericity of Theorem 3(b), and the non-degeneracy of the noise that Theorem 4 needs.
  4. Robustness of the sampling law (HNeB). New Appendix F.12 and Figure 6: dense and sparse Gaussian reservoirs, two input channels, and two nonlinear autoregressions with Gaussian innovations against their time reversals. Below, d2d^2 is the measured separation of the class means, and the reported ranges and slopes are over cells with d2≥0.2d^2\geq0.2. On the orthogonal and the two Gaussian reservoirs, d2/(λ4Lκ⋆)d^2/(\lambda^4L\kappa_\star), with the exact constant of each reservoir, lies between 0.95 and 1.14 below a crossover scale. With two channels and on an exponential autoregression, d2d^2 grows as λ4L\lambda^4L at small scales. Wherever two or more lengths reach this threshold, d2d^2 grows linearly in LL (log-log slope 0.94 to 1.01). On a threshold autoregression with non-zero mean, the ratio to λ4L\lambda^4L is still rising at the smallest scale at which d2d^2 can be measured.
  5. Standard deviations and tuned hyper-parameters (HNeB). Table 7 reports standard deviations over reservoir draws, and new Table 8 chooses nn, ρ\rho, the ridge, and λ\lambda by cross-validation. Most errors fall, and the conclusions stand. On the AR / AR-GARCH pair, whose classes share the law of their autoregressive coefficients and differ in the volatility of the innovations, the readout leaves chance under LDA only with both remedies (0.112 against 0.499 and 0.485).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.