- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Sequence models are often used in time-series forecasting by autoregressively generating long trajectories, with each one-step prediction fed back in as the next input. However, they are only trained to predict one-step ahead, which this paper argues is an unreliable indicator of long horizon behaviour. That is, having low one-step loss does not tell you whether the generated trajectory is stable. Each one-step prediction has a small error that nudges the state off the correct trajectory. The paper distinguishes the accumulation of these errors from the question it studies, which is whether the dynamical system governing the state has an attracting cycle that pulls nearby states back onto it. Without such a cycle, the errors persist or grow, typically causing the state to decay or explode. The existence of such a cycle depends mostly on the choice of architecture, with some models, such as linear SSMs, never having one, regardless of how well they are trained and how close their loss gets to zero.
For LTI SSMs with input sequence , the state is updated by with . When the model runs with its own predictions used as inputs, the state is updated by (ignoring bias terms). Since this is a linear dynamical system, its long-term behaviour is governed by the spectral radius of , with causing the signal to decay, causing it to explode and keeping it bounded at an amplitude proportional to its starting size. This case implies that if an error nudges the trajectory, it will remain there unless nudged again. A linear SSM (with no non-linearities elsewhere in the model) therefore never has an attracting cycle. The paper claims that whether a trained model has less than or greater than one is not decided by the loss and thus that in the LTI SSM setting, one should study rather than the loss to understand the long-horizon behaviour of the forecast.
For non-linear models (including LTI SSMs that have non-linearities introduced elsewhere in the model), there is no single to compute. Instead, the paper proposes a simple test to identify whether a trained model has an attracting cycle. This only requires running the model forward, and unlike Lyapunov exponents and Floquet multipliers, it does not require computing derivatives or knowing the period of the orbit. The test is as follows: (1) burn the model in on real data, (2) scale the resulting state by several factors, (3) run each scaled copy on its own predictions, and (4) compare the sizes of the final oscillations. If all copies end at the same non-zero size, the model has an attracting cycle whereas if each copy’s final size is proportional to its scaling factor, it has none. The latter is the outcome for models in which doubling the input doubles the output, such as linear maps and ReLU networks without biases.
The paper verifies that the test behaves as expected on known dynamical systems, compares it with Lyapunov exponents and Floquet multipliers and applies it to a range of architectures on synthetic and ECG data.
Critical strengths
-
The paper addresses a real practical problem in time-series forecasting models. Most of these models are usually selected on one-step loss but deployed on long horizons; the paper makes it clear that such losses are not guarantees for good long-term behaviour. It does not propose a fix, but stating the problem clearly is useful in itself.
-
The test proposed in the paper is cheap and simple, with only forward passes required. Therefore it can be used as a quick diagnostic for stability.
-
Since the test scales the state by large factors, it can show whether states far from the cycle are also attracted back to it, whereas Lyapunov exponents and Floquet multipliers can only measure whether states very close to it get attracted back.
Critical weaknesses
-
The model tested is not the model described. The introduction describes the SSM as an autoregressive recurrence whose long-term behaviour is given by . However, the experiments use an SSM with a sliding window; so the SSMs state is restarted from zero at every step and run using only the recent inputs (betwen and in experiments). Thus, for a LTI SSM, this is a finite autoregressive model that only approximates the full one when the window is large. The appendix reports that changing the window length between , and points can move by up to . This is comparable to how far many trained seeds are from one. The exact recurrent closed loop is only compared with the companion, and only for six seeds, whereas the 64-seed is computed at . Seeds with close to could therefore switch sides if the SSM were run recurrently, which would change whether they decay or diverge. This should be made more clear in the main text, and SSMs without sliding window inputs should be tested.
-
Limitations of the test. i) The attractor verdict depends only on the final "size" of the various scaled copies. Hence, copies that settle onto different orbits of similar sizes would be read as converging to a single attracting orbit, even if none of those orbits is attracting. ii) The test scales the state about the origin but measures size about the mean. Thus, if the same dynamics are expressed in coordinates shifted by a constant, the test's verdict might change. While this does not affect the paper’s datasets (since they are centred), it limits the test’s use more generally. iii) The test provides no information for models whose first operation is normalisation without skip connections. While this is acknowledged in the paper, it is still quite a big drawback. This includes models like the normalise-then-replace S4 and LinOSS stacks used in the paper. This wiring is not standard (the paper itself uses pre-norm for ECG), so the test gives no evidence about attractors for the stacked rows of Table . iv) The test's conclusion depends on multiple thresholds that seem to have been chosen by hand/arbitrarily. It is unclear how robust the test is to changes in these thresholds. The appendix attempts to check this but only for standard known dynamical systems and not with trained models on real or synthetic data.
-
The claims go beyond the evidence. The paper claims to identify "a failure no drift remedy can touch". This only follows from the paper’s results for affine models or for models in which scaling the input scales the output by the same factor. It is not shown for the nonlinear models where failures are observed, such as Mamba . The paper also does not test remedies such as multi-step losses (such as path-based metrics), noise injection or scheduled sampling. The related works section argues against some of these by appealing to Proposition , which only applies to homogeneous loops. Apart from TimesFM, all models are small and only trained for a short number of steps, on one simulated oscillator and one real dataset (ECG one). More up to date selective SSMs are barely used, with the only one being a single-layer Mamba example. Therefore it is hard to conclude that the evidence strongly supports the statements.
-
Theory and novelty. It is well known that the long-term behaviour and stability of linear or affine dynamical systems is governed by the spectral radius. The remaining mathematical results are relatively simple. Proposition is false as stated for degrees other than or since scaling compounds as rather than (although only these degrees are used in the paper). Proposition is titled "Teacher forcing pins near one but not to a side. The "not to a side" part is proven, but the "near one" part is not; part (b) only shows that a model with exactly zero loss has , and does not say how close is to one, which is only shown empirically.
-
Clarity. Overall the paper is very hard to follow. The introduction does not say that the synthetic experiments use the van der Pol oscillator, which is only first named in the title of of the plots of Figure and only explained in the appendix. Many terms are used before they are defined or not defined at all such as "host", "informative trained loops" and "isolated normalisers". Similarly, many terms have multiple meanings such as , and . The figures are dense, and key evidence is typically only included in the appendix rather than in the main paper.
Questions for authors
- Can the -seed experiment be repeated without the sliding windows as input?