acceptodds
Withdrawn from ICLR 2027

Sustained Is Not Stable: A Reference-Free Test for Attractors in Free-Running Sequence Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon generative sequence models are trained and scored under teacher forcing, which resets the state to ground truth at every step, and are deployed in closed loop, where nothing does. We show the objective is structurally silent on the closed loop: for a linear recurrence the one-step loss is flat, at a perfect fit, along readout directions that move the closed-loop spectral radius, and its sublevel sets contain both decaying and diverging loops at any residual. Across 64 identically trained single-layer state-space models the held-out loss is unrelated to that radius (Spearman +0.06, CI [-0.21, +0.34]), the radius alone classifies all 64 free-running outcomes, and free-running amplitude spans ten orders of magnitude. We give a reference-free test: scale the burned-in state by , run the model on its own output, and fit settled amplitude against on log-log axes. Slope 0 is an attractor. Slope 1 is a theorem: every positively homogeneous closed loop—LTI recurrences, bias-free ReLU stacks—returns it at any width, depth or training length. Calibrated on twenty known systems (25/26 verdicts), the test agrees with Lyapunov exponents and Floquet multipliers on every trained loop inside its basin, needs no gradients, Jacobians, reference or period, and reports finite basins that the local methods cannot see. Across eight architectures it locates the attractor: not in depth but in whether a non-homogeneous operation lies on the closed loop—a biased readout fed back at depth one supplies it, the same readout outside the loop does not, a two-layer bias-free ReLU stack returns exactly 1.000, depth 2 to 8 changes nothing—and normalisation does not create the attractor but decides whether it is global. On MIT-BIH a two-layer stack collapses on 26 of 32 seeds while beating persistence, and the collapsed seeds include the seven best by loss. The test certifies existence, not correctness; and existence cannot be assumed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
state space models, recurrent neural networks, dynamical systems, attractors, closed-loop stability, teacher forcing, autoregressive rollout, Lyapunov exponents, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TCwwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Sequence models are often used in time-series forecasting by autoregressively generating long trajectories, with each one-step prediction fed back in as the next input. However, they are only trained to predict one-step ahead, which this paper argues is an unreliable indicator of long horizon behaviour. That is, having low one-step loss does not tell you whether the generated trajectory is stable. Each one-step prediction has a small error that nudges the state off the correct trajectory. The paper distinguishes the accumulation of these errors from the question it studies, which is whether the dynamical system governing the state has an attracting cycle that pulls nearby states back onto it. Without such a cycle, the errors persist or grow, typically causing the state to decay or explode. The existence of such a cycle depends mostly on the choice of architecture, with some models, such as linear SSMs, never having one, regardless of how well they are trained and how close their loss gets to zero.

For LTI SSMs with input sequence uu, the state is updated by st+1=Ast+But+1s_{t+1} = As_t + B u_{t+1} with u^t+1=Wst\hat u_{t+1} = Ws_t. When the model runs with its own predictions used as inputs, the state is updated by st+1=(A+BW)sts_{t+1} = (A+BW)s_t (ignoring bias terms). Since this is a linear dynamical system, its long-term behaviour is governed by the spectral radius ρ\rho of A+BWA+BW, with ρ<1\rho <1 causing the signal to decay, ρ>1\rho >1 causing it to explode and ρ=1\rho = 1 keeping it bounded at an amplitude proportional to its starting size. This ρ=1\rho =1 case implies that if an error nudges the trajectory, it will remain there unless nudged again. A linear SSM (with no non-linearities elsewhere in the model) therefore never has an attracting cycle. The paper claims that whether a trained model has ρ\rho less than or greater than one is not decided by the loss and thus that in the LTI SSM setting, one should study ρ\rho rather than the loss to understand the long-horizon behaviour of the forecast.

For non-linear models (including LTI SSMs that have non-linearities introduced elsewhere in the model), there is no single ρ\rho to compute. Instead, the paper proposes a simple test to identify whether a trained model has an attracting cycle. This only requires running the model forward, and unlike Lyapunov exponents and Floquet multipliers, it does not require computing derivatives or knowing the period of the orbit. The test is as follows: (1) burn the model in on real data, (2) scale the resulting state by several factors, (3) run each scaled copy on its own predictions, and (4) compare the sizes of the final oscillations. If all copies end at the same non-zero size, the model has an attracting cycle whereas if each copy’s final size is proportional to its scaling factor, it has none. The latter is the outcome for models in which doubling the input doubles the output, such as linear maps and ReLU networks without biases.

The paper verifies that the test behaves as expected on known dynamical systems, compares it with Lyapunov exponents and Floquet multipliers and applies it to a range of architectures on synthetic and ECG data.

Critical strengths
  1. The paper addresses a real practical problem in time-series forecasting models. Most of these models are usually selected on one-step loss but deployed on long horizons; the paper makes it clear that such losses are not guarantees for good long-term behaviour. It does not propose a fix, but stating the problem clearly is useful in itself.

  2. The test proposed in the paper is cheap and simple, with only forward passes required. Therefore it can be used as a quick diagnostic for stability.

  3. Since the test scales the state by large factors, it can show whether states far from the cycle are also attracted back to it, whereas Lyapunov exponents and Floquet multipliers can only measure whether states very close to it get attracted back.

Critical weaknesses
  1. The model tested is not the model described. The introduction describes the SSM as an autoregressive recurrence whose long-term behaviour is given by A+BWA+BW. However, the experiments use an SSM with a sliding window; so the SSMs state is restarted from zero at every step and run using only the recent inputs (betwen 3232 and 256256 in experiments). Thus, for a LTI SSM, this is a finite autoregressive model that only approximates the full one when the window is large. The appendix reports that changing the window length between 1616, 6464 and 128128 points can move ρ\rho by up to 0.010.01. This is comparable to how far many trained seeds are from one. The exact recurrent closed loop is only compared with the W=256W = 256 companion, and only for six seeds, whereas the 64-seed ρ\rho is computed at W=32W = 32. Seeds with ρ\rho close to 11 could therefore switch sides if the SSM were run recurrently, which would change whether they decay or diverge. This should be made more clear in the main text, and SSMs without sliding window inputs should be tested.

  2. Limitations of the test. i) The attractor verdict depends only on the final "size" of the various scaled copies. Hence, copies that settle onto different orbits of similar sizes would be read as converging to a single attracting orbit, even if none of those orbits is attracting. ii) The test scales the state about the origin but measures size about the mean. Thus, if the same dynamics are expressed in coordinates shifted by a constant, the test's verdict might change. While this does not affect the paper’s datasets (since they are centred), it limits the test’s use more generally. iii) The test provides no information for models whose first operation is normalisation without skip connections. While this is acknowledged in the paper, it is still quite a big drawback. This includes models like the normalise-then-replace S4 and LinOSS stacks used in the paper. This wiring is not standard (the paper itself uses pre-norm for ECG), so the test gives no evidence about attractors for the stacked rows of Table 11. iv) The test's conclusion depends on multiple thresholds that seem to have been chosen by hand/arbitrarily. It is unclear how robust the test is to changes in these thresholds. The appendix attempts to check this but only for standard known dynamical systems and not with trained models on real or synthetic data.

  3. The claims go beyond the evidence. The paper claims to identify "a failure no drift remedy can touch". This only follows from the paper’s results for affine models or for models in which scaling the input scales the output by the same factor. It is not shown for the nonlinear models where failures are observed, such as Mamba . The paper also does not test remedies such as multi-step losses (such as path-based metrics), noise injection or scheduled sampling. The related works section argues against some of these by appealing to Proposition 11, which only applies to homogeneous loops. Apart from TimesFM, all models are small and only trained for a short number of steps, on one simulated oscillator and one real dataset (ECG one). More up to date selective SSMs are barely used, with the only one being a single-layer Mamba example. Therefore it is hard to conclude that the evidence strongly supports the statements.

  4. Theory and novelty. It is well known that the long-term behaviour and stability of linear or affine dynamical systems is governed by the spectral radius. The remaining mathematical results are relatively simple. Proposition 11 is false as stated for degrees other than 00 or 11 since scaling compounds as αkt\alpha^{k^t} rather than αk\alpha^k (although only these degrees are used in the paper). Proposition 44 is titled "Teacher forcing pins ρ\rho near one but not to a side. The "not to a side" part is proven, but the "near one" part is not; part (b) only shows that a model with exactly zero loss has ρ≥1\rho \geq 1, and does not say how close ρ\rho is to one, which is only shown empirically.

  5. Clarity. Overall the paper is very hard to follow. The introduction does not say that the synthetic experiments use the van der Pol oscillator, which is only first named in the title of of the plots of Figure 44 and only explained in the appendix. Many terms are used before they are defined or not defined at all such as "host", "informative trained loops" and "isolated normalisers". Similarly, many terms have multiple meanings such as WW, AA and GG. The figures are dense, and key evidence is typically only included in the appendix rather than in the main paper.

Questions for authors
  1. Can the 6464-seed experiment be repeated without the sliding windows as input?
Reviewer_R5qZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper examines why low teacher forced prediction error does not guarantee stable autonomous dynamics. It proposes a reference free attractor diagnostic based on the dependence of settled rollout amplitude on initial state scaling. The method is supported by theoretical arguments involving homogeneity and closed-loop spectral radius, and evaluated on dynamical systems, multiple sequence architectures, TimesFM,and ECG recordings.

Critical strengths

The distinction between sustained oscillations and genuinely attracting dynamics is important. The analysis connecting teacher forced loss, spectral radius, and autonomous prediction failure is insightful.

Critical weaknesses

There is a mathematical concern with Proposition 1. For a degree k homogeneous recurrence, repeated composition generally produces scaling by powers of k, rather than the claimed fixed exponent k at every iteration. The statement appears valid for the considered cases k = 0 and k = 1, but not for arbitrary k as written. More fundamentally, a zero amplitude scaling slope does not establish the existence of an attracting orbit. Distinct nonconverging trajectories can have identical RMS amplitudes, and perturbations along one radial direction cannot detect instability in other directions. Thus, the proposed test is a useful diagnostic but not a general certificate of attraction. The empirical evaluation is also concentrated on oscillatory systems, limiting conclusions about general sequence generation.

Questions for authors

Can Proposition 1 be corrected or restricted to k = 0 and k = 1? What additional assumptions make a zero amplitude scaling slope sufficient to establish an attractor rather than merely amplitude invariance? How does the proposed test distinguish trajectories with identical amplitudes but different asymptotic dynamics?

Reviewer_anbnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a new way to test whether a model's closed loop will be stable. The main diagnostic tool is a test that works without needing any ground truth trajectories. First, the model's internal state is "burned in" and then scaled by a factor called alpha. Next, the model is run with a loop where the output from the previous step becomes the input for the next step,. After the initial transient dies out, the final amplitude of the output is measured. This measurement is then plotted against alpha on a log-log scale. If the slope is zero, the loop has an attractor (it will sustain a stable orbit.) If the slope is positive, the loop diverges (the amplitude grows over time.) If the slope is negative, the loop decays (the amplitude shrinks to zero.) The test is backed by four mathematical propositions that explain why the slope indicates attractor existence. This framework was tested on 64 different single-layer state-space models that were trained identically except for their random seeds. Each model was run freely for many steps, and the alpha-scaling test was applied to all of them. The method gives practitioners a way to check whether their long-horizon generative model will actually work when deployed and provides a theoretical guarantee that the test result is meaningful.

Critical strengths
  • The test certifies that an attractor exists, not that it is the right one. This distinction is a new conceptual insight: a model may have a stable orbit that still produces incorrect behavior, and the test explicitly acknowledges this gap.
  • There is mathematical analysis. The paper introduces four propositions that link the alpha-scaling test result to dynamical properties of the closed loop. This provides a theoretical guarantee that the test is not arbitrary.
  • The test works on black-box checkpoints without needing gradients or ground-truth trajectories.
  • The paper shows that Mamba collapses despite low loss.
  • Experiments span eight architectures.
Critical weaknesses
  • Most experiments focus on single-layer state-space models and a few transformer variants. Results may not generalize to other model families like diffusion models.
  • The paper focuses on discrete-time models. The extension to continuous-time dynamics is mentioned in passing.
  • The alpha-scaling test was designed for systems that settle to a periodic orbit. Its applicability to non-oscillatory or transient behaviors is unclear.
  • The test requires running the model for many free steps and measuring amplitude, which may be expensive for large models. It is not clear to me that the detected attractors correspond to useful or correct behavior.
Questions for authors
  • How generalizable is the alpha-scaling test to non-oscillatory or transient dynamics?

  • How does the test handle noisy or stochastic systems? Does the slope remain interpretable?

  • Why does Mamba achieve the lowest training loss yet collapse in deployment? Is this a general property of state-space models with gating?

  • How does the depth of the network affect the existence of an attractor?

  • Can the propositions be extended to stochastic or partially observed systems?

  • How could the test be integrated into a model development pipeline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.