acceptodds
Withdrawn from ICLR 2027

Learning Chaos Without Seeing Chaos: Extrapolation of Period-Doubling Cascades in Autoregressive Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Autoregressive models are trained to predict a system's behavior one step at a time, and recursive generation allows the learned dynamics to unfold over long horizons. To what extent can such dynamics learned from local observations recover broader organization of an underlying system that was only partially observed during training? We study small autoregressive transformers trained from scratch on trajectories sampled from restricted parameter regimes of several non-linear dynamical systems, including logistic and sine maps, the Lorenz system, and the generalized Hopf system, with control parameters and state trajectories represented as sequences of continuous tokens. Under closed-loop evaluation at parameter ranges far outside the training distribution, the models recover correct self-similar period-doubling cascades, chaotic dynamics, and attractor structures with remarkable visual and numerical fidelity. For the logistic map, a transformer reproduces successive period doublings up to period 128, yielding a finite-order scaling ratio of 4.6687, matching the Feigenbaum constant to within . We further investigate how these structures emerge over the course of training, and reveal with causal interventions how control-parameter information is processed through attention to shape the learned dynamics. These results suggest that a surprisingly narrow window into a system's local behavior may suffice for autoregressive transformers to generalize to its unseen global dynamical organization.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
dynamical systems, autoregressive models, transformers, bifurcation, chaos, non-linear dynamics, machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_c2BwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Learning Chaos Without Seeing Chaos investigates whether autoregressive transformers trained on short observations from restricted dynamical regimes can recover unseen long-term behaviour. It demonstrates period-doubling cascades, chaos and multistability using scalar maps, the three-variable Lorenz system and the two-variable generalized Hopf system. Its main contribution is structural extrapolation beyond the training regime, although scalability to spatially extended PDEs remains untested. The small models suggest modest training requirements, but computational simplicity does not show much effort to this paper. I would like to reject this paper accordingly.

Critical strengths

The evaluation examines long-term dynamical structure beyond one-step prediction accuracy. Numerical bifurcation verification and controlled interventions on parameter-conditioning pathways provide useful evidence about the learned models’ behaviour.

Critical weaknesses
  1. Insufficiently established additional insight over simpler models. Appendix D shows that an MLP reproduces a period-doubling cascade of the same order, while Table 3 reports substantially lower distributional errors for polynomial NVAR. These results do not invalidate the transformer observations, but leave the significance of the transformer-focused contribution insufficiently established. The intervention analysis demonstrates that parameter information affects predictions, without fully explaining why extrapolation succeeds.

  2. Restricted experimental scope and uneven quantitative fidelity. Validation is confined to small maps and low-dimensional ODEs, leaving applicability to spatially extended or higher-dimensional systems untested. Moreover, reproducing Feigenbaum scaling does not establish accurate bifurcation placement: the reported transition locations exhibit seed-dependent displacement. Although the paper acknowledges this distinction, stronger evidence is needed to establish the broader significance of its structural-extrapolation results.

I recommend rejection. The observations are interesting, but the additional insight beyond simpler predictors and the generality of the conclusions are not sufficiently established.

Questions for authors
  1. Contribution beyond simpler models: The MLP reproduces a cascade of the same order, while polynomial NVAR achieves lower distributional errors. What additional scientific insight does the transformer analysis establish beyond these baselines?
  2. Scope and limitations: With validation restricted to scalar maps and low-dimensional ODEs, which conclusions do you expect to extend to higher-dimensional, spatially extended systems? What factors would limit this extension?
Reviewer_ZG2HOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether autoregressive Transformers trained on local, one-step predictions within restricted parameter regimes can recover unseen global dynamical structures through recursive generation. Experiments on logistic and sine maps, the Lorenz system, and a Hopf system examine extrapolation to period-doubling cascades, chaotic behavior, and multistability. The authors evaluate the generated dynamics using distributional comparisons, numerical bifurcation analysis, scaling ratios, and Lyapunov exponents. The results demonstrate that a surprisingly narrow window into a system’s local behavior may suffice for autoregressive transformers to generalize to its unseen global dynamical organization.

Critical strengths
  1. The paper investigates extrapolation of dynamical structure beyond the training parameter range, a relevant problem in data-driven dynamical modeling. The experiments cover logistic and sine maps, the Lorenz system, and a generalized Hopf system, examining period doubling, chaos, and multistability.

  2. Parameter-label permutation, activation patching, and attention-blocking experiments help establish that parameter conditioning affects the generated dynamics. The appendices also report baseline results and failure cases, making the scope and limitations of the findings more transparent.

Critical weaknesses
  1. The conceptual novelty is limited, and the main new scientific insight remains unclear. Learning dynamics from restricted observations and extrapolating bifurcations or attractor structures to unseen parameter regimes have been investigated previously. The main difference here appears to be the use of a different architecture and more detailed numerical verification. The paper does not sufficiently explain what substantial new understanding it provides. In other words, the paper does not offer either a novel methodological contribution or a theoretical insight.

  2. The distinction from the closest prior work is not clearly established. Kim et al. (2021) studied RNNs that infer and extrapolate bifurcation and period-doubling structures from local examples. Köglmayr and Räth (2024) and van Tegelen et al. (2025) investigated parameter extrapolation using reservoir computing and Neural ODEs, respectively. Casert et al. (2024) also demonstrated that Transformers can generate emergent behavior absent from training under unseen conditions. Although these studies are cited, the paper lacks a sufficient comparison of the available training information, extrapolation tasks, and resulting scientific conclusions. Consequently, it is difficult to determine whether the contribution extends beyond demonstrating previously observed capabilities in another architecture and several benchmark systems.

Questions for authors
  1. What is the key novel scientific conclusion relative to the closest prior studies? Please distinguish between reproducing established phenomena, providing more detailed validation, and establishing findings that were not previously known.

  2. Given that the MLP already reproduces the core period-doubling results, what are the specific reasons for emphasizing Transformers?

  3. If no clear performance advantage is established, would the authors consider framing the contribution as an empirical study of structural extrapolation across architectures?

Reviewer_siaHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the ability of surrogate maps learned from parametric dynamical systems data to generalize to unseen settings of the system parameters. The authors show that for both simple discrete maps and a couple continuous dynamical systems that, when provided the system parameters as input, MLPs and transformers can reliably extrapolate to dynamics beyond bifurcation transitions not observed during training. The authors also show that the transformers learn to causally use the system parameter input to perform forecasting via activation patching.

Critical strengths
  • The authors show that one-step-ahead training and system parameter conditioning reliably allows the learned maps to extrapolate over bifurcations for unseen system parameter settings. The results are especially strong and quantified for the discrete time maps.
Critical weaknesses
  1. The discrete time maps make up a majority of the quantitative results. As noted by the authors, the logistic map and the sine map are separable in the control parameter i.e. r * g(x). For the logistic map, although the models are not trained in any regime beyond the first period doubling, the learning problem is still extremely simple and highly constrained as the map is a degree 3 polynomial separable in the control parameter. The sine map is less straightforward but the NVAR results show that a low-degree polynomial is capable of fitting the forward map so I would expect that the 8K+ parameter MLP and transformer baselines can also do that (to some degree) if they are capable of modeling the logistic map. All to say, that there should be more quantitative results or mechanistic studies for the continuous systems which should be harder given that their forward maps come from an RK scheme. Moreover, the continuous time systems are the more practically relevant systems.
  2. The focus on transformers is not very well supported. The tables in the appendix show that MLPs can achieve the same extrapolation capability (even slightly better than the transformer for the logistic map). I understand that transformers enable the use of counterfactual interventions such as activation patching to establish the causal effect of the system parameter embedding, but given that there is only one axis of variation being probed (the effect of input parameter embedding), a simple ablation where the model is trained without that feature is another equally valid empirical signal. So it just seems like the transformer is a more complicated learned map that enables a more complicated analysis where an MLP could have sufficed.
Questions for authors
  • Why are there no non-transformer baselines for the continuous time systems?
  • Beyond MSE and estimated lyapunov exponents are there any other quantitative results that can be presented for the continuous time systems that reveal mechanistically how extrapolation happens? The results in the main text for the continuous time systems are purely qualitative.
  • Why is there a focus on transformers for the discrete time systems specifically? Couldn't a similar blocking intervention be done for the MLPs, or even a simple parameter embedding ablation as mentioned in Weakness 2?
Reviewer_zYRcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether autoregressive transformers trained on trajectories from restricted parameter regimes can reproduce dynamical structures outside those regimes. Experiments cover logistic and sine maps, the Lorenz system, and a generalized Hopf system. The principal scalar result is a numerically verified period-doubling cascade reaching period 128, with finite-order scaling close to the Feigenbaum constant. The flow experiments demonstrate unseen chaotic behavior and coexistence of attractors. Additional analyses investigate training dynamics, training-window sensitivity, and the role of parameter information transmitted through attention. The paper provides interesting evidence that local predictive training can produce autonomous models with meaningful unseen dynamical organization.

Critical strengths
  1. Diverse metrics. The paper measures bifurcation locations, finite-order scaling ratios, periodic-orbit multipliers, long-run distributional distances, largest Lyapunov exponents, and multistable outcomes. These diagnostics address different aspects of learned dynamics and strengthen the empirical contribution.

  2. Experiments on different chaotic systems. The Lorenz model exhibits positive estimated largest Lyapunov exponents at tested chaotic parameters. The Hopf experiments demonstrate different long-time outcomes from different initial conditions at the same parameter values, supporting learned coexistence of attractors.

Critical weaknesses
  1. The incremental scientific contribution. Parameter-conditioned extrapolation of bifurcations and unseen dynamics has precedents in recurrent and reservoir models. Within this paper, the MLP also reaches the same verified logistic cascade order, while the polynomial baseline achieves substantially smaller distributional errors. These comparisons do not diminish the value of demonstrating the phenomenon in transformers, and outperforming baselines is not required for an empirical contribution. However, the paper should explain more precisely whether its main advance is the verification methodology, an observation about transformer inductive biases, or a mechanistic account of successful extrapolation.

  2. The available diagnostics establish selected aspects of fidelity, rather than comprehensive dynamical agreement. The largest Lyapunov exponent is meaningful evidence of chaotic instability, but it leaves the remaining stability spectrum untested. Agreement in the largest exponent does not establish agreement in contraction rates, neutral directions, or the number of unstable directions in higher-dimensional settings.

  3. The interventions identify parameter mediation more clearly than the mechanism of extrapolation. Attention blocking and activation patching show that parameter information affects predictions and autonomous behavior. However, the architecture already makes attention the communication route between separate parameter and state tokens. These experiments do not yet explain why the learned parameter dependence extrapolates successfully, or when it should fail.

  4. Relevant prior work on statistical accuracy. The discussion of local prediction accuracy versus autonomous statistical fidelity would benefit from engagement with Park et al. (NeurIPS 2024) and Jiang et al. (NeurIPS 2023), which explicitly investigate this distinction and the preservation of invariant measures. The Lyapunov analysis should also be situated relative to Pathak et al. (Chaos, 2017), which studies recovery of chaotic attractors and Lyapunov spectra, including for Kuramoto–Sivashinsky dynamics. Please clarify how the present results extend this literature through extrapolation across unobserved parameter regimes.

[1] Ruoxi Jiang, Peter Y. Lu, Elena Orlova, and Rebecca Willett. “Training neural operators to preserve invariant measures of chaotic attractors.” Advances in Neural Information Processing Systems, 36, 2023.

[2] Jaideep Pathak, Zhixin Lu, Brian R. Hunt, Michelle Girvan, and Edward Ott. “Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data.” Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(12):121102, 2017.

[3] Zongyi Li, Miguel Liu-Schiaffini, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. “Learning chaotic dynamics in dissipative systems.” Advances in Neural Information Processing Systems, 35, 2022.

[4] Jeongjin (Jayjay) Park, Nicole Tianjiao Yang, and Nisha Chandramoorthy. “When are dynamical systems learned from time series data statistically accurate?” Advances in Neural Information Processing Systems, 37, 2024.

Questions for authors
  1. What specific new conclusion about transformer learning remains after accounting for the MLP’s successful cascade and the quartic-family failure?

  2. Can you compare the remaining Lyapunov exponents of the learned and reference Lorenz systems, using consistent physical-time normalization and a convergence check?

Comments on the paper

1 comment

We have decided to withdraw this submission from ICLR. We sincerely thank all the reviewers for their time and detailed feedback, and before closing the discussion, we would like to clarify several points about the focus of our paper and what our results establish.

  1. We train transformers to predict the next state using trajectories from restricted parameter regimes, with the control parameter and system state represented as separate continuous tokens. We ask whether their autoregressive rollouts can recover dynamical structures absent from training. Our contribution is mainly an empirical investigation of this capability in transformers. Its value rests on what the findings reveal about transformer learning and generalization, rather than on whether transformers outperform MLPs or other predictors.
  2. We further examine how the recovered structures emerge during training, how the trained model uses control-parameter information to shape its generated dynamics, and the boundaries of this capability.

We will take the time to revise and strengthen the work before submitting it elsewhere. We appreciate all the reviewers’ efforts, which have helped us identify substantial opportunities to clarify our contribution and further improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.