acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Prediction

Abstract

Foundation models for dynamical systems can forecast unseen partial differential equations (PDEs), ordinary differential equations (ODEs), and time series during inference. However, it remains an open question whether these foundation models truly learn generalizable in-context forecasting strategies. We show that looped transformers are capable of emulating propagators for dynamical systems in-context by first establishing a link between linear attention and (extended) dynamic mode decomposition. We then generalize this insight into the (TTF) framework in which a one-step forecaster leverages iterations of gradient descent as recurrent depth. We demonstrate that lightweight TTF models trained on ODEs and PDEs generalize to out-of-distribution system dynamics during autoregressive rollout after being trained for one-step prediction. Our results reinforce TTF as a theoretically motivated framework for designing recurrent depth models capable of forecasting unseen dynamical systems at test-time.

Primary area:
learning on time series and dynamical systems
Keywords:
in-context learning, scientific machine learning, recurrent depth models, dynamic mode decomposition

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_X7x6OpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Test-Time Forecasting (TTF), a framework that improves dynamical system forecasting through recurrent computation and test-time optimization. It introduces two variants, iTTF and eTTF, which update predictions or forecaster parameters, respectively. Experiments on ODEs and PDEs show improved out-of-distribution forecasting with increased iterations, while highlighting additional computational costs and limitations in generalization.

Critical strengths
  1. The paper establishes a theoretically motivated connection between recurrent linear attention and Dynamic Mode Decomposition (DMD), providing a principled foundation for the proposed framework.
  2. It unifies implicit and explicit test-time forecasting into a flexible framework that supports different forecasters and optimization methods.
  3. Experiments demonstrate improved out-of-distribution forecasting as recurrent iterations increase across several dynamical systems.
Critical weaknesses
  1. Experiments are primarily conducted on synthetic ODEs and PDEs, leaving performance on real-world time series unverified.
  2. Increasing recurrent iterations introduces additional inference costs, while explicit TTF also incurs substantial training overhead.
  3. The novelty of the proposed TTF framework is somewhat limited, as test-time optimization and adaptation have been extensively explored in other domains.
Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to reviewer

[W1.] In our revised text, we train iTTF models on the Eigenworms dataset [1] which consists of real experimental measurements of C. elegans posture motion under different stimuli and demonstrate the efficacy of basic TTF models in Figure. 5,16,17 in the revised text. Additionally, we train eTTF models on real PIV data of a 2D fluid flow and show in Figure. 8 that minimal eTTF models can beat the standing FNO baseline for this task (see response to reviewer BHyf). We prefer to focus our paper's main experiments on nonlinear dynamics since they better illustrate the connection of the TTF framework to Koopman theory.

[W2.] For iTTF models, LL recurrent iteration costs the same as an analogous LL layer fixed depth model since the recurrent model structurally mirrors the fixed depth model but with tied layer weights. Additional inference cost is only incurred if the user chooses to spend more recurrent iterations to achieve lower error. For eTTF models however, we agree that both training and inference is more expensive and recognize this limitation. Despite the cost and performance discrepancy between eTTF and iTTF models, we choose to present eTTF since it is the natural theoretical generalization of the idea that relying on Koopman operator theory to forecast via regression in-context directly corresponds to looped linear attention; this finding is the core idea we intend to highlight in this work rather than presenting evaluations of production-ready surrogate models.
Relatedly, it is also well known that TTT and looped transformers typically require higher pretraining overhead [2] but gain 1) adaptive test-time compute via loop iterations, and 2) reduced model size (parameters); which we demonstrate in our work.

[W3.] We emphasize that our novelty comes from the link between Koopman operator theory and test-time training in a forecasting setting, and also the theoretical connections to linear attention design. We briefly discuss the broader test-time training field in the related works section which we have updated to include a more detailed discussion.

Our Appendix D theoretically motivates multiple common transformer architecture design choices such as short convolutions, sharpness control, gated linear units, etc. which are implemented in open frontier models due to their empirical benefits, but which have previously eluded clear motivation [3,4,5]. Moreover, prior works primarily justify the use of transformer layers in dynamics forecasting and TSFMs based on their empirical performance, rather than principled motivation. Thus, TTF represents a novel framework that both guides the design of forecasting models capable of in-context learning, and theoretically motivates modern transformer architecture design more broadly, using the lens of optimization dynamics.

References

[1] Greg J. Stephens, Bethany Johnson-Kerner, William Bialek, and William S. Ryu. Dimensionality and Dynamics in the Behavior of C. elegans. PLoS Computational Biology, 4(4):e1000028, 2008.

[2] Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv:2510.26692, 2025.

[3] Kimi Team. Kimi K3: Open Frontier Intelligence. arXiv:2607.24653, 2026.

[4] William Merrill et al. Olmo Hybrid: From Theory to Practice and Back. arXiv:2604.03444, 2026.

[5] Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, and Daniel Y. Fu. Parcae: Scaling Laws For Stable Looped Language Models. COLM, 2026.

Thanks for your feedback, which has addressed most of my concerns. I have raised my score in response.

We thank the referee for their review which has allowed us to improve the related works and experiments.

Reviewer_BHyfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies how transformer-based foundation models in-context forecast dynamical systems by focusing on looped transformers. The paper first reveals that solving the dynamic mode decomposition by gradient descent is equivalent to applying one linear attention layer repeatedly. The authors further generalize this into test-time forecasting, in which each loop iteration is an optimization step on the in-context one-step forecasting loss. The theoretical results are validated with experiments on systems, like linear ODEs, the forced Duffing oscillator, etc.

Critical strengths
  1. The relation between looped linear attention and dynamic mode decomposition is quite novel and interesting. Its proof is clean and correct.

  2. Experiments are well designed and give solid evidence that the proposed construction is learned. Figure 7, 8 and 9 show that the learned convolution filters and projections match the predicted construction.

Critical weaknesses
  1. Some theory is wrong, like Prop.E.2, see Questions For Authors for details.

  2. The PDEs studied in this paper are one-dimensional, which is a bit limited.

  3. Some explanations are not clear enough, like duffing γ\gamma range differs between Sec. 4.1 and App. B.1.

Questions for authors
  1. Are fast weights fit per trajectory or shared across a batch? In eq. 10 says that the in-context trajectory is the "full batch" while in Section 3.2, it says that inner optimizers induce batch size dependence. Please elaborate that.

  2. Is xTx_T included in the context used to compute θL\theta_L and θ∗\theta^*? If yes, Prop. 3.1 and Prop. E.2 bound in-sample fit and should not be presented as forecasting guarantees. I think xTx_T not included is the correct and meaningful version, which makes the latent model risk a real out-of-sample generalization term.

  3. I think there is a mistake in Prop. E.2. This proof asserts ϕi(xT)=Kdϕi(xT−1)\phi_i(x_T) = K_d \phi_i(x_{T-1}) with Kd=Pdμ U∣GdK_d = P^{\mu}_d \, \mathcal{U}|_{G_d}. In fact

ϕi(xT)=(Uϕi)(xT−1),Uϕi−Kdϕi=(I−Pdμ) Uϕi.\phi_i(x_T) = (\mathcal{U}\phi_i)(x_{T-1}), \qquad \mathcal{U}\phi_i - K_d \phi_i = (I - P^{\mu}_d)\,\mathcal{U}\phi_i .

The result in Korda–Mezić only shows Kd,T→KdK_{d,T} \to K_d. Therefore,

lim⁡T→∞E ∥ϕ(xT)−Fθ∗ϕ(xT−1)∥2=∑i∥(I−Pdμ) Uϕi∥L2(μ)2,\lim_{T\to\infty} \mathbb{E}\,\bigl\|\phi(x_T) - F_{\theta^*}\phi(x_{T-1})\bigr\|^2 = \sum_i \bigl\|(I - P^{\mu}_d)\,\mathcal{U}\phi_i\bigr\|^2_{L^2(\mu)},

which is generally nonzero. This limit vanishes only if GdG_d is Koopman-invariant. But this hypothesis is missing.

  1. This paper studies Hurwitz LDS, where the only invariant ergodic measure is δx∗\delta_{x^*}. Then L2(μ)L^2(\mu) is one-dimensional and for d>1d > 1 linearly independent ϕi\phi_i cannot exist. Could please check this and explain it?
5 comments
AuthorsOpenReview ↗

Response to reviewer

Weaknesses

[W1.] Thank you for pointing out the issues here, see question responses below.

[W2.] In our revised text, we train eTTF models on the 2D Cylinder flow data in RealPDEBench [1] which is experimental particle image velocimetry (PIV) data of flow past a cylindrical obstacle. We show in our revised text in Figure 8 that a 6M parameter FNO eTTF model, evaluated with inner batch size B=1, can match the pointwise error and improve the spectral error of the released 100M parameter FNO baseline in the setting of pretraining on real flows and evaluating on real OOD higher Reynolds number flows. We emphasize that our paper is primarily a theoretical and conceptual contribution of the TTF framework, we favor motivating examples like ODEs and 1D PDEs because of their well-studied dynamics and propagators; we will consider larger scale empirical evaluations such as PDEs on unstructured meshes in future work.

[W3.] We have clarified in the text and Appendix B.1 that we chose the Duffing γ\gamma range to capture a bifurcation over attractor geometries and ensure that more difficult dynamics are unseen during training. Specifically, the ID/OOD split on this range generates primarily single well orbits for ID training below γ=0.375\gamma = 0.375, and fully chaotic double well dynamics above γ=0.375\gamma = 0.375 for OOD evaluation. We've additionally added a figure exhibiting the P25, P50, and P75 values for the ID and OOD range for Duffing in the appendix (Figure. 14).

Questions

[Q1.] For iTTF models, the fast weights, which are implicit through the trajectory prediction, are fit per trajectory. For eTTF models, the fast weights are fit to a batch of trajectories from the same system, so each individual trajectory in a batch serves as additional in-context signal for eTTF models. Our eTTF results in our original submission trained and evaluated with specified batch sizes for each system, but we update the results with eTTF models trained and evaluated so that the batch size for the fast weights was B=1 for all systems for a fair comparison and notice consistent performance (Fig. 4,6,16,7,8). Note that we include the full batch B>1B>1 evaluations, as before, for KS, KdV, and the newly added Cylinder flow in the Appendix (Fig. 17,20,22,23).

[Q2.] Thank you for pointing this out, xTx_T should not be included in the context. The original submission had an indexing error which we have corrected in the theorems and proofs of Appendix E.

[Q3.] We agree, and thank the referee for flagging this. We updated the proof and discussion in the Appendix E.2 to explicitly use Koopman-invariance. Additionally, imposing the Koopman invariance condition allows us to weaken the condition T→∞T\to \infty to a finite number of time points, which is updated in Prop. E.1.

[Q4.] You are correct that asymptotically the invariant ergodic measure for a Hurwitz LDS is the delta distribution on the fixed point. However, our experiments are conducted in the transient regime which is not captured by the equilibrium invariant measure. We revised Appendix A.1 to mention that over 75% of systems retain more than 10% of the initial transient (also see Figure 9 for a visualization of typical context and forecast horizons).

References

[1] Peiyan Hu et al. RealPDEBench: A Benchmark for Complex Physical Systems with Real-World Data. ICLR, 2026.

Reviewer_BHyfOpenReview ↗

Response to Authors

Dear authors,

Thank you very much for the explanation. I went through the revised version. Most of the problems and issues in my previous review comments are resolved.

The only remaining is [Q4]. I see your point on the experiments. But to my understanding that Proposition E.1 still needs an assumption of an ergodic measure. Probably you can think about it in the future.

I will increase the score of soundness to 3 and keep the rating of this paper as 3.

AuthorsOpenReview ↗

Reply to Reviewer Response

Thank you very much for replying. And thanks for pointing out the issue with Q4---we misinterpreted that you were asking about the transients. We agree with you that Proposition E.1 does not apply to the LDS experiments. Our goal of Prop. E.1 is to demonstrate an application of the risk bound in Prop. 3.1, and analyze TTF with an EDMD objective for ergodic dynamics, as you mention.

However, we note that iTTF for linear dynamics follows directly given full rank trajectory data X\mathbf{X} and the usual conditions of gradient descent with a convex objective.

On a separate note, we would like to draw your attention to our new experimental results in the revision, in which we apply eTTF to real experimental measurements of a fluid flow around a cylinder (we added this in response to Weakness 2). We think this is a compelling addition because it shows that the parameter efficient TTF framework even applies to real data (∼15×\sim15\times smaller than the standing FNO baseline), which doesn't strictly obey the assumptions of the theory (e.g. experimental data is noisy and non-ergodic). We think these new real-2D-data experiments are a substantial new contribution, and we thank the referee for suggesting them. We kindly ask the referee to consider them when assigning their final rating. Thank you again for your review.

Reviewer_gCnkOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

Test-Time Forecasting (TTF) connects DMD/EDMD regression with recurrent linear attention. In iTTF, repeated blocks refine predictions while the learned network weights stay fixed. In eTTF, selected forecasting weights are updated at test time using observed input transitions; the pretrained encoder and decoder stay fixed. The experiments test how extra inference iterations affect rollouts on held-out affine systems, Duffing oscillators, and PDE parameter settings.

Critical strengths
  1. The connection to DMD is useful because it explains how the repeated block fits a predictor to the input sequence. It also explains the role of input injection and short convolutions.
  2. The learned filters and inferred operators in Figures 7-10 make the linear-system results easier to interpret.
  3. The KdV and KS results show that extra inference steps can improve forecasts without adding model parameters. This is useful when model size is limited but more computation is available.
Critical weaknesses
  1. In Proposition E.2, invertibility of ϕ\phi lets us recover the state from its features, but does not make their evolution linear. Eq. 48 uses that stronger property. Please state it as an additional assumption or retain the approximation error from the fixed feature space.

  2. Please check the activation in Proposition D.3. For the stated σ=ReLU\sigma=\mathrm{ReLU}, the squared derivative is 0 or 1, while Eq. 41 uses a ReLU2\mathrm{ReLU}^2 gate. The activation assumption and the resulting update need to agree.

  3. Section 3 describes fitting to one context, but Section 3.2 mentions batch-size dependence. Please give the evaluation batch size and clarify whether each trajectory has its own adapted forecasting weights or shares them with other trajectories.

Minor related-work comment: Since Section 5 distinguishes TTF from earlier EDMD methods by allowing nonlinear forecasters, some discussion of nonlinear forecasting such as based on state reconstruction motivated by Takens' theorem would enrich this section. e.g., LETS Forecast: Learning Embedology for Time Series Forecasting (ICML 2025) and Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspective (NeurIPS 2024).

Questions for authors

See Weaknesses

4 comments
AuthorsOpenReview ↗

Response to reviewer

[W1.] Your issue here is correct, we implicitly assumed that ϕ\mathbf{\phi} gave a basis for a Koopman invariant subspace (the Koopman operator maps this subspace to itself). We updated the proof and discussion in Appendix E to explicitly impose Koopman-invariance. Additionally, imposing the Koopman invariance condition allows us to weaken the condition T→∞T\to \infty to a finite number of time points TT, which we updated in Prop. E.1.

[W2.] Thank you for pointing this out. This was a typo and should have been σ(x)=ReLU2(x)\sigma(x) = \texttt{ReLU}^2(x). We have reworked Appendix D and present a variation of the statement with σ\sigma being the softplus activation instead since the resulting update aligns very closely to gated attention [1]. We note the old result with ReLU2\texttt{ReLU}^2 in the remark immediately following it (App. D.4).

[W3.] We clarify that all iTTF models implicitly fit a single trajectory context, while eTTF models exhibit batch size dependence due to aggregation in the inner-objective. For our original submission the eTTF models were trained with full batch inner optimization and evaluated full batch i.e. a batch size equal to the number of initial conditions generated for each system described for each ODE/PDE (see the data generation paragraphs in Appendix B.1,C.1.1,C.2.1,C.3.1). Since each context trajectory in a batch corresponds to a single system parameter, the eTTF training and evaluation batch size may be interpreted as an additional test-time scaling axis, analogous to the few shot ICL setting in language modeling [2,3]. For clarity, we replace our eTTF evaluations in the main text with the same models trained with inner batch size B=1 and evaluated at B=1 (a single trajectory) (Fig. 4,6,16,7,8), with the full batch evaluations (multiple trajectories from the same system), pushed to the appendix (Fig. 17,20,22,23). We generally observe rollout error improving with batch size as expected.

We thank the reviewer for pointing out the additional related works. We have added them to our references and include a discussion in the related works section (Section 5).

References

[1] Zihan Qiu et al. Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention- Sink-Free. NeurIPS, 2025.

[2] Jonas Geiping et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. NeurIPS, 2025.

[3] Tom Brown et al. Language Models are Few-Shot Learners. NeurIPS, 2020.

We thank the reviewer for their constructive comments. As the discussion period draws to a close, we would be happy to provide any further clarification on the theoretical results or the additional experiments.

Thank you authors for the detailed response. Most of my concerns have been addressed. I have updated my rating.

We sincerely thank the reviewer for updating the score in light of the revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.