acceptodds
Under review as a conference paper at ICLR 2027

Fixed-Point Reasoning: Learning Optimal Physical Representations of Dynamical Systems

Abstract

Learning dynamical systems from observations is fundamental to scientific modeling, prediction, and control, while recurring structures across heterogeneous systems motivate foundation models that learn general representations. However, existing representation learning approaches lack a principled way to preserve physical fidelity. We introduce a core concept, optimal physical representation, defined by minimizing the inconsistency between the learned representation and the inferred physical dynamics. Specifically, we endow tokens with explicit physical meanings via a novel physics-aware encoding constructed from the estimated dynamics. This creates a critical feedback link for representation refinement, formulated as a cooperative Refiner–Estimator loop. To guarantee convergence, we further develop a novel Fixed-Point Transformer with explicit contractive constraints, driving the loop toward a self-consistent equilibrium. More broadly, this establishes fixed-point reasoning as a new paradigm for physical representation learning, where the model reasons over physical estimates through recurrent refinement until equilibrium. Extensive experiments across diverse dynamical systems demonstrate improved predictive accuracy, zero-shot generalization, stable long-horizon rollout, and progressive refinement toward the fixed-point equilibrium.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Fixed-Point Reasoning, Physical Representation Learning, Dynamical Systems, Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eyMROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LASS-ODE, a framework that jointly refines physical representations and estimates dynamics through a Fixed-Point Transformer. It combines physics-aware feedback, cross-system knowledge sharing, and structured local ODE operators with adaptive stability. Theoretical analyses establish convergence and stability conditions, while extensive experiments demonstrate strong predictive accuracy, zero-shot generalization, and stable long-horizon rollout.

Critical strengths
  • The paper introduces an interesting Estimator–Refiner framework that iteratively incorporates inferred dynamics into latent representations. Unlike conventional physics-informed approaches that primarily use physical knowledge as conditioning or regularization, this framework explicitly couples representation learning with dynamics estimation. This is a meaningful contribution to transferable modeling across heterogeneous dynamical systems.

  • The extensive experiments demonstrate competitive performance in in-domain prediction, zero-shot generalization, and long-horizon rollout. The structured piecewise-linear ODE operators provide an effective balance between computational efficiency and predictive accuracy. The ablation studies and real-world evaluations further support the practical effectiveness of the proposed framework.

Critical weaknesses
  • Although the paper establishes convergence of the Estimator–Refiner iterations, the theoretical guarantee primarily concerns convergence to a self-consistent representation rather than recovery of physically correct dynamics. A converged representation may still correspond to inaccurate physical estimates. This distinction weakens the theoretical justification for the claimed optimal physical representation.

  • The proposed framework combines several components, including recurrent refinement, physics-aware encoding, cross-system attention, and structured ODE operators. Although ablation studies are provided, it remains unclear whether the improvements primarily arise from explicit physical feedback or from increased model capacity and iterative computation. More controlled comparisons would strengthen the central methodological claim.

The paper presents a well-motivated framework for jointly learning physical representations and dynamical models through fixed-point refinement. The empirical results are promising, and the structured local dynamics offer practical computational advantages. However, the theoretical connection between fixed-point convergence and physical correctness remains insufficiently established. Furthermore, the current experiments do not adequately isolate the benefits of physics-grounded reasoning from those of recurrent computation and architectural complexity. These limitations weaken the central claims of the paper.

Questions for authors
  1. Does convergence to the proposed fixed point imply any guarantee regarding the accuracy of the inferred dynamics? Can the authors clarify the relationship between representation self-consistency and physical correctness?

  2. How does the proposed method compare against a recurrent Transformer with matched parameters and computational budget but without physics-aware feedback? Such a comparison would help establish whether the improvement originates from physical reasoning rather than recurrent computation alone.

  3. How are the contraction conditions enforced during training, and how sensitive are predictive accuracy and convergence speed to the anchoring coefficient and contraction constraints?

11 comments

Q1. Connection to Physical Correctness.

We thank the reviewer for this important comment. We first clarify that our objective is a "self-consistent optimal physical representation," as defined in Definition 1, rather than a globally optimal representation that guarantees recovery of the true underlying physics. We will add the qualifier "self-consistent" throughout the manuscript to eliminate this ambiguity. We use this term because, after training, (1) the learned representations are largely enriched with intrinsic physical information and extrinsic cross-system dynamical information; (2) the fixed-point iterations are guaranteed to converge to a unique self-consistent equilibrium under the conditions of Theorem 1; and (3) the trajectory prediction loss encourages the learned self-consistent optimal representation to support accurate estimation of local affine dynamic parameters in the latent space, enabling the model to capture the dynamics of heterogeneous ODE systems. These properties define the scope of our self-consistent optimality, without implying guaranteed recovery of the true governing dynamics. The effectiveness of the overall framework is further supported by extensive experiments, particularly its strong zero-shot performance on previously unseen ODE systems, providing empirical evidence that LASS-ODE learns transferable capabilities to infer and refine dynamical parameters in the latent space without any access to the ground-truth governing equations or physical parameters.

(1) Theoretical formulation and guarantees. Our work addresses data-driven representation learning across heterogeneous ODE systems without assuming known governing equations, ground-truth dynamical parameters, or system-specific physical priors. Under these general conditions, strict guarantees on prediction accuracy, physical parameter identification errors, or identifiability cannot be established without additional assumptions on the underlying dynamics and available observations. Our theoretical contribution instead concerns the self-consistency of the representation learning process.

We define a self-consistent optimal physical representation in Definition 1 by considering three essential variables in a triplet: the learned internal representation E⋆E^\star, the predicted state trajectory x^⋆\hat{\boldsymbol{x}}^\star, and the inferred dynamical parameters θ^⋆\hat{\boldsymbol{\theta}}^\star. We establish three pairwise consistency requirements among these variables. The first two established concerns in physics-informed learning are captured by coordinate consistency and governing consistency. Coordinate consistency addresses potential spatiotemporal coordinate mismatches among the physical system inputs, learned representations, and predicted outputs. Governing consistency, closely related to the physical residual constraints commonly employed in PINNs, requires the predicted trajectories to satisfy the governing dynamics described by the inferred physical parameters.

Beyond these established considerations, we introduce learn-estimate consistency to address a fundamental problem in internal representation learning: how can inferred physical information directly improve learned representations, and how can the refined representations subsequently support more accurate physical inference? Specifically, the Estimator infers dynamical parameters Est(E)\mathrm{Est}(E) from the current representation EE, while the Refiner incorporates these inferred properties to enrich the representation through Ref(E,Est(E))\mathrm{Ref}(E,\mathrm{Est}(E)). We characterize the inconsistency between physical inference and representation refinement as ∣∣E−Ref(E,Est(E))∣∣F2||E-\mathrm{Ref}(E,\mathrm{Est}(E))||_F^2.

Our fixed-point reasoning framework aims to minimize this learn-estimate inconsistency through iterative Estimator-Refiner updates. Theorem 1 establishes sufficient conditions under which these iterations converge to a unique equilibrium E⋆E^\star, satisfying $\hat{\boldsymbol{\theta}}^\star=\mathrm{Est}(E^\star), E^\star=\mathrm{Ref}(E^\star,\hat{\boldsymbol{\theta}}^\star). $ At this equilibrium, further refinement using the inferred physical parameters leaves the representation unchanged, providing a principled criterion for self-consistency and termination of the reasoning process. This guarantee concerns the convergence and mutual consistency of physical inference and representation refinement, rather than recovery of the unknown true physical dynamics.

Q1. Connection to Physical Correctness. (Continued)

(2) Why fixed-point reasoning benefits representation learning. The central motivation is to enrich internal representations with two complementary sources: intrinsic physical information inferred within each ODE system and extrinsic dynamical information learned across heterogeneous ODE systems. First, intrinsic physical information is obtained from inferred local dynamical operators, including Ωk\Omega_k, QkQ_k, and αk\boldsymbol{\alpha}_k, which characterize local rotational behavior, directional expansion or contraction, and growth or decay rates. Through physics-aware encoding (Eq. (8)), these properties enrich the physical context of internal tokens and inform subsequent representation refinement. Second, extrinsic cross-system information is retrieved through the Common Structure Hub (CSH), which captures recurring dynamical patterns across heterogeneous ODE systems (Eqs. (9)--(10)). Together, these two sources enable the model to refine internal representations using both system-specific dynamical information and transferable structures learned across systems.

(3) Empirical evidence supporting the proposed mechanism. Beyond the theoretical convergence guarantees, we provide extensive empirical evidence to evaluate five key questions regarding the effectiveness of our framework.

(i) Does fixed-point reasoning improve internal representations and inferred dynamics? Fig. 4(a,b) and Appendix H show that internal representations progressively approach clusters associated with recurring dynamical patterns, including oscillation, dissipation, and fast--slow dynamics. More importantly, Appendix M (Fig. 19) demonstrates that intermediate trajectory predictions progressively recover system-specific dynamical behaviors across reasoning iterations. These results provide empirical evidence that our representation enrichment mechanism not only organizes internal representations around meaningful dynamical structures but also improves their ability to capture physical dynamics.

(ii) Does the reasoning process converge in practice? Fig. 4(c) shows that the normalized representation-update magnitude decreases rapidly across reasoning iterations, indicating that the Estimator--Refiner process approaches a stable equilibrium. This empirical observation is consistent with the convergence guarantee established in Theorem 1.

(iii) Does the framework learn transferable dynamical reasoning capabilities across heterogeneous ODE systems? Our evaluation covers approximately 40 GB of heterogeneous dynamical-system data, including 38 ODE systems for training and 14 unseen ODE systems for zero-shot evaluation. As shown in Fig. 2, LASS-ODE achieves approximately 90% and 85% average MSE reductions relative to the best competing model for each representative in-domain and zero-shot system, respectively. These reductions are averaged across three representative systems in each setting, with each system evaluated at three observation ratios. Importantly, the strong zero-shot performance provides empirical evidence that LASS-ODE learns a transferable capability to infer and refine local affine dynamical parameters in the latent space, enabling it to capture the dynamics of previously unseen ODE systems without access to their ground-truth governing equations or physical parameters. Complete numerical comparisons are provided in Appendices I and J.

(iv) Can the framework accurately predict realistic physical dynamics? We evaluate power-system frequency dynamics following disturbances and AI data-center voltage transients. As shown in Table 1, LASS-ODE reduces prediction MSE by approximately 85% and 90%, respectively, compared with the second-best methods. Fig. 5 further demonstrates accurate reconstruction of post-disturbance frequency trajectories across different buses and disturbance conditions. These results support the effectiveness of our framework beyond classical ODE benchmarks.

(v) How important is the Fixed-Point Transformer to the overall predictive performance? Fig. 7(a) shows that removing the Fixed-Point Transformer leads to the largest performance degradation among the architectural ablations, increasing prediction MSE by approximately 2.6--2.7 times. This provides empirical evidence for the importance of the overall fixed-point reasoning module in dynamical prediction. To further isolate the contributions of physics-aware feedback and recurrent refinement from model capacity and computation, we have conducted additional controlled ablations, which we discuss in our subsequent responses to your related questions.

Q2. Additional ablation study.

We thank the reviewer for this constructive suggestion. To isolate the contributions of explicit physical feedback from model capacity and iterative computation, we conduct additional controlled ablations. Importantly, we control model capacity at the architectural level by preserving the same hidden dimensions, FFN dimensions, attention configurations, CSH, MoE architecture, prediction head, and number of refinement layers/steps across the variants. In particular, all variants retain the same attention and MoE configurations, including the 8-expert architecture with top-2 routing. Thus, no variant is given additional network width, refinement depth, or architectural components. We also maintain comparable overall model sizes and computational budgets.

Specifically, we compare LASS-ODE against three variants: (i) Looped (No Phys.), which retains the same recurrent, weight-shared Transformer blocks and contraction constraints but removes only the physical feedback P(θ)P(\theta); (ii) Untied (Phys.), which replaces the recurrent Fixed-Point Transformer with a stack of independently parameterized Transformer blocks while retaining physical feedback; and (iii) Untied (No Phys.), which further removes physical feedback from the untied architecture. Importantly, the Physics/No-Physics variants within each architecture retain identical model configurations and parameter counts, differing only in whether P(θ)P(\theta) is incorporated. For the recurrent/untied comparisons, we preserve the same refinement depth and per-block dimensions, while changing only whether Transformer parameters are shared across refinement steps.

LASS-ODE achieves the lowest MSE of 0.043×10−20.043\times10^{-2}, reducing prediction errors by 37.2%, 20.2%, and approximately 45.8% compared with Looped (No Phys.), Untied (Phys.), and Untied (No Phys.), respectively. In particular, the comparison between LASS-ODE and Looped (No Phys.) isolates the contribution of physical feedback under the same recurrent architecture. Similarly, Untied (Phys.) reduces MSE by 19.6% compared with Untied (No Phys.), independently demonstrating the benefit of physical feedback without recurrent weight sharing. Furthermore, LASS-ODE outperforms Untied (Phys.), demonstrating the benefit of recurrent weight sharing when physical feedback is retained. Likewise, Looped (No Phys.) reduces MSE by 11.9% compared with Untied (No Phys.), demonstrating the contribution of recurrent weight sharing without physical feedback.

Together, these controlled comparisons provide evidence that the predictive improvements arise from explicit physical feedback and recurrent weight sharing, rather than simply increased model capacity or additional iterative computation. In particular, the consistent improvements from physical feedback under both recurrent and untied architectures support its independent contribution to predictive performance. We will include these additional ablation results and implementation details in the revised manuscript.

Q3. Fixed point and dynamics.

We thank the reviewer for this question. As clarified in our response to Question 1, convergence to the proposed fixed point guarantees a unique self-consistent equilibrium under the conditions of Theorem 1, but does not necessarily guarantee the accuracy or identifiability of the true underlying dynamics. Without known governing equations, ground-truth physical parameters, or additional assumptions, such guarantees are generally difficult to establish.

Our framework instead combines representation enrichment with intrinsic and cross-system dynamical information, fixed-point convergence, and trajectory prediction loss. The prediction loss encourages the learned self-consistent representations to support accurate estimation of local affine dynamical parameters in the latent space, enabling prediction of heterogeneous ODE dynamics from observed trajectories. Importantly, the effectiveness of this mechanism is supported by extensive empirical results, particularly strong zero-shot performance on previously unseen ODE systems without access to their ground-truth governing equations or physical parameters. Please see our response to Question 1 for the detailed theoretical formulation, the distinction between self-consistency and physical correctness, and the supporting experimental evidence.

Reviewer_ArcaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a new model for predicting time series coming from real-world physical systems and toy mathematical ODE models. The authors propose a model that uses a fixed-point convergence mechanism at inference and provide a theoretical guarantee of convergence. They then adapt existing benchmarks to compare their model to re-implementations of competing models.

Critical strengths

The paper shows strong empirical results where their model strongly outperforms competing models. They also show through ablations that most of their design choices actually play a role in improving performance.

Critical weaknesses

The document is poorly written with non-rigorous statements, and grandiose claims of "optimal physical representation" that are not backed by solid theoretical or empirical results.

On the theoretical side, I don't understand the relevance of fixed points for improving the prediction of physical dynamics. As mentioned below, the paper starts from a fixed-point equation without explaining how it helps with prediction of physical dynamics.

On the empirical side, the evaluation protocol is chosen by the authors and lacks crucial details. The description is nearly entirely deferred to the appendix, and the appendix does not explain the split between train, validation and test and does not have any numerical description even for basic information like the number of samples. The fact that the authors construct their own evaluation protocol and re-implement the baselines is hard to trust. Other crucial information is missing on the evaluation side, for instance, Section 4.5 presents scores without any context: which task? which dataset?

There's a lot of novel component in their architecture that makes it hard to understand what is the core contribution at the architecture level.

Misc poorly written details:

  • l.060, "one-way physical conditioning": not clear
  • l.071–073, "progressively improves learning": vague statement. Nothing in the main text actually describes the learning process, only inference is described.
  • On the topic of learning, the authors don't describe how they differentiate through the fixed point, which is basic information. title: "fixed-point reasoning": reasoning is a poor choice of words, which makes sense in the context of LLMs that learn to imitate human reasoning in chain-of-thought but not in non-reasoning physical prediction tasks.
  • l.065 "cooperative Refiner-Estimator": cooperative here is vaguely introduced and can be mistaken for the rigorous concept in game theory.
  • l.089–090. "Recover {x̂_i}" recovers the estimate itself; it should be the true x_i.
  • "Time-aware GRU" (l.120–122). Only Δt is added as input: it's only discretization-aware, l.152 "with an explicit dynamical identity and grounding it in physical meaning rather than generic features." a very hard-to-understand sentence.
  • Fig. 3 is hard to read. The ground truth is hidden under many overlapping dashed predictions.
  • Definition 1 is very vague and doesn't justify the wording of "optimal physical representation" there's nothing physical about it, it only applies to their Estimator/Refiner architecture and it seems to be trivially satisfied by many fixed-point systems.
Questions for authors

My central concern is about the evaluation methodology. Why didn't you re-use evaluation methods and results from the existing literature? It is harder to trust your evaluation methodology on tasks that are not standards.

There are many dynamics that can reach a fixed point; why did you focus on contractive dynamics?

19 comments

We sincerely thank the reviewer for the critical feedback. After carefully reviewing the manuscript and all your comments, we would like to first clarify two points: (1) We acknowledge that some terminology and descriptions may be imprecise or misleading, and we appreciate the reviewer for identifying these issues. We address them in our subsequent responses and will make corresponding revisions. (2) We respectfully emphasize that the core concepts of self-consistent optimal physical representation and fixed-point reasoning are well-defined in our paper, with explicit mathematical formulations and theoretical convergence guarantees under stated conditions.

We first briefly clarify the central idea. Existing physics-informed learning methods often incorporate physical knowledge through regularization, physical constraints, or input, thereby indirectly influencing internal representations after each Transformer block. This motivates our central question: Can inferred physical knowledge directly refine these internal representations during the learning process?

Intuitively, inferred physics can provide valuable dynamical information to enrich internal representations. In our model, the inferred local dynamical operator (Eqs. (5)–(6)) contains Ωk\Omega_k, which characterizes rotational dynamics; QkQ_k, which defines the orthogonal directions of expansion or contraction; αk\boldsymbol{\alpha}_k, which specifies the corresponding growth or decay rates; and bk\boldsymbol{b}_k, which represents local drift. These properties help characterize dynamical behaviors such as oscillation, directional expansion, and dissipation. Through our physics-aware encoding (Eq. (8)), such dynamical information is incorporated into the internal token representations, enriching their physical context and enabling more informed representation refinement in subsequent Transformer blocks.

Essentially, our approach incorporates reasoning into internal representation learning. Reasoning is broadly understood as the process of drawing inferences and conclusions from available information [1]. A growing research direction treats repeated computation over internal representations, including during training, as latent reasoning. For example, Saunshi et al. [2] explicitly describe looped Transformer computation as "reasoning with latent thoughts," while Geiping et al. [3] refer to recurrent internal computation as "latent reasoning" and incorporate it into model training. These studies illustrate that reasoning is not restricted to inference-time chain-of-thought generation. In our work, the inferred dynamical parameters (Ωk\Omega_k, QkQ_k, and αk\boldsymbol{\alpha}_k) provide intermediate physical information that guides internal representation learning.

Q1. Theoretical and Empirical support to the "Optimal physical representation".

Given our goal of using physical reasoning to improve internal representation learning, a key question is when this reasoning process should terminate and what constitutes a desirable representation. Since we do not assume known governing equations or supervision of physical parameters, it is difficult to define global optimality with respect to the true underlying physics. Instead, our work focuses on self-consistent optimal physical representations.

In this response, we address the reviewer's concerns regarding the theoretical and empirical support for optimal physical representations.

Specifically, in Definition 1, we consider three essential variables: the learned representation E⋆E^\star, the predicted state trajectory x^⋆\hat{\boldsymbol{x}}^\star, and the inferred dynamical parameters θ^⋆\hat{\boldsymbol{\theta}}^\star. We define three pairwise consistency requirements among them. The first and third are well-recognized consistency issues in physics-informed learning. Coordinate consistency ensures that the learned representation E⋆E^\star, the input observations from which it is constructed, and the predicted state trajectory x^⋆\hat{\boldsymbol{x}}^\star are aligned in the same spatiotemporal coordinate system, while governing consistency requires the predicted trajectory x^⋆\hat{\boldsymbol{x}}^\star to satisfy the governing dynamics defined by θ^⋆\hat{\boldsymbol{\theta}}^\star.

We introduce the second requirement, learn–estimate consistency, which establishes consistency between physical inference and representation learning in our fixed-point reasoning process. For a given internal representation EE, the Estimator Est(E)\mathrm{Est}(E) infers local dynamical parameters, while the Refiner Ref(E,Est(E))\mathrm{Ref}(E,\mathrm{Est}(E)) uses these inferred dynamics to enrich and refine the representation. We characterize their inconsistency through the fixed-point residual $ ||E-\mathrm{Ref}\big(E,\mathrm{Est}(E)\big)||_F^2.$

Importantly, this residual characterizes learn-estimate consistency rather than serving as an additional explicit training loss. We train the Estimator and Refiner using only the trajectory prediction loss, while the constrained fixed-point iterations drive the representation toward a self-consistent equilibrium. At this equilibrium, the residual vanishes, yielding $\hat{\boldsymbol{\theta}}^\star =\mathrm{Est}(E^\star),\qquad E^\star=\mathrm{Ref}(E^\star,\hat{\boldsymbol{\theta}}^\star).$

These conditions mean that the inferred dynamics are obtained from the current representation, while the representation remains unchanged under refinement using these inferred dynamics. Our key contribution is to formulate this coupled Estimator–Refiner process as fixed-point reasoning (Eq. (1)) and impose sufficient contraction conditions to guarantee convergence to a unique equilibrium for a fixed input representation (Theorem 1). This establishes a principled termination criterion for physical reasoning and supports learning the self-consistent optimal physical representation defined by the three consistency requirements.

In summary, we would like to emphasize three points: (1) Our work addresses a representation learning problem, aiming to use inferred physical parameters as additional context to enhance internal representations through an iterative physical reasoning process. (2) We define self-consistent optimal physical representations and establish a fixed-point framework with convergence guarantees for the coupled Estimator–Refiner mapping under stated conditions. (3) Importantly, we do not claim to learn globally optimal representations of the true physical dynamics. Definition 1 explicitly introduces the concept of a "self-consistent optimal physical representation." We acknowledge that we omitted "self-consistent" in some parts of the manuscript, which may have caused confusion. We will consistently use the full term "self-consistent optimal physical representation" throughout the revised paper to clarify the scope of our claims.

Q1. Theoretical and Empirical support to the "Optimal physical representation". (Continued)

Regarding empirical evaluation, no single universally adopted protocol exists across heterogeneous dynamical systems. Existing studies typically use task-specific settings. For example, [4] evaluates continuous-time trajectory prediction on periodic and MuJoCo dynamics; [5] evaluates zero-shot forecasting across 135 chaotic systems against methods including N-BEATS and TiDE; and [6] evaluates trajectory forecasting on Lorenz-63, Lorenz-96, and Kuramoto--Sivashinsky systems using prediction-validity and statistical metrics. Accordingly, we evaluate classical ODE systems and public nonlinear benchmarks using trajectory prediction MSE. We use established baseline architectures and implementations, evaluating most on the same datasets, partitions, and comparable training budgets. Necessary architectural adaptations are documented in Appendix G. We provide further details on data sources, data splits, baseline implementations, and evaluation settings in our responses to Questions 3 and 6. Specifically, our empirical study uses approximately 40 GB of heterogeneous dynamical-system data, combining publicly available nonlinear benchmarks with simulated trajectories from classical ODE systems (Appendix E). We systematically evaluate five key questions:

(1) Does fixed-point reasoning improve internal representations and inferred dynamics? Since this is a data-driven learning problem, it is difficult to theoretically guarantee that inferred physical parameters always improve representation quality and prediction accuracy. Instead, we provide empirical evidence to evaluate this effect. Specifically, Fig. 4(a,b) and Appendix H show that internal tokens progressively approach clusters associated with recurring dynamical behaviors, such as oscillation, dissipation, and fast-slow dynamics. These results suggest that fixed-point reasoning helps organize internal representations around physically meaningful and reusable dynamical structures. Furthermore, Appendix M (Fig. 19) demonstrates that intermediate trajectory predictions progressively recover system-specific dynamics.

(2) Does the reasoning process converge, and are the proposed fixed-point constraints effective? Fig. 4(c) shows that the normalized representation-update magnitude decreases rapidly across reasoning iterations, indicating that the constrained Fixed-Point Transformer approaches a stable equilibrium in the tested systems, consistent with our Theorem 1.

(3) How well does the model generalize across heterogeneous dynamical systems? Fig. 2 demonstrates substantial prediction improvements in both in-domain and zero-shot settings. Across the three representative in-domain systems shown in Fig. 2, LASS-ODE achieves approximately 90% average MSE reduction relative to the best competing model for each system. For the three representative zero-shot systems, the corresponding average reduction is approximately 85%. These percentages are averaged across the three systems in each setting, with each system evaluated over three observation ratios. Complete numerical results across additional systems and observation ratios are provided in Appendices I and J.

(4) Can the framework accurately predict realistic physical dynamics? We evaluate power-system frequency dynamics following disturbances and AI data-center voltage transients. As shown in Table 1, LASS-ODE reduces prediction MSE by approximately 85% and 90%, respectively, compared with the second-best model. Fig. 5 further demonstrates accurate prediction of post-disturbance frequency trajectories across different buses and disturbance conditions. Additional experimental details are provided in Appendix O.

(5) How important is the Fixed-Point Transformer to the overall predictive performance? Fig. 7(a) shows that removing the Fixed-Point Transformer leads to the largest performance degradation among the architectural ablations, increasing prediction MSE by approximately 2.6--2.7 times. This provides empirical evidence for the importance of the overall fixed-point reasoning block in dynamical prediction.

[1] Ke et al., A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems. TMLR, 2025.

[2] Saunshi et al., Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR, 2025.

[3] Geiping et al., Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. NeurIPS, 2025.

[4] Rubanova, Y., Chen, R. T. Q., and Duvenaud, D. Latent Ordinary Differential Equations for Irregularly-Sampled Time Series. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.

[5] Zhang, Y., and Gilpin, W. Zero-Shot Forecasting of Chaotic Systems. International Conference on Learning Representations (ICLR), 2025.

[6] Mandal, P., and Gottwald, G. A. Learning Dynamical Systems with Hit-and-Run Random Feature Maps. Nature Communications, 16, 5961, 2025.

Reviewer_avS4OpenReview ↗
Rating
4(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper approaches representation learning for modeling dynamical systems as a fixed-point problem, where representations serve to infer unknown physical processes, while these inferred processes should further refine the representations. The result is LASS-ODE, a Refiner-Estimator architecture that iteratively updates representations and predictions through a recurrent architecture. LASS-ODE is tested against a variety of models on a variety of time series prediction tasks and makes notable improvements.

Critical strengths

S1: Timely problem and interesting theoretical contribution. ML for physical modeling is an important and difficult problem with a variety of impactful downstream applications, and the authors propose an interesting theoretical methodology based fixed-point optimization for learning unknown physical dynamics.

S2: Complete set of baselines. The experiments include a large variety of relevant model types, like physical foundation models, time series foundation models, etc.

Critical weaknesses

W1: Some experiments or experimental design documentation are missing for appropriate support of the paper's claims. For example, it is unclear what the relative FLOP requirements/parameter counts are for LASS-ODE vs the baselines, only a plot of inference time against baselines which is appreciated. Also, the claims about stable long-horizon rollout (L078) is only weakly supported by the reported experiment in Section 4.4. Further details in the Questions section.

Questions for authors

Q1: How were the physical foundation models (eg MPP, PDE-FM) adapted to the datasets in this paper?They are designed and pretrained on spatiotemporal fields but the tests here are all multivariate time series with few variables, so the comparison may not hold up. The adaptation strategy should be explicitly stated, and the fact that these models were trained from scratch (eg Poseidon-0D) should be discussed as a limitation.

Q2: Some baselines perform worse than trivial predictions. The data lie in [−1, 1], so predicting zero gives MSE ≤ 100 (×10⁻²). Chronos reports 297.8, 243.7, and 144.7. Timer-XL reports 136.8. Informer scores about 45–65 almost everywhere. This is close to what a constant prediction gives on a unit sine.

Q3: Missing ablations: The “w/o Fixed-Point Transformer” ablation (L409, L411–413) removes the full Refiner: the attention with CSH, the 8-expert MoE, and the physics encoder. The model loses parameters and compute. The lost capacity alone can cause the 2.6–2.7× error increase. I would recommend:

  • a DEQ or looped Transformer with the same blocks and constraints, and an untied layer stack with the same compute
  • feedback-only ablation (loop without P(θ̂)), to disentangle the physics feedback from a generic loop

Q4: Can the authors report compute differences for all the models? E.g., differences in architectures, training data, training length. There are also no seeds or error bars for LASS-ODE.

Q5: Section 4.3/Appx O don't describe where these real-world datasets come from (e.g., are these datasets taken from previous work or is it original data taken by the authors). Additionally, some of the strongest baselines were not tested here (Mamba, Latent MoS seem the strongest from Fig 2).

Q6: The current test for rollout stability (Sec. 4.4, Fig. 6) only tracks perturbations in the model’s latent rollout, only inside the training window, and with no baselines. Please consider these tests:

  • Report prediction error vs. time after the prefix, for LASS-ODE and the main baselines.
  • Test rollout beyond the training window. For example, feed each predicted window back as the next prefix. Report error growth and divergence rate. For chaotic systems, report valid prediction time or long-term statistics, because pointwise error grows for any model.

If extended rollout is outside the intended scope, please narrow the “long-horizon” claim.

9 comments

Q1. Parameter count.

We thank the reviewer for this suggestion. For our main experiments, LASS-ODE uses a default configuration with approximately 15M parameters. To ensure fair comparisons, we use publicly available implementations of the non-foundation-model baselines (see answers to questions 2-3) and scale their configurations to comparable model capacities, typically ranging from 10M to 20M parameters, while preserving their original architectures. For pretrained foundation models, we retain their publicly available large-scale checkpoints, such as TimesFM (200M parameters), Chronos-T5-Large (710M), and MPP-AViT-L (409M), without additional training on our dataset. We will further report the exact parameter counts and inference FLOPs for LASS-ODE and all baselines under consistent evaluation settings, including the computational costs associated with fixed-point iterations and sparse MoE activation. These additions will complement the existing latency and scaling analyses and provide a more comprehensive comparison of computational efficiency.

Q2. Adaptation strategy.

We thank the reviewer for this important comment. To clarify the evaluation and adaptation strategies, we organize our 18 baselines into five groups:

(A1) Pretrained time-series foundation models. TimesFM, Chronos, and Timer-XL use publicly available pretrained weights without retraining. These models are evaluated through their existing forecasting interfaces to assess zero-shot generalization across heterogeneous ODE systems.

(A2) Pretrained scientific foundation models. MPP and PDE-FM use publicly available pretrained weights without retraining their backbones. Since these models were developed for spatially structured physical fields, their application to ODE trajectories requires input/output adaptation. Specifically, ODE state variables can be represented as separate channels of spatially constant fields, rather than as different spatial locations, and the predicted fields can be mapped back to ODE states. This preserves the original field-based architectures while enabling cross-domain evaluation.

The following baseline groups (B, C1, and C2) are retrained using our ODE dataset. Our dataset covers 38 ODE systems, with trajectories generated under diverse initial conditions, physical parameters, and system configurations. For each system, we allocate 8,192 trajectories for training, 1,024 for validation, and 1,024 for in-domain testing, totaling 311,296 training, 38,912 validation, and 38,912 test trajectories. For zero-shot evaluation, we additionally include 14 unseen ODE systems, each with 1,024 independent test trajectories, yielding 14,336 zero-shot test trajectories.

(B) Existing model implementations with retraining. Mamba, CfC, LTC, Latent ODE, Latent MoS, and ContiFormer retain their established architectures, with necessary input/output adjustments, and are retrained on our ODE dataset under a common prediction setting.

(C1) Forecasting and state-space architectures with ODE adaptations and retraining. Informer and Autoformer retain their original forecasting architectures with adapted input/output interfaces, while S4D-style SSM incorporates discretization based on observation intervals. These models are retrained for ODE trajectory prediction.

(C2) Scientific operator architectures with ODE adaptations and retraining. CoDA-NO-1D, PROSE-ODE, PROSE-PDE, and Poseidon-0D use ODE-adapted versions of their original numerical architectures and are trained from scratch without the original pretrained PDE weights. CoDA-NO-1D treats time as a one-dimensional operator domain; PROSE-ODE processes observed time-state pairs; PROSE-PDE maps state variables onto an artificial one-dimensional grid; and Poseidon-0D represents ODE states as spatially constant fields, following its original field-based architecture.

We acknowledge that scientific foundation models such as MPP and PDE-FM were originally designed for spatiotemporal field prediction, whereas our evaluation focuses on heterogeneous ODE trajectories without explicit spatial grids. We include them as complementary cross-domain baselines to investigate the transferability of learned dynamical representations to our task. However, differences in data representations and spatial inductive biases may limit their performance, while scratch-trained adaptations do not fully reflect the capabilities of the original pretrained models. We will explicitly clarify these adaptation strategies and discuss their limitations in the revised manuscript.

Q3. Basline worse than trivial solutions.

We thank the reviewer for raising this important concern. We agree that comparisons with trivial predictors are useful sanity checks. However, an MSE exceeding that of a zero predictor does not necessarily indicate constant predictions or numerical instability; it can also arise from substantial mismatches between predicted and true dynamical trajectories.

First, we clarify the interpretation of the reported MSE values. For target trajectories normalized to [−1,1][-1,1], a zero predictor has an MSE no greater than 11 (or 100100 in our reported units of ×10−2\times 10^{-2}). However, this is not an upper bound on the MSE of other predictions. For example, if the true trajectory is a unit-amplitude sine wave but the prediction has the opposite phase, the MSE is 22 (or 200200 in our reported units), even though both trajectories remain within [−1,1][-1,1]. More generally, incorrect oscillation phases, frequencies, amplitudes, or dynamical trends can produce substantially larger errors than predicting zero.

Importantly, Appendix Figs.~12--16 provide direct visualizations of the predicted trajectories. These examples show that poor predictions are not predominantly constant outputs. Instead, several baselines generate nontrivial temporal variations that deviate substantially from the ground truth in their oscillatory patterns, phase relationships, amplitudes, or state evolution. Such severe trajectory mismatches can produce large averaged MSE values, particularly when the predicted evolution is inconsistent with the underlying dynamics.

Second, we explain the observed performance differences in relation to the architectural designs and evaluation settings of five baseline categories.

(A) Pretrained time-series foundation models (TimesFM, Chronos, Timer-XL). These models learn transferable temporal representations through large-scale pretraining and are evaluated without retraining on our ODE dataset. However, short observed prefixes may not sufficiently reveal the underlying dynamics, particularly for nonlinear oscillatory and coupled systems. For example, at a 30%30\% prefix, Chronos reports MSEs of 297.80297.80 on Lotka--Volterra and 136.04136.04 on FitzHugh--Nagumo, which decrease to 1.001.00 and 1.381.38 at a 90%90\% prefix, respectively. Timer-XL similarly decreases from 136.72136.72 to 11.9811.98 on FitzHugh--Nagumo. These results indicate substantial sensitivity to the available observations, consistent with the difficulty of recovering complex dynamical patterns from limited temporal context.

(B) Informer. Informer is primarily designed for efficient long-sequence forecasting, using ProbSparse attention and self-attention distilling to reduce computational complexity. Our task instead emphasizes extracting informative dynamical patterns from limited observations to predict physical state evolution. ProbSparse attention may underemphasize less dominant but dynamically important observations, while temporal distilling may compress local oscillatory details, nonlinear transitions, and phase relationships. These mechanisms may limit the preservation of information needed for accurate ODE prediction. For example, Informer's MSEs remain nearly unchanged across 30%30\%, 60%60\%, and 90%90\% prefixes on Hopf normal (49.9049.90, 50.1050.10, 49.7049.70) and Pendulum (48.6048.60, 48.4048.40, 48.4048.40). This provides a possible explanation for its persistently high errors despite additional observations.

(C) ContiFormer. ContiFormer is primarily designed for irregular time-series modeling, using Neural ODEs to construct continuous-time attention representations and capture temporal dependencies across irregular observations. However, its continuous-time modeling mainly enhances representation learning, while the predicted physical states are not explicitly generated by integrating inferred governing dynamics. In this respect, its forecasting mechanism still relies on learned representations, similar to conventional forecasting Transformers such as Autoformer. Our task instead requires accurately recovering the underlying state evolution from limited observations. This distinction may explain why ContiFormer exhibits relatively high prediction errors on several nonlinear systems, whereas Latent ODE and Latent MoS, which explicitly model latent state evolution through continuous-time dynamics, achieve substantially lower errors. For example, at a 30%30\% prefix on Lotka--Volterra, ContiFormer obtains an MSE of 56.4056.40, compared with 1.901.90 and 0.910.91 for Latent ODE and Latent MoS, respectively. Even at a 90%90\% prefix on Van der Pol (μ=10\mu=10), ContiFormer reports 21.7021.70, compared with 2.602.60 and 0.820.82 for Latent ODE and Latent MoS.

Q3. Basline worse than trivial solutions. (Continued)

(D) Pretrained scientific foundation models (MPP, PDE-FM). These models were originally pretrained on spatially structured physical fields, where their architectures are particularly effective at capturing spatial correlations and interactions among neighboring field variables. However, our task involves heterogeneous, low-dimensional ODE trajectories without an explicit spatial field structure. Consequently, their spatial inductive biases and learned representations may not directly translate into effective modeling of temporal state evolution and nonlinear dynamical coupling. For example, on Hamiltonian SHO, PDE-FM and MPP report MSEs of 133.00133.00 and 101.00101.00, respectively, even with a 90%90\% observed prefix. We include MPP and PDE-FM as complementary cross-domain baselines to evaluate the transferability of pretrained scientific representations, without retraining on our ODE dataset.

(E) Task-adapted scientific architectures (CoDA-NO-1D, PROSE-ODE, PROSE-PDE, Poseidon-0D). These models are adapted to our ODE prediction task and trained from scratch. However, some adaptations cannot fully exploit the mechanisms underlying their original designs. For example, PROSE-ODE uses only numerical observations without the symbolic information supported by its original framework; PROSE-PDE and Poseidon-0D operate on artificial or spatially constant fields, where the spatial structures exploited by their original architectures are absent. These differences may contribute to their system-dependent prediction performance. For instance, at a 30%30\% prefix on Lotka--Volterra, CoDA-NO-1D, PROSE-ODE, PROSE-PDE, and Poseidon-0D report MSEs of 23.5223.52, 21.2921.29, 25.1425.14, and 18.0618.06, respectively, whereas Latent ODE and Latent MoS achieve 1.901.90 and 0.910.91. Nevertheless, several adapted architectures improve substantially with additional observations; for example, PROSE-PDE and Poseidon-0D decrease to 7.847.84 and 4.354.35 at a 90%90\% prefix on the same system.

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your thoughtful and insightful comments. During the revision period, we have devoted considerable effort to addressing your key concerns, including conducting additional controlled ablation and sensitivity studies, clarifying the theoretical formulation and physical significance of the self-consistent optimal physical representation, resolving ambiguities in the methodology, and strengthening the fairness and reproducibility of our experimental comparisons.

As the discussion deadline is approaching, we would greatly appreciate any further questions or feedback you may have. Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.