acceptodds
Under review as a conference paper at ICLR 2027

Self-Supervised Learning of Task-Agnostic Representations for Physical Dynamics

Abstract

Foundation models for spatio-temporal physical systems seek representations that generalize across systems, parameters, and operating conditions. We explore a self-supervised route to this goal, decoupling representation learning from task-specific prediction to capture reusable properties of physical dynamics. We study masked reconstruction in physical space and JEPA-style prediction in latent space, and introduce regime fingerprinting to capture trajectory-level properties of the dynamics: an early partial observation must identify a broader view of the same trajectory, thereby encouraging the preservation of information that persists across time. The learned representations provide explicit, compact conditioning signals for downstream models and support efficient in-context adaptation to previously unseen physical systems. Experiments on heterogeneous 2D and 3D spatio-temporal dynamics across multiple downstream tasks show strong performance in few-shot settings, together with computationally efficient cross-physics generalization.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Foundation Models, Dynamical Systems, PDE, Self-supervised, Latent representations, JEPA

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_kPbKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TRACE, a self-supervised representation learning framework for heterogeneous spatio-temporal physical systems. is paper presents TRACE, a self-supervised representation learning framework for heterogeneous spatio-temporal physical systems. The main idea is to separate the learning of physical dynamics representations from downstream forecasting tasks. This framework introduces a unified patch-based encoder for physical systems with different spatial dimensions, resolutions and fields. It combines JEPA-style hidden potential prediction targets and state fingerprint targets, which align the representation of early partial observations with that of the complete trajectory. The author evaluates the learned representations through physical parameter regression, dynamic prediction, and cross-system generalization. Experiments conducted on The Well dataset show improvements in parameter recovery of several physical systems, competitive prediction performance, and the ability to adjust downstream models on previously unseen systems.

Critical strengths
  1. The paper indicates that TRACE can learn representations related to physical properties. It can be seen from Figures 3,8 and 9 in the paper that the visualization of t-SNE and PCA shows the meaningful association between potential representations and physical parameters, and the parameter regression experiment quantitatively supports the validity of these representations.
  2. The Repeat Patchifier proposed by the author in the paper provides a concise method for processing two-dimensional and three-dimensional physical data using shared projection weights, avoiding the need to design input layers separately for different dimensions. The ablation experiment in the paper also demonstrated that this design can achieve comparable or relatively good performance, so it has practical value for the unified modeling of heterogeneous physical systems.
Critical weaknesses

1.Although RF has improved the recovery accuracy of physical parameters, it is still unclear whether the learned representations capture features at the dynamic mechanism level or information specific to trajectories. The existing experiments have not directly verified the invariance of the characterization to the initial conditions, and this property is precisely the important basis for the authors to propose the RF method. 2.The Zero-shot experiment only involves two unseen physical systems and lacks a baseline model for direct comparison. Furthermore, the independent contributions of RF and dynamic Field Embedding mechanisms have not been fully distinguished, making it difficult to determine the main source of the model's generalization ability.

Questions for authors

How does the RF objective function handle trajectories with the same physical mechanism but different initial conditions? How can it be proved that the learned representations mainly capture information at the dynamic mechanism level rather than information specific to trajectories?

3 comments

Thank you for your detailed and constructive review. Following your comments and suggestions, we conducted several additional experiments to investigate the distinction between regime-level and trajectory-specific information, extend zero-shot evaluation to additional physical systems, and disentangle the contributions of regime fingerprinting and dynamic field embeddings. We also analyzed the RF negative sampling strategy and performed additional ablations to clarify its effects on parameter inference. We address your concerns below.

W1 & Q2 - Testing invariance to initial conditions

It is still unclear whether the learned representations capture features at the dynamic mechanism level or information specific to trajectories - evaluating invariance to initial conditions

Let us consider the vectors inferred by TRACE (denoted TRACE vectors in the following) used to condition the baseline forecasters.

We measured how much these learned conditioning representations depend on the initial condition in two different ways.

Measuring similarities between TRACE conditioning vectors. First we compare centered TRACE vectors from three settings:

  1. Same trajectory: same trajectory at different time windows,
  2. Same regime: distinct trajectories sharing regime parameters but with different initial conditions,
  3. Different regimes: trajectories from distinct regimes. The metric is mean cosine similarity.
DatasetSame trajectorySame regimeDifferent regimes
Active matter0.730.75-0.04
Euler periodic0.24-0.08-0.09
Rayleigh-Bénard0.140.13-0.09
TRL 2D0.580.57-0.09
Gray-Scott0.190.14-0.15
MHD 640.970.94-0.10
TGC-0.06-0.10-0.21

The results show that, on most datasets, vectors from different trajectories sharing the same regime are substantially more similar than vectors from different regimes. This indicates that TRACE captures regime-level information, although the degree of invariance to initial conditions varies across datasets. Euler periodic is an exception, where the vectors distinguish trajectories but show little separation between regimes.

Note that Acoustic Scattering Maze and Supernova are not shown because their parameters do not vary across trajectories.

Counterfactual conditioning-vector replacement. In a second, forecasting experiment, we replace the conditioning vector with:

  1. Own vector: the vector from the original trajectory (no replacement),
  2. Same regime: the vector from a trajectory of the same regime with another initial condition,
  3. Different regimes: the vector comes from a trajectory from another regime,
  4. Gaussian: the vector comes from a Gaussian draw with the dataset's mean and standard deviation.

Measures were performed on the first window of every validation trajectory (this is a different setting from table 20). The metric is step-1 NRMSE.

ModelDatasetOwn vectorSame regimeDifferent regimesGaussian
PDE-T (TRACE)Active Matter0.10850.1690 (+58%)0.2869 (+162%)0.2366 (+117%)
Euler periodic0.26330.4002 (+52%)0.4969 (+89%)0.4687 (+78%)
Rayleigh-Bénard0.01020.0843 (+726%)0.0874 (+757%)0.0660 (+547%)
TGC0.18410.2328 (+27%)0.2657 (+45%)0.2345 (+29%)
TRL 2D0.25100.2546 (+1.4%)0.3582 (+43%)0.2942 (+17%)
MHD0.33360.3359 (+0.6%)0.4226 (+26%)0.3817 (+14%)
MPP (TRACE)Active Matter0.14100.1832 (+28%)0.3359 (+130%)0.3166 (+114%)
Euler periodic0.13250.1894 (+43%)0.3286 (+148%)0.2743 (+107%)
Rayleigh-Bénard0.13510.1482 (+8.5%)0.1925 (+31%)0.1845 (+29%)
TGC0.25440.2739 (+8.1%)0.3169 (+26%)0.3001 (+20%)
TRL 2D0.26060.2623 (+0.7%)0.3412 (+31%)0.2994 (+15%)
MHD0.47240.4518 (-4.7%)0.5031 (+6.1%)0.5320 (+13%)

The results are consistent with the previous experiment. Replacing the conditioning vector with one from the same regime generally leads to a smaller increase in forecasting error than using a vector from a different regime or a Gaussian vector. This confirms that the forecasters exploit regime-level information. However, the remaining performance degradation indicates that the vectors also contain trajectory-specific information.

W2 — More zero-shot experiments and the contributions of RF versus field embeddings

W2.1 The Zero-shot experiment only involves two unseen physical systems and lacks a baseline model for direct comparison

Zero-shot forecasting on unseen physical systems. As suggested, we ran zero-shot forecasting on three new datasets from The Well, with PDE-Transformer and MPP, with and without TRACE conditioning, and with the PDE-Transformer base as a direct baseline. Zero-shot is now evaluated on five unseen systems instead of two in the original paper: PlanetSWE, Helmholtz staircase and Euler with open boundaries (Euler with periodic boundaries is in the training set), in addition to Rayleigh-Taylor and Shear Flow. The metric is step-1 NRMSE.

ModelRayleigh TaylorShear FlowPlanetSWEHelmholtzEuler open BC
PDE-T (base)0.4020.3851.0280.9690.875
PDE-T (TRACE)0.3820.3600.9820.8370.637
MPP (base)-----
MPP (TRACE)0.7580.4540.7350.6800.384

MPP (base) cannot handle zero-shot on unseen datasets: it encodes each input with a learned embedding specific to each dataset and channel.

PDE-Transformer (base) can run on unseen sets: its conditioning takes each field's embedding from its name, so fields seen in training keep their trained embedding and new fields get an untrained one.

There are two main conclusions from these experiments:

  • Improved zero-shot forecasting: TRACE reduces PDE-Transformer's step-1 NRMSE on all five unseen systems, with particularly substantial improvements on Euler with open boundaries (0.875 → 0.637).
  • Transfer to previously unseen physical systems: TRACE's dynamic field embeddings allow MPP to operate on unseen systems for which its native dataset-specific embeddings are unavailable. Moreover, MPP+TRACE achieves competitive accuracy with PDE-T+TRACE, outperforming it on three of the five datasets.

These results support the usefulness of TRACE for cross-physics generalization, both by improving forecasting accuracy and by enabling architectures with dataset-specific field embeddings to handle unseen systems.

W2.2 the independent contributions of RF and dynamic Field Embedding mechanisms have not been fully distinguished

RF and field embeddings zero-shot ablation. For the zero-shot forecasting experiment, we compared the vanilla PDE-Transformer baseline, TRACE (JEPA + RF) with only the regime vector (no field embeddings), full TRACE (JEPA + RF) with both regime vector and field embeddings, and a JEPA-only model also with both components. The experiments are performed on Rayleigh–Taylor and Shear Flow. The metric is step-1 NRMSE.

PDE-TRayleigh TaylorShear Flow
TRACE$_\text{r}^\text{f}$ (RF + regime and field embeddings)0.3820.360
JEPA-only$_\text{r}^\text{f}$ (regime and field embeddings)0.369 (-3%)0.378 (+5%)
TRACE$_\text{r}$ (RF + regime embedding)0.432 (+13%)0.397 (+10%)
base PDE-T 1 frame0.412 (+8%)0.458 (+27%)

On these two unseen systems, dynamic field embeddings provide the main contribution to zero-shot forecasting accuracy. Removing RF has only a limited effect, whereas removing dynamic field embeddings increases the error on both datasets. Thus, the zero-shot improvements appear to rely primarily on the dynamic field embeddings rather than on the RF objective.

Q1 — What does RF discriminate, and what is actually invariant?

Q1.1 How does the RF objective function handle trajectories with the same physical mechanism but different initial conditions?

In Eq. 6, all other windows in the batch are treated as negatives, including windows from the same trajectory and trajectories sharing the same regime parameters. We measured the composition of the negative pool during training: on average, 0.4% of negatives come from the same trajectory and 8.1% share the anchor's regime. The detailed statistics are reported below.

DatasetSame trajectory (%)Same regime (%)
Acoustic maze0.05--
Active matter0.562.3
Euler periodic0.0210.0
Gray-Scott0.1016.7
Rayleigh-Bénard0.072.9
TRL 2D1.3511.1
MHD 641.2910.1
Supernova 640.16--
TGC0.043.7
Average0.408.1

We performed experiments on the parameter regression task, by removing the same regime trajectories from the denominator in Eq. 6. Results are provided in the table below (All trajectories for Eq. 6 with all the trajectories as negative and No same regime trajectory when excluding examples from the same regime). The setup and parameters of this expetriment correspond to table 10 in the manuscript. The metric is probe NRMSE.

NegativesPrRaα\alphaζ\zetaT0T_0zρ0\rho_0MaMs
All trajectories (Table 10)4.89E-21.46E-21.88E-27.79E-28.77E-11.47E-14.16E-13.34E-31.83E-2
No same regime trajectory4.74E-21.81E-21.99E-29.40E-28.71E-11.69E-14.32E-12.55E-34.13E-2

Removing same-regime negatives does not improve parameter inference under the tested protocol: probe errors increase on six of the nine parameters. These results do not indicate a benefit from excluding same-regime trajectories from the RF negative pool. Note that this corresponds to a realistic setting where multiple conditions of an underlying phenomenon have been captured.

Q1.2 How can it be proved that the learned representations mainly capture information at the dynamic mechanism level rather than information specific to trajectories?

See W1. The evidence does not prove that the representations mainly encode mechanisms rather than trajectories; it demonstrates sensitivity to shared regimes alongside residual trajectory-specific content.

Reviewer_WH3UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes TRACE, a self-supervised encoder for heterogeneous 2D/3D PDE trajectories. A universal patchifier that lifts 2D data into 3D allows a single set of weights to handle arbitrary dimensionality and numbers of fields, while axial attention and a set of learnable global tokens jointly produce local and trajectory-level representations. Pretraining combines JEPA-style masked latent prediction with an auxiliary objective called regime fingerprinting, which uses InfoNCE to align the global tokens of a student that sees only an initial portion of a trajectory with those of a teacher that sees the full trajectory. The frozen encoder is evaluated through physical parameter probing, t-SNE visualizations of the global tokens, and experiments conditioning two forecasting backbones on a single frame together with a regime embedding and per-field embeddings extracted by TRACE from the same 10-frame history, in place of feeding the history directly. Vector replacement controls check whether the forecasters actually use the extracted embeddings.

Critical strengths
  1. The problem setting is clear and well motivated. The goal of separating regime inference from state prediction, computing conditioning only once, and avoiding rereading the history during each rollout is concrete. The architectural choices that implement this goal, particularly the shared 2D/3D projection and field embeddings dynamically generated from context, are practical. The latter enables zero-shot evaluation on unseen systems that the baselines are structurally unable to handle.

  2. The downstream results are substantive. Replacing a 10-frame history with a single frame plus a compact vector yields substantial speedups for MPP with comparable first-frame accuracy, and improves first-frame accuracy for PDE-Transformer. Frozen TRACE features achieve lower parameter regression errors than the encoders of two foundation models trained for forecasting, and the stable-target control is a reasonable attempt to separate the signal from the additional machinery.

Critical weaknesses
  1. The central claim that “regime fingerprinting captures persistent regime characteristics” is not supported by the objective design, and the experimental evidence does not distinguish regime information from trajectory-specific information. The loss performs instance discrimination. Although the paper explicitly defines trajectories that share the same regime but differ in initial conditions, all such trajectories are treated as negatives in Eq. (6). The paper does not specify the negative pool, and given the single-system batch construction and the stride-1 windows used in several datasets, nothing prevents overlapping windows of the same source trajectory from serving as negatives. Thus, minimizing the loss encourages encoding trajectory-specific nuisances such as phase or starting time, directly conflicting with the stated goal. The results are consistent with this interpretation. The auxiliary objective helps only at small weights, increasing its weight degrades performance on most parameters, and its weight is decayed to zero toward the end of final pretraining; Section E.1 states that the corresponding ablations are in Section F.2, but F.2 contains only the λ sweep and no ablation of the decay schedule. The utilization controls are defined inconsistently: the main text describes a same-regime replacement, while the Table 5 caption and Appendix G.4 define it as a same-dataset shuffle; likewise, Table 5 describes the Random control as a vector from another trajectory in the batch, whereas G.4 defines it as a Gaussian draw. Consequently, the only experiment capable of separating regime information from trajectory information was not actually performed. The finding that replacing the vector with one from another real trajectory hurts PDE-Transformer more than Gaussian noise does also suggests that the vector encodes trajectory-specific information. Long-term rollouts using the trajectory’s own vector diverge on several datasets as well, although the paper does not rule out the single-frame input as the cause.

  2. The experimental design does not isolate the sources of the performance gains. The external comparisons in Table 2 differ from TRACE in architecture, capacity, training objective, and probe size, and the three Turbulence Gravity Cooling probes were evaluated before completing their training budget. In the final configuration, there is no model using the TRACE architecture trained with a forecasting objective, no JEPA-only model, and no reported numbers for the random-weight reference described in Appendix F.4. The t-SNE visualizations and all forecasting and control experiments use only the full model. Consequently, the contribution of regime fingerprinting over JEPA is measured only in a small-scale parameter regression ablation on a single pretraining run, where the reported three seeds are probe seeds, non-default settings use a single probe seed, and no variance is reported.

Questions for authors
  1. Please perform the replacement control as described in the main text, replacing the vector with one from a trajectory of the same regime but a different initial condition, on the four datasets with known regime parameters, and report it alongside the same-dataset shuffle and Gaussian controls. Neither the t-SNE proximity of same-regime trajectories (Figures 3b, 8, 9) nor the stable-target control (Table 11) substitutes for this test, since neither distinguishes regime information from trajectory information. Please also report, under the Table 10 protocol, regime fingerprinting variants that exclude same-source-trajectory windows and same-regime trajectories from the negatives.

  2. To isolate the sources of the Table 2 margins, please report in the final configuration a random-weight TRACE reference, a JEPA-only encoder, and the TRACE architecture trained with the baselines’ ten-frame next-step forecasting objective, with matched probe sizes. The causal temporal objective of Table 9 is not equivalent, as it predicts the within-window tendency rather than the next frame from a ten-frame history.

  3. Section E.1 points to F.2 for the decay-schedule ablation, but F.2 contains only the λ sweep. Why is the weight decayed to zero, and what are the results without decay? Section 2.2 states that the regime-level representation is extracted from z_global, whereas F.4 states that the probe reads both patch and global tokens, and Table 12 shows that removing the global tokens barely changes performance. Please report probing results reading only the global tokens and only the local tokens, for both the JEPA-only encoder and TRACE, and the t-SNE of Figures 3b and 9 for the JEPA-only encoder.

  4. What is the negative pool in Eq. (6)? On the stride-1 datasets, can overlapping windows of the same source trajectory appear together in a batch, and what proportion of negatives share the anchor’s regime?

  5. For PDE-Transformer, long-horizon errors with the trajectory’s own vector far exceed the base model’s on several datasets (Table 16), fine-tuning with shuffled vectors is more stable (Table 21), and TRACE_r diverges as well, so the field embeddings are not the cause. Please report PDE-Transformer with ten input frames plus TRACE conditioning, and with one input frame without conditioning, to separate the conditioning from the single-frame input as the cause. If the divergence is attributed to the backbone rather than the representation, given that MPP does not show it (Table 17), please explain why the same vector destabilizes one backbone and not the other.

7 comments

Thank you very much for your detailed review. We have carefully considered your comments and performed several additional experiments to address your concerns. In particular, we investigated the distinction between regime-level and trajectory-specific information, conducted additional ablations of regime fingerprinting and its training objectives, introduced new controls for parameter inference and forecasting, and analyzed and substantially reduced long-horizon forecasting instability. We clarified several methodological details and corrected inconsistencies in the manuscript. We address your remarks below, grouping related questions when appropriate.

W1 — Regime information versus trajectory-specific information

W1.1 the experimental evidence does not distinguish regime information from trajectory-specific information.

Let us consider the vectors inferred by TRACE (denoted TRACE vectors in the following) used to condition the baseline forecasters.

We measured how much these learned conditioning representations depend on the initial conditions in two different ways.

Measuring similarities between TRACE conditioning vectors. First we compare centered TRACE vectors from three settings:

  1. Same trajectory: same trajectory at different time windows,
  2. Same regime: distinct trajectories sharing regime parameters but with different initial conditions,
  3. Different regimes: trajectories from distinct regimes. The metric is mean cosine similarity.
DatasetSame trajectorySame regimeDifferent regimes
Active matter0.730.75-0.04
Euler periodic0.24-0.08-0.09
Rayleigh-Bénard0.140.13-0.09
TRL 2D0.580.57-0.09
Gray-Scott0.190.14-0.15
MHD 640.970.94-0.10
TGC-0.06-0.10-0.21

On all datasets except Euler, the Same trajectory and Same regime similarities are close to each other, and both are clearly higher than Other regime: the vectors group by regime and depend little on the initial condition. On Euler, the vector separates trajectories but not regimes.

Acoustic Scattering Maze and Supernova are not shown because their parameters do not vary across trajectories.

Counterfactual conditioning-vector replacement. In a second, forecasting experiment, we replace the conditioning vector with:

  1. Own vector: the vector from the original trajectory (no replacement),
  2. Same regime: the vector from a trajectory of the same regime with another initial condition,
  3. Different regimes: the vector comes from a trajectory from another regime,
  4. Gaussian: the vector comes from a Gaussian draw with the dataset's mean and standard deviation.

Measures were performed on the first window of every validation trajectory (this is a different setting from table 20). The metric is step-1 NRMSE.

ModelDatasetOwn vectorSame regimeDifferent regimesGaussian
PDE-T (TRACE)Active Matter0.10850.1690 (+58%)0.2869 (+162%)0.2366 (+117%)
Euler periodic0.26330.4002 (+52%)0.4969 (+89%)0.4687 (+78%)
Rayleigh-Bénard0.01020.0843 (+726%)0.0874 (+757%)0.0660 (+547%)
TGC0.18410.2328 (+27%)0.2657 (+45%)0.2345 (+29%)
TRL 2D0.25100.2546 (+1.4%)0.3582 (+43%)0.2942 (+17%)
MHD0.33360.3359 (+0.6%)0.4226 (+26%)0.3817 (+14%)
MPP (TRACE)Active Matter0.14100.1832 (+28%)0.3359 (+130%)0.3166 (+114%)
Euler periodic0.13250.1894 (+43%)0.3286 (+148%)0.2743 (+107%)
Rayleigh-Bénard0.13510.1482 (+8.5%)0.1925 (+31%)0.1845 (+29%)
TGC0.25440.2739 (+8.1%)0.3169 (+26%)0.3001 (+20%)
TRL 2D0.26060.2623 (+0.7%)0.3412 (+31%)0.2994 (+15%)
MHD0.47240.4518 (-4.7%)0.5031 (+6.1%)0.5320 (+13%)

The results are consistent with the previous experiment. Replacing the conditioning vector with one from the same regime generally leads to a smaller increase in forecasting error than using a vector from a different regime or a Gaussian vector. This confirms that the forecasters exploit regime-level information. However, the remaining performance degradation indicates that the vectors also contain trajectory-specific information.

In summary, both experiments show that TRACE vectors capture substantial regime-level information: representations from different trajectories sharing the same regime are generally more similar, and replacing a conditioning vector with one from the same regime is less detrimental than using a different regime. However, the replacement experiment also reveals some trajectory-specific information. Thus, TRACE learns representations that are sensitive to the underlying dynamical regime, without being strictly invariant to initial conditions.

W1.2 & Q4 The paper does not specify the negative pool - What is the negative pool in Eq. (6)?

In the denominator of Eq. 6, all other windows in the batch are treated as negatives, whether they come from the same trajectory, the same regime or another regime. On average, on all the datasets, there are about 8% of the trajectories that belong to the same regime. This means that on average, in the denominator of Eq. 6, one finds the same percentage. The statistics for each dataset are provided in the table below.

DatasetSame trajectory (%)Same regime (%)
Acoustic maze0.05--
Active matter0.562.3
Euler periodic0.0210.0
Gray-Scott0.1016.7
Rayleigh-Bénard0.072.9
TRL 2D1.3511.1
MHD 641.2910.1
Supernova 640.16--
TGC0.043.7
Average0.408.1

In a complementary experiment we did a negative pool ablation: we performed experiments on the parameter regression task, by removing the same regime trajectories from the denominator in Eq. 6. This is addressed in the answer to Q1.2. The conclusion is that performances are affected but do not change significantly (please see the detailed answer below).

W1.3 no ablation of the decay schedule

See answer to Q3.1.

W1.4 The utilization controls are defined inconsistently

You are right. "Context" in Table 5 is the "Shuffled" control of Table 20: the vector of another trajectory of the same dataset, of any regime. "Random" is a Gaussian draw matched to the dataset's statistics, as G.4 says. Only Table 5's caption described them wrongly; the values are unchanged. We corrected Sec. 3.5 and Table 5 so that all three agree.

W1.5 Long-term rollouts using the trajectory’s own vector diverge on several datasets as well

Following the suggestion, we analyzed the cause of this instability for PDE-Transformer. We found that the input normalization method, RevIN (Kim et al., 2022), is the main cause for this instability. At each rollout step, RevIN normalizes the input fields (the model's last prediction) by their own spatial mean and std. The output is rescaled back using the same statistics. With this process normalization errors accumulate over time. We found out that the following simple alternative corrects this instability: at test time, without retraining the model we use for the normalization the mean and std computed once from the first input frame instead of the RevIN normalization.

The table below gives the mean NRMSE for the 8-dataset at three horizons (1, 1:10, 10:20). The new normalization significantly reduces the error accumulation over time.

Normalization1st1:1010:20
computed at each step0.2580.8444.350
computed once from the first frame0.2580.7201.120

The next table provides detailed results per dataset on the long-term range (10:20) (NRMSE), and shows a large improvement over the per-step normalization of Table 16

Dataset ↓ \ PDE-T →(TRACE) each step norm(TRACE) first frame norm(base) first frame norm
Acoustic maze1.4121.0671.162
Active matter1.7411.2132.793
Euler periodic6.4991.3901.435
Rayleigh-Bénard16.4050.7581.278
TRL 2D4.4221.2902.061
MHD 642.3171.3021.946
Supernova 641.0881.1294.215
TGC0.9150.8091.595
Rayleigh Taylor0.7760.7610.999
Shear Flow1.8660.9781.498
Average4.3501.1202.061

Fixing normalization statistics throughout the rollout substantially reduces the previously observed instability, lowering average NRMSE over steps 10–20 from 4.350 to 1.120 (-74%), without retraining. Importantly, under the same normalization, TRACE also substantially outperforms the PDE-Transformer baseline (1.120 vs. 2.061). These results show that the original long-horizon instability was largely associated with the normalization procedure, and provide stronger evidence of TRACE's benefits for long-term forecasting. We will update Table 16 accordingly in the revised manuscript

W2 — Isolating pretraining effects and evaluating variability

W2.1 In the final configuration, there is no [...] JEPA-only model

See answer to Q2.2.

W2.2 In the final configuration, there is no [...] reported numbers for the random-weight reference

See answer to Q2.1.

Q1 — Same-regime replacement and RF negative-pool ablations

Q1.1 Please perform the replacement control as described in the main text

See answer to W1.1 second experiment (Counterfactual conditioning vector replacement).

Q1.2 Please also report, under the Table 10 protocol, regime fingerprinting variants that exclude same-source-trajectory windows and same-regime trajectories from the negatives.

RF negative-pool ablation. We performed two pretrainings, All trajectories for Eq. 6 with all the trajectories as negative and No same regime trajectory when excluding examples from the same regime. The setup and parameters of this expetriment correspond to table 10 in the manuscript. The metric is probe NRMSE.

NegativesPrRaα\alphaζ\zetaT0T_0zρ0\rho_0MaMs
All trajectories (Table 10)4.89E-21.46E-21.88E-27.79E-28.77E-11.47E-14.16E-13.34E-31.83E-2
No same regime trajectory4.74E-21.81E-21.99E-29.40E-28.71E-11.69E-14.32E-12.55E-34.13E-2

Removing same-regime negatives does not improve parameter inference under the tested protocol: probe errors increase on six of the nine parameters. These results do not indicate a benefit from excluding same-regime trajectories from the RF negative pool. Note that this corresponds to a realistic setting where multiple conditions of an underlying phenomenon have been captured.

Reviewer_EYmHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes TRACE, a self-supervised encoder for learning task-agnostic representations of physical dynamics. It combines JEPA-style masked latent prediction with a regime fingerprinting (RF) objective to capture persistent information from physical trajectories. The learned representations are used for physical parameter estimation and conditioning downstream forecasting models. Experiments are conducted on different 2D and 3D physical systems.

Critical strengths
  1. The paper is well motivated and clearly written. Separating the identification of underlying dynamics from downstream prediction is a reasonable idea. The RF objective also has a clear motivation and a simple design.

  2. The experiments are relatively comprehensive, covering different self-supervised objectives, parameter estimation, and forecasting with two backbones. The results show promising performance in parameter recovery and improved inference efficiency for MPP.

Critical weaknesses
  1. The contribution of RF to forecasting is unclear. The forecasting experiments change several factors at the same time, including the input history, conditioning mechanism, field embeddings, and some training settings. There is also no direct comparison between JEPA-only and JEPA+RF in downstream forecasting. It is therefore hard to tell how much RF contributes to the improvements.

  2. The short-term forecasting improvements sometimes come with serious long-term instability. For example, in Appendix Table 16, the NRMSE on Rayleigh-Bénard increases from 1.278 to 16.405 over rollout steps 11–20 when using TRACE. Similar problems appear on other datasets. The main text briefly mentions this issue, but does not discuss its severity enough. This is an important limitation for practical applications.

Questions for authors
  1. Could the authors compare JEPA-only and JEPA+RF in downstream forecasting, using the same backbone, conditioning mechanism, and training settings? This would help show whether RF actually improves forecasting performance.

  2. Could the authors explain the severe long-term instability on some datasets? What causes this problem, and when does TRACE help or hurt long-term forecasting? I also think these results should be shown and discussed more clearly in the main text.

3 comments

Thank you for the review. Following your remarks and suggestions, we conducted additional experiments to (1) disentangle the contributions of TRACE's conditioning components, (2) isolate the effect of regime fingerprinting on parameter inference and forecasting, and (3) investigate and substantially reduce the long-horizon forecasting instability observed with PDE-Transformer. We detail these new results below.

W1 — Confounding factors in forecasting comparisons

The forecasting experiments change several factors at the same time, including the input history, conditioning mechanism, field embeddings, and some training settings

Thanks for the suggestion. We performed ablations on PDE-Transformer starting from a base model and incorporating the two components of TRACE (regime embedding and dynamic field embeddings) one at a time, keeping the history fixed at one frame. The metric is average step-1 NRMSE on the eight datasets.

Model1st step NRMSEChange
PDE-T (base) 1 frame0.320
+ regime embedding (TRACE$_\text{r}$)0.268-16.3%
+ field embeddings (TRACE$_\text{r}^\text{f}$)0.258-3.7%

At a fixed input history, the regime embedding provides the largest improvement (-16.3%), while dynamic field embeddings bring a smaller additional gain (-3.7%). These results highlight the respective contributions of TRACE's two conditioning components, with regime conditioning accounting for most of the improvement in forecasting accuracy.

Q1 — Does regime fingerprinting improve downstream forecasting?

Could the authors compare JEPA-only and JEPA+RF in downstream forecasting

We performed this experiment for the parameter regression and forecasting tasks.

Parameter regression task. On the parameter regression task, we performed comparisons between JEPA-only and JEPA+RF encoders. Everything is kept constant except for their respective objective. The metric is normalized RMSE for nine physical parameters.

Pretraining objectivePrRaα\alphaζ\zetaT0T_0zzρ0\rho_0MaMs
JEPA + RF (Table 2)7.00E-28.25E-31.02E-19.69E-29.43E-11.98E-14.71E-13.70E-34.68E-2
JEPA only7.64E-21.77E-21.08E-12.00E-19.47E-11.90E-14.41E-13.10E-21.04E-1

Adding RF improves parameter inference on all six parameters associated with Rayleigh-Bénard, Active Matter, and MHD, with particularly large gains for Ra, ζ\zeta, and Ms. This supports the role of RF in learning representations that retain information about the underlying physical parameters. Results on TGC are mixed, however, indicating that the benefit is not uniform across all systems.

Forecasting task. We compared the JEPA-only and JEPA+RF encoders using the PDE-Transformer forecasting setup. The metric is NRMSE at step 1, over steps 1–10 and 10–20 on the eight datasets, computing the normalization once as specified in Q2.

1st1:1010:20
PDE-T →JEPA onlyJEPA+RFJEPA onlyJEPA+RFJEPA onlyJEPA+RF
Dataset ↓
Acoustic maze0.1360.1460.7860.7871.0721.067
Active matter0.1410.1380.6530.6591.2021.213
Euler periodic0.2860.2920.8230.8631.2991.390
Rayleigh-Bénard0.1300.1320.3900.3970.7530.758
TRL 2D0.2940.2951.3210.9261.5461.290
MHD 640.3290.3280.8430.8411.3031.302
Supernova 640.5930.5961.0080.8631.3031.129
TGC0.1430.1410.4280.4230.8100.809
Average0.2570.2580.7810.7201.1611.120

JEPA-only and JEPA+RF achieve nearly identical first-step forecasting accuracy. However, RF improves the average NRMSE over longer rollouts, by 8% over steps 1–10 and 4% over steps 10–20. Thus, RF primarily improves the recovery of physical parameters, while providing a more modest, dataset-dependent benefit for long-horizon forecasting.

Q2 — Long-horizon instability and its origin

Could the authors explain the severe long-term instability on some datasets?

Long-horizon forecasting stability. Following the suggestion, we analyzed the cause of this instability for PDE-Transformer. We found that the input normalization method, RevIN (Kim et al., 2022), is the main cause for this instability. At each rollout step, RevIN normalizes the input fields (the model's last prediction) by their own spatial mean and std. The output is rescaled back using the same statistics. With this normalization errors accumulate over time. We found out that the following simple alternative corrects this instability: at test time, without retraining the model we use for the normalization the mean and std computed once from the first input frame instead of the RevIN normalization.

The table below gives the mean NRMSE at three horizons (1, 1:10, 10:20) for the 8-datasets. The new normalization significantly reduces the error accumulation over time.

Normalization1st1:1010:20
computed at each step0.2580.8444.350
computed once from the first frame0.2580.7201.120

The next table provides detailed results per dataset on the long-term range (10:20) horizon (NRMSE), and shows a large improvement over the per-step normalization of Table 16.

Dataset ↓ \ PDE-T →(TRACE) each step norm(TRACE) first frame norm(base) first frame norm
Acoustic maze1.4121.0671.162
Active matter1.7411.2132.793
Euler periodic6.4991.3901.435
Rayleigh-Bénard16.4050.7581.278
TRL 2D4.4221.2902.061
MHD 642.3171.3021.946
Supernova 641.0881.1294.215
TGC0.9150.8091.595
Rayleigh Taylor0.7760.7610.999
Shear Flow1.8660.9781.498
Average4.3501.1202.061

Fixing normalization statistics throughout the rollout substantially reduces the previously observed instability, lowering average NRMSE over steps 10–20 from 4.350 to 1.120 (-74%), without retraining. Importantly, under the same normalization, TRACE also substantially outperforms the PDE-Transformer baseline (1.120 vs. 2.061). These results show that the original long-horizon instability was largely associated with the normalization procedure, and provide stronger evidence of TRACE's benefits for long-term forecasting. We will update Table 16 accordingly in the revised manuscript.

Reviewer_qNiQOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper outlines a path towards building foundation model using JEPA style prediction in latent space, instead of the more "traditional" auto-regressive rollouts in the pixel space, via comparisons in predictive accuracy (in the physical parameters of the underlying dynamics), inference cost, generalization, etc.

The paper has made significant contributions in 1/ creating a universal patchification (how do you turn trajectories of different dimensions (2D, 3D) and types (different number of channels) into a unified tokens format) 2/ recovery of physical parameters using a combination of JEPA and "regime fingerprinting" loss. 3/ Investigation of the learned representations through this proposed method and comparing it to the other pre-trained models.

However, while the method described recovered the physical parameters best (compared to two baselines), however, it is not obvious it will do well with dynamics roll outs given the various experiments. For example rolling out with MPP without conditioning it with TRACE representations frequently outperform rolling out with MPP with TRACE representations. It is worth noting that PDE-transformer rollouts seems to benefit from conditioning it with TRACE representation across all datasets included in the paper.

Critical strengths

The paper provides several significant contributions:

1/ a universal patchification (how do you turn trajectories of different dimensions (2D, 3D) and types (different number of channels) into a unified tokens format), this allows direct evaluation on unseen system without dataset specific look up tables for their field embeddings. 2/ a method (TRACE) to find persistent representations that is non-changing across time (usually represented by some kind of physical parameters) 3/ Pre-"conditioning" some roll out models with TRACE seems to help 1 model, but not the other over 9 different datasets.

Critical weaknesses

I would suggest a couple weakness that may need to be addressed:

Given the current experimental results, it is unclear whether the performance of TRACE will be generalized beyond PDE transformer. It would be nice to understand what makes TRACE work so much better for PDE-transformer than MPP (which is a very small model) in terms of roll out performance.

Questions for authors

Would you consider:

1/ Is it possible to try more than MPP, or PDE-transformer, given the large number of more recent foundation models given the time frame of the review period. Given that TRACE seems to really help PDE-transformer and not so much MPP in terms of roll out performance, it would be really nice to understand why that's the case, and when TRACE will be more useful.

2/ Is it possible to try more than two dataset beyond Rayleigh-Taylor and Shear Flow? It would be good to try something quite different in Mach number

3 comments

Thank you for the positive review. Following your suggestions, we added a third forecasting backbone (DPOT), analyzed the respective behaviors of our two initial baselines MPP and PDE-Transformer, and performed experiments on additional datasets for the 0-shot setting.

Q1 — Generality across forecasting backbones and why TRACE helps PDE-T more than MPP

Q1.1 Is it possible to try more than MPP, or PDE-transformer?

Additional forecasting backbone. We performed experiments with a new foundation backbone, DPOT (Hao et al., 2024), trained from scratch on eight training datasets in the same setting as for the two other baselines; This comes in complement to tables 16 (PDE-Transformer) and 17 (MPP) in the paper. As done for these tables we compare the base backbone DPOT versus its TRACE-conditioned version. The Metric is NRMSE at step 1 and over rollout steps 1–10 and 10–20.

1st1:1010:20
DPOT →baseTRACEbaseTRACEbaseTRACE
Dataset ↓
Acoustic maze0.0770.0630.1620.1200.3360.292
Active matter0.1120.0770.5110.2491.2210.814
Euler periodic0.2080.1290.5210.3531.0570.675
Rayleigh-Bénard0.1440.1010.3230.2190.5980.440
TRL 2D0.2780.2110.4690.4020.7310.699
MHD 640.4400.3290.8051.0991.1561.014
Supernova 640.7380.5780.8100.7581.0071.091
TGC0.1910.1190.5050.3611.0080.757

The results are aligned with the ones obtained with PDE-Transformer. TRACE consistently improves DPOT forecasting, reducing step-1 NRMSE on all 8 datasets and rollout NRMSE on 7 of 8 datasets at both horizons. This extends the improvements observed with PDE-Transformer to a third forecasting backbone, supporting the generality of TRACE conditioning across architectures.

Q1.2 Given that TRACE seems to really help PDE-transformer and not so much MPP in terms of roll out performance, it would be really nice to understand why that's the case

We do not have a full explanation for this difference, but the additional experiments rule out several possibilities. The benefit is not specific to PDE-T, since DPOT also improves on all 8 datasets (mean step-1 NRMSE: 0.274 → 0.201). Model size does not explain the difference (MPP: 221M, PDE-T: 210M, DPOT: 218M), nor does the location of conditioning (every block in PDE-T and MPP, once in DPOT). It is also not explained by the learning rate: at the baseline learning rate, TRACE improves PDE-T from 0.320 to 0.279 and DPOT from 0.274 to 0.223. Finally, MPP does use TRACE conditioning: replacing its conditioning vector increases its error by 52–74% (Table 5).

The results suggest that TRACE is particularly beneficial when the forecasting backbone does not already extract sufficient information about the dynamics from its inputs. MPP achieves strong forecasting accuracy without TRACE, which may explain its limited additional benefit from conditioning. This interpretation remains a hypothesis, but the improvements observed with PDE-T and DPOT show that TRACE's forecasting benefits generalize beyond a single architecture.

Q2 — Zero-shot generalization beyond two unseen systems

Q2.1 Is it possible to try more than two dataset beyond Rayleigh-Taylor and Shear Flow?

Zero-shot forecasting on unseen physical systems. As suggested, we ran zero-shot forecasting on three new datasets from The Well, with PDE-Transformer and MPP, with and without TRACE conditioning, and with the PDE-Transformer base as a direct baseline. Zero-shot is now evaluated on five unseen systems instead of two in the original paper: PlanetSWE, Helmholtz staircase and Euler with open boundaries (Euler with periodic boundaries is in the training set), in addition to Rayleigh-Taylor and Shear Flow. Metric is step-1 NRMSE.

ModelRayleigh TaylorShear FlowPlanetSWEHelmholtzEuler open BC
PDE-T (base)0.4020.3851.0280.9690.875
PDE-T (TRACE)0.3820.3600.9820.8370.637
MPP (base)-----
MPP (TRACE)0.7580.4540.7350.6800.384

MPP (base) cannot handle zero-shot on unseen datasets: it encodes each input with a learned embedding specific to each dataset and channel.

PDE-Transformer (base) can run on unseen sets: its conditioning takes each field's embedding from its name, so fields seen in training keep their trained embedding and new fields get an untrained one.

There are two main conclusions from these experiments:

  • Improved zero-shot forecasting: TRACE reduces PDE-Transformer's step-1 NRMSE on all five unseen systems, with particularly substantial improvements on Euler with open boundaries (0.875 → 0.637).
  • Transfer to previously unseen physical systems: TRACE's dynamic field embeddings allow MPP to operate on unseen systems for which its native dataset-specific embeddings are unavailable. Moreover, MPP+TRACE achieves competitive accuracy with PDE-T+TRACE, outperforming it on three of the five datasets.

These results support the usefulness of TRACE for cross-physics generalization, both by improving forecasting accuracy and by enabling architectures with dataset-specific field embeddings to handle unseen systems.

Reviewer_qNiQOpenReview ↗

Brief response to author's new comments

I thank authors' recent experiments confirm my expectation that this method should generalize beyond the two datasets selected by the authors originally.

I update my score accordingly

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

We sincerely thank the Area Chair and all four reviewers for their careful evaluation and constructive feedback. Their comments motivated substantial additional experiments that strengthen our results and clarify several aspects of TRACE, particularly the information captured by its representations and their contribution to downstream forecasting.

We highlight five main additions to the paper

  1. Generality across forecasting architectures. (qNiQ) We evaluated TRACE with a third backbone, DPOT, and found consistent improvements across the eight training datasets.

  2. Extended zero-shot generalization. (qNiQ, kPbK) We expanded the evaluation from two to five unseen physical systems and introduced direct baseline comparisons. TRACE improves PDE-Transformer on all five systems, while its dynamic field embeddings enable MPP to operate on previously unseen systems.

  3. Understanding the learned representations. (EYmH, WH3U, kPbK) New similarity measurements and conditioning-vector replacement experiments show that TRACE captures substantial information about the underlying dynamical regimes, while retaining some trajectory-specific information. Additional ablations distinguish the contributions of regime fingerprinting and dynamic field embeddings.

  4. Isolating the contribution of regime fingerprinting. (EYmH, WH3U, kPbK) We introduced matched JEPA-only comparisons, random-weight controls, negative-sampling ablations, and analyses of local and global representations. These experiments clarify that RF contributes most clearly to physical-parameter inference, with more modest benefits for forecasting.

  5. Improved long-horizon forecasting stability. (EYmH, WH3U) We identified the repeated recomputation of normalization statistics as a major source of instability in PDE-Transformer. Fixing these statistics throughout the rollout reduces average long-horizon NRMSE from 4.350 to 1.120 without retraining, while preserving a substantial advantage over the baseline.

We have also revised the manuscript with changes highlighted in blue, to incorporate these findings, clarify methodological details, and correct inconsistencies identified by the reviewers.

Several comments raised related questions. We address them in the individual responses, providing the corresponding experiments and conclusions for each reviewer.

We hope that these additional results and clarifications address the main concerns and provide a more complete assessment of TRACE's contributions. We would be happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.