- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents TRACE, a self-supervised representation learning framework for heterogeneous spatio-temporal physical systems. is paper presents TRACE, a self-supervised representation learning framework for heterogeneous spatio-temporal physical systems. The main idea is to separate the learning of physical dynamics representations from downstream forecasting tasks. This framework introduces a unified patch-based encoder for physical systems with different spatial dimensions, resolutions and fields. It combines JEPA-style hidden potential prediction targets and state fingerprint targets, which align the representation of early partial observations with that of the complete trajectory. The author evaluates the learned representations through physical parameter regression, dynamic prediction, and cross-system generalization. Experiments conducted on The Well dataset show improvements in parameter recovery of several physical systems, competitive prediction performance, and the ability to adjust downstream models on previously unseen systems.
Critical strengths
- The paper indicates that TRACE can learn representations related to physical properties. It can be seen from Figures 3,8 and 9 in the paper that the visualization of t-SNE and PCA shows the meaningful association between potential representations and physical parameters, and the parameter regression experiment quantitatively supports the validity of these representations.
- The Repeat Patchifier proposed by the author in the paper provides a concise method for processing two-dimensional and three-dimensional physical data using shared projection weights, avoiding the need to design input layers separately for different dimensions. The ablation experiment in the paper also demonstrated that this design can achieve comparable or relatively good performance, so it has practical value for the unified modeling of heterogeneous physical systems.
Critical weaknesses
1.Although RF has improved the recovery accuracy of physical parameters, it is still unclear whether the learned representations capture features at the dynamic mechanism level or information specific to trajectories. The existing experiments have not directly verified the invariance of the characterization to the initial conditions, and this property is precisely the important basis for the authors to propose the RF method. 2.The Zero-shot experiment only involves two unseen physical systems and lacks a baseline model for direct comparison. Furthermore, the independent contributions of RF and dynamic Field Embedding mechanisms have not been fully distinguished, making it difficult to determine the main source of the model's generalization ability.
Questions for authors
How does the RF objective function handle trajectories with the same physical mechanism but different initial conditions? How can it be proved that the learned representations mainly capture information at the dynamic mechanism level rather than information specific to trajectories?
3 comments
W2 — More zero-shot experiments and the contributions of RF versus field embeddings
W2.1 The Zero-shot experiment only involves two unseen physical systems and lacks a baseline model for direct comparison
Zero-shot forecasting on unseen physical systems. As suggested, we ran zero-shot forecasting on three new datasets from The Well, with PDE-Transformer and MPP, with and without TRACE conditioning, and with the PDE-Transformer base as a direct baseline. Zero-shot is now evaluated on five unseen systems instead of two in the original paper: PlanetSWE, Helmholtz staircase and Euler with open boundaries (Euler with periodic boundaries is in the training set), in addition to Rayleigh-Taylor and Shear Flow. The metric is step-1 NRMSE.
| Model | Rayleigh Taylor | Shear Flow | PlanetSWE | Helmholtz | Euler open BC |
|---|---|---|---|---|---|
| PDE-T (base) | 0.402 | 0.385 | 1.028 | 0.969 | 0.875 |
| PDE-T (TRACE) | 0.382 | 0.360 | 0.982 | 0.837 | 0.637 |
| MPP (base) | - | - | - | - | - |
| MPP (TRACE) | 0.758 | 0.454 | 0.735 | 0.680 | 0.384 |
MPP (base) cannot handle zero-shot on unseen datasets: it encodes each input with a learned embedding specific to each dataset and channel.
PDE-Transformer (base) can run on unseen sets: its conditioning takes each field's embedding from its name, so fields seen in training keep their trained embedding and new fields get an untrained one.
There are two main conclusions from these experiments:
- Improved zero-shot forecasting: TRACE reduces PDE-Transformer's step-1 NRMSE on all five unseen systems, with particularly substantial improvements on Euler with open boundaries (0.875 → 0.637).
- Transfer to previously unseen physical systems: TRACE's dynamic field embeddings allow MPP to operate on unseen systems for which its native dataset-specific embeddings are unavailable. Moreover, MPP+TRACE achieves competitive accuracy with PDE-T+TRACE, outperforming it on three of the five datasets.
These results support the usefulness of TRACE for cross-physics generalization, both by improving forecasting accuracy and by enabling architectures with dataset-specific field embeddings to handle unseen systems.
W2.2 the independent contributions of RF and dynamic Field Embedding mechanisms have not been fully distinguished
RF and field embeddings zero-shot ablation. For the zero-shot forecasting experiment, we compared the vanilla PDE-Transformer baseline, TRACE (JEPA + RF) with only the regime vector (no field embeddings), full TRACE (JEPA + RF) with both regime vector and field embeddings, and a JEPA-only model also with both components. The experiments are performed on Rayleigh–Taylor and Shear Flow. The metric is step-1 NRMSE.
| PDE-T | Rayleigh Taylor | Shear Flow |
|---|---|---|
| TRACE$_\text{r}^\text{f}$ (RF + regime and field embeddings) | 0.382 | 0.360 |
| JEPA-only$_\text{r}^\text{f}$ (regime and field embeddings) | 0.369 (-3%) | 0.378 (+5%) |
| TRACE$_\text{r}$ (RF + regime embedding) | 0.432 (+13%) | 0.397 (+10%) |
| base PDE-T 1 frame | 0.412 (+8%) | 0.458 (+27%) |
On these two unseen systems, dynamic field embeddings provide the main contribution to zero-shot forecasting accuracy. Removing RF has only a limited effect, whereas removing dynamic field embeddings increases the error on both datasets. Thus, the zero-shot improvements appear to rely primarily on the dynamic field embeddings rather than on the RF objective.
Q1 — What does RF discriminate, and what is actually invariant?
Q1.1 How does the RF objective function handle trajectories with the same physical mechanism but different initial conditions?
In Eq. 6, all other windows in the batch are treated as negatives, including windows from the same trajectory and trajectories sharing the same regime parameters. We measured the composition of the negative pool during training: on average, 0.4% of negatives come from the same trajectory and 8.1% share the anchor's regime. The detailed statistics are reported below.
| Dataset | Same trajectory (%) | Same regime (%) |
|---|---|---|
| Acoustic maze | 0.05 | -- |
| Active matter | 0.56 | 2.3 |
| Euler periodic | 0.02 | 10.0 |
| Gray-Scott | 0.10 | 16.7 |
| Rayleigh-Bénard | 0.07 | 2.9 |
| TRL 2D | 1.35 | 11.1 |
| MHD 64 | 1.29 | 10.1 |
| Supernova 64 | 0.16 | -- |
| TGC | 0.04 | 3.7 |
| Average | 0.40 | 8.1 |
We performed experiments on the parameter regression task, by removing the same regime trajectories from the denominator in Eq. 6. Results are provided in the table below (All trajectories for Eq. 6 with all the trajectories as negative and No same regime trajectory when excluding examples from the same regime). The setup and parameters of this expetriment correspond to table 10 in the manuscript. The metric is probe NRMSE.
| Negatives | Pr | Ra | z | Ma | Ms | ||||
|---|---|---|---|---|---|---|---|---|---|
| All trajectories (Table 10) | 4.89E-2 | 1.46E-2 | 1.88E-2 | 7.79E-2 | 8.77E-1 | 1.47E-1 | 4.16E-1 | 3.34E-3 | 1.83E-2 |
| No same regime trajectory | 4.74E-2 | 1.81E-2 | 1.99E-2 | 9.40E-2 | 8.71E-1 | 1.69E-1 | 4.32E-1 | 2.55E-3 | 4.13E-2 |
Removing same-regime negatives does not improve parameter inference under the tested protocol: probe errors increase on six of the nine parameters. These results do not indicate a benefit from excluding same-regime trajectories from the RF negative pool. Note that this corresponds to a realistic setting where multiple conditions of an underlying phenomenon have been captured.
Q1.2 How can it be proved that the learned representations mainly capture information at the dynamic mechanism level rather than information specific to trajectories?
See W1. The evidence does not prove that the representations mainly encode mechanisms rather than trajectories; it demonstrates sensitivity to shared regimes alongside residual trajectory-specific content.
Thank you for your detailed and constructive review. Following your comments and suggestions, we conducted several additional experiments to investigate the distinction between regime-level and trajectory-specific information, extend zero-shot evaluation to additional physical systems, and disentangle the contributions of regime fingerprinting and dynamic field embeddings. We also analyzed the RF negative sampling strategy and performed additional ablations to clarify its effects on parameter inference. We address your concerns below.
W1 & Q2 - Testing invariance to initial conditions
It is still unclear whether the learned representations capture features at the dynamic mechanism level or information specific to trajectories - evaluating invariance to initial conditions
Let us consider the vectors inferred by TRACE (denoted TRACE vectors in the following) used to condition the baseline forecasters.
We measured how much these learned conditioning representations depend on the initial condition in two different ways.
Measuring similarities between TRACE conditioning vectors. First we compare centered TRACE vectors from three settings:
The results show that, on most datasets, vectors from different trajectories sharing the same regime are substantially more similar than vectors from different regimes. This indicates that TRACE captures regime-level information, although the degree of invariance to initial conditions varies across datasets. Euler periodic is an exception, where the vectors distinguish trajectories but show little separation between regimes.
Note that Acoustic Scattering Maze and Supernova are not shown because their parameters do not vary across trajectories.
Counterfactual conditioning-vector replacement. In a second, forecasting experiment, we replace the conditioning vector with:
Measures were performed on the first window of every validation trajectory (this is a different setting from table 20). The metric is step-1 NRMSE.
The results are consistent with the previous experiment. Replacing the conditioning vector with one from the same regime generally leads to a smaller increase in forecasting error than using a vector from a different regime or a Gaussian vector. This confirms that the forecasters exploit regime-level information. However, the remaining performance degradation indicates that the vectors also contain trajectory-specific information.