- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether pretrained physics tokenizers can be reused for autoregressive emulation across different dynamics. Using a conv-tokenizer and a Transformer across three fluid benchmarks, the authors show that in-domain pretraining boosts both accuracy and data efficiency, but out-of-domain transfer calls for tokenizer fine-tuning. Interestingly, keeping an in-domain pretrained tokenizer mostly frozen improves long-term rollout stability on Euler and RB compared with fully fine-tuning it. Finally, borrowing initial weights from related physical fields facilitates adaptation to unseen variables.
Critical strengths
By varying tokenizer initialization while keeping the downstream processor architecture unchanged and training it from scratch, the authors isolate the benefits of tokenizer pretraining across three physical benchmarks. In addition, the paper offers insightful guidance on adaptation strategies, revealing that freezing an in-domain tokenizer can enhance rollout correlation despite higher next-step error, while out-of-domain transfer demands fine-tuning. This effectively highlights the limitations of relying solely on short-term prediction metrics to evaluate autoregressive emulators.
Critical weaknesses
The paper lacks sufficient experimental rigor regarding both rollout stability and data-scarcity claims. In Figure 3, relying on a single correlation threshold overlooks amplitude and mean drift, while an evaluation sample of only nine TRL trajectories provides weak corroboration for stability gains. Furthermore, Figure 4 provides uncertainty estimates solely for the full-data regime, leaving seed and subset-selection variability unquantified in the data-scarce regimes. These omissions weaken confidence in the paper's core empirical findings.
Questions for authors
See weaknesses
2 comments
Thank you for the clarifications and additional analyses, which are helpful in assessing the paper. I maintain my original rating of Weak Accept.
Response to reviewer kh8L
We thank the reviewer for their thoughtful assessment and for highlighting the insights from our controlled comparisons, particularly regarding tokeniser adaptation and long-term rollout stability. We address the concerns about rollout evaluation and data-scarcity robustness below.
1. Rollout stability metrics
The reviewer suggests extending the rollout evaluation beyond decorrelation time and a single threshold. We agree that these additional perspectives would provide a more comprehensive assessment of rollout performance. We have therefore included two complementary analyses in Appendix E:
Rollout VRMSE (Figures 11-13), which captures prediction errors over the autoregressive rollout, including differences in amplitude and mean. The findings agree with the observed patterns in the decorrelation time analysis (Figure 3).
Decorrelation time at an additional threshold of 0.95 (Figure 14), which establishes that the observed benefits of in-domain pretraining and the effects of tokeniser freezing are robust to the choice of correlation threshold.
We thank the reviewer for their suggestion and contribution to improving the study.
2. Limited number of TRL validation trajectories
The nine trajectories correspond to the entire TRL validation split available in The Well. We follow the same approach for Euler and RB, computing rollout metrics on all available validation trajectories. Although the TRL validation split covers all nine cooling-time parameter values in the dataset, representing a range of physical conditions, we acknowledge that the small number of trajectories limits how confidently we can draw conclusions from the rollout evaluation of TRL.
Importantly, our findings regarding the benefits of in-domain pretraining for rollout performance are also supported by Euler and RB. We will clarify in the revised paper that rollout evaluation uses all available validation trajectories.
3. Training-seed and subset-selection variability in data-scarcity experiments
We thank the reviewer for their valuable suggestion regarding subset-selection variability, and agree that investigating the sensitivity of our data-scarcity results to different training subsets would strengthen our analysis.
We have therefore added data-scarcity experiments using two new training subsets, alongside the original subset. These experiments cover the most data-scarce settings for RB (0.3%) and TRL (10%), across all five initialisation and adaptation settings. We have incorporated the results into Figure 4 and have also included rollout VRMSE learning curves for data-scarce experiments in Appendix F (Figures 15 and 16).
Regarding training-seed variability, we acknowledge that additional repetitions would provide a more comprehensive uncertainty characterisation. Within the compute budget available for our study, we prioritised exploring a broader range of physical systems, pretraining strategies, tokeniser adaptation settings, and data-scarcity levels, while conducting multiple-seed evaluations for selected experiments. This approach allowed us to investigate the behaviour of pretrained tokenisers across a wider range of conditions while keeping the experiments computationally feasible. We have provided further details on the rationale behind these experimental choices and our computational budget in the “Experimental scope and compute budget” section of our common response.