acceptodds
Under review as a conference paper at ICLR 2027

Taming Pretrained Physics Tokenisers under Domain Shift

Abstract

Pretrained tokenisers offer a promising paradigm for making physics foundation models more modular and flexible, yet it remains unclear when their representations help downstream emulation and how robustly they transfer across physical systems. We systematically study these questions across three physical systems from The Well and a range of tokeniser pretraining settings, comparing emulators trained from scratch with those initialised using pretrained tokenisers. We find that pretraining improves next-step prediction and autoregressive rollout quality, while also making emulators more robust to downstream data scarcity. Under domain shift, successful transfer depends on allowing the pretrained tokeniser to adapt to the downstream system. We further find that reusing weights from a semantically related field improves transfer to physical fields unseen during pretraining. Together, these results provide practical principles for adapting and reusing pretrained physics tokenisers in downstream emulation.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
physics foundation models, physics tokenisers, representation learning, pretraining, transfer learning, domain shift, PDE emulation, surrogate modelling, autoregressive modelling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kh8LOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether pretrained physics tokenizers can be reused for autoregressive emulation across different dynamics. Using a conv-tokenizer and a Transformer across three fluid benchmarks, the authors show that in-domain pretraining boosts both accuracy and data efficiency, but out-of-domain transfer calls for tokenizer fine-tuning. Interestingly, keeping an in-domain pretrained tokenizer mostly frozen improves long-term rollout stability on Euler and RB compared with fully fine-tuning it. Finally, borrowing initial weights from related physical fields facilitates adaptation to unseen variables.

Critical strengths

By varying tokenizer initialization while keeping the downstream processor architecture unchanged and training it from scratch, the authors isolate the benefits of tokenizer pretraining across three physical benchmarks. In addition, the paper offers insightful guidance on adaptation strategies, revealing that freezing an in-domain tokenizer can enhance rollout correlation despite higher next-step error, while out-of-domain transfer demands fine-tuning. This effectively highlights the limitations of relying solely on short-term prediction metrics to evaluate autoregressive emulators.

Critical weaknesses

The paper lacks sufficient experimental rigor regarding both rollout stability and data-scarcity claims. In Figure 3, relying on a single correlation threshold overlooks amplitude and mean drift, while an evaluation sample of only nine TRL trajectories provides weak corroboration for stability gains. Furthermore, Figure 4 provides uncertainty estimates solely for the full-data regime, leaving seed and subset-selection variability unquantified in the data-scarce regimes. These omissions weaken confidence in the paper's core empirical findings.

Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Response to reviewer kh8L

We thank the reviewer for their thoughtful assessment and for highlighting the insights from our controlled comparisons, particularly regarding tokeniser adaptation and long-term rollout stability. We address the concerns about rollout evaluation and data-scarcity robustness below.

1. Rollout stability metrics

The reviewer suggests extending the rollout evaluation beyond decorrelation time and a single threshold. We agree that these additional perspectives would provide a more comprehensive assessment of rollout performance. We have therefore included two complementary analyses in Appendix E:

  1. Rollout VRMSE (Figures 11-13), which captures prediction errors over the autoregressive rollout, including differences in amplitude and mean. The findings agree with the observed patterns in the decorrelation time analysis (Figure 3).

  2. Decorrelation time at an additional threshold of 0.95 (Figure 14), which establishes that the observed benefits of in-domain pretraining and the effects of tokeniser freezing are robust to the choice of correlation threshold.

We thank the reviewer for their suggestion and contribution to improving the study.

2. Limited number of TRL validation trajectories

The nine trajectories correspond to the entire TRL validation split available in The Well. We follow the same approach for Euler and RB, computing rollout metrics on all available validation trajectories. Although the TRL validation split covers all nine cooling-time parameter values in the dataset, representing a range of physical conditions, we acknowledge that the small number of trajectories limits how confidently we can draw conclusions from the rollout evaluation of TRL.

Importantly, our findings regarding the benefits of in-domain pretraining for rollout performance are also supported by Euler and RB. We will clarify in the revised paper that rollout evaluation uses all available validation trajectories.

3. Training-seed and subset-selection variability in data-scarcity experiments

We thank the reviewer for their valuable suggestion regarding subset-selection variability, and agree that investigating the sensitivity of our data-scarcity results to different training subsets would strengthen our analysis.

We have therefore added data-scarcity experiments using two new training subsets, alongside the original subset. These experiments cover the most data-scarce settings for RB (0.3%) and TRL (10%), across all five initialisation and adaptation settings. We have incorporated the results into Figure 4 and have also included rollout VRMSE learning curves for data-scarce experiments in Appendix F (Figures 15 and 16).

Regarding training-seed variability, we acknowledge that additional repetitions would provide a more comprehensive uncertainty characterisation. Within the compute budget available for our study, we prioritised exploring a broader range of physical systems, pretraining strategies, tokeniser adaptation settings, and data-scarcity levels, while conducting multiple-seed evaluations for selected experiments. This approach allowed us to investigate the behaviour of pretrained tokenisers across a wider range of conditions while keeping the experiments computationally feasible. We have provided further details on the rationale behind these experimental choices and our computational budget in the “Experimental scope and compute budget” section of our common response.

Thank you for the clarifications and additional analyses, which are helpful in assessing the paper. I maintain my original rating of Weak Accept.

Reviewer_juiuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the effect of pretraining on autoregressive models that predict the time evolution of physics fields. The studied architecture is based on the combination of a tokenizer that learns an embedding and a processor that predicts the state of the field at a given time step. The paper analyzes different scenarios for the pretraining and fine-tuning, considering on-domain and off-domain datasets with respect to the downstream task and freezing parts of the network architecture when training the emulators. It also covers 3 different datasets and several evaluation metrics.

Its main contributions are the identification that pretrained tokenizers improve emulator performance, extend the decorrelation tasks and produce good results even when data is scarce. Furthermore, the paper demonstrates that out-of-domain pretraining can work when there is some physical overlap between the considered datasets.

Most of the results in this paper are properly validated. The main weak spots are that the results are provided with a low number of training seeds, no comparison with other methods from the literature is provided, in-domain pretraining relies on samples from the downstream training data distributions, and some details about the architecture and the experiments are not presented clearly.

Critical strengths
  • The paper studies the effect of pretraining for physics field autorregressive models in detail. It does a good job in covering the effects of weight freezing and out-of-domain pretraining on relevant tasks for the field. A documentation of the behavior of the models under data scarcity is also important.
  • The paper explores the effect of considering physics similarities between the pretraining and downstream datasets. This observation is positive for the development of useful foundation models for physics.
Critical weaknesses
  • There are no comparisons with approaches from the literature. The message of the paper would be strengthened by comparing its results with other baselines trained on the same datasets or by doing the fine-tuning with weights from a non-physical dataset pretraining.
  • The in-domain pretraining setup is built by splitting the dataset in two. The pretraining is done with one half and the emulator training with the other. This undermines the results obtained with the training from scratch, specially in the data scarcity experiments, since the model has already seen a large amount of data through the in-domain pretraining before the emulator training starts. This should be explicitly addressed, and a comparison of the results with a network trained from scratch with the full dataset should be provided. If possible, the pretraining should be repeated with in-domain datasets that do not contain samples from the data distribution that is used for the downstream tasks.
Questions for authors
  • You use 2 out-of-domain datasets for the TRL dataset but you only consider the two for the first experiment. Why? And which pretrained model do you consider for the other two?
  • What motivates the loss choice? Why do you use MSE for the processor and MAE for the tokenizer? Why don't you use the VRMSE that you use for evaluation?
  • What is the value of the rollout length? You introduce it but never specify it.
  • I am struggling a bit to understand the structure of your architecture. How do you extract field-specific output head after mapping them into a shared representation in the tokenizer? What is the structure of the tokens? Does each field and time step get its own token with 16 latent dimension?
6 comments
AuthorsOpenReview ↗

Response to reviewer juiu - Part 1

We thank the reviewer for their constructive feedback and for recognising the value of our work on pretraining strategies, data scarcity, and the role of physical similarities in cross-domain transfer. We address the concerns and questions raised below.

1. Comparisons with existing approaches and running more random seeds

We thank the reviewer for these suggestions and agree that comparisons with additional architectures, as well as evaluations across more random seeds, would further strengthen the study.

Our primary objective is to isolate the effect of tokeniser pretraining on downstream emulation. To this end, we compare emulators with different pretraining settings while keeping the architecture and training procedure fixed. Extending this investigation to other architectures would require repeating the controlled pretraining comparisons within each architecture family (across three emulation domains and five training settings). Are there particular comparisons that the reviewer considers especially informative but would not require a similarly extensive experimental evaluation?

Regarding the number of random seeds, we acknowledge that additional seeds would provide more robust uncertainty estimates. Within our available compute budget, we prioritised experimental breadth across domains, pretraining settting, adapdation approaches, and data-scarce scenarios, which would have been difficult to maintain with more extensive random-seed evaluations. We discuss these trade-offs in greater detail in the “Experimental scope and compute budget” section of our common response.

We find the reviewer's suggestion of pretraining on non-physical data particularly interesting, as it could provide further insights into the transferability of pretrained representations. However, pretraining on non-physical datasets raises some methodological considerations. Our tokeniser's input and output heads correspond to physical quantities such as pressure and velocity. When pretraining on natural videos with three RGB channels, there is no direct correspondence between the colour channels and physical fields. Introducing RGB-specific heads would therefore result in no field overlap with the downstream physical quantities, whereas our findings suggest that field overlap plays an important role in the transferability of out-of-domain tokenisers. We believe this is an interesting direction for future investigation and would appreciate the reviewer's perspective on how best to make such a comparison meaningful.

2. Dataset splitting and its implications for model comparisons and data-scarcity experiments

The reviewer highlights an important observation that the in-domain pretrained model has access to additional data during tokeniser pretraining, whereas the from-scratch baseline is trained only on the emulator-training split. They suggest that this difference may disadvantage the from-scratch baseline, particularly in the data-scarcity experiments.

The data-scarcity experiments are designed to reflect a scenario where a downstream user has limited task-specific data and must choose between training a model from scratch or reusing a tokeniser. The pretrained tokeniser is assumed to be available beforehand, potentially trained by someone else. This is consistent with the foundation model paradigm, where pretrained models are reused across downstream tasks. To replicate this setting, we split the data at the trajectory level. This separation also prevents the in-domain tokeniser from seeing and potentially memorising the trajectories used for emulation, making the in-domain transfer and data-scarcity results more robust. Although the in-domain tokeniser has seen additional examples from the same domain during pretraining, these were used for a different task and, importantly, none of the emulation trajectories were seen during pretraining.

Importantly, comparisons across panels of Figure 4 are informative in addressing the reviewer's concern about the amount of data available to the from-scratch model. For instance, we can compare the RB 2% in-domain run with the RB 100% no-pretraining run. The latter has access to 50 times more emulation training examples, yet achieves higher loss.

Beyond the data-scarcity scenario considered in our study, the reviewer's suggestion raises an interesting representation learning question, particularly relevant to the development of foundation models. Considering the tokeniser and emulator as a single system with access to the entire dataset, is it more beneficial to pretrain the tokeniser before training the emulator, or to train the entire system from scratch?

To investigate this question, we have initiated additional experiments in which the from-scratch emulator is trained on the combined pretraining and emulation splits. We thank the reviewer for bringing up this point and will also revise the paper to clarify the motivation behind our data-scarcity experiments.

AuthorsOpenReview ↗

Response to reviewer juiu - Part 2

3. In-domain pretraining under distribution shift

In our current in-domain setup, pretraining and emulation use disjoint trajectories (initial conditions) from the same physical system, with all physical parameter configurations represented in both splits. We would appreciate clarification on what the reviewer means by "in-domain datasets that do not contain samples from the data distribution used for downstream tasks." Are you suggesting pretraining on the same physical system but with different physical parameter configurations, or a different form of distribution shift?

4. Out-of-domain pretraining mixtures for TRL

In the TRL full-data experiments, we considered two out-of-domain pretrained tokenisers to see whether including RB in the pretraining mixture makes a substantial difference. We found no substantial difference between the two. Therefore, for the TRL data-scarcity experiments, we used the tokeniser pretrained without RB (on Euler, shear flow, and active matter). This is the same out-of-domain tokeniser used for RB emulation, allowing more direct comparisons between TRL and RB emulation, such as assessing which of the two datasets is harder to learn, without differences in the pretrained tokeniser being a confounding factor. We appreciate the reviewer's question and will clarify this choice in the paper.

5. Choice of loss functions

The primary motivation was to remain consistent with the established training objectives of the models our method relies on. We use MSE for tokeniser pretraining, following the L2 pixel reconstruction term in MAGVIT (Yu et al., 2023), and MAE for downstream dynamics prediction, following Walrus (McCabe et al., 2026, Sec. 4.2).

Beyond consistency with the original implementations, there is an additional reason for using MAE in dynamics forecasting. Squared-error objectives favour conditional-mean predictions, which can lead to oversmoothing and loss of fine-scale structures under uncertainty. This limitation has been documented in physics emulation, including turbulent-flow reconstruction and forecasting (Oommen et al., 2026; Renn et al., 2026). While this effect can also arise in reconstruction, it is particularly relevant to the downstream prediction task, where future states must be inferred from potentially incomplete information. MAE provides a less outlier-sensitive alternative.

We use VRMSE for evaluation because it provides a dimensionless, interpretable measure of prediction error relative to the spatial variability of each physical field. In particular, VRMSE > 1 indicates that the overall prediction error exceeds the spatial variation in the ground truth. However, directly optimising VRMSE,

\frac{\mathrm{RMSE}} {\sqrt{\sigma_{\mathbf{x}}^2+\epsilon}} ,$$ would weight errors inversely by the target's spatial standard deviation. This could disproportionately prioritise nearly constant fields during training, regardless of their relative importance to the forecasting task. Since our training already employs RMS normalisation to account for differences in field magnitudes, we retain the established MAE objective for emulator training.
AuthorsOpenReview ↗

Response to reviewer juiu - Part 3

6. Autoregressive rollout length

Thank you for pointing this out. In our evaluation, the quantity of interest is the decorrelation time, i.e., the number of autoregressive steps until the prediction's spatial correlation with the ground truth falls below the specified threshold. Thus, the effective rollout length is determined by when the prediction decorrelates. We use a maximum evaluation horizon of 40 steps and will clarify this distinction in the paper.

7. Tokeniser architecture and token structure

Each physical field (including scalars and vector components) corresponds to a channel with its own weight slice in the encoder's first convolution. The number of input channels of the encoder and output channels of the decoder equals the total number of physical fields across all datasets. When processing each dataset, we only select the convolutional weight slices corresponding to the fields it contains.

These channels are mapped into a shared feature representation and processed jointly by the encoder. The number of channels progressively increases from the number of physical fields to 16, 32 and 64, followed by a reduction to 18 in the bottleneck. Here, 16 refers to the number of channels in the first convolutional feature representation, not the token dimension. There are three stages of downsampling, which compress both in space and time. The encoder hence produces one continuous, 18-dimensional token per location on the compressed spatiotemporal grid, jointly encoding information from all physical fields within its spatiotemporal receptive field. These tokens are projected to 1088 dimensions for transformer processing, then projected back to 18 dimensions before passing to the decoder. The decoder reconstructs the fields, and its final convolution uses field-specific output-channel weight slices to recover the physical fields.

We appreciate the reviewer's feedback on clarity and will further streamline the explanation of the architecture and token structure in Appendix B.

References

Yu et al. (2023). MAGVIT: Masked Generative Video Transformer. CVPR 2023. Paper

McCabe et al. (2026). Walrus: A Cross-domain Foundation Model for Continuum Dynamics. ICML 2026. Paper

Oommen et al. (2026). Learning Turbulent Flows with Generative Models for Super Resolution and Sparse Flow Reconstruction. Nature Communications. Paper

Renn et al. (2026). Data-driven Surrogate Models for Forecasting Experimentally Measured Fluid Flows. arXiv:2606.10848. Paper

Reviewer_UtiAOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper provides systematic study of tokenizer pretraining for neural physical simulators. Specifically, it studies if using the pretrained tokenizer is more preferable of jointly training an encoder alongside physical models. The author uses an off-the-shelf processor architecture combined with MagViT-style tokenizer, results show that pretrained tokenizer improves upon baseline on one step prediction and longer rollout. It also shows cross-domain transferability.

Critical strengths
  1. Introducing the idea of a pretrained tokenizer is an interesting idea. The author studied four aspects of pretrained tokenizer for physical simulation, which can be valuable to the community
Critical weaknesses
  1. The adoption of tokenizer (autoencoder) for physical simulation is not a new idea, as already discussed in [1] [2], I do not see any new contributions from this work that is novel and distinct. These related works are not discussed and compared.
  2. Experiments are limited. Specifically, the original well benchmark has 16 diverse domains and this paper only covers 3. The results do not sufficiently establish that pretrained tokenizer is generally good across different physical domains.
  3. The author only explored one design of model architecture while the well included baselines of 4 architectures. It is unclear if the observed gain is limited to the specific chosen architecture or can be generalized.
  4. The compute is unfair for experiments with pretrained tokenizer. The author should consider increasing training time for the from-scratch run so that the total compute budget is matched. It may well be the case that spending to compute used to train the tokenizer directly to training the processor longer will be more beneficial.

[1] Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation [2] Generative Latent Neural PDE Solver using Flow Matching

Questions for authors

See Weaknesses. Please explain

  1. how is this work different from existing works using a latent tokenizer/VAE for PDE
  2. does the result generalize beyond 3 domains on the chosen architecture
3 comments
AuthorsOpenReview ↗

Response to reviewer UtiA - Part 1

We thank the reviewer for their feedback and for recognising the value of systematically investigating different aspects of tokeniser pretraining for physical simulation. We address the concerns regarding novelty, generalisability, and the training budget of the from-scratch baseline below.

1. Novelty and distinction from existing work

Our central contribution concerns two aspects of tokeniser pretraining for physics emulation: (i) controlled evaluation of its benefits within the same emulator architecture; and (ii) transfer under domain shift and data scarcity. Given the growing interest in physics foundation models, a systematic investigation that isolates the effects of tokeniser pretraining from other architectural and training choices is particularly timely.

Neither [1] nor [2] explores these avenues: they don’t compare pretrained and randomly initialised tokenisers within the same emulator architecture, and both use tokenisers pretrained on the same physical system as their downstream emulator. We acknowledge the similarity identified by the reviewer: both works employ domain-specific autoencoder pretraining for physics emulation, an approach also reflected in our in-domain, mostly-frozen setting. We do not claim novelty in the use of in-domain autoencoder pretraining itself.

More broadly, latent-space approaches to PDE emulation have used learned representations as a modelling approach for efficient and accurate prediction. Our study takes a different perspective: rather than proposing a new tokeniser or latent-space emulator that uses pretrained representations, we empirically investigate the effects of tokeniser pretraining through controlled comparisons, examining how these effects depend on domain alignment, downstream data availability, and tokeniser adaptation strategies. We believe this approach constitutes a distinct contribution to the literature.

We thank the reviewer for raising this point and will revise the Related Work section to more clearly position our contributions within the literature.

2. Generalisability across physical domains

We choose domains from The Well based on the number of spatial dimensions, physical field overlap, and the size of the training split. First, we exclude 3D domains, since extending the chosen architecture to 3D would require 4D spatiotemporal convolutions, substantially increasing computational and memory costs, and introducing architectural changes beyond the scope of this work. This results in nine 2D domains. Among these, we select domains with meaningful physical field overlap and number of training examples, leaving us with five domains: four used for tokeniser pretraining (Euler, RB, shear flow, and active matter) and three as emulation targets (Euler, RB, and TRL).

We choose our emulation targets such that they cover a range of difficulties (quantified by the VRMSE achieved by Walrus [3]), spatial resolutions, and physical field overlap with out-of-domain tokeniser. Table 1 compares the three emulation domains along these dimensions.

Table 1: Comparison of the three emulation domains in terms of spatial resolution, physical field overlap with the respective OOD tokeniser, and emulation difficulty (VRMSE achieved by Walrus). Field overlap denotes the fraction of emulation-domain fields also present in the OOD tokeniser's pretraining domains. Euler VRMSE values are averaged across the open and periodic boundary condition datasets.

Emulation domainSpatial resolutionField overlap with OOD tokeniserWalrus rollout VRMSE (t=1–20)Walrus rollout VRMSE (t=21–60)
Euler512 × 5121/4 (limited)0.0654 (easy)0.3116
RB512 × 1282/3 (high)0.0992 (medium)0.6441
TRL128 × 3843/3 (full)0.3393 (difficult)0.8648

Given that the main findings of the paper, including the advantages of in-domain and out-of-domain pretraining and the effects of different freezing strategies, hold across these systems despite their differences, we expect these findings to generalise beyond the three evaluated domains, although the relative benefits of pretraining may vary across emulation domains.

AuthorsOpenReview ↗

Response to reviewer UtiA - Part 2

3. Generalisability across model architectures

We agree with the reviewer that investigating the benefits of tokeniser pretraining across different architecture families would be an interesting empirical study in its own right, as also discussed in Subsections 5.2 and 5.3. We do not claim that the observed gains are architecture-independent.

We focus on the effect of tokeniser pretraining within a fixed architecture, allowing us to make controlled comparisons between training from scratch, in-domain pretraining, and out-of-domain pretraining. As explained in the “Experimental scope and compute budget” section of our common response, we made a deliberate choice to prioritise the breadth of pretraining and transfer experiments within our available compute budget.

Extending the study to additional architectures would require repeating the full set of comparisons for each architecture. This would have substantially limited our ability to investigate other aspects of tokeniser transfer, including three distinct downstream physical systems, multiple random initialisations for RB and TRL, different tokeniser freezing strategies, data-scarce settings, and the role of semantic field overlap in cross-domain transfer. We therefore chose to investigate these complementary aspects of tokeniser pretraining in greater depth, while keeping the architecture fixed. We agree that extending this analysis to other tokeniser and emulator architectures is an important and exciting direction for future work.

4. Fairness of compute for experiments with pretrained tokeniser

There are two important considerations here:

1. Downstream user perspective: Our setup considers a downstream user who has access to an already-pretrained tokeniser, potentially released publicly by another researcher or institution. For this user, the pretraining cost has already been incurred, and the relevant comparison is between training an emulator with a pretrained tokeniser and training the same emulator from scratch, under the same downstream training budget. This is the comparison presented in our paper.

2. Amortisation across downstream tasks: A pretrained tokeniser can be reused across multiple downstream tasks, without repeating its pretraining. Therefore, even when accounting for the initial pretraining cost, matching the full pretraining and downstream training budget for each from-scratch baseline would not reflect the total compute required when the tokeniser is reused across several tasks.

We agree that the reviewer raises a separate and interesting question: whether allocating the pretraining compute directly to longer emulator training would lead to better performance when considering a single task and the total end-to-end compute budget. The suggested comparison would address a complementary research question, while our experiments focus on establishing the benefits of tokeniser pretraining under a fixed downstream training budget. We will clarify this distinction in the paper.

References

[1] Rozet et al. (2025). Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation. NeurIPS 2025. Paper

[2] Li et al. (2026). Generative Latent Neural PDE Solver using Flow Matching. Machine Learning: Science and Technology. Paper

[3] McCabe et al. (2026). Walrus: A Cross-domain Foundation Model for Continuum Dynamics. ICML 2026. Paper

Thank the authors for the response. Some of my concerns are addressed, and recognize the limitation of compute and cost. That being said, I still consider this work as borderline leaning towards a rejection. Hence, I have raised my score to weak reject.

One of the key concern about generalizability is discussed and explained by the cost of compute, however that does not change my judgment that the work has limited impact and does not show the findings is generalizable, significantly impact the value of this work and the soundness of the conclusion beyond this specific setting.

In general, scientific research should produce generalizable insights as opposed to an engineering report of a specific system. As an analogy, pretraining studies in LLMs generally cover multiple model sizes/families/data scales. Even the MAGVIT-2, on which this paper's tokenizer is based on, explored using MLM and ViViT for different downstream tasks. I understand that there is considerable cost of training, but I think the authors could have designed the experiment differently.

Currently, it is unclear if the conclusions 1) "out-of-domain tokenizer cannot be used as is", 2) "jointly finetuning tokenizer by unfreeze it leads to better performance",3) "but frozen tokenizer sometimes benefits long rollout stability" holds in general, or is caused by certain biases in this specific tokenizer-processor combination. It may well be the case that a stronger, larger processor solves issue 1) because it learns to "fix imperfect tokenization". It may also fix 3) because stronger model can properly achieve stable rollout without a frozen tokenizer.

Even if training larger processor can be expensive, the authors can still conduct compute-matched studies with fewer steps. Further, 1) can be resolved with a better experiment design and analysis. Specifically, the authors can report the out-of-domain reconstruction quality of the tokenizer, which should be an upper bound of the processor. If the reconstruction fails, then it would be clear that 1) holds. Additionally, detailed studies on the reconstruction quality of OOD and in-domain tokenizer would also help understand the gap between these to setups independent of the choice of processor.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common response to all reviewers - Part 1

We thank all reviewers for their constructive feedback and for recognising the value of our systematic investigation of tokeniser pretraining for physics emulation.

 

Summary of contributions

Our main contribution is a controlled empirical study of when and how tokeniser pretraining benefits downstream emulation, comparing in-domain pretraining, out-of-domain pretraining, freezing strategies, adaptation techniques, data availability, and random initialisation within a fixed architecture. Our experiments establish several important findings:

  • Next-step prediction accuracy: Pretrained tokenisers improve next-step prediction accuracy compared with training from scratch, with benefits from both in-domain and out-of-domain pretraining (UtiA, juiu, kh8L).

  • Transfer across physical systems: Out-of-domain pretraining can improve downstream performance, but requires a trainable tokeniser (UtiA, juiu, kh8L). We further demonstrate that transferring weights between physically related fields facilitates adaptation to previously unseen quantities (juiu, kh8L).

  • Adaptation and rollout stability: Mostly freezing an in-domain tokeniser can improve long-term rollout stability despite higher next-step error, highlighting the importance of evaluating autoregressive emulators beyond short-term prediction accuracy (kh8L).

  • Data scarcity: Pretrained tokenisers retain substantial benefits even when downstream training data is limited, an important setting for the development of reusable physics representations (juiu, kh8L).

We appreciate the reviewers' recognition of these findings, particularly their relevance to understanding the transferability of pretrained representations and the development of useful physics foundation models.

 

Additional experiments

To address the reviewers' concerns and strengthen the paper, we have initiated several additional experiments. We prioritised experiments that would provide new insights or improve the soundness of the study while making the most of the additional compute available.

  • Data-scarcity robustness (kh8L): Repeat the most data-scarce experiments (RB 0.3% and TRL 10%) using two additional training-subset selections to quantify sensitivity to subset selection.
  • Full-data from-scratch baseline (juiu): Train additional from-scratch baselines using the combined pretraining and emulation splits to assess the benefits of pretraining when the baseline has access to all available training data.

We have incorporated the new data-scarcity experiments into the paper (Subsection 3.3 and Appendix F). The full-data baseline experiments are ongoing.

 

Paper revisions

In addition to the experiments above, we have updated the paper with the following revisions:

  • Related work (UtiA): Expanded the discussion of existing latent-tokeniser approaches and clarified how our contributions differ, particularly regarding controlled pretraining comparisons and cross-domain transfer.
  • Compute fairness (UtiA): Clarified why we compare pretrained and randomly initialised tokenisers under the same downstream training budget, distinguishing this from comparisons under a matched total compute budget that includes tokeniser pretraining.
  • Data-scarcity scenario (juiu): Clarified the motivation for using disjoint pretraining and emulation trajectories and the downstream data-scarcity scenario considered.
  • Experimental details (juiu): Specified the maximum autoregressive rollout horizon and clarified the choice of out-of-domain tokeniser for TRL.
  • Architecture (juiu): Improved the explanation of field-specific input/output heads and the structure of the shared latent tokens in Appendix B.
  • Rollout metrics and sample size (kh8L): Added decorrelation time at an additional correlation threshold and rollout VRMSE to provide a more comprehensive assessment of rollout performance (Appendix E). Clarified that rollout evaluation uses all available validation trajectories.

The new version is available to download from OpenReview. To facilitate reviewing the changes, we have highlighted the revised and newly added text in purple. The highlighting is for review purposes only and is not intended to appear in the final version.

AuthorsOpenReview ↗

Common response to all reviewers - Part 2

Experimental scope and compute budget

The additional experiments outlined in part 1 of the common response address several of the reviewers' concerns. We also recognise the value of further experiments, particularly across additional architectures and random seeds. Here, we provide some context for the experimental choices and computational constraints that shaped the scope of our study.

Our experimental design was motivated by the goal of isolating the effects of tokeniser pretraining. To achieve this, we designed our experiments to compare different pretraining and adaptation strategies within a fixed architecture. Furthermore, we chose to run experiments at the original data resolution and with reasonably sized models, rather than conducting more exhaustive experiments using much smaller configurations. We believe this makes the results more relevant to the community, particularly as physics foundation models continue to develop, and conclusions drawn from smaller-scale experiments may not necessarily generalise to larger settings.

Within our compute budget, these choices meant prioritising experimental breadth across different physical domains, adaptation strategies, and data-availability regimes over comparisons across additional architectures and more extensive random-seed evaluations. To put these trade-offs into perspective, a single emulator training run on 8 GPUs requires approximately 30 hours for Euler, 10 hours for RB, and 16 hours for TRL. The experiments reported in Figure 1 involve 5 Euler, 15 RB, and 21 TRL runs, amounting to approximately 5000 GPU-hours (USD 40,000 at representative commercial cloud rates). Adding just one random seed across the experimental settings of Figure 1 would require another 2500 GPU-hours (USD 20,000). Assuming other tokeniser architectures require comparable compute, reproducing Figure 1 with just one additional tokeniser family would require six tokeniser pretraining runs and the corresponding emulator experiments, amounting to approximately 8500 GPU-hours (USD 68,000). These estimates do not account for the additional experiments on data scarcity and semantic weight transfer, which further contribute to the overall computational cost of the study. While exploring additional architectures, downstream tasks, and random seeds would be valuable, the computational cost of these extensions limits what we can explore within the current study.

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage/continue to engage during this period so that we have our issues discussed thoroughly.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.