acceptodds
Under review as a conference paper at ICLR 2027

Pretraining Design Drives Gains in Zero-Shot PDE Rollouts

Abstract

Generalization beyond the pretraining distribution remains a major challenge for pretrained neural PDE solvers. We study zero-shot out-of-distribution (OOD) rollout prediction in fluid dynamics, where a pretrained model must predict future trajectories from a single initial condition without downstream finetuning, contextual demonstrations, or test-time adaptation. We uncover a surprisingly large performance gap: a lightweight ViT-style spatiotemporal model with only about 25M parameters outperforms the 629M-parameter pretrained Poseidon-L on several OOD systems, with approximately - lower error on two of them, despite its weaker in-distribution accuracy. We examine how pretraining design affects zero-shot transfer through controlled comparisons of autoregressive rollout, teacher forcing, non-autoregressive full-trajectory prediction, and different temporal conditioning schemes. We find that training formulation can change OOD errors by large factors. With full-history attention, teacher forcing improves our ViT's source-domain accuracy and yields better overall source-domain and OOD performance than AR rollout training, despite the training-inference mismatch. Comparisons with FNO reveal similar benefits in source-domain prediction and on several OOD systems, showing that the advantages of teacher forcing extend beyond our ViT architecture.

Primary area:
learning on time series and dynamical systems
Keywords:
Neural PDE Solvers, Scientific Machine Learning, Pretraining, Out-of-Distribution Generalization, Zero-Shot Generalization, Spatiotemporal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UjBuOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how training and prediction formulations affect zero-shot out-of-distribution (OOD) trajectory prediction in pretrained neural PDE solvers. The authors develop a lightweight spatiotemporal ViT and compare four formulations: local-history autoregressive rollout, full-history autoregressive rollout, teacher forcing, and non-autoregressive full-trajectory prediction. The models are pretrained on six fluid dynamics datasets and evaluated without finetuning on five OOD tasks involving different initial-condition distributions within the same PDE families. The experiments show that the choice of formulation can substantially affect OOD performance, with teacher forcing generally outperforming full-history autoregressive training despite the training-inference mismatch. The authors further investigate the effect of teacher forcing using a smaller ViT and an FNO. Finally, they report that their 25M-parameter ViT achieves substantially lower errors than the much larger Poseidon-L on two OOD tasks, even when Poseidon uses direct prediction.

Critical strengths

Systematic comparison of training and prediction formulations. The paper compares four formulations within the same ViT architecture, including local-history autoregressive rollout, full-history autoregressive rollout, teacher forcing, and full-trajectory prediction. The comparison between autoregressive training and teacher forcing is further extended to a smaller ViT and an FNO, providing a useful empirical analysis of how training strategy and temporal conditioning affect zero-shot OOD performance across different architectures.

Critical weaknesses
  1. Misleading baseline comparison in the central motivation. A central motivation of the paper is the surprisingly strong performance of a lightweight ViT compared with the much larger Poseidon-L model. However, the initial comparison in Table 1 evaluates Poseidon using autoregressive rollout, despite direct prediction being an established inference strategy for Poseidon. In fact, the original Poseidon paper (Herde et al., 2024, Appendix D.6.2) explicitly mentions comparing both strategies and selecting the better one for each task. It is therefore not surprising that using a less suitable inference strategy can substantially degrade Poseidon's performance. Indeed, the authors' own results show that direct prediction reduces Poseidon-L's error from 1.2350 (Table 1) to 0.2576 (Table 3) on NS-BB and from 1.4902 to 0.7152 on NS-VS. Although the paper later reports comparisons using direct prediction, the much larger performance gaps under autoregressive rollout receive considerable emphasis in motivating the work. This weakens the central narrative because part of the observed advantage comes from the baseline's inference strategy rather than the proposed pretraining design.

  2. Limited scope of OOD generalization. The paper investigates how pretraining design affects zero-shot OOD performance, but the evaluation is limited to distribution shifts within the same governing equation families. Specifically, all five OOD tasks retain the compressible Euler or incompressible Navier–Stokes equations, as well as the same physical variables and computational domain as the pretraining datasets, and primarily differ in their initial-condition distributions and resulting flow patterns (Section 4; Appendix C). While this is a meaningful form of OOD generalization, it provides limited evidence of whether the observed benefits of different pretraining formulations extend to more substantial shifts in the underlying physics. Evaluating additional settings involving different governing equations, physical parameters, or geometries would help establish the generality of the conclusions.

Questions for authors

Questions:

  1. Choice of inference strategy for Poseidon and framing of the paper. The original Poseidon paper (Herde et al., 2024, Appendix D.6.2) discusses both direct prediction and autoregressive rollout and selects the better strategy for each downstream task. Given that direct prediction substantially improves Poseidon-L's performance in your experiments, what motivated the decision to use autoregressive rollout in the initial comparison in Table 1? More importantly, since this comparison plays a central role in motivating the paper, I am concerned that it creates a misleading impression of the performance gap between the models.

    I suggest reconsidering the central framing to place greater emphasis on the investigation of training and inference strategies, rather than primarily on outperforming Poseidon. For example, the paper could first highlight how the same pretrained Poseidon model performs substantially differently under autoregressive rollout and direct prediction. This observation could then motivate a broader investigation into how training and inference strategies affect zero-shot OOD performance. The authors could present their controlled comparisons within the proposed ViT architecture, followed by a comparison with Poseidon using appropriate inference strategies. This would better reflect the paper's main empirical contribution.

  2. Selection of training duration. Section 5 uses 80 epochs for the controlled comparisons because longer training improves source-domain accuracy. However, the final comparison in Table 6 uses a teacher-forced ViT trained for 40 epochs. Could the authors clarify the rationale for this choice and report the corresponding results for the 80-epoch teacher-forced model? In particular, what criterion was used to select the training duration for the final comparison?

  3. Interpretation of the final comparison with Poseidon. Table 6 shows that the proposed ViT substantially outperforms Poseidon-L on two of the five OOD tasks (NS-BB and NS-VS), while Poseidon-L performs better on CE-RKH and NS-SL. On CE-RM, both models have relative L1 errors exceeding 1, meaning their errors are larger than those of a trivial zero-field predictor under the reported metric. These mixed results seem to provide more limited support for the paper's emphasis on outperforming Poseidon than the overall narrative suggests. Could the authors explain why the proposed ViT substantially outperforms Poseidon-L on some OOD tasks but performs considerably worse on others, particularly within the Navier–Stokes family, and discuss what this suggests about the generalization capabilities and limitations of the proposed approach?

  4. Discussion of related work. The related work section could benefit from discussing Walrus: A Cross-domain Foundation Model for Continuum Dynamics (McCabe et al., ICML 2026). Walrus is particularly relevant given its focus on cross-domain pretraining and stable long-horizon prediction of physical dynamics. How do the findings of the present paper relate to the training and prediction strategies studied in Walrus?

Comments and Suggestions:

  1. Clarify the use of "rollout" and "trajectory prediction." The paper frequently uses "rollout prediction" to describe the overall task, including in the title and Section 3. However, one of the four formulations, full-trajectory prediction, generates all future states jointly rather than through autoregressive rollout. This makes it somewhat unclear whether "rollout" refers specifically to autoregressive prediction or more generally to predicting a future trajectory. I suggest distinguishing these terms more consistently, using "trajectory prediction" or "emulation" for the general task and "autoregressive rollout" when referring to recursive prediction.

  2. Clarity of Figure 1(b). Figure 1(b) is difficult to follow without consulting the subsequent sections and Appendix B, despite being the main figure introducing the four formulations. In particular, the relationship between the evolution blocks ξτ\xi_\tau, their embedded representations, and the physical states utu_t is not immediately clear. The figure also mixes training and inference formulations without explicitly distinguishing them. For example, teacher forcing differs from full-history AR in whether ground-truth or predicted histories are used during training, but both use the same autoregressive inference procedure. I suggest replacing the equations in Figure 1(b) with schematic diagrams illustrating the inputs, predictions, and feedback connections for each formulation, using different colors or line styles to distinguish ground-truth and predicted states. The differences between training and inference should also be made explicit. The equations could instead be presented in the main text, following Equations 1-4. This would make the four formulations and the focus of the paper easier to understand.

  3. Clarity and organization of experimental results. The organization of the results tables could be improved to make the purpose of each comparison clearer. Some tables combine multiple comparison factors, while closely related results are spread across different tables. For example, Table 2 mixes differences in architecture, training duration, and inference strategy, while assessing the effect of Poseidon's inference strategy requires comparing Tables 1 and 3. Similarly, the final comparisons involving ViT and FNO are separated between Table 6 and Appendix Table 14, and the different comparison settings in Tables 4 and 5 could be made more explicit. I suggest reorganizing or consolidating some of the tables around clearly defined experimental questions, making it clear which factors vary and which are held fixed. This would make the results easier to follow and interpret.

  4. Inconsistency in source-domain results. The relative L1 and L2 errors for the 80-epoch ViT under local-history AR on CE-Gauss appear to be swapped between Tables 2 and 17 and Tables 10 and 11. Specifically, Tables 2 and 17 report 0.2289 (L1) and 0.1878 (L2), whereas Tables 10 and 11 report 0.1878 (L1) and 0.2289 (L2). Could the authors verify which values are correct and ensure consistency across the tables?

4 comments
AuthorsOpenReview ↗

Response to Reviewer UjBu

(Part 1/2)

We thank you for your detailed and constructive review! Your suggestions helped us substantially improve the paper, and we have revised it following most of them. As you suggested, the paper now starts from the observation that the same pretrained Poseidon checkpoint performs substantially differently under AR rollout and direct prediction, uses it to motivate the controlled comparisons of training and inference formulations, and compares with Poseidon under both inference modes. We have also revised the terminology, Figure 1(b), the organization of the tables, and the CE-Gauss values, as detailed at the end of this response.

Poseidon's inference strategies and Walrus (W1, Q1, Q4). We would like to add one clarification on Appendix D.6.2 of the Poseidon paper. In addition to selecting the better strategy for each task, it shows that after finetuning, neither AR rollout nor direct prediction is consistently better across downstream tasks. We also checked the released Poseidon codebase and found that finetuning is likewise performed with all2all training. This is why we initially evaluated Poseidon with AR rollout in Table 1, which also matches the inference procedure of our ViT. In contrast, our evaluation of the pretrained models themselves, without finetuning, shows a clear and consistent gap between the two inference mode. This difference from the finetuned models is the core motivation of our paper: the training and inference formulations can matter for the zero-shot OOD generalization of pretrained models, even when they make little difference after finetuning. This is also the main difference from Walrus, which we have added to the related work. Like Poseidon, Walrus evaluates OOD generalization after finetuning on the downstream tasks. Strictly speaking, after finetuning, the downstream task is no longer unseen by the model, whereas we evaluate the pretrained models on unseen tasks zero-shot. In addition, Walrus is trained with one-step prediction from ground-truth histories and uses patch jittering to improve rollout stability, but it does not directly compare teacher-forced and rollout-based training.

Scope of the OOD evaluation (W2). Our OOD setting follows from our research question, which focuses on the zero-shot generalization of the pretrained models themselves. The five OOD tasks keep the governing equation families, physical fields, and computational domain of pretraining, so all models can be evaluated without modification. Shifts to other governing equations, physical fields, or geometries would require modifying the input and output embeddings and finetuning the models, which would no longer evaluate the pretrained models themselves. We have clarified this in Section 4 and Appendix C. Extending the study to such settings is an interesting direction for future work.

Training duration (Q2). There was no specific criterion: in the original submission, we used 80 epochs for the controlled comparisons simply because the source-domain accuracy was still improving at that point. The revised paper reports all formulations at both 40 and 80 epochs (Table 3), and the conclusions are consistent across the two training lengths. For the comparison with Poseidon in Table 1, we use 40 epochs to match the released Poseidon-B and Poseidon-T checkpoints.

AuthorsOpenReview ↗

Response to Reviewer UjBu

(Part 2/2)

Interpretation of the comparison with Poseidon (Q3). First of all, you are right that all models have errors above 1 on CE-RM. CE-RM is no longer marked in the revised tables. Secondly, the errors on the CE systems tend to be higher than those on the NS systems. In our early exploration, we also tried training separate foundation models for CE and NS, and the CE-only model did not converge well, which suggests that the CE tasks may be more challenging. On the three NS systems, the OOD performance is generally better.

In the revision, we also evaluated Poseidon-B and Poseidon-T under direct prediction, which we had not done in the original submission. To be honest, we had not noticed before how strong Poseidon-B is under direct prediction: it achieves a particularly low error on NS-BB, lower than our ViT, and with 158M parameters, it achieves lower error than the 629M Poseidon-L on four of the five OOD systems, which likely suggests that zero-shot OOD performance does not simply improve with model size. Meanwhile, our ViT achieves the lowest NS-VS error among all models (Table 1).

For NS-SL, we do not yet have a fully satisfactory explanation of why our ViT performs worse than Poseidon-L. Figure 9 suggests that our ViT does not capture well the small vortices emerging in the later stage of the shear-layer evolution, whereas Poseidon-L partially captures them. One possibility is that Poseidon's direct prediction, which maps the initial condition to each future time in a single step, helps preserve such small-scale structures. However, Figure 3 shows that the error gap between our ViT and Poseidon-L on NS-SL is already present in the first prediction steps, so error accumulation during AR rollout alone does not seem to explain it, and the fine-scale reconstruction of our patch-based ViT may also play a role.

Comments and suggestions.

  1. Terminology. We changed the text to use “trajectory prediction” for the general task and “autoregressive rollout” only for recursive prediction. As noted in the general response, the title is unchanged for now, as we are confirming whether it can be changed during the discussion period.
  2. Figure 1(b). We redrew Figure 1(b) to show the inputs, predictions, and feedback connections of each formulation, with separate training and inference columns and distinct ground-truth and predicted states. The equations are now in the main text (Eq. 5).
  3. Organization of results. Table 1 merges the original Tables 1, 3, and 6, and Table 3 consolidates the formulation comparisons across model sizes and training lengths together with FNO (original Tables 3, 4, 5, and 14). The captions state which factors vary.
  4. CE-Gauss values. Thank you for catching this. The correct values are 0.1878 (relative L1L_1) and 0.2289 (relative L2L_2). We corrected the tables and verified all reported values against our evaluation outputs.

Additional experiments and analyses. Beyond your suggestions, we also strengthened the empirical study:

  1. We added a 92M ViT, so the formulation comparisons now cover three model sizes (10M, 25M, and 92M).
  2. We report all formulations and model sizes, including FNO, at both 40 and 80 epochs, under both relative L1L_1 and L2L_2 (Tables 3 and 11) and on the source domain (Tables 7 and 8).
  3. We added a per-step error analysis (Figure 3 and "Where Do the Gains of Teacher Forcing Arise?" in Section 5), with the per-step errors of all formulations on all five OOD systems in Figure 8 (Appendix G).

We hope these revisions address your concerns, and we thank you again for your constructive suggestions.

I thank the authors for their detailed response and revisions. I believe they have largely addressed my concerns, particularly regarding the framing of the paper, the comparison with Poseidon, and the presentation results. I think the revisions have substantially improved the paper, and I am happy to upgrade my rating to weak accept.

We thank you very much for upgrading the rating and for being positive about our study! Thanks again for the constructive suggestions and comments.

Reviewer_A7tTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The article focuses on out-of-distribution rollout prediction on models that have been trained on several fluid dynamics datasets. In particular, it tests how the pretraining objective affects this prediction (local-history AR, full-history AR with or without teacher forcing, and full-trajectory prediction). The paper provides measurements to prove that a smaller ViT model can outperform a larger Poseidon-L baseline on several datasets, and that teacher forcing with full history conditioning is generally better than AR rollout training.

Critical strengths
  • The paper is easy to follow.

  • The paper addresses a practical question the field has not settled about several training design choices: predicting the full trajectory at once or training on single-step prediction and rolling out auto-regressively at inference; conditioning on the last state only or on the full history; and, when training through an autoregressive rollout, feeding the model its own predictions or the ground truth.

  • The experiments are cheap to reproduce.

Critical weaknesses
  • I find the comparison against Poseidon unfair. The authors state that they beat Poseidon in several datasets with fewer parameters according to Table 1. But for this comparison they took a checkpoint of Poseidon that was pretrained using direct prediction of the entire sequence, and they evaluated it using step by step rollout, which Poseidon was not trained for. On table 2 we actually see that Poseidon beats their model on most datasets when used correctly.

  • The paper misses some ablations on how the ViT components they chose are useful.

  • For an empirical study, the setup is too limited. The paper does not provide any architectural novelty, which is not a weakness by itself since it investigates a fundamental training question. However, the setup is very narrow with 2D datasets coming from just 2 equations, at low, fixed resolutions. Given the variability on resolution, number of channels, number of datasets, and even number of dimensions we see in the state of the art models (like PDE-Transformer (Holzschuh et al., 2025), Walrus (McCabe et al., 2026), MORPH (Rautela et al., 2025), DPOT (Hao et al., 2024)…), this study should include much more variability in these terms in order to provide insights on the current state of the art. Furthermore, it should also provide some comparisons on these state of the art models.

Benjamin Holzschuh et al. PDE-transformer: Efficient and versatile transformers for physics simulations. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 23562–23602. PMLR, 2025.

Michael McCabe et al. Walrus: A cross-domain foundation model for continuum dynamics. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2026.

M. S. Rautela et al. MORPH: PDE foundation models with arbitrary data modality, 2025. URL https://arxiv.org/abs/2509.21670.

Zhongkai Hao et al. Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training. In Proceedings of the 41st International Conference on Machine Learning, 2024.

Questions for authors

Some questions that could raise my score are:

  • Can you expand the backbones and datasets? Is it even possible to provide heterogeneous datasets with different resolutions and numbers of channels? Providing an architecture that can handle such variability.

  • With a tubelet size of 4 frames on the rollout horizon, teacher forcing outperforms the other strategies. Is it the same situation with a tubelet size of 1?

  • Do you have any ablation on the custom ViT components?

  • Why do you have different batch sizes on table 7? I guess you fixed the computational cost or some similar criteria but I don't find the details.

2 comments
AuthorsOpenReview ↗

Response to Reviewer A7tT

We thank you for your review and for recognizing that our paper addresses a practical question the field has not settled! We apologize that our presentation may not have been clear enough, and we have reorganized the paper. To address your concern about fairness, we have combined the original Tables 1, 3, and 6 into the new Table 1, so that the fair comparison is clear immediately. We would also like to restate the scope of this paper more clearly: we study how training and inference formulations affect the zero-shot OOD generalization of pretrained PDE models when only a single initial condition is given. To isolate this effect, we fix the backbone and vary only the formulation. We do not aim to propose a new architecture or to claim state-of-the-art performance.

Fairness of the comparison with Poseidon (W1). In the original submission, Table 1 only served to motivate the study, and the comparisons with Poseidon under direct prediction were reported in Tables 3 and 6. The new Table 1 reports all three released Poseidon checkpoints under both AR rollout and direct prediction, side by side with our ViT. Table 2 of the original submission reports source-domain (in-distribution) accuracy, where Poseidon under direct prediction is indeed more accurate than our ViT, as also stated in our original abstract. Our study concerns OOD generalization, where this relation no longer holds on some systems (Table 1).

Choice of comparison models (W3). We selected the comparison models based on our research question, zero-shot trajectory prediction from a single initial condition. Several recent models address a different setting: MPP and DPOT condition on a history of multiple input frames (16 and 10 frames, respectively), and Walrus likewise predicts from a multi-frame context, which is similar to the reason we did not adopt VICON (Appendix A). PDE-Transformer and MORPH propose architectures for heterogeneous resolutions, channels, and dimensions, and are mainly evaluated after downstream finetuning. Poseidon is a representative pretrained model that predicts a trajectory from a single initial condition, so we use it to motivate the study and conduct the controlled comparisons on our ViT. Following your suggestion, we have added these works to the related work section.

Architecture, heterogeneous data, and component ablations (W2, Q1, Q3). Handling heterogeneous resolutions, channels, and dimensions is an architectural research direction independent of our research question. Recent designs such as per-channel tokenization (PDE-Transformer) and modality-agnostic data formats (MORPH) are compatible with a ViT-style backbone like ours, but our focus is the training and inference formulation, which we compare with a fixed architecture. Our backbone is based on ViViT, and the convolutional stem and refinement are standard components (Wu et al., 2021; Ranftl et al., 2021). Because all formulations are compared on the same backbone, these component choices do not confound our conclusions. Moreover, FNO, which has no tubelet patchification, stem, or refinement, shows the same teacher-forcing benefit on NS-BB and NS-VS. In addition, the revised paper adds a 92M ViT, so three model sizes and two training lengths now consistently support the favorable OOD generalization of teacher forcing on NS-BB, NS-VS, and CE-RKH (Table 3).

Tubelet size (Q2). Regarding the tubelet size of our ViT, the main pattern we observe under the current setting is that the per-step errors in the new Figure 8 (Appendix G) show a common trend within each 4-step block. In our early exploration, we also tried tubelet sizes of Δk=1\Delta k = 1 and 22: the main difference was that this within-block pattern disappeared, while the overall prediction trend and accuracy over the 20 steps did not change substantially. We therefore chose Δk=4\Delta k = 4 for its lower computational cost.

Batch sizes (Q4). You are right: batch sizes were chosen to fit memory for each formulation and model size. Teacher forcing and full-history AR, the key comparison, use the same effective batch size at each model size (192, 125, and 96 for the 10M, 25M, and 92M ViTs). We added this to the caption of Table 4 (Table 7 in the original submission).

We hope these clarifications address your concerns.

Thanks for the answer and the revised paper. The new Table 1 makes the Poseidon comparison much fairer, and the batch size explanation is clear. I also find the observation about Poseidon's sensitivity to the inference mode interesting, and the teacher-forcing gain on NS-BB is consistent across both architectures.

However, Table 1 also shows that Poseidon-B under direct prediction outperforms the ViT on 3 of the 5 OOD systems (CE-RKH, NS-BB, NS-SL), while the ViT is better on NS-VS and all models fail on CE-RM. This is not reflected in the abstract, which still compares only with Poseidon-L.

Some of my other points were not really addressed. No ablations were added, so the claims about the ViT itself are still not supported. The answer about the tubelet size is vague, with no numbers. And the experimental setup is still narrow: one dataset collection, few OOD systems and single runs. Since the paper does not propose a new method, its value depends on the strength of the evidence.

I will therefore keep my score.

Reviewer_HLNHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates how pretraining design affects zero-shot out-of-distribution (OOD) rollout prediction for neural PDE solvers. The authors develop a lightweight spatiotemporal ViT with approximately 25M parameters and evaluate its ability to predict unseen dynamics from a single initial condition. The model achieves comparable prediction errors with the 629M-parameter Poseidon-L on OOD benchmarks. The paper also systematically compares four training and prediction formulations: local-history autoregressive rollout, full-history autoregressive rollout, teacher forcing, and non-autoregressive full-trajectory prediction. The experiments suggest that training formulation substantially influences OOD performance. Besides, teacher forcing can outperform rollout training despite the training–inference mismatch in some cases. Additional experiments with smaller ViT and FNO models show that teacher forcing benefits are not limited to the proposed architecture.

Overall, the paper provides an empirical investigation of pretraining choices for zero-shot PDE prediction, highlighting that model scale and source-domain accuracy alone do not determine OOD generalization performance.

Critical strengths
  1. The proposed 25M-parameter ViT achieves competitive zero-shot OOD performance against the much larger Poseidon-L model. In particular, the improvements on NS-BB and NS-VS are substantial and remain evident under both relative L1 and L2 errors. These results somehow could support the idea that larger pretrained models do not necessarily outperform smaller alternatives under distribution shifts.

  2. The paper systematically compares four training and prediction formulations using the same 25M ViT backbone. These experiments provide direct evidence that training formulation and temporal conditioning can considerably affect OOD prediction performance, independently of changes to the backbone architecture.

  3. The comparison between full-history autoregressive training and teacher forcing is particularly interesting, as it challenges a training consensus and examines whether matching the training and inference input distributions is necessarily beneficial for zero-shot PDE prediction.

Critical weaknesses
  1. A major concern is the lack of statistical evaluation across independent training runs. Appendix B explicitly states that each configuration is pretrained only once. The paper does not report the mean and standard deviation across multiple random seeds. This is particularly concerning because several improvements attributed to teacher forcing are relatively small. Since the paper's primary contribution is empirical, stronger evidence of reproducibility is particularly important.

  2. The advantages of teacher forcing vary across tasks and evaluation metrics. For example, on NS-BB, teacher forcing improves L1 error but worsens L2 error by approximately 14.7%. For FNO on CE-RKH, the L2 error increases by approximately 128%. Performance on different systems is also not consistent. These inconsistencies weaken the claim that teacher forcing is a generally preferable pretraining strategy.

  3. Although the 25M ViT outperforms the much larger Poseidon-L on some OOD tasks, this does not establish that pretraining design is more important than model size, as the models differ in architecture, training formulation, and prediction strategy. Moreover, Table 1 shows that larger Poseidon variants achieve lower errors on 3 of 5 of the five OOD tasks, so the authors cannot conclude that model scale has little impact on zero-shot OOD performance. More controlled experiments are needed to clarify the relative contributions of model scale and pretraining design.

  4. The proposed architecture mainly adapts existing ViT components, while the training strategies are well-established. The contribution is primarily empirical, but the paper provides limited insights into when and why different pretraining strategies should be preferred, reducing the broader significance of its findings.

Questions for authors
  1. Could the authors provide results across multiple random seeds for the key teacher-forcing versus autoregressive comparisons to assess the statistical reliability of the reported improvements?

  2. Can the authors provide further insights into why teacher forcing substantially improves performance on some OOD systems but degrades performance on others? Are there identifiable properties of the underlying dynamics that explain these differences?

  3. Could the authors clarify the intended conclusions regarding model scale versus pretraining design? Are there additional controlled comparisons that could help disentangle their respective effects on OOD generalization?

2 comments
AuthorsOpenReview ↗

Response to Reviewer HLNH

We thank you for your review and for raising the statistical robustness concern! We apologize that our presentation may not have been clear enough, and we have reorganized the paper. We want to clarify that we never claimed that scale is unimportant, in either the original or the revised submission. This paper aims to study how the training formulation affects the OOD generalization of pretrained models.

Statistical robustness (W1, W2, Q1). Regarding statistical robustness, we added the new Table 3, which compares the four training formulations across three ViT sizes (10M, 25M, and a newly added 92M model), each pretrained independently for 40 and 80 epochs, together with FNO at both training lengths, and we report all comparisons under both relative L1 and L2 (Tables 3 and 11). Out of these six independently pretrained ViT settings, the table below reports the number of settings in which teacher forcing (TF) achieves the lowest error (1st) and the lowest or second-lowest error (top 2) among the four formulations:

OOD systemL1: 1stL1: top 2L2: 1stL2: top 2
NS-BB6/66/65/66/6
NS-VS6/66/64/66/6
CE-RKH3/66/62/66/6
NS-SL3/63/62/62/6
CE-RM (all errors > 1)1/61/61/61/6

These results are consistent with the claims in our paper. On NS-BB and NS-VS, teacher forcing(TF) consistently improves OOD prediction: it ranks first or second in all six settings under both metrics, and it also reduces the NS-BB and NS-VS errors of FNO at both training lengths. On CE-RKH, TF also ranks first or second in all six settings, although its advantage is less pronounced. On NS-SL, all formulations perform similarly (Table 3), and on CE-RM, all models have errors above 1 with small differences among formulations. The two examples raised in the review are exceptions rather than the overall trend: the NS-BB L2 increase occurs only for the 25M ViT at 80 epochs, where TF still ranks second, and the CE-RKH increase occurs only for FNO.

Why teacher forcing helps (W4, Q2). We added a per-step error analysis (Figure 3 and "Where Do the Gains of Teacher Forcing Arise?" in Section 5). The analysis supports our main conclusion that, unlike prior PDE studies that often favor AR rollout training, teacher forcing can benefit the OOD generalization of pretrained models on some tasks. The advantage of TF arises mainly in the early stage of the trajectory, where its errors are substantially lower than those of AR rollout training, and Poseidon-L's direct prediction shows the same pattern relative to its AR rollout. Within the 20-step horizon, this early advantage largely persists, and on NS-VS the errors of both our ViT and FNO saturate after roughly 15 steps, so the early-stage advantage determines the time-averaged error. This is consistent with the objective of each formulation: TF learns the mapping from each ground-truth state to the next, whereas AR rollout training targets long-horizon prediction, so when errors tend to saturate, larger early errors can outweigh the benefit of a more stable long-term prediction.

Model scale (W3, Q3). As clarified above, this paper does not study the relative importance of model scale and pretraining design. Our controlled comparisons fix the architecture and model size and vary only the training formulation. We only note that, within our experiments, larger models do not show a consistent OOD advantage: increasing the ViT from 25M to 92M does not consistently reduce the OOD error (Table 3). Also, under direct prediction, the lowest Poseidon errors on CE-RKH and NS-SL come from the 21M Poseidon-T and on NS-BB from the 158M Poseidon-B, rather than the 629M Poseidon-L (Table 1).

We hope the revised paper and these clarifications address your concerns.

Thank you for the additional experiments and clarifications. While the rebuttal clarifies some concerns, there are still concerns left:

  1. Statistical robustness remains unresolved. Varying model sizes and training epochs does not replace multiple random seeds under identical settings, especially when numbers differ slightly.

  2. Teacher forcing is not consistently beneficial. For a PDE foundation model, I consider consistent generalization across diverse physical systems particularly important, but the reported results and supplemented experiments do not demonstrate such consistency. They also do not explain why the benefits vary substantially across PDE systems. This largely limits the contribution and informativeness that bring to the community.

I therefore maintain my original score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their careful reviews. We carefully read all review comments, and we think that our presentation may not have been clear enough, making the fairness of our comparison with Poseidon and the main contribution of this paper easy to misread. We apologize for this. It was not intentional. Table 1 was meant only to motivate the study, its caption points to the stronger direct-prediction results reported later (Table 6 in the original submission), and none of our claims rely on it. We have reorganized the paper accordingly.

In the revision, we combine the original Tables 1, 3, and 6, which were previously spread across the paper, so that the fair comparison is clear immediately from the start. Also, the abstract and introduction now explicitly point out our main question: how training and inference formulations affect zero-shot OOD generalization.

To make the revisions easy to track, we have marked the main changes in blue in the revised PDF. For new or redrawn figures and tables, the figure or table label is marked in blue. We will change them back to black in a later revision. Minor corrections and wording adjustments, such as typo fixes and the consistent use of "trajectory prediction", are not marked, so that the blue text highlights the substantive changes rather than appearing throughout the paper. The main revisions are summarized below.

  • Abstract and introduction. The paper is now framed around how training and inference formulations affect the zero-shot OOD generalization of pretrained PDE models. The motivating observation is that the same pretrained Poseidon checkpoint performs substantially differently under AR rollout and direct prediction (AR rollout increases the OOD error by up to 4.8× for Poseidon-L and 55× for Poseidon-B). The text now uses “trajectory prediction” for the general task and “AR rollout” only for recursive prediction. The only exception is the title. The author guidelines are unclear on whether the title can be changed during the discussion period, so we have kept the original title for now while we confirm this. If permitted, we will change it to “Pretraining Design Drives Gains in Zero-Shot PDE Trajectory Prediction”.
  • Methods. We redrew Figure 1(b) to show the inputs, predictions, and feedback of each formulation, with separate training and inference columns and distinct ground-truth and predicted states, and moved the equations of the four formulations into Eq. 5.
  • Results and discussion. Section 5 is reorganized around explicit questions:
    • Table 1 merges the original Tables 1, 3, and 6.
    • Figure 2 is redrawn to show the pretraining convergence of teacher forcing and local-history AR at 40 and 80 epochs.
    • A new paragraph discusses training–inference mismatch.
    • The new Table 3 consolidates the formulation comparisons (original Tables 3, 4, 5, and 14) across three ViT sizes (10M, 25M, and a newly added 92M model) and two training lengths, together with FNO as a non-Transformer backbone, totaling 28 settings, each pretrained independently.
    • The new Figure 3 and the accompanying per-step error analysis examine where the gains of teacher forcing arise.
  • Related work. We added Walrus, PDE-Transformer, and MORPH.
  • Appendices. All comparisons are reported under both relative L1L_1 and L2L_2 (Appendix J), source-domain results are provided for every model (Appendix F), per-step errors for all formulations are given in Appendix G, the implementation details now include the batch-size criterion (Appendix B), and the inconsistent CE-Gauss value has been corrected.

We respond to each reviewer individually below.

Dear Reviewers,

We are writing to follow up on our responses. We thank all reviewers for their time and effort, and we believe our revised paper has addressed most of the concerns, especially those regarding the clarity of the paper and the fairness and robustness of the experiments. As the discussion period will close soon, we would appreciate it if you could let us know whether you have any further concerns or suggestions.

Thanks and all the best, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.