acceptodds
Under review as a conference paper at ICLR 2027

Tadpole-2D: From PDE Foundation Model to Global Weather Forecasting

Abstract

The development of AI-based weather forecasting models generally demands significant computational resources and extensive data storage. Recent advances in foundation models for partial differential equations (PDEs) have enabled efficient synthetic datasets and storage-free training pipelines, which may address these challenges. In this work, we present Tadpole-2D, a versatile foundation model for two-dimensional (2D) PDEs. It is pretrained to reconstruct 2D slices from diverse three-dimensional PDE data generated online, and its downstream performance is verified on a challenging PDE dynamics learning task. We extend the model to global weather prediction with adaptations enabling it to process atmospheric data on the Hierarchical Equal Area isoLatitude Pixelization (HEALPix) grid. We choose HEALPix for its superior numerical properties over equiangular grids used in most flagship weather systems. In short-range weather forecasting, the resulting model outperforms state-of-the-art PDE foundation models of comparable size. To our knowledge, ours is the first work in which a PDE foundation model is demonstrated to transfer successfully to global weather forecasting.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
pde, fluid, weather, forecast, healpix, foundation, model, surrogate, online, spherical

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hFXvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Tadpole-2D, a PDE foundation model pretrained on online synthetic simulations and adapted to global weather forecasting on the HEALPix grid. The direction is interesting, and the results demonstrate the potential of transferring PDE foundation models to real-world weather prediction.

Critical strengths

The paper establishes a meaningful connection between PDE foundation models and weather forecasting. The online pretraining pipeline is resource-conscious, and the HEALPix adaptation is carefully implemented.

Critical weaknesses

The main concern is that the contribution of PDE pretraining is not sufficiently isolated. The pretrained model uses a frozen backbone with LoRA, whereas the from-scratch model trains all parameters under the same optimization budget. Thus, the reported improvements may partly reflect different adaptation and optimization regimes rather than the benefit of PDE pretraining alone.

The comparison with other PDE foundation models is also confounded. Tadpole-2D uses HEALPix-aware cross-face padding, positional embeddings, and a weather-specific weighted loss, while the baselines process faces independently and use different objectives. Their visible boundary artifacts may therefore result from unequal spherical interfaces rather than weaker pretrained representations.

Overall, this is a promising proof of concept, but stronger matched controls are needed to establish that synthetic PDE pretraining is the source of the reported gains.

Questions for authors

How much of Tadpole-2D’s improvement comes from PDE pretraining rather than its HEALPix-aware padding, positional embeddings, weather-specific loss, and LoRA fine-tuning? Could the authors provide matched ablations that isolate these factors and equip the baseline PDE foundation models with comparable cross-face processing?

1 comment

Dear reviewer,

Thank you for your valuable feedback and for recognizing the link between PDE foundation models and weather forecasting. Below, we address the weaknesses (W) and the questions (Q). The new results are in our anonymous repository in rebuttal_results.md and will be added in the revised manuscript.

W1: The pretrained model uses a frozen backbone with LoRA, whereas the from-scratch model trains all parameters.

Q1: How much of the improvement comes from PDE pretraining rather than LoRA fine-tuning?

We use LoRA because Tadpole is pretrained as an autoencoder and never sees temporal dynamics. Liu et al. (2026, Sec. 5.3 and Appendix A.4) found that LoRA on the frozen backbone keeps the pretrained representation and lets the latent subnetwork learn the dynamics. To separate LoRA from the improvements, we added three controls. In all of them, both models use all HEALPix adaptations.

  1. Both models train all parameters (Additional Results 1.1): The pretrained model has 2 to 11% lower RMSE and a higher ACC for all five variables and lead times up to 24 hours. The only exception is t850 at 24 hours, where the RMSEs differ by 0.5%.
  2. Both models use our fine-tuning method (Additional Results 1.2): a frozen backbone with rank-32 LoRA, plus the latent subnetwork and skip connections (6.2M trainable parameters in both). In the from-scratch model, the frozen backbone keeps its random weights. The pretrained model has 1 to 7% lower RMSE, except t850 at 24 hours (3% higher).
  3. Pretrained model, two fine-tuning methods (Additional Results 1.3): compared to full parameter fine-tuning, the LoRA method achieved 15 to 40% lower RMSE.

Q2: Matched ablations of the HEALPix-aware padding, positional embeddings and weather-specific loss.

We fine-tune three more variants of our model - each HEALPix adaptation is removed one at a time to evaluate its contribution (Additional Results 2).

  • HEALPix padding has the largest effect: without it, the RMSE of every variable is 4% to 30% higher at 6 hours and 9% to 60% higher at 24 hours.
  • Loss weighting, on the other hand, shifts accuracy between slower vs. faster-changing variables; without it, q700, u850 and v850 are 2 to 9% better, while z500 and t850 are 6 to 20% worse. This is expected based on the six-hour time-difference statistics discussed in Appendix A.7.2, from which we derive the loss weights.
  • The HEALPix coordinate embedding changes the RMSE by less than 1% for every variable and lead time. We assessment is that global coordinates matter more at longer lead times, when interactions over large distances become important.

W2 and Q3: The baselines process faces independently and use different objectives, so their boundary artifacts may come from unequal spherical interfaces rather than weaker pretrained representations. Could the authors equip them with comparable cross-face processing?

We trained Poseidon-T, MORPH-S and PDEformer-2-S again with the same HEALPix adaptations as Tadpole-2D: HEALPix padding in every convolution, the loss weighting and the HEALPix coordinate embedding (for PDEformer-2-S, the latitude and the distance from the face corner as extra input fields). Additional Results 4.1 shows the 24-hour forecast of Fig. 5 for all five variables, and a close-up of q700 at the poles, where four HEALPix faces meet.

  • MORPH-S still shows square artifacts from its 8×8 patches, for example in z500 and t850.
  • PDEformer-2-S still shows seams along the face boundaries, for example in q700 and u850, and where the faces meet at the poles.
  • Poseidon-T, like Tadpole-2D, shows no clear seams, but it has an irregular small-scale texture in q700 near the poles.

The artifacts of MORPH-S and PDEformer-2-S therefore remain even when all models use the same cross-face padding.

Reviewer_jSJLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Tadpole-2D is a VAE pretrained to reconstruct 2D slices sampled online from seven families of 3D PDEs. A latent dynamics subnetwork trained with LoRA and a HEALPix interface adapt it to six-hourly ERA5 forecasting of five variables; the model is also fine-tuned on transonic cylinder flow. The paper reports gains over the same architecture trained from scratch and over three PDE foundation models of comparable size, and compares q700 ACC with GraphCast, Pangu-Weather and Aurora.

Critical strengths
  1. The paper asks a worthwhile question: can synthetic data from generic PDEs help a real weather task? Online data generation avoids storing a large simulation corpus, and the HEALPix adaptation is well suited to global fields.
  2. The evaluation covers five ERA5 variables, several PDE foundation models, different weather-data budgets and an additional transonic-flow task, and the implementation details are documented clearly (Appendix A.4).
Critical weaknesses
  1. The central claim is that PDE pretraining improves weather forecasting, but pretraining only reconstructs spatial slices and learns no temporal evolution (Sec. 3.1). The comparison in Sec. 4.1 also changes initialization and adaptation together: the pretrained model freezes the backbone and trains rank-32 LoRA (6.16M trainable parameters), whereas the scratch model trains all 23.63M weights end to end (Appendix A.4.3). The results therefore show that this recipe works, but cannot tell whether the gain comes from PDE representations, from generic spatial reconstruction, or from the fine-tuning setup.
  2. Evidence for the global-forecasting claim is limited. The comparison with GraphCast, Pangu-Weather and Aurora reports q700 ACC only, and Tadpole-2D falls behind after the six-hour lead (Fig. 6). The five-variable evaluation is against PDE foundation models only, on a single test year (2022) and up to 48 hours (Sec. 4.1, Appendix A.4.4).
Questions for authors
  1. (Clarification) How does a randomly initialized model perform with the same rank-32 LoRA, trainable-parameter budget and schedule? A non-PDE spatial-reconstruction pretraining control would also show whether the transfer is specific to PDE data.
  2. (Clarification) Can you compare with specialist weather models on all five variables and several lead times using RMSE and ACC? Do the gains hold on another test year and for longer rollouts?
1 comment

Dear reviewer,

Thank you for the constructive review and for recognizing the motivation and scope of our work. We answer each weakness (W) and question (Q) below. The new results are in our anonymous repository: rebuttal_results.md and will be added to the revised manuscript.

W1.1: Pretraining only reconstructs spatial slices and learns no temporal evolution.

Whether to learn temporal evolution during pretraining or only downstream is an important design question. As discussed by Liu et al. (2026), reconstruction is a more transferable pretraining target than time stepping: it only requires learning the low-dimensional manifold of admissible solutions. Conversely, with dynamics pretraining, the same state can evolve into different future states depending on the PDE, parameters, and boundary conditions, so the network must also identify the system e.g. with parameter embeddings. We therefore learn the dynamics downstream, for the target system.

A reconstruction target also lets the encoder and decoder process each physical field as an independent scalar field (Sec. 3.1). This decoupling allows the same weights to be fine-tuned on systems with different numbers of fields. During fine-tuning (Sec. 3.2.1), the latent subnetwork couples the fields and learns the dynamics, while the frozen VAE keeps the pretrained spatial representation.

W1.2 / Q1: Initialization and adaptation change together. How does a randomly initialized model perform with the same rank-32 LoRA, parameter budget and schedule?

We added this control in Additional Results 1.2. With the same LoRA, the pretrained initialization lowers the 6-hour RMSE by 4-6% on all variables versus the randomly initialized model.

W1.3: Does the gain come from PDE representations, generic spatial reconstruction, or the fine-tuning setup?

The PDE representation, spatial reconstruction, and the fine-tuning setup are all important components of our framework:

  • PDE data: We chose PDEs because it is the closest simulated data to weather. ERA5 combines observations with forecasts of the ECMWF IFS, which solves the hydrostatic primitive equations with parametrized physics (IFS Documentation CY41R2, Part III). Each of our five variables follows one of these equations: momentum (u850, v850), thermodynamic (t850), moisture (q700) and hydrostatic (z500). Secondly, PDE-pretrained models have also shown transfer beyond their pretraining equations (Herde et al., 2024; Liu et al., 2026.
  • Spatial reconstruction: as we explain for W1.1, it lets the model learn PDE representations independently of their dynamics and channel count.
  • Fine-tuning setup: under our training schedule, training LoRA adapters and the latent subnetwork on the frozen backbone is much more effective than training all weights. Additional Results 1.3 shows that LoRA (rank 64) lowers RMSE by 15-40% on all five variables at 6-24 hours, e.g., by 37% for z500 at 24 hours.

W2: The comparison with GraphCast, Pangu-Weather and Aurora: Tadpole2D falls behind after six hours (Fig. 6).

We agree and do not claim to outperform these models. With Fig. 6 we intend to present context: it shows where a PDE foundation model currently stands relative to specialist weather models. These models forecast 69 to 227 variables on 13 to 37 pressure levels and are trained with multi-step objectives and far more compute. Aurora (1.3B parameters) and Pangu-Weather (about 256M) are also 9 to 46 times larger. Tadpole-2D has 28M parameters, forecasts five variables and is fine-tuned on one-step targets on a single GPU. Our contribution is a PDE foundation model and a framework for extending it to global forecasting on HEALPix. We therefore compare it with the same architecture without pretraining and with PDE foundation models of similar size.

W2 & Q2: Fig. 6: All five variables, several lead times, RMSE and ACC? Another test year and longer rollouts?

All variables and both metrics: we have added them in Additional Results 3. All variables follow the trend from the manuscript's Fig. 6: Tadpole-2D is closest to the specialist models at six hours, and the gap grows with lead time.

Another test year: Additional Results 3 also reports 2020, which lies outside our training (1979-2018) and validation (2019) years. Compared with 2022, Tadpole-2D's scores relative to GraphCast and Pangu-Weather change by at most 3%.

Longer rollouts: As we discuss in the paper, enabling long rollouts is not the main focus of our work; nevertheless, we outline strategies for longer rollouts in the outlook as a possible extension of the work. We target short-range forecasts up to 24 hours (Sec. 1), and accordingly fine-tune on only one-step targets and evaluated on short-term forecasting. Fig. 4 in the manuscript and Additional Results 3 show a longer rollout of up to 48 hours.

Reviewer_3W8QOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Tadpole-2D, a two-dimensional PDE foundation model pretrained using spatial reconstruction of synthetic PDE solutions generated online. The method extends the previous Tadpole framework to two dimensions and adapts it to global weather forecasting through a HEALPix interface, including geometry-aware convolutional padding and positional embeddings. The model is evaluated on short-range weather forecasting using ERA5 and on transonic cylinder-flow dynamics.

The experimental results are encouraging. The pretrained model substantially outperforms its from-scratch counterpart and achieves better forecasting accuracy than Poseidon-T, MORPH-S, and PDEformer-2-S. The transonic flow experiments also demonstrate promising generalization to dynamics outside the pretraining distribution.

Overall, the paper provides an interesting demonstration of transfer learning from synthetic PDE data to atmospheric forecasting. However, several experimental confounders and missing ablations make it difficult to determine which components are responsible for the reported improvements.

Critical strengths
  1. Interesting application and transfer-learning results.

Demonstrating that a PDE foundation model pretrained without atmospheric data can be successfully adapted to global weather forecasting is an interesting contribution. The empirical improvements over training from scratch are substantial, particularly in the low-data regime.

  1. Strong empirical performance across two downstream tasks.

The model performs consistently well relative to the selected PDE foundation model baselines in weather forecasting. The additional transonic cylinder-flow experiment provides useful evidence of versatility beyond atmospheric dynamics.

  1. Efficient pretraining pipeline and detailed experimental setup.

The online synthetic data generation pipeline eliminates the need to store a large pretraining dataset. The appendix provides extensive implementation details, including architectures, optimization settings, and evaluation protocols, which is valuable for reproducibility.

Critical weaknesses
  1. The weather forecasting comparisons do not adequately control for differences in spherical geometry handling.

Section 3.2.2 introduces HEALPix-aware convolutional padding, enabling spatial information to propagate across neighboring HEALPix faces. However, Appendix A.4.4 indicates that the baseline models process the twelve HEALPix faces independently.

Consequently, Tadpole-2D benefits from spatial communication across face boundaries, whereas the baselines do not have an equivalent mechanism. This is a potentially important advantage unrelated to the effectiveness of the pretrained PDE representations.

Furthermore, Table 7 shows that the compared models use different loss functions and optimization schedules. While using the original fine-tuning recipes is reasonable, these differences make it difficult to isolate the effects of the architectures and pretraining strategies.

A more controlled comparison, in which the baselines receive comparable geometric information and are evaluated under a common training objective, would substantially strengthen the empirical conclusions.

  1. The experiments do not fully establish that the improvements originate from learning transferable PDE representations.

The pretrained model uses a reconstruction-capable encoder–decoder, and the latent dynamics network is initialized with a residual structure approximating the identity transformation (Appendix A.3, Equation 5).

For short-range weather forecasting, where consecutive atmospheric states are strongly correlated, this initialization may provide a substantial advantage independently of whether the model has learned transferable PDE-specific features.

The comparison with training from scratch is informative, but the two settings also differ in their optimization strategies: the pretrained backbone is frozen and adapted using LoRA, whereas the from-scratch model is trained end-to-end.

Additional controls, such as a persistence baseline, matched full fine-tuning experiments, or reconstruction pretraining on non-PDE spatial fields, would help clarify the source of the improvements.

  1. Important architectural and pretraining ablations are missing.

The proposed method combines several existing techniques with adaptations for the downstream problem, including reconstruction pretraining, 3D simulation slicing, HEALPix-aware padding, and global positional embeddings.

However, the experiments do not isolate their individual contributions. In particular, the claimed advantages of the HEALPix interface would be more convincing with ablations of the geometry-aware padding and positional embeddings, as well as a comparison against an otherwise identical model operating on a latitude–longitude grid.

Similarly, the benefit of pretraining on slices from 3D PDE systems, as opposed to directly generated 2D fields, has not been established experimentally.

These ablations are especially important because the methodological novelty primarily concerns the combination and adaptation of existing approaches.

  1. Limited statistical evaluation and incomplete computational cost comparison.

The reported experiments use a single random seed, and no variability estimates across independent training runs are provided. In the transonic cylinder-flow experiment, the reported 1,150 evaluation rollouts are derived from only five test trajectories and contain overlapping temporal windows. Consequently, they should not be interpreted as independent test realizations.

The paper also emphasizes parameter-efficient fine-tuning and inexpensive pretraining. However, the number of trainable parameters does not fully characterize computational efficiency. The pretrained backbone still contributes to inference cost, and the substantial computational expenditure of pretraining should also be considered.

Reporting total model parameters, training and inference costs, and performance variation across seeds would make the efficiency and robustness claims more convincing.

Questions for authors

Questions for the Authors

  1. How much of the observed performance improvement over Poseidon-T, MORPH-S, and PDEformer-2-S remains when the baselines receive comparable cross-face information on the HEALPix grid? Can the authors provide a controlled comparison under a common loss function?

  2. To what extent are the improvements due to PDE-specific pretraining rather than reconstruction-based initialization? In particular, how does the model compare against persistence and alternative reconstruction-pretraining strategies?

  3. What are the individual contributions of HEALPix-aware padding, global positional embeddings, and the use of 3D simulation slices during pretraining?

  4. Can the authors report variability across independent seeds and provide a more complete comparison of total parameter counts, computational costs, and inference efficiency?

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 3W8Q (Part 1)

Thank you for the extensive feedback. We appreciate that you found the PDE to weather transfer interesting and the pre-training strategy efficient. We address the weaknesses (W) and questions (Q) as follows. The referred additional results are available in the accompanying anonymized GitHub repo at rebuttal_results.md and we will accordingly update the revised manuscript.

W1 & Q1: A more controlled comparison, in which the baselines receive comparable geometric information and are evaluated under a common training objective

We present our contribution as a combination of a model, a pre-training pipeline, and an adaptation framework for HEALPix weather prediction. We similarly treat each PDE foundation model baseline with the fine-tuning framework proposed in their respective works. Hence, the manuscript presents evaluations of the combined system. Nevertheless, applying the same HEALPix adaptations to different models is an interesting ablation. We have added the requested experiments under Additional Results 4. Here, all models receive the HEALPix adaptations and are trained on the same weighted MSE loss.

With the adaptations, Tadpole-2D still performs significantly better than MORPH-S and PDEformer-2-S (additional results 4.2), and when fine-tuned with LoRA, it is on par with Poseidon-T (additional results 4.3). Our online pre-training pipeline lets Tadpole achieve these results without the data and I/O requirements of large fixed pre-training datasets as for other PDE foundation models.

From visual inspection of the 24 h forecasts (Additional Results 4.1), the patch seams and face seams in the predicted fields of MORPH-S and PDEformer-2-S appear in spite of the cross-face processing. Poseidon-T adds patterns in the q700 field prediction near the poles; Tadpole-2D's q700 field remains relatively smoother where the ground truth is smooth.

W2.1: The pretrained model uses a reconstruction-capable encoder–decoder, and the latent dynamics network is initialized with a residual structure approximating the identity transformation (Appendix A.3, Equation 5). For short-range weather forecasting, where consecutive atmospheric states are strongly correlated, this initialization may provide a substantial advantage independently of whether the model has learned transferable PDE-specific features.

W2.3: Additional controls, such as a persistence baseline, matched full fine-tuning experiments, or reconstruction pretraining on non-PDE spatial fields, would help clarify the source of the improvements.

Q2: To what extent are the improvements due to PDE-specific pretraining rather than reconstruction-based initialization? In particular, how does the model compare against persistence and alternative reconstruction-pretraining strategies?

Thank you for this question. The table below compares Tadpole-2D with the persistence forecast (published by WeatherBench 2 for 2022). Each cell gives Tadpole-2D / persistence. Tadpole-2D outperforms persistence by a wide margin on every variable and lead time we checked, so its forecast skill is learned during fine-tuning and is not left over from a near-identity starting point.

However, training the backbone with alternative and non-PDE reconstruction-pre-training strategies is a longer experiment that is out of scope for the rebuttal. Drawing meaningful conclusions from this test would entail an extensive analysis and comparison of each dataset's spatial patterns and spectra. We will consider your feedback as an outlook for our reconstruction-based pipeline.

VariableRMSE 6 h ↓RMSE 24 h ↓RMSE 48 h ↓ACC 6 h ↑ACC 24 h ↑ACC 48 h ↑
z500 (m² s⁻²)51 / 235154 / 606296 / 8460.998 / 0.9580.982 / 0.7200.932 / 0.454
t850 (K)0.67 / 1.471.24 / 3.031.81 / 3.910.980 / 0.9090.931 / 0.6070.853 / 0.345
q700 (g kg⁻¹)0.42 / 0.860.86 / 1.611.22 / 1.920.965 / 0.8540.843 / 0.4850.673 / 0.269
u850 (m s⁻¹)1.24 / 2.762.28 / 5.503.39 / 6.680.976 / 0.8810.915 / 0.5260.809 / 0.300
v850 (m s⁻¹)1.29 / 3.242.34 / 6.323.42 / 7.310.971 / 0.8260.904 / 0.3370.792 / 0.114
AuthorsOpenReview ↗

Rebuttal to Reviewer 3W8Q (Part 2)

W2.2: The comparison with training from scratch…: the pretrained backbone is frozen and adapted using LoRA, whereas the from-scratch model is trained end-to-end.

We use LoRA (Hu et al., 2021) as it is a well-established strategy that allows parameter-efficient fine-tuning for pre-trained models. Tadpole is pre-trained as an autoencoder and never sees temporal dynamics, and Liu et al. (2026) found that LoRA on the frozen backbone maintains the spatial representations learned during pre-training and lets the latent subnetwork learn the dynamics. On the other hand, when training from scratch, there are no learned spatial structures to preserve.

To separate LoRA from the gains, we have added an experiment in Additional Results 1.1 where both the pre-trained and from-scratch models tune all their parameters. In this setting, the pre-trained model has 2 to 11% lower RMSE and a higher ACC for all five variables and lead times up to 24 hours. The only exception is t850 at 24 hours, where the RMSEs differ by 0.5%.

W3.1 & Q3: The proposed method combines several existing techniques with adaptations for the downstream problem … However, the experiments do not isolate their individual contributions.

We have added an ablation study that isolates the individual contributions of each HEALPix adaptation. The results and discussion are available in Additional Results 2. Following is a summary:

  • HEALPix-aware padding contributes most significantly: without it, the RMSE of every variable is 4% to 30% higher at 6 hours and 9% to 60% higher at 24 hours.
  • The loss weighting redistributes accuracy across slower vs. faster-changing variables. Without it, q700, u850, and v850 are 2 to 9% better, while z500 and t850 are 6 to 20% worse. This is expected based on the six-hour time-difference statistics discussed in the manuscript's Appendix A.7.2, from which we derive the loss weights.
  • The HEALPix coordinate embedding had little to no effect. Our assessment is that global coordinates matter more at longer lead times, when interactions over large distances become important.

W3.2: the benefit of pretraining on slices from 3D PDE systems, as opposed to directly generated 2D fields, has not been established experimentally.

In the case of our online pre-training pipeline, this study would require tuning the distribution of a 2D simulator's PDE parameters and initial conditions to achieve a dataset regime that is comparable to our current 3D simulator. Similar to our reply for W2.3 (alternate reconstruction pre-training strategies), quantifying the benefit of 3D PDE systems over 2D fields is a longer study that we will consider as an outlook for our online framework.

AuthorsOpenReview ↗

Rebuttal to Reviewer 3W8Q (Part 3)

W4.1: Limited statistical evaluation and incomplete computational cost comparison.

Q5: Can the authors report variability across independent seeds and provide a more complete comparison of total parameter counts, computational costs, and inference efficiency?

We note first that we did not yet tune any of the models for speed (for example, with memory optimizations or just-in-time compilation), so the cost numbers below serve as an initial assessment. To assess robustness, we repeated the comparison with PDE foundation models (Sec. 4.1 in the manuscript) with three further seeds per model; the seed changes the initialization of all newly trained weights and the order of the training samples. The following table contains the test RMSE at 24 h, mean and standard deviation over four seeds (test year 2022):

z500 (m² s⁻²)t850 (K)q700 (g kg⁻¹)u850 (m s⁻¹)v850 (m s⁻¹)
Tadpole-2D159.7 ± 1.51.281 ± 0.0070.873 ± 0.0022.354 ± 0.0082.413 ± 0.009
Poseidon-T276.2 ± 0.81.542 ± 0.0030.967 ± 0.0032.766 ± 0.0082.888 ± 0.007
MORPH-S368.0 ± 7.72.085 ± 0.0081.031 ± 0.0013.305 ± 0.0033.348 ± 0.003
PDEformer-2-S369.6 ± 3.92.171 ± 0.0281.168 ± 0.0073.669 ± 0.0153.683 ± 0.004

Over all variables and lead times (6 to 48 h), the standard deviation is at most 1.0 % of the mean for Tadpole-2D and Poseidon-T and 2.3 % for MORPH-S and PDEformer-2-S; every Tadpole-2D run surpassed every baseline run in RMSE and ACC.

The following table shows parameter counts and costs on one GH200 GPU for the weather setting (one forecast: 12 HEALPix faces, 5 variables; one time step; inference in single precision, training in bfloat16 mixed precision). Note: PDEformer-2 is released in MindSpore; its numbers are for our PyTorch reimplementation (outputs agree with the original to about 1e-5, relative), which recomputes decoder activations during training to save memory.

Tadpole-2D (LoRA 64)Poseidon-TMORPH-SPDEformer-2-S
Parameters at inference23.6M20.8M32.8M27.7M
Trainable parameters8.4M20.8M32.8M27.7M
GFLOPs per forecast step2637.954464
Inference time per forecast step, 1 forecast (ms)23466.031
Inference time per forecast step and forecast, 8 forecasts per batch (ms)7.65.91.828
Training step, batch 12 (ms)20022252472
Peak training memory, batch 12 (GiB)17.64.22.448.5
Fine-tuning time, 40,000 updates, including data loading and validation (h)3.03.42.35.3

W4.2 In the transonic cylinder-flow experiment, the reported 1,150 evaluation rollouts are derived from only five test trajectories and contain overlapping temporal windows. Consequently, they should not be interpreted as independent test realizations.

The evaluation on the transonic cylinder-flow experiment takes 230 frames each from five simulations at different mach numbers (Appendix A.4.7). This gives 1,150 unique initial fields and the models emulate each field's time-stepping autoregressively, yielding 1,150 unique rollouts. However, since the 230 frames in each set are consecutive, we agree with the reviewer's assessment that the temporal windows of these rollouts overlap. We will update the manuscript with this clarification and add the following results to the appendix.

We repeated the evaluation with 11 start frames per simulation, spaced 21 frames apart (frames 500, 521, …, 710), so that no frame belongs to two rollouts. This gives 55 non-overlapping rollouts. Compared to the manuscript's numbers, the rollout-mean NRMSEs changed by at most 1.6 % and the assessment stays the same.

Tadpole-2DPoseidon-TMORPH-S
NRMSE, 1,150 overlapping rollouts (manuscript)0.04400.05030.0963
NRMSE, 55 non-overlapping rollouts, mean ± std over rollouts0.0447 ± 0.00780.0504 ± 0.00700.0954 ± 0.0096
Spectral error (manuscript definition), 1,150 overlapping0.00710.02620.2146
Spectral error (manuscript definition), 55 non-overlapping0.00880.02620.2146
Reviewer_3W8QOpenReview ↗

Response to rebuttal

Thank you for the detailed rebuttal and additional experiments. I think most of my concerns have been addressed.

I still think the benefits of PDE-specific pretraining could be better isolated. However, I don't consider these major issues at this point.

I'll keep my my Weak Accept recommendation.

As for the AI reviews flag, I have read and assessed the paper before using AI. I have compared my own points to to the generated assessment, considered further poiints made by it, and further used it to polish the final review.

Comments on the paper

1 comment

Dear Reviewers,

We thank you all for your reviews and valuable feedback. We are glad you found the transfer from PDEs to weather forecasting interesting, the online pretraining framework efficient, and our experiments well detailed with encouraging results.

Based on your feedback, we have run additional experiments and added them to the revised manuscript, where all changes are highlighted in blue. Below is a summary of the additions:

  • We added an ablation of the HEALPix adaptations in the main text (Sec. 4.1). HEALPix-aware padding contributes most: without it, the RMSE of every variable is 4–29% higher at 6 hours and 9–59% higher at 24 hours.

  • PDE foundation models with our HEALPix adaptations: We clarified that our contribution combines a pretrained model with a fine-tuning framework for weather forecasting on the HEALPix grid, and that the main comparison fine-tunes each baseline with its own framework. We additionally trained all three baselines with our HEALPix adaptations and objective. Tadpole-2D still has a clearly lower RMSE than MORPH-S and PDEformer-2-S. Poseidon-T closes the gap: when all parameters are trained, it is ahead at 24 hours, and when both models use rank-64 LoRA, Tadpole-2D is better at 6 hours and on par at 24 hours. LoRA lowers Tadpole-2D's RMSE by 4–14% but raises Poseidon-T's by 1–5%, which suggests that our reconstruction-pretrained representations transfer more readily. Visually, Tadpole-2D's forecasts show neither the patch artefacts of MORPH-S, the face seams of PDEformer-2-S, nor the small-scale texture that Poseidon-T adds near the poles.

  • Separating pretraining gains from LoRA (low-rank adaptation): We clarified why LoRA applies to fine-tuning a pretrained model, and added controls where the pretrained and from-scratch models use the same adaptation. Pretraining lowers the 6-hour RMSE of all five variables by 4.5–10.7% when both models train all parameters, and by 4–6% when both use rank-32 LoRA. For the pretrained model, LoRA lowers the RMSE by 15-40% compared with training all parameters.

  • For robustness, we repeated the PDE foundation model comparison with four seeds per model: every Tadpole-2D run has a lower RMSE and a higher ACC than every baseline run, for every variable and lead time. In addition, we have expanded the comparison with specialist weather models to all five variables using RMSE and ACC, included the persistence forecast, and added a second test year (2020).

  • Outlook: Following the reviewers' suggestions, the outlook now discusses alternate reconstruction-pretraining strategies.

We will continue refining the paper for the camera-ready version and updating the source-code release. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.