- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper works on a practical and critical limitation of current existing CFD surrogates, the limited generalization issue. Authors propose a cross domain pretraining method (by pooling several existing datasets) that can reduce additional CFD simulation data needed to adapt the pretrained model to a new domain. The results show that cross-domain pretraining improves zero-shot and few-shot performance on held-out datasets compared with training from scratch and transferring from domain-specific experts. The authors further study sample efficiency during fine-tuning and how transfer performance changes with pretraining dataset diversity and model size. Authors also plan to release a unified collection publicly available CFD datasets, which could help the community to keep working on the CFD surrogates generalization.
Critical strengths
- This paper addresses a practical limitation of current CFD surrogates, the cross domain pretraining method is intuitive and straightforward to adopt across existing surrogate architectures, while the performance improvements are good over training from scratch when there are limited target domain data and training budget.
Critical weaknesses
- It is still unclear how well the method generalizes across different Re numbers independently of geometry and other dataset differences, considering the main contribution is to help generalize CFD surrogates. Current work apply only to steady-state and time-averaged flows, whether the method can be extended to unsteady predictions still uncertain and can be discussed.
Questions for authors
- Could authors discuss more about how to select datasets for cross-domain pretraining when a new target domain has few or no CFD simulatios yet in practical scenario? When adding a dataset might reduce transfer performance? Are there any general criteria?
- Could the authors provide training and validation curves for joint and single-domain pretraining to show how cross-domain pretraining affects convergence? And curves for target-domain training would help check whether the scratch baselines have converged or undertrained.
2 comments
Thanks for the experiments, explanation and additional training info. My questions are addressed.
Thank you for taking the time to review our manuscript. We have incorporated your feedback into a revised version, but feel free to let us know if there are any remaining questions or concerns.
Reynolds Number
Thank you for pointing this out. One way to examine generalization to different Reynolds numbers is by considering a single dataset, which prevents exploiting geometric/dataset differences as a proxy for the Reynolds number. Our study has results for this setting, where the Emmi Wing dataset has a range of Reynolds numbers (5-20e6) for randomly generated wing geometries. This is due to different Mach numbers and length-scales for each wing, which result in a random sweep of Reynolds numbers. We find that expert and joint models trained on Emmi Wing do not have issues predicting flows for different Reynolds numbers, since the necessary information is provided to the model (inlet velocity + length-scale is provided, and density/viscosity is held constant). However, we agree that this should be better discussed and that generalizing across large ranges in the Reynolds number is not investigated (i.e., 1e3 vs. 1e6). We discuss this in the revised conclusion section (Line 507).
Apart from this setting, it does not seem possible to test different Reynolds numbers independent of geometric or dataset differences. Within a single dataset, the geometry is restricted to a parametric family and we can vary the Reynolds number without confounding factors (which is done for Emmi Wing mentioned above). Combining datasets would unfortunately re-introduce dataset differences (discretization, base geometry, etc.) that serve as a proxy for flow regimes. We further verify this by performing an ablation study where the Reynolds number is provided as extra input during cross-domain pretraining, and find that it does not contribute extra information (discussed in Appendix C.6, Table 17):
Table 17: Pretraining validation error of the joint model with and without the Reynolds number Re.
Extensions to Transient Systems
Thank you for this suggestion. We have discussed potential extensions to transient systems in the revised conclusion (around line 503). Predicting mesh-based, time-dependent systems is a very interesting direction, however, current datasets don’t exist to be able to examine cross-domain pretraining in this setting. Despite this, we would expect that cross-domain pretraining would also benefit transient systems, since there is also the added benefit of being able to use context frames to infer different regimes.
Dataset Selection
Thank you for the question. In general, we observe that pretraining on datasets closer to the downstream task is the most beneficial. For example, experts that are closer to the target domain do better in zero/few-shot evaluations.
For zero-shot generalization, there are cases where adding datasets can reduce performance. For example, if we add a dataset that is significantly different from the target domain, then models may overfit to this during pretraining. We have found that few-shot samples during finetuning serve an important role to align pretrained models to a new domain; without this, models cannot adapt to arbitrary shifts in the target regime.
Training Curves
Thank you for the suggestion. We have provided training and validation curves for both pretraining and finetuning in Figures 10, 11, 12, 13 in Appendix C.7.
In general, we find that all models converge during pretraining and finetuning. There are no significant differences in the convergence between models, although the cross-domain model may be a little more unstable during pretraining. During finetuning, the validation error of the scratch models also tends to converge before the training error, likely due to overfitting at low sample counts.