- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents SHIFT (Simulation for High Fidelity Training), an extensive benchmark designed for studying 3D industrial fluid dynamics. It features both transient Delayed Detached Eddy Simulations (DDES) and steady Reynolds Averaged Navier Stokes (RANS) simulations for external aerodynamics. It is split into two primary segments: an automotive dataset (SHIFT-SUV) featuring highly accurate vehicle shape and wake flow simulations, and an aerospace dataset (SHIFT-Wing) focusing on transonic wing-body configurations using adaptive meshes.
This paper is not about an algorithm, instead it focuses on creating a robust infrastructure and dataset. They outline testing protocols for both standard and unfamiliar data distributions, offer validation through computational fluid dynamics (CFD) and mesh-convergence metrics, and test several recent geometry-aware neural surrogate models.
Critical strengths
Important benchmark contribution: Large-scale, three-dimensional CFD data with industrially relevant geometries remain limited. The paper addresses a significant impediment to progress in data-driven engineering simulation. Combination of scale, fidelity, and geometric variation: The benchmark combines properties that are rarely available together: a sizable simulation collection, three-dimensional geometries, high-resolution CFD, and engineering-relevant design variation. Coverage of two materially different CFD domains: Automotive wake flows and transonic aerospace flows present different challenges, including separated wakes, wall quantities, compressibility, and localized shock structures. Attention to numerical credibility: Validation against established reference cases and the reported mesh-convergence evidence are major strengths. The value of a learning benchmark depends directly on the credibility of the underlying CFD solutions. Standardized ID and OOD evaluation: Explicit splits can help the community distinguish geometric interpolation from extrapolation and compare surrogate models under common conditions. Useful baseline evaluation: The initial results establish that the benchmark is practically usable and indicate that near-wall and localized flow quantities remain difficult for current architectures. Potential long-term impact: With accessible data, standardized preprocessing, and maintained evaluation infrastructure, SHIFT could become a useful community resource for learned CFD.
Critical weaknesses
Limited methodological and diagnostic analysis: The paper is primarily a dataset contribution. The baseline section reports accuracy but provides relatively limited analysis of why particular architectures succeed or fail. Dependence on a proprietary simulation pipeline: Researchers may be able to reproduce surrogate-learning experiments without being able to reproduce the original CFD data generation. Limited emphasis on engineering quantities of interest: Field-level errors are important, but practical CFD evaluation also depends on drag, lift, moments, pressure distributions, shock location, separation behavior, and wall quantities. These should be central benchmark outputs. Aggregate metrics may conceal localized failures: Low global normalized errors can coexist with large errors near shocks, walls, separation regions, and wake structures. Region-specific and physics-relevant metrics are needed.
Questions for authors
What are the exact licensing and access conditions? Will academic and commercial researchers receive the same geometries, meshes, fields, and metadata? The information in the Appendix is not complete. Will the release include fixed train-validation-test indices, standardized preprocessing, evaluation scripts, and reference model configurations? How were the OOD cases selected, and how far do they lie from the training distribution in geometric and physical parameter space? Were any simulations removed because of meshing, convergence, or solver failures? If so, are the exclusion criteria and failure cases documented? Are integrated quantities such as drag, lift, and moments included in the public data and official evaluation protocol? Are localized metrics provided for shocks, wakes, boundary layers, separated regions, and other high-gradient areas?
3 comments
Response to Reviewer 6qmL: Part II
Q7) Are localized metrics provided for shocks, wakes, boundary layers, separated regions, and other high-gradient areas?
Response: Currently there is no segmentation of the field data into different physics regimes, though this information is embedded within the solution files themselves. To facilitate extracting such information, we will include a python script with each dataset that allows users to post-process the provided data into such regions, which users can adapt for their needs.
Weakness: Field-level errors are important, but practical CFD evaluation also depends on drag, lift, moments, pressure distributions, shock location, separation behavior, and wall quantities. These should be central benchmark outputs.
Response: The integrated Quantities of Interest are part of the dataset. As asked by the reviewer, we benchmark the models on these, outlined in the tables below. These cover a single run. We will repeat these runs over multiple random seeds and include the final results in the camera ready manuscript.
Table 1: $C_D$; MAE in drag counts (R² in parentheses):
| Model | SUV | Wing 0.5 | Wing 0.85 |
|---|---|---|---|
| Transolver | 31.5 (0.986) | 0.55 (0.9999) | 0.75 (1.0000) |
| GeoTransolver | 26.0 (0.991) | 1.14 (0.9996) | 0.68 (1.0000) |
| DoMINO | 93.1 (0.914) | 2.97 (0.9963) | 3.92 (0.9995) |
| AB-UPT | 27.1 (0.991) | 1.07 (0.9996) | 1.41 (0.9999) |
Table 2: $C_L$; MAE (R² in parentheses):
| Model | SUV | Wing 0.5 | Wing 0.85 |
|---|---|---|---|
| Transolver | 0.0101 (0.973) | 0.00073 (1.0000) | 0.00086 (1.0000) |
| GeoTransolver | 0.0082 (0.975) | 0.00041 (1.0000) | 0.00080 (1.0000) |
| DoMINO | 0.0225 (0.898) | 0.0041 (0.9989) | 0.0050 (0.9987) |
| AB-UPT | 0.0087 (0.969) | 0.00084 (1.0000) | 0.0018 (0.9998) |
Table 3: $C_M$ (SHIFT-Wing only); MAE (R² in parentheses):
| Model | Wing 0.5 | Wing 0.85 |
|---|---|---|
| Transolver | 0.00039 (0.9999) | 0.00066 (0.9998) |
| GeoTransolver | 0.00027 (1.0000) | 0.00059 (0.9998) |
| DoMINO | 0.0054 (0.9859) | 0.0055 (0.9921) |
| AB-UPT | 0.00058 (0.9999) | 0.00125 (0.9994) |
Thanks to the authors for answering the questions that provide added required clarity for researchers who would like to use these data.
I will however, keep my rating the same.
Response to Reviewer 6qmL: Part I
Q1) What are the exact licensing and access conditions?
Response: Both SHIFT-Wing and SHIFT-SUV are released under the CC-BY-NC-4.0 (Creative Commons Attribution Non-Commercial 4.0) license. This enables users to use the dataset to train their own models, augment the dataset with their own samples, etc. The license requires the users to provide attribution and prevents commercial use.
With respect to access, we chose credentialed access. The physical reasoning capabilities enabled by large-scale 3D CFD datasets present inherent dual use realities. Additionally, the broader SHIFT dataset ecosystem includes datasets possessing direct defense and military applications [eg, SHIFT-CCA (UAV design) and SHIFT-Missile (defense payloads)]. Thus, we utilize credentialed access requiring contact sharing for the release of the SHIFT datasets, to ensure responsible data stewardship. This credentialed access model also provides vital telemetry on dataset utilization, directly informing our long-term resource allocation strategy. By verifying legitimate scientific inquiry and tracking usage trends, we can identify which domains provide the most value to the Scientific Machine Learning community, allowing us to prioritize supercomputing resources to expand the most highly utilized datasets in future releases.
These details are outlined in Appendix A of the original manuscript. We will move this discussion to the main text in the camera ready manuscript.
Q2) Will academic and commercial researchers receive the same geometries, meshes, fields, and metadata?
Response: Yes, all researchers will receive the same dataset, irrespective of the nature of their employment.
Q3) Will the release include fixed train-validation-test indices, standardized preprocessing, evaluation scripts and reference model configurations?
Response: Yes, the release includes standardized train, validation and test dataset splits; standardized preprocessing; evaluation scripts along with reference model configurations.
Q4) How were the OOD cases selected, and how far do they lie from the training distribution in geometric and physical parameter space?
Response: For the SHIFT-SUV dataset, the OOD set consists of ten truck geometries simulated with their corresponding operating conditions identical to the SUV simulations. This represents a distribution shift in geometry space. As SHIFT-SUV does not have any global parameters, this is the only distribution shift that is feasible.
For the SHIFT-Wing dataset, the OOD set includes 20 CFD simulations carried out at Mach Numbers of 0.5 and 0.85, where all the geometry parameters are extended beyond the range of the training data by 101-105%.
Q5) Were any simulations removed because of meshing, convergence, or solver failures? If so, are the exclusion criteria and failure cases documented?
Response: Yes, during the data generation phases of the datasets there are portions of the generated simulations that are rejected. For SHIFT-SUV, ~2% of the simulations are rejected due to the solver not completing the full iteration count (and thus statistical averaging time) requested. Solver residual convergence was monitored and was deemed converged for each of the samples provided in the dataset.
For SHIFT-Wing, the solver iterates through a mesh adaption sequence (iterating between solution←→mesh). Convergence of these samples was determined through successful completion of the full grid sequence. Stability of the sequence requires adequate solver convergence in each individual stage. About 10% of runs were rejected due to failure to complete.
Q6) Are integrated quantities such as drag, lift, and moments included in the public data and official evaluation protocol?
Response: Yes, the integrated quantities like lift and drag are included in the datasets. a) The SHIFT-SUV dataset includes csv files with the [drag, drag_coefficient, lift,lift_coefficient]. These contain the transient history and final time-averaged scalar metrics, along with a residuals_and_forces.png b) The SHIFT-Wing dataset includes a forces.json file mapping out the exact final drag (CD), lift (CL), normalized coefficients and also moments (CM). These integrated quantities are also part and parcel of all the datasets in the broader SHIFT dataset ecosystem.