- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce KaleidoFlow that utilises LLM to generate 2D incompressible laminar flow datasets with no human in the loop. The process can be divided into 3 general steps: (i) using LLM to propose geometries, which is one set of shape parameters, (ii) designing the mesh and verifying it via a standard grid-independence test, and (iii) simulating via opensource solver (OpenFOAM) under larminar and transient settings. Finally, the datasets contain a total of 3948 simulation cases. With that, the authors carried out 4 test splits for benchmarking under 9 architectures.
Critical strengths
- I like the illustrations presented in this paper. They are consistent and informative, though they are clearly LLM-assisted. Overall,this paper is quite comprehensive and well written.
Critical weaknesses
- In the context of this paper, the main contribution is to use an LLM to generate a large dataset. It is essentially a dataset generation method, nothing more than that. Current commercial LLMs can easily do this work for non-experts, but for experts, it is useless. Hence, the novelty and contribution are minimal. The authors should consider whether "datasets and benchmarks" is the better primary area to submit.
- As per the title "flows on diverse geometries", I suppose the dataset contains different flow conditions other than just varying the Reynolds number. This dataset currently contains only 2D, laminar, and internal flows. Also, the Reynolds number is up to 40000, yet there is no turbulence model, and for any internal flow with Re> 2100, it is no longer considered to be laminar. Thus, the claim in App. A.3 that "every released trajectory is a physically consistent solution" is too strong for the high-Re part of the dataset.
- Regarding the power law claim, the testing K=32, 64, 128 is only two folds between each, not an order of magnitude difference as many smooth decaying curves look linear in log-log over that range.
Questions for authors
- What fraction of trajectories has Re > 2300? Do the diversity and G3 results hold if the high-Re cases are removed? Is a 2D laminar solution at Re up to 40,000 considered physical, and how should users interpret these trajectories?
- Why doesn't the divergence excess improve with K? Does this suggest purely data-driven scaling cannot fix physical consistency?
- Why did the authors choose to map the simulated fields onto the uniform grids? Isn't that going to corrupt the continuity of the flow, reducing the fidelity? Why no preverse the grids and retain the physical information in them?
4 comments
[Q1]
(1) "What fraction of trajectories has Re > 2300?" Only 1.35% of the trajectories have a Reynolds number above 2,300. This work focuses on geometry, so the dataset varies the geometry most widely. Many applications, such as biomedical devices and fuel cells, operate at low or moderate Reynolds numbers.
(2) "Do the results hold without the high-Re cases?" Without the high-Re trajectories, the diversity statistics remain essentially unchanged, and the G3/G2 error ratio of a retrained UNet-c remains similar. Every family has trajectories at or below Re 2,300, so removing the trajectories above Re 2,300 keeps all 459 families and 3,924 of the 3,948 geometries. The between-family share of the shape variance stays at 92%, and the nearest-family distance relative to the within-family spread stays at 3.8 (Table 1). We retrained UNet-c once without the SST families and the trajectories above Re 2,300 (2.9% of the training data), and the G3/G2 ratio changes less than between two training seeds (Table 2).
| Training set | G2 | G3 | G3/G2 |
|---|---|---|---|
| Without high-Re | 0.0912 | 0.3874 | 4.25 |
| All trajectories | 0.0879 | 0.3904 | 4.44 |
(3) "How should users interpret these trajectories?" The high-Re trajectories are verified solutions of the 2D equations of their flow model, and learning such solutions is an accepted practice in operator learning. We understand your concern that turbulence in the standard sense is three-dimensional. However, our understanding is that these trajectories are still meaningful in three respects. First, 2D turbulence is an established subject of fluid dynamics, which describes thin-layer flows in the atmosphere and the ocean [6]. Second, the trajectories above Re 5,000 use the k-omega SST model, which is also used in 2D by AirfRANS [4]. Third, operator learning targets the verified solution of given equations, and many works learn 2D turbulent or chaotic flows, such as FNO [7]. Therefore, these trajectories are suitable for operator learning, and in the revised Appendix A.3, every trajectory is labelled with its Reynolds number and flow model.
[Q2] "Why doesn't the divergence excess improve with K?"
We would like to clarify that the divergence excess does improve with . It decreases from to , although it does not follow the power law as strictly as the relative error. The table reports the excess (, mean ± std over three seeds) and the of each power law (Table 20).
| Architecture | Divergence | Relative error | |||
|---|---|---|---|---|---|
| UNet-c | 4.0±1.6 | 3.0±0.5 | 1.1±0.1 | 0.78 | 0.98 |
| HNO | 8.8±2.1 | 3.0±0.4 | 2.6±0.2 | 0.85 | 0.95 |
| F-FNO | 9.2±0.1 | 5.4±0.1 | 4.0±0.3 | 0.89 | 0.95 |
This deviation from the power law is expected, because our training objective is the relative error of the velocity, while the divergence depends on its spatial derivatives, which this loss does not constrain (Appendix D.2). Language models show the same pattern: the training loss follows a power law [8], while metrics that are not optimized directly can deviate from it [9]. Therefore, data-driven scaling does improve physical consistency, but it does not enforce it. A constraint in the architecture or the loss can enforce it, which we leave to future work.
[Q3]
(1) "Why map the fields onto uniform grids?" The uniform grid enables a systematic and fair comparison of all major classes of neural operator architectures, because convolutional, spectral, and transformer architectures all operate on it (Appendix D).
(2) "Isn't that going to corrupt the continuity?" The mapping is designed to keep both the continuity and the fidelity of the verified solution at the resolution of the uniform grid. The verified solution satisfies mass conservation, checked for every trajectory by the cumulative continuity error of the solver (Appendix B.2). The mapping does not interpolate point values, which would break this conservation. Each uniform-grid value is the exact area-weighted average of the finite-volume solution over the fluid part of its grid cell, so the fluid-area integral of every field is preserved and every value remains within the range of the mesh solution (Appendix A.4).
(3) "Why not preserve the grids?" We retain the solutions on the original meshes and will release them to support future research.
References
[1] Poseidon: Efficient Foundation Models for PDEs
[2] Multiple Physics Pretraining for Spatiotemporal Surrogate Models
[3] Foam-Agent: Towards Automated Intelligent CFD Workflows
[4] AirfRANS: High Fidelity Computational Fluid Dynamics Dataset for Approximating Reynolds-Averaged Navier-Stokes Solutions
[5] Perspective: A Method for Uniform Reporting of Grid Refinement Studies
[6] Two-Dimensional Turbulence
[7] Fourier Neural Operator for Parametric Partial Differential Equations
[8] Scaling Laws for Neural Language Models
[9] Are Emergent Abilities of Large Language Models a Mirage?
Thanks for the answer. I will maintain my score for the following reasons:
- "current architectures generalize poorly" is a common issue, most researchers accept that, so it adds no value to quantify it. What is really added value is being able to solve it while identifying the mechanism behind.
- "cannot readily generate diverse and verified flow data at very large scale", in what application or situation does a user need to generate vast and diverse geometries? Like design optimisation, which usually requires a user to refine and optimise a geometry within a similar family.
- "engine judges every mesh strategy by a grid convergence test", this criterion is a very common practice in CFD, letting an LLM agent obey this convergence test would not add any value because it is a must-do for a valid solution.
Thus, I agree with the reviewer DucG that the novelty of this work is limited and should be more suitable for a dataset and benchmark track, for which I maintain my score for this paper.
Thank you for your reply. We respond to your points below and then discuss the choice of track.
(1) Regarding your first and second points, we wish to explain the broader context. In classical numerical simulation, we usually solve each problem with expert-designed ad hoc methods. Recently, a line of work [1, 2, 3, 4] pursues general PDE solvers that handle diverse problems with a single model. Such general solvers require diverse and scalable data on many different geometries and boundary conditions, but data with sufficiently diverse geometries are still lacking. As a result, existing work cannot evaluate or pretrain such solvers across a rich set of geometry families.
In this paradigm of general solvers, quantifying how neural operators generalize across such diverse geometries clearly adds value, rather than "it adds no value to quantify it" (Q1). Most current PDE benchmarks, such as PDEBench [5], PDEArena [6], and CFDBench [7], measure how well a model fits a single geometry family. In contrast, we quantify how current architectures generalize as the geometries become significantly more diverse. In addition, this work provides the first scaling law for neural operators across geometries: the error on new geometry families decreases as a power law in the number of training families. This law also clearly adds value, because it provides a scalable path towards such general solvers that generalize across geometries, as scaling laws have done in other domains [8, 9].
For the same reason, datasets of vast and diverse geometries are needed, which directly answers "in what application or situation does a user need to generate vast and diverse geometries" (Q2). A general solver must handle unseen geometries, which our results show requires many geometry families in training, and our engine provides such data systematically.
(2) "This criterion is a very common practice in CFD." (Q3) The grid convergence test is indeed a standard practice, but organizing it into an automated workflow that produces large-scale and diverse datasets in a standardized format has not been done before. In common practice, whether or not LLMs are used, an expert designs the mesh for each geometry and verifies it with this test. In contrast, this work designs and verifies meshes fully automatically across hundreds of diverse geometry families. In the engine, the agent iteratively refines the mesh strategy from structured diagnostics that identify the failed quantity and summarize all earlier attempts (Sec. 3.2 and Appendix A.2). At the same solver budget, this repair passes 70% of sampled families, versus 20% for a conventional mesh-size search (Appendix C.5). This workflow enables the scaling study above, and it allows future work to expand verified data to the same standard.
On the choice of track, we clarify why we believe this work fits its current track, applications to physical sciences, rather than a dataset and benchmark track. This work is not merely a fixed collection of data with an evaluation protocol, but also contributes an infrastructure and scaling-law findings beyond the dataset and benchmark. (i) An infrastructure: an autonomous engine that continuously produces flow data on new geometry families, and future work can use it to build larger datasets for its own problems under the same verification standard and data format. Methodologically, it is the first autonomous data engine for large-scale flow simulation, in which an LLM agent both proposes new geometry families and iteratively repairs their mesh strategies (see (2)). (ii) Scaling-law findings: the error on new families decreases approximately as a power law in the number of training families, a law not reported before for neural operators across geometries. Similar works have been published at ICLR in application areas rather than datasets and benchmarks: GenSim [10] (ICLR 2024, spotlight) synthesizes diverse simulation tasks with LLMs to improve robot generalization, and Lin et al. [11] (ICLR 2025, oral) report a power law between robot generalization and data diversity. Our work extends this paradigm to neural operator learning in physical simulation.
References
[1] PDEformer: Towards a Foundation Model for One-Dimensional Partial Differential Equations
[2] Multiple Physics Pretraining for Spatiotemporal Surrogate Models
[3] DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training
[4] Poseidon: Efficient Foundation Models for PDEs
[5] PDEBench: An Extensive Benchmark for Scientific Machine Learning
[6] Towards Multi-spatiotemporal-scale Generalized PDE Modeling
[7] CFDBench: A Large-Scale Benchmark for Machine Learning Methods in Fluid Dynamics
[8] Scaling Laws for Neural Language Models
[9] Scaling Vision Transformers
[10] GenSim: Generating Robotic Simulation Tasks via Large Language Models
[11] Data Scaling Laws in Imitation Learning for Robotic Manipulation
Thanks for your thoughtful reviews and insightful questions. They have significantly improved this work. We respond point-by-point below.
We respectfully clarify that this work is not only a dataset generation method. Our main contribution is the first systematic study of how neural operators generalize across diverse geometry families, an important open problem in operator learning [1, 2]. It shows quantitatively that current architectures generalize poorly to new families (Sec. 4.3) and identifies an approximate power law between their error and the number of training families (Sec. 4.4). These findings require verified flow data on hundreds of geometry families, which existing tools and agents [3] cannot readily produce, so we built an autonomous data engine (Appendix A).
State-of-the-art LLMs are powerful, but they cannot readily generate diverse and verified flow data at very large scale. (i) An LLM asked for many geometries tends to repeat familiar shapes, so the engine requires each proposal to target sparsely covered flow mechanisms and to differ from its closest existing families (Appendix A.1). (ii) A simulation can run normally and still have a large discretization error, which the solver output does not reveal, so the engine judges every mesh strategy by a grid convergence test and repairs failed strategies (Appendices A.2 and B.1). The engine is thus a harness that lets an LLM agent produce a large, diverse, and verified dataset, and a stronger LLM yields a better dataset. It is therefore useful to both experts and non-experts.
Therefore, we believe that this work makes significant contributions in neural operator studies and fits the current primary area. We have revised the abstract and Sec. 1 accordingly.
Regarding "only 2D, laminar, and internal flows": We acknowledge that the current dataset is 2D and consists mainly of laminar flows in bounded domains, as now stated in Sec. 6. However, the dataset is highly diverse within this scope (Sec. 3.1). (1) It has 459 geometry families from 10 application categories, 40 times more families than any existing flow dataset. (2) In 54.0% of the families the flow passes obstacles and forms wakes, and in 28.1% it has several inlets or outlets. (3) The boundary conditions include moving or rotating walls, symmetry planes, periodic pairs, and time-varying inlets. The Reynolds number ranges from 1 to 40,000, and the three highest-Re families use the k-omega SST turbulence model. This scope is sufficient for our findings, whose question does not depend on the dimension. Even on 2D, mostly laminar flows, all nine architectures generalize poorly to new families (Sec. 4.3), so this gap is not an artifact of a simple setting, and we expect that 3D or turbulent flows would make it even more challenging rather than close it. The engine has not yet been tested in 3D.
Regarding "there is no turbulence model" and the claim in Sec. A.3 being "too strong for the high-Re part": We agree that the original version did not state the flow model and that this claim was too strong. (1) We clarify that the dataset does use a turbulence model. The three families with the highest Reynolds numbers, 152 trajectories including all 39 above Re 5,000, use the k-omega SST model. (2) We have revised the claim to the form that we can verify: every released trajectory is a verified solution of the equations of its flow model. Each mesh strategy passes the grid convergence test [5] on its hardest trajectory with an index below 5% for the time-averaged fields (Appendix B.1), and every trajectory is checked for mass conservation, the Courant number, and time-step collapse (Appendix B.2). In CFD terminology, this is solution verification. Validating a flow model for a specific physical flow is beyond the scope of this work.
Following your suggestion, we extended K to 24–271, with six architectures and three seeds. The table gives the G3 error (10−2, mean ± std over three seeds), α, and R2 (Table 16). Over this wider range, the error of every architecture follows an approximate power law, error∝K−α, with R2 from 0.95 to 0.98. It also extrapolates: fitted on K≤128 only, it predicts the error at K=192 and 271 within 6% for five architectures. In contrast, an exponential decay, which would mean early saturation, underestimates these errors by 9 to 41% (Appendix E.4).