- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a new dataset of laminar flame dynamics simulations, with the goal of testing the performance of different forecasting methods (including traditional autoregressive models, recurrent neural networks and neural operators) on this complex dataset. The data is obtained by computational fluid dynamics and takes into account chemical reactions. 8 different time rollouts are included with different boundary conditions of the fuel speed. Two such rollouts are used for training and the rest for testing. Testing decouples the latent representation from the forecasting method (except for end-to-end models). The results seem to show that neural operators perform well in terms of forecast accuracy on average, but poorly in terms of predicting the correct heat release. Meanwhile, linear methods outperform several neural networks and achieve the best accuracy in predicting the heat release.
Critical strengths
- A realistic benchmark for flame dynamics was devised, providing open clean data for others to test their methods on. Several methods were tested with a clear protocol.
- The evaluation protocol seems to distinguish different qualities of the predictive methods (i.e. just field error or heat release) which may be crucial for final physical accuracy. While the analysis is not fully fleshed out, this could be quite important.
Critical weaknesses
- The data is limited in scope to a single geometry, to a single flame chemistry, and to laminar flows. Therefore, while filling a gap in existing benchmarks, I believe this data falls short of providing a comprehensive benchmark for flame dynamics.
- There is very little analysis of the results. Beyond stating that linear models are simpler and thus may overfit the data less, the critical limitation of e.g. recurrent or convolutional models is unclear. Furthermore the very high heat release error of the neural operator remains unexplained given the good performance on the field prediction.
Questions for authors
- HP selection for neural operators seems to choose a very small (2 compared to at least 8 for other methods). Could this be a problem behind the poor results on the heat release error?
- Could you include a further analysis of the errors for representative methods for each class?
- Could you expand on the motivation for developing such a benchmark and what are the applications of surrogate models specifically on this problem? In particular I am curious about the standard efficiency motivation: are all tested models faster at inference than the CFD, and by how much?
Reviewer ziiS
1. HP selection for neural operators seems to choose a very small Nx (2 compared to at least 8 for other methods). Could this be a problem behind the poor results on the heat release error?
We thank the reviewer for this interesting observation. The FNO value Nx=2 was selected by the Optuna hyperparameter search, rather than fixed a priori. The search considered the same range of candidate values used for the other neural models and selected the configuration with the lowest validation MSE. Although we cannot exclude an effect of Nx on heat-release prediction, the POD+DMD results Nx=0, which also yield one of the lowest ϵQ values, suggest that a larger Nx does not necessarily lead to better heat-release predictions.
A closer inspection of the FNO predictions (Appendix E, "Additional Result") reveals that larger errors are concentrated mainly in the upper-right region of the domain, where the cells have larger volumes, as visible in the new Figure 1. This helps explain the discrepancy between the field-wise nRMSE and the integrated heat-release error: nRMSE assigns equal weight to every cell, whereas the calculation of total heat release weights each cell contribution by its volume. Consequently, errors in larger cells have a greater impact on the predicted total heat release. We have added this analysis to the Appendix E.
2. Could you include a further analysis of the errors for representative methods for each class?
We thank the reviewer for this suggestion. We added a more detailed error analysis for one representative method per model class in Appendix E ("Additional Results") of the manuscript. The chosen representative methods are POD–ARX (closed-form forecaster on a linear compressor), CAE–ARX (closed-form forecaster on a nonlinear compressor), POD–Transformer (the neural latent forecaster with the lowest nRMSE) and FNO (neural operator on the full fields), with the constant forecast as reference. The analysis includes the error growth along the closed-loop rollout, to assess whether the errors accumulate over the forecast horizon, and the spatial distribution of the heat-release error, to identify where the heat-release error originates and why FNO combines the lowest field error with the largest heat-release error.
3. Could you expand on the motivation for developing such a benchmark and what are the applications of surrogate models specifically on this problem? In particular I am curious about the standard efficiency motivation: are all tested models faster at inference than the CFD, and by how much?
We thank the reviewer for the observation. FireMark addresses the lack of open benchmarks for forecasting spatially resolved reacting-flow fields under controlled, time-varying forcing. Such benchmarks are needed to compare forecasting methods under consistent conditions. In the longer term, surrogate models could support digital twins of combustion systems, enabling real-time prediction and control under changing operating conditions. Potential applications include instabilities and explosions detection, real-time fuel consumption optimization and emission reduction. These applications motivate the benchmark but are not demonstrated by the present single-configuration dataset. We have sytressed this motivation in the introduction, section 1 of the manuscript.
Regarding computational efficiency, the eight CFD simulations required approximately 127–508 wall-clock hours each on 20 cores. This information about the computational cost are reported in section 3 and further details were reported in section B of the appendix.
It is not possible to calculate a speed up factor from CFD simulations to surrogate models as we used two different machines to do the calculations. However, a rough comparison already shows the order of magnitude. The fastest CFD simulation (step, A=0.5) required about 4.6×105 s (≈127 h on 20 CPU cores) to produce the 2000 snapshots of one trajectory, i.e. about 230 s per snapshot. In contrast, every tested surrogate requires less than 6 ms per step (see the inference-cost table in Appendix F), so a full 2000-step rollout takes 12 s maximum, depending on the model. This corresponds to a speed-up of roughly 4×104 for the slowest surrogate (exact DMDc) and 4×105 for the fastest.
We have added this in the paragraph Surrogate model speed-up, Appendix F.