- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CTF4Nuclear, a common task framework for systematically evaluating scientific machine learning methods in nuclear systems. The framework includes four datasets covering different nuclear and nuclear-adjacent systems and evaluates models across twelve metrics, including forecasting, noise robustness, limited-data performance, and parametric generalization. It also introduces a sparse-observation state-estimation task to assess the reconstruction of full physical states from limited sensor measurements.
Critical strengths
-
The framework considers multiple aspects of model performance, including forecasting, noise robustness, limited-data learning, and parametric generalization. In particular, the introduction of sparse-observation state estimation is practically meaningful, as reconstructing full physical states from limited sensor measurements is an important challenge in real-world nuclear system monitoring.
-
The paper evaluates a wide range of approaches, including classical dynamical systems methods, neural operators, deep learning models, and time-series foundation models. The results provide useful insights into their relative strengths and limitations, particularly highlighting the competitiveness of simpler methods under the considered data constraints.
-
The benchmark brings together four datasets covering different nuclear and nuclear-adjacent systems, physical processes, and modeling fidelities. This diversity provides a valuable foundation for future research on machine learning for complex multiphysics systems.
Critical weaknesses
-
Insufficient cross-dataset evaluation. Although the proposed benchmark includes four datasets covering different nuclear systems and physical processes, comprehensive evaluations of machine learning methods are conducted primarily on the MSFR dataset. The other three datasets (HPMR, DYNASTY, and MHD) are evaluated only with a few simple baselines. Since one of the main motivations of the benchmark is to provide a unified evaluation framework across diverse nuclear systems, the limited experimental coverage weakens this contribution. I would expect more comprehensive comparisons across multiple datasets to demonstrate the generality of the framework and reveal how different methods perform under different physical regimes.
-
Insufficient justification of the aggregate scoring scheme. The paper ranks methods by taking the arithmetic mean of twelve evaluation scores, but the rationale behind this aggregation strategy is not sufficiently discussed. In particular, the four task categories contain different numbers of metrics, implicitly assigning different weights to forecasting, noise robustness, limited-data performance, and parametric generalization. Furthermore, a high average score may conceal substantial failures on individual tasks, which is particularly concerning for safety-critical nuclear applications. The authors should provide a stronger justification for the scoring scheme and investigate whether the model rankings remain consistent under alternative aggregation strategies, such as equal weighting across task categories or greater emphasis on critical tasks.
-
Insufficient experimental validation of the proposed sparse-observation task. The introduction of sparse-observation state estimation is presented as an important contribution of the benchmark. However, its experimental evaluation is relatively limited, relying primarily on the Empirical Interpolation Method (EIM) to investigate the effects of sensor number and measurement type. There is a lack of systematic comparisons with other relevant approaches, particularly learning-based state reconstruction and data assimilation methods. Since this task is intended to reflect realistic monitoring scenarios in nuclear systems, more comprehensive experiments involving different models, sensor configurations, and observation noise levels would help demonstrate the usefulness and discriminative capability of the proposed evaluation paradigm.
-
Questions regarding the benchmark difficulty and discriminative capability. An interesting but somewhat concerning observation is that the simple last-value baseline achieves the second-highest average score, outperforming most of the evaluated machine learning methods, including several neural operators and foundation models. While this may reflect the slowly evolving dynamics of the MSFR system, it also raises questions about whether the current tasks and scoring metrics are sufficiently challenging and discriminative. Alternatively, the result may indicate that the evaluated methods are not adequately adapted or optimized for the benchmark. The authors should provide a more detailed analysis to distinguish these possibilities. In particular, comparisons across different prediction horizons, more challenging transient regimes, and stronger task-specific baselines would help clarify whether the observed ranking reflects the inherent difficulty of the benchmark or limitations of the evaluated methods.
Questions for authors
See above.
We thank the reviewer for the careful review. We have uploaded a revised PDF; page numbers below refer to it.
W1 (cross-dataset evaluation). We ran four methods representative of different families on DYNASTY and HPMR with the MSFR protocol. The results are preliminary and may be updated as further runs complete, and the remaining methods are not yet included; full E1–E12 scores are in Appendix C.6 (Table 7, p. 35). Average score over the twelve tasks:
On HPMR all four methods improve on the last-value baseline; on DYNASTY only Reservoir and Panda do, and PyKoopman, first on the MSFR, reaches the minimum score on one task. The ranking does not transfer across systems (Section 3.1, p. 8; Limitations, p. 10), which is expected given the different physical characteristics and fidelity of the systems. MHD cannot be evaluated within the discussion period because of its computational cost and will be included in the final version.
W2 (aggregation). The composite score is an unweighted mean meant as an overview; the paper says so, and task-group averages and the number of tasks at −100 of each method are reported (Appendix C.1, Table 3, p. 30), so that failures are not hidden in the mean and readers can weight the groups according to their application. We recomputed the ranking with equal weight per group, with −100 replaced by 0, and with the median (Appendix C.2, p. 29): the ordering is stable, but the first position is not. PyKoopman is first with the original score, the last-value baseline with equal group weights (68.11 versus 66.01), Reservoir with −100→0 or the median. Further checks (interquartile mean, win rate, geometric mean of the errors) show that the last-value baseline ranks between 1st and 5th across aggregators and is never the best method on a single task. Removing the denoising group changes the ordering far more than removing any other group, because of the −100 of the foundation models on E3. Ultimately, the aggregation method is meant to be a summary to show the versatility of a method across tasks, not a definitive ranking; users should look at the individual scores to select a method for their application (Appendix C.2, p. 29).
W3 (sparse-observation task). Besides EIM we compare Gappy POD, POD-DeepONet, an echo state network, an LSTM and SHRED, with the same pre-computed temperature measurements (first 5, 10 or 20 sensors of an EIM-greedy batch and of a random batch, both released with the data; noisy at test time for the noisy tasks, whose noise level is not varied in this study), one configuration per method, trained on the released training data, three seeds when stochastic. Average score over the twelve tasks, with EIM-greedy sensors and 5 to 20 sensors: EIM 32.9 to 64.0, the five other estimators 52–65 to 65–74; with random sensors EIM is competitive with few sensors (54.6 against 35–55 with 5 sensors) but does not improve with more (54.6 with 20 sensors against 57–65), so placement matters as much as the estimator. POD-DeepONet, ESN and SHRED are within seed variability of one another (up to 6 points). On DYNASTY (four experimental thermocouples, 2 to 4 sensors) all methods score between 79 and 96, since the transients are dominated by slow trends. Full tables are in Appendix C.4 (Tables 4 and 5, p. 33; Section 3.2, p. 9). A broader study of placement across datasets is stated as a limitation (p. 10).
W4 (difficulty). (i) The strength of persistence on the MSFR has a physical origin stated in the paper (Section 3.1, p. 8): the RANS model filters high-frequency fluctuations. (ii) The benchmark discriminates once the physics changes: the last-value baseline scores 80.8 on DYNASTY, −26.9 on MHD and −49.0 on HPMR (Appendix C.5, Table 6, p. 34), and the runs of W1 show that the margin over persistence depends on the system. (iii) The horizon matters: on the MSFR the short-time scores of PyKoopman and the last-value baseline are close (87.5 versus 81.2) but the long-time ordering reverses (31.1 versus 71.8), while Reservoir is best on both (92.4, 87.4). (iv) The result is not due to untuned methods: each method is tuned for each task on the released training data (search spaces in Appendix D), and the best-performing methods are task-adapted dynamical-system methods (Koopman, reservoir) rather than generic foundation models. PyKoopman improves on persistence on eight of the twelve tasks and is the only method with a positive mean margin (+8.29), whereas Moirai-2 has a margin of −15.43 because of two outright failures (Appendix C.1, Table 3, pp. 29–30).