acceptodds
Under review as a conference paper at ICLR 2027

CTF4Nuclear: a Common Task Framework for Scientific Machine Learning in Nuclear Systems

Abstract

Designing, monitoring, and operating nuclear systems is exceptionally challenging because of the complex physical phenomena and their interactions that govern the system dynamics. While high-fidelity simulations can resolve these non-linear, multi-physics interactions (e.g., between neutronics and thermal-hydraulics), they are computationally expensive and rarely suitable for multi-query and real-time applications. Furthermore, model-based approaches inherently rely on simplifying assumptions, leading to inevitable discrepancies with real-world measurements. Machine Learning (ML) surrogates offer a promising alternative; however, systematic comparisons of their performance across relevant operating conditions and data constraints remain limited. In safety-critical settings such as nuclear engineering, a fair and rigorous comparison of different ML methods, and a clear understanding of their advantages and limitations, is of paramount importance. To address this gap, we introduce a Common Task Framework (CTF) for the application of ML in nuclear engineering, building upon previous efforts in dynamical systems and seismology. This CTF considers a curated set of datasets spanning different nuclear and nuclear-adjacent systems, including molten salt, micro-reactors, electrically conducting fusion coolants, and experimental thermal-hydraulics facilities. To demonstrate the rigour and fairness of the CTF, we apply it to the one of the most strongly coupled system, the molten salt reactor, evaluating the performance of all methods on twelve established metrics (forecasting, noise robustness, limited data, and parametric generalisation), and identifying substantial differences in their robustness and generalisation. Additionally, we introduce a sparse-observation monitoring paradigm to evaluate whether models can reconstruct and forecast system dynamics from limited sensor measurements, providing a common setting for comparing forecasting and state-estimation methods. Overall, the proposed CTF provides a reproducible basis for systematic comparison of data-driven dynamical models in nuclear engineering and establishes a framework for assessing ML capability under realistic data and observation constraints in complex, safety-critical physical systems.

Primary area:
datasets and benchmarks
Keywords:
Nuclear Reactors, Multi-Physics, Scientific Machine Learning, Common Task Framework

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zNwhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CTF4Nuclear, a common task framework for systematically evaluating scientific machine learning methods in nuclear systems. The framework includes four datasets covering different nuclear and nuclear-adjacent systems and evaluates models across twelve metrics, including forecasting, noise robustness, limited-data performance, and parametric generalization. It also introduces a sparse-observation state-estimation task to assess the reconstruction of full physical states from limited sensor measurements.

Critical strengths
  1. The framework considers multiple aspects of model performance, including forecasting, noise robustness, limited-data learning, and parametric generalization. In particular, the introduction of sparse-observation state estimation is practically meaningful, as reconstructing full physical states from limited sensor measurements is an important challenge in real-world nuclear system monitoring.

  2. The paper evaluates a wide range of approaches, including classical dynamical systems methods, neural operators, deep learning models, and time-series foundation models. The results provide useful insights into their relative strengths and limitations, particularly highlighting the competitiveness of simpler methods under the considered data constraints.

  3. The benchmark brings together four datasets covering different nuclear and nuclear-adjacent systems, physical processes, and modeling fidelities. This diversity provides a valuable foundation for future research on machine learning for complex multiphysics systems.

Critical weaknesses
  1. Insufficient cross-dataset evaluation. Although the proposed benchmark includes four datasets covering different nuclear systems and physical processes, comprehensive evaluations of machine learning methods are conducted primarily on the MSFR dataset. The other three datasets (HPMR, DYNASTY, and MHD) are evaluated only with a few simple baselines. Since one of the main motivations of the benchmark is to provide a unified evaluation framework across diverse nuclear systems, the limited experimental coverage weakens this contribution. I would expect more comprehensive comparisons across multiple datasets to demonstrate the generality of the framework and reveal how different methods perform under different physical regimes.

  2. Insufficient justification of the aggregate scoring scheme. The paper ranks methods by taking the arithmetic mean of twelve evaluation scores, but the rationale behind this aggregation strategy is not sufficiently discussed. In particular, the four task categories contain different numbers of metrics, implicitly assigning different weights to forecasting, noise robustness, limited-data performance, and parametric generalization. Furthermore, a high average score may conceal substantial failures on individual tasks, which is particularly concerning for safety-critical nuclear applications. The authors should provide a stronger justification for the scoring scheme and investigate whether the model rankings remain consistent under alternative aggregation strategies, such as equal weighting across task categories or greater emphasis on critical tasks.

  3. Insufficient experimental validation of the proposed sparse-observation task. The introduction of sparse-observation state estimation is presented as an important contribution of the benchmark. However, its experimental evaluation is relatively limited, relying primarily on the Empirical Interpolation Method (EIM) to investigate the effects of sensor number and measurement type. There is a lack of systematic comparisons with other relevant approaches, particularly learning-based state reconstruction and data assimilation methods. Since this task is intended to reflect realistic monitoring scenarios in nuclear systems, more comprehensive experiments involving different models, sensor configurations, and observation noise levels would help demonstrate the usefulness and discriminative capability of the proposed evaluation paradigm.

  4. Questions regarding the benchmark difficulty and discriminative capability. An interesting but somewhat concerning observation is that the simple last-value baseline achieves the second-highest average score, outperforming most of the evaluated machine learning methods, including several neural operators and foundation models. While this may reflect the slowly evolving dynamics of the MSFR system, it also raises questions about whether the current tasks and scoring metrics are sufficiently challenging and discriminative. Alternatively, the result may indicate that the evaluated methods are not adequately adapted or optimized for the benchmark. The authors should provide a more detailed analysis to distinguish these possibilities. In particular, comparisons across different prediction horizons, more challenging transient regimes, and stronger task-specific baselines would help clarify whether the observed ranking reflects the inherent difficulty of the benchmark or limitations of the evaluated methods.

Questions for authors

See above.

1 comment

We thank the reviewer for the careful review. We have uploaded a revised PDF; page numbers below refer to it.

W1 (cross-dataset evaluation). We ran four methods representative of different families on DYNASTY and HPMR with the MSFR protocol. The results are preliminary and may be updated as further runs complete, and the remaining methods are not yet included; full E1–E12 scores are in Appendix C.6 (Table 7, p. 35). Average score over the twelve tasks:

MethodDYNASTYHPMR
Reservoir91.459.9
Panda86.72.8
Last-value baseline80.8−49.0
Average baseline77.2−35.6
Opt DMD72.8−2.4
PyKoopman61.536.6
Zero baseline0.0−38.9

On HPMR all four methods improve on the last-value baseline; on DYNASTY only Reservoir and Panda do, and PyKoopman, first on the MSFR, reaches the minimum score on one task. The ranking does not transfer across systems (Section 3.1, p. 8; Limitations, p. 10), which is expected given the different physical characteristics and fidelity of the systems. MHD cannot be evaluated within the discussion period because of its computational cost and will be included in the final version.

W2 (aggregation). The composite score is an unweighted mean meant as an overview; the paper says so, and task-group averages and the number of tasks at −100 of each method are reported (Appendix C.1, Table 3, p. 30), so that failures are not hidden in the mean and readers can weight the groups according to their application. We recomputed the ranking with equal weight per group, with −100 replaced by 0, and with the median (Appendix C.2, p. 29): the ordering is stable, but the first position is not. PyKoopman is first with the original score, the last-value baseline with equal group weights (68.11 versus 66.01), Reservoir with −100→0 or the median. Further checks (interquartile mean, win rate, geometric mean of the errors) show that the last-value baseline ranks between 1st and 5th across aggregators and is never the best method on a single task. Removing the denoising group changes the ordering far more than removing any other group, because of the −100 of the foundation models on E3. Ultimately, the aggregation method is meant to be a summary to show the versatility of a method across tasks, not a definitive ranking; users should look at the individual scores to select a method for their application (Appendix C.2, p. 29).

W3 (sparse-observation task). Besides EIM we compare Gappy POD, POD-DeepONet, an echo state network, an LSTM and SHRED, with the same pre-computed temperature measurements (first 5, 10 or 20 sensors of an EIM-greedy batch and of a random batch, both released with the data; noisy at test time for the noisy tasks, whose noise level is not varied in this study), one configuration per method, trained on the released training data, three seeds when stochastic. Average score over the twelve tasks, with EIM-greedy sensors and 5 to 20 sensors: EIM 32.9 to 64.0, the five other estimators 52–65 to 65–74; with random sensors EIM is competitive with few sensors (54.6 against 35–55 with 5 sensors) but does not improve with more (54.6 with 20 sensors against 57–65), so placement matters as much as the estimator. POD-DeepONet, ESN and SHRED are within seed variability of one another (up to 6 points). On DYNASTY (four experimental thermocouples, 2 to 4 sensors) all methods score between 79 and 96, since the transients are dominated by slow trends. Full tables are in Appendix C.4 (Tables 4 and 5, p. 33; Section 3.2, p. 9). A broader study of placement across datasets is stated as a limitation (p. 10).

W4 (difficulty). (i) The strength of persistence on the MSFR has a physical origin stated in the paper (Section 3.1, p. 8): the RANS model filters high-frequency fluctuations. (ii) The benchmark discriminates once the physics changes: the last-value baseline scores 80.8 on DYNASTY, −26.9 on MHD and −49.0 on HPMR (Appendix C.5, Table 6, p. 34), and the runs of W1 show that the margin over persistence depends on the system. (iii) The horizon matters: on the MSFR the short-time scores of PyKoopman and the last-value baseline are close (87.5 versus 81.2) but the long-time ordering reverses (31.1 versus 71.8), while Reservoir is best on both (92.4, 87.4). (iv) The result is not due to untuned methods: each method is tuned for each task on the released training data (search spaces in Appendix D), and the best-performing methods are task-adapted dynamical-system methods (Koopman, reservoir) rather than generic foundation models. PyKoopman improves on persistence on eight of the twelve tasks and is the only method with a positive mean margin (+8.29), whereas Moirai-2 has a margin of −15.43 because of two outright failures (Appendix C.1, Table 3, pp. 29–30).

Reviewer_b7X6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CTF4Nuclear, a common task framework for evaluating machine learning methods in nuclear systems. The framework provides four datasets from different nuclear systems and 12 evaluation metrics covering forecasting, noise robustness, limited training data, and parameter generalization. The authors also introduce a new task to reconstruct the full system state from sparse sensor observations.

To validate the framework, the authors compare various machine learning methods, including traditional regression methods, deep learning models, and foundation models, mainly on the MSFR dataset. The results show that PyKoopman achieves the best average score, while a simple last-value baseline also performs well. The authors further evaluate sparse state estimation using the Empirical Interpolation Method (EIM). For the other three datasets, only basic baseline results are provided.

Critical strengths

(I) The authors collect and release four datasets covering different nuclear systems and simulation settings. Since publicly available datasets are limited in this field, these datasets can be useful for future research and comparisons of machine learning methods.

(II) The authors provide a common evaluation framework with 12 metrics and several baseline methods. In particular, the simple last-value baseline outperforms many complex machine learning models on the MSFR dataset. This result shows the importance of using appropriate baselines and consistent evaluation protocols for fair comparisons.

Critical weaknesses

(I) Although the authors provide four datasets, the comprehensive comparison of machine learning methods is conducted only on the MSFR dataset. The other three datasets are evaluated only with simple baselines. Also, the new sparse state estimation task is evaluated only with EIM. Therefore, it is difficult to understand how useful the proposed framework is across different nuclear systems and machine learning methods.

(II) The four datasets are generated using numerical simulations rather than real measurements. Although this is understandable because real nuclear data is difficult to obtain, it is unclear how well the benchmark reflects real operating conditions. This limits the practical significance of the evaluation results.

Questions for authors

(I) The authors provide four datasets, but the detailed comparison of machine learning methods is conducted only on MSFR. Why were the other three datasets evaluated only with simple baselines? Are there any results with representative machine learning methods on these datasets?

(II) The proposed sparse state estimation task is evaluated only with EIM. Why are other machine learning methods not included in this evaluation? How can we assess the usefulness of this new benchmark task without comparing different methods?

(III) All four datasets are based on numerical simulations. Were the simulation data or the model predictions validated using real experimental measurements? If not, how can we assess whether the benchmark reflects real operating conditions?

2 comments

We thank the reviewer for the review and for recognising the value of the datasets and of the sparse-observation task. We have uploaded a revised PDF; page numbers below refer to it.

W1 (breadth of the evaluation).

Other datasets (Q1). The first release gave reference baselines on the remaining datasets to establish the evaluation pipeline on one dataset, since tuning every method on the twelve tasks of each dataset is the dominant cost. We now ran four methods representative of different families on DYNASTY and HPMR with the MSFR protocol (tuning on the released training data only). The results are preliminary and may be updated as further runs complete, and the remaining methods are not yet included; full E1–E12 scores are in Appendix C.6 (Table 7, p. 35). Average score over the twelve tasks:

MethodDYNASTYHPMR
Reservoir91.459.9
Panda86.72.8
Last-value baseline80.8−49.0
Average baseline77.2−35.6
Opt DMD72.8−2.4
PyKoopman61.536.6
Zero baseline0.0−38.9

On HPMR (chaotic) all four methods improve on the last-value baseline; on DYNASTY only Reservoir and Panda do, and PyKoopman, first on the MSFR, reaches the minimum score on one task. The ranking therefore differs from the MSFR (Section 3.1, p. 8; Limitations, p. 10). MHD cannot be evaluated within the discussion period because of its computational cost and will be included in the final version.

Sparse-observation task (Q2). Besides EIM we compare Gappy POD, POD-DeepONet, an echo state network (ESN), an LSTM and SHRED, with the same pre-computed temperature measurements (first 5, 10 or 20 sensors of an EIM-greedy batch and of a random batch, both released with the data), one configuration per method, trained on the released training data, three seeds when stochastic. Average score over the twelve tasks (mean ± std over seeds; windowed methods read past measurements):

MethodEIM-greedy, 5EIM-greedy, 10EIM-greedy, 20Random, 5Random, 10Random, 20
EIM32.948.364.054.659.154.6
Gappy POD52.554.267.735.145.654.1
POD-DeepONet64.7 ± 1.172.3 ± 0.873.9 ± 0.947.5 ± 0.550.7 ± 5.864.5 ± 2.8
ESN (windowed)64.3 ± 1.266.1 ± 0.170.7 ± 0.250.5 ± 1.952.8 ± 0.762.5 ± 1.0
LSTM (windowed)55.5 ± 2.557.8 ± 5.964.7 ± 1.250.5 ± 2.950.6 ± 1.157.2 ± 3.8
SHRED (windowed)63.8 ± 3.366.1 ± 1.569.5 ± 3.054.8 ± 2.053.8 ± 6.064.2 ± 1.5

With EIM-greedy sensors EIM is far behind with 5 sensors and catches up at 20; with random sensors it is competitive with few sensors but does not improve with more, so placement matters as much as the estimator. POD-DeepONet, ESN and SHRED are within seed variability of one another. On DYNASTY (four experimental thermocouples, 2 to 4 sensors) all methods score between 79 and 96, since the transients are dominated by slow trends, and only EIM degrades markedly with two sensors (Appendix C.4, pp. 32–33; Section 3.2, p. 9). MHD is not evaluated for the same reason as above, while HPMR is an ODE system and state reconstruction has low engineering interest.

W2, Q3 (simulated data). Public measured datasets covering the twelve tasks are rare or unavailable for these systems, so the benchmark is simulation-based. The simulators span the range of fidelity used in nuclear engineering (Introduction; Limitations, Section 4, p. 10): HPMR is a lumped point-kinetics model of the kind used for control studies; DYNASTY is simulated with RELAP5, a system code widely used for reactor licensing, whose model of the facility is validated against its experiments (maximum RMSE 7.25 °C; Missaglia et al., 2025); MSFR and MHD use multi-physics finite-volume solvers, increasingly adopted in the community, for which no measurements exist (the MSFR has not been built); the MSFR solver has been cross-verified against other codes (Tiberga et al., 2020) and the MHD solver against other solvers and analytical solutions. The transients are deliberately more challenging than normal operation, since nuclear reactors are designed to operate in stable regimes. Validation against real measurements is therefore possible only for DYNASTY, and the scores are relative measures under a common protocol, not predictions for physical plants.

Thank you for your detailed response and additional experiments.

The new results on DYNASTY and HPMR show that different machine learning methods perform differently across the datasets. Also, the additional experiments on sparse state estimation address my main concern about the limited evaluation.

I also understand that obtaining real measurement data is difficult in nuclear systems. Your explanation about the validation of the simulators is reasonable.

Overall, your response addresses my main concerns. Therefore, I have decided to change my rating from Weak Rejection to Weak Acceptance.

Reviewer_ENecOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces CTF4Nuclear, a common task framework for evaluating data-driven modelling methods in nuclear engineering. It collects four datasets generated from simulations which span different physical systems - HPMR, DYNASTY, MHD Channel flow and MSFR. Authors use different evaluation scores covering forecasting, noise robustness, limited data, and parametric generalization, and introduce a new sparse-observation state-estimation setting. The main result evaluates a broad selection of methods on the molten salt fast reactor (MSFR) dataset, while results on the other 3 datasets are present in the appendix section and cover the baselines only namely - zero prediction, average, last value.

Critical strengths
  1. The motivation of the paper is sound. Application domains like this do need systematic evaluations of ML based methods in nuclear engineering.
  2. Dataset covers different physically distinct systems and multiple modelling fidelities.
  3. Sparse-observation state estimation is a good addition, connecting model evaluation to a monitoring problem.
  4. The authors also acknowledge several important limitations, including simulator assumptions, synthetic noise, and the scope of the sparse-observation study.
Critical weaknesses
  1. Simulated data and reliability: The benchmark relies on simulated data. While the paper acknowledges modelling assumptions, it should more clear in explaining the evidence supporting the selected simulation configurations and regimes, and explain the resulting limits on interpreting benchmark performance.
  2. Limited Evaluations and information : Full results are only provided for the MSFR dataset, while only simple baselines are provided for other 3 datasets. Moreover, there is no information about standard deviation of models for each test, independent run counts, random seeds, algorithm tuning details. While we see SD being reported across the metrics, there is no information about runs in each metric calculation.
  3. Paper writing : The main text would benefit from a compact specification of dataset sizes, task inputs and outputs, evaluation horizons, and score aggregation. These details are central to assessing a benchmark contribution.
  4. Failure cases : The −100 penalty can substantially affect aggregate rankings. The paper mentions memory and training-time limitations for some methods, it does not provide a complete breakdown of failures by method and task. Please justify the penalty and specify resource limits and failure-handling rules.
Questions for authors
  1. Is there a difference in observations between forecasting methods and reconstruction methods ? Do reconstruction methods take in present-time noisy information ?
  2. What are the metrics standard deviations/errors for each task ? Are they well separated or do the results overlap ?
  3. Could you report forecasting and reconstruction summaries separately, alongside failure counts, and specify the hardware, resource limits, and hyperparameter-selection budgets?
  4. Can you report broader results for the other 3 tasks as well ?
  5. A compact dataset-and-task specification table would improve reproducibility: trajectory counts, snapshot counts, sampling intervals, forecast horizons, noise levels, parameter regimes, and normalization conventions. Per-field errors and representative trajectories would also help connect the aggregate scores to physical behaviour.
1 comment

We thank the reviewer for the detailed review. We have uploaded a revised PDF; page numbers below refer to it.

W1 (simulated data). Experimental datasets for these systems are rare, unavailable or not allowed to be shared, so the benchmark is simulation-based. As stated in the introduction, the simulators adopted spread over the spectrum of fidelity usually used in nuclear engineering: HPMR is a lumped model mostly adopted for control studies, DYNASTY is simulated with RELAP5 (validated against experimental data in Missaglia et al., 2025) and this code is widely adopted for licensing of reactors from the thermal-hydraulic perspective, MHD and MSFR adopt multi-physics finite-volume solvers, which are increasingly used in the community; the MSFR solver has been cross-verified against other codes (Tiberga et al., 2020) and the MHD solver against other solvers and analytical solutions (Limitations, Section 4, p. 10). Regarding regimes, the operating conditions are chosen to be more challenging for the ML evaluation than what is usually encountered in reality, since nuclear reactors are designed to operate in a stable regime. The scores therefore measure the relative ability of methods to learn simulated dynamics under a common protocol, not their performance on physical plants.

W2 (runs, tuning, other datasets). The "±" in the Average column is the standard deviation across the twelve tasks, not across runs; the caption now makes this explicit (Table 2, p. 8). Scores come from one tuned configuration per method and task on the released training data (search spaces in Appendix D); per-method seed variability, hardware and tuning cost are not reported in the current version. On the other datasets (HPMR and DYNASTY) we report preliminary results for four methods, which may be updated as further runs complete (Appendix C.6, Table 7, p. 35). MHD cannot be evaluated within the discussion period because of its computational cost and will be included in the final version.

W3, Q1, Q5 (specification, inputs). Table 1 (p. 5) lists for each dataset model, fields, state dimension, sampling interval, training snapshots, horizon, scoring windows and noise levels; the aggregation (unweighted mean, scores clipped to [−100, 100]) is stated in the metrics subsection (p. 5). Forecasting methods receive the training matrix (clean or noisy); the denoising tasks (E3, E5) take the noisy window as input and the clean one as target; the sparse-observation task receives only the sensor values at the instant to be reconstructed (noisy for the noisy tasks), so its scores are not comparable with the forecasting table (p. 7). Per-field errors and representative trajectories are not included in this revision: the scores are defined on the full state, and animated previews of every dataset are in the released repository.

W4, Q3 (−100 penalty). −100 is the lower bound of the score range: it is assigned either to a diverging prediction or to a run that fails (mainly out of memory); the metrics subsection and the table caption now say so (pp. 5, 8). This is a deliberate choice: a method that diverges or cannot produce a result is not usable for that task, and the composite score should reflect it. To keep failures visible rather than hidden in the average, the per-task scores of Table 2 (p. 8) show every method and task at −100, and Table 3 (p. 30) reports the average of each task group and the number of tasks at −100. Resource limits of the runs are not reported in this revision. We also checked that the conclusions do not hinge on this convention: with −100 replaced by 0, with equal weight per task group, or with the median instead of the mean, the overall ordering changes little, but the first position moves among PyKoopman, Reservoir and the last-value baseline (Appendix C.2, p. 29). We therefore present these three as a group of comparable methods rather than as a strict ranking.

Q2 (separation of results). The "±" in Table 2 is the spread across tasks, so it does not measure how well methods are separated, and per-task run-to-run deviations are not available for the forecasting methods (single tuned run per task, see W2). To check this, we resampled the twelve tasks with replacement (bootstrap, Appendix C.2, p. 29): PyKoopman is first in 55% of the resamples, Reservoir in 25% and the last-value baseline in 11%, while the other methods are clearly below. The top three are thus not separated, consistently with W4. For the sparse-observation estimators, run with three seeds, the seed-to-seed standard deviation of the average score is between 0.1 and 6 points (Appendix C.4, Tables 4 and 5, p. 33).

Comments on the paper

3 comments
AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

We thank the reviewers for their constructive feedback. In our revised manuscript, we have added representative method comparisons on HPMR and DYNASTY, expanded the sparse-observation evaluation beyond EIM, and analysed sensitivity to the aggregation scheme.

Before the discussion period closes, we would be grateful if you could review our responses and share any remaining concerns or points requiring clarification.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Summary of the revision

As the discussion period is approaching its end, we wanted to summarise how the revised manuscript addresses the concerns, and we report the complete results on DYNASTY and HPMR, just finalised.

Main concerns and changes

  • Evaluation beyond the MSFR (all reviewers). Methods are now run on DYNASTY and HPMR with the MSFR protocol, and the complete results are reported below. MHD (122400 variables) is in the process of being evaluated and will be included in the final version; due to its high computational cost, it could not be completed during the rebuttal period.
  • Sparse-observation task beyond EIM (b7X6, zNwh). Gappy POD, POD-DeepONet, ESN, LSTM and SHRED are compared with EIM on the MSFR (EIM-greedy and random sensors, 5 to 20 sensors) and on DYNASTY (the four experimental thermocouples): Section 3.2 and Appendix C.4.
  • Aggregation and the −100 penalty (ENec, zNwh). Task-group averages and failure counts (Table 3), alternative aggregations and a bootstrap over the tasks (Appendix C.2): the overall ordering is stable, while the first position is shared among PyKoopman, Reservoir and the last-value baseline.
  • Specification and reproducibility (ENec). Dataset and task table (Table 1), clarified captions and inputs, and release of the sensor measurements and of the state-estimation code.
  • Simulated data (ENec, b7X6). Evidence behind each simulator and limits of interpretation in the Limitations (Section 4).

The scores of all methods on the two datasets are also released in the anonymous repository, and the tables below will replace Table 7 in the final version, together with the MHD results. Finally, we highlight the following observations:

  1. The ranking is physics/fidelity-dependent. Reservoir is first on both datasets (third on the MSFR), whereas PyKoopman, first on the MSFR, is twelfth of nineteen methods on DYNASTY and third on HPMR; conversely, NeuralODE is third on DYNASTY but close to the last-value baseline on HPMR.
  2. The benchmark discriminates once the physics/fidelity changes. On DYNASTY, whose transients are slow and nearly monotonic, only 4 of the 19 methods exceed the last-value baseline (80.8); on HPMR, whose reactivity forcing is chaotic, all methods that produce a result exceed it (−49.0) and 15 of 19 exceed the zero prediction. Together with the MSFR, this confirms that the strength of persistence is a property of the system, not of the metrics.
  3. Time-series foundation models, which mostly failed on the high-dimensional MSFR, can run on these low-dimensional states and are competitive on DYNASTY (Panda is second); TabPFN on both datasets and LLMTime on DYNASTY did not produce a result and are assigned −100 on every task.
AuthorsOpenReview ↗

Complete results on DYNASTY and HPMR

Below we report the complete results on DYNASTY and HPMR, with the same metrics as in the MSFR: average score over the twelve tasks and average of each task group; #(−100) is the number of tasks at the minimum score; baselines in italics.

DYNASTY

MethodAvgForecastNoisyLimitedParam.#(−100)
Reservoir91.495.595.982.995.50
Panda86.894.782.184.692.40
NeuralODE83.484.885.981.281.30
LSTM82.491.473.980.194.90
Last-value baseline80.887.880.577.581.10
SINDy78.887.870.379.785.00
Average baseline77.273.672.979.285.20
ODE-LSTM77.155.890.680.465.00
Sundial76.489.655.980.696.00
Chronos-275.691.252.680.396.70
Chronos-T574.492.654.079.886.20
Opt DMD72.889.976.277.439.60
Moirai-270.486.050.375.185.90
PyKoopman61.593.924.779.367.11
DeepONet58.074.446.179.822.01
FNO53.0−4.090.728.583.92
KAN49.482.378.5−9.075.32
Moirai-MoE38.832.71.360.276.91
Spacetime15.1−9.3−9.019.080.14
Zero baseline0.00.00.00.00.00
TabPFN−100.0−100.0−100.0−100.0−100.012
LLMTime−100.0−100.0−100.0−100.0−100.012

HPMR

MethodAvgForecastNoisyLimitedParam.#(−100)
Reservoir59.995.162.621.296.80
ODE-LSTM43.260.270.7−18.394.30
PyKoopman36.671.047.750.6−48.20
LSTM32.474.941.2−28.794.50
Sundial25.739.5−4.428.267.00
FNO11.557.97.00.3−3.50
DeepONet11.1−43.820.64.160.80
Chronos-23.518.8−77.249.358.42
Panda2.826.8−33.9−4.466.70
Chronos-T52.75.1−58.223.081.31
SINDy−1.863.7−54.26.021.91
Opt DMD−2.4−43.7−22.133.36.90
KAN−31.8−53.20.1−70.93.72
Average baseline−35.6−46.5−45.5−48.020.00
LLMTime−37.0−17.3−96.6−28.545.32
Moirai-2−37.3−37.3−90.0−18.129.92
Zero baseline−38.9−46.5−45.5−48.00.00
Moirai-MoE−41.4−43.3−96.8−26.842.12
Spacetime−47.3−9.3−51.0−100.027.35
NeuralODE−48.1−85.9−69.1−36.58.70
Last-value baseline−49.0−66.4−72.0−46.18.51
TabPFN−100.0−100.0−100.0−100.0−100.012

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.