acceptodds
Under review as a conference paper at ICLR 2027

TokaReconBench: A Benchmark for Transferable AI in Tokamak Equilibrium Reconstruction

Abstract

Developing reliable and transferable AI models for plasma equilibrium reconstruction is essential yet challenging for tokamak fusion reactors. Existing AI-based equilibrium reconstruction methods have shown promising performance as fast surrogates for traditional solvers. However, their predominantly in-distribution evaluation provides limited evidence of reliability under domain and device shifts. To address this limitation, we introduce TokaReconBench, a benchmark for transferable tokamak equilibrium reconstruction. It integrates heterogeneous experimental and synthetic datasets, including public MAST measurements, ITER-like simulations, and our newly contributed, configuration-aligned EXL-50U simulation and experimental data. We define unified task formulations and evaluation protocols covering three levels of generalization: simulation-to-simulation (sim2sim), simulation-to-real (sim2real) transfer, and cross-device generalization. We benchmark representative neural surrogate models, operator learning approaches, and physics-based baselines under diverse training loss settings. Metrics assess field accuracy, geometry, and physics consistency against reference source terms. Experiments reveal significant bottlenecks in cross-domain and cross-device transfer for existing learning-based reconstruction models. We hope TokaReconBench provides a unified platform for developing transferable AI methods towards future multi-device fusion applications. Our benchmark, datasets, and evaluation protocols are available upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Scientific Machine Learning, AI for Science, Fusion Energy, Tokamak Equilibrium Reconstruction, Benchmark, Sim-to-Real Transfer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_77kfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces TokaReconBench, a benchmark that targets transferability of tokamak equilibrium reconstruction. Compared to existing benchmarks, TokaReconBench takes magnetics as reconstruction main inputs, supporting simulation-to-real transferring on the same device, cross-device evaluation, and target-device supervision. For cross-domain evaluation (e.g., from ITER-like simulation to EXL-50U simulation), fine-tuning on target domain is utilized to establish transferability.

Critical strengths
  1. The authors promised to release their datasets, which can be valuable for tokamak fusion research.
  2. This work is easy to follow, avoiding overly technical details in fusion science.
Critical weaknesses
  1. Insufficient ablation. For tasks T4 and T5, the usefulness of pre-training is not justified. The transferability may come solely from the 5% in-domain fine-tuning.
  2. The significance of this work is not fully justified. The fundamental improvement from the three earlier benchmarks in Table 1 should be discussed.
  3. This work relies on existing models and does not include model design specifically for the task. Despite the loss functions are designed for the task, they do not show convincing improvement in Table 5.
Questions for authors
  1. For T4 and T5, could you remove the pre-training phase while keeping other settings the same?
  2. Could you justify the fundamental differences between TokaReconBench and the three previous benchmarks? Can these benchmarks be easily (like simply asking an AI agent to do the job) modified or extended to support magnetics inputs, same-device sim2real, cross-device evaluation, and target-device supervision?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 77kf (1/3)

We thank the reviewer for the thoughtful review. We appreciate the reviewer’s positive assessment of the manuscript’s clear and accessible presentation, and the recognition that the release of our datasets could provide a valuable resource for the tokamak fusion research community.

Concern 1

Weakness1: Insufficient ablation. For task T4\mathbb{T}_4 and task T5\mathbb{T}_5, the usefulness of pre-training is not justified. The transferability may come solely from the 5% in-domain fine-tuning.

Question1: For task T4\mathbb{T}_4 and task T5\mathbb{T}_5, could you remove the pre-training phase while keeping other settings the same?

Reply:

We thank the reviewer for this constructive suggestion. We agree that directly comparing source domain pretraining followed by fine-tuning against training from scratch on the same target domain data helps clarify the exact contribution of pretraining versus target domain supervision.

To clarify this, we conducted a new control experiment during the rebuttal period, covering task T4\mathbb{T}_4 (ITER simulation to EXL-50U simulation) and task T5\mathbb{T}_5 (MAST experiment to EXL-50U experiment).

Taking the U-Net architecture described in Appendix E.1 as an example, we compared source pretrained models fine-tuned on 5% of the target domain data against randomly initialized models trained from scratch on the exact same 5% target domain subset. The results are reported in following Table 9.

Table 9. Comparison between source pretrained fine-tuning and training from scratch, both using 5% target domain data, on cross-device task T4\mathbb{T}_4 and task T5\mathbb{T}_5 with U-Net. Both strategies use the same target-training subset and preprocessing. Bold values indicate the better result within each task.

TaskTraining strategyEψRMSEE_{\psi}^{\mathrm{RMSE}} (Wb/rad) ↓\downarrowEψrelE_{\psi}^{\mathrm{rel}} ↓\downarrowConditional mean ELCFS95E_{\mathrm{LCFS}}^{95} (cm) ↓\downarrowLCFS extraction success rate ↑\uparrow
task T4\mathbb{T}_4: ITER Sim. →\to EXL-50U Sim.Source pretrained + 5% target fine-tuning0.0015840.076675.2960,206 / 60,218 (99.98%)
task T4\mathbb{T}_4: ITER Sim. →\to EXL-50U Sim.Random init. + 5% target from scratch0.0015970.077225.1259,781 / 60,218 (99.27%)
task T5\mathbb{T}_5: MAST Exp. →\to EXL-50U Exp.Source pretrained + 5% target fine-tuning0.0087020.4463723.8530,041 / 31,849 (94.32%)
task T5\mathbb{T}_5: MAST Exp. →\to EXL-50U Exp.Random init. + 5% target from scratch0.0078090.4211319.7329,987 / 31,849 (94.15%)

The experimental results show that in task T4\mathbb{T}_4, the field reconstruction errors of the two strategies are very close, differing by less than 1%. In task T5\mathbb{T}_5, training from scratch reduces the relative flux error (EψrelE_{\psi}^{\mathrm{rel}}) from 0.44637 to 0.42113 (a 5.65% reduction) and reduces the conditional mean LCFS 95th percentile Hausdorff distance (ELCFS95E_{\mathrm{LCFS}}^{95}) from 23.85 cm to 19.73 cm (a 17.27% reduction). Thus, in this U-Net control experiment, source-domain pretraining followed by target-domain fine-tuning did not provide a consistent advantage over training from scratch with the same target-data budget.

This behavior may be related to the substantial differences in physical scales and internal electromagnetic environments across tokamak devices [Casper et al., 2014; Shi et al., 2025]. For example, in sampled training data for task T4\mathbb{T}_4, the median reference plasma current is 12.41 MA for ITER-like simulation and 0.417 MA for EXL-50U simulation; the corresponding median centered flux RMS values are approximately 4.25 and 0.02149 Wb/rad, respectively, differing by more than two orders of magnitude.

Although we applied dimensionless normalization and data standardization during preprocessing to remove unit inconsistencies, fundamental differences in vessel geometry, poloidal field coil layouts, and magnetic topology make it difficult for priors learned on the source device to generalize directly.

This finding is also consistent with our discussion in Section 3.3 (lines 430 to 431 and lines 463 to 466) of the original manuscript regarding simulation to real and cross-device transfer: the tested two-stage pretraining and fine-tuning recipe remains insufficient to bridge the physical distribution gap across devices.

We have included these experimental results and the physical mechanism analysis in Appendix G.

We will extend these experiments to additional architectures (such as FNO) and include the results in the final version of the paper.

(Continued in Part 2.)

AuthorsOpenReview ↗

Response to Reviewer 77kf (2/3)

(Continued from Part 1.)

Concern 2

Weakness2: The significance of this work is not fully justified. The fundamental improvement from the three earlier benchmarks in Table 1 should be discussed.

Question2: Could you justify the fundamental differences between TokaReconBench and the three previous benchmarks? Can these benchmarks be easily (like simply asking an AI agent to do the job) modified or extended to support magnetics inputs, same-device sim2real, cross-device evaluation, and target-device supervision?

Reply:

We thank the reviewer for raising this question.

In the Related Work section of the original manuscript, lines 730 to 740 in Appendix A, we provided an overview of the distinctions between the three existing related benchmarks and our work.

In general, a scientific benchmark generally comprises three core elements:

  • Dataset: In TokaReconBench, this encompasses public MAST experimental data, ITER like simulation data, and our newly contributed EXL-50U simulation and experimental datasets (Section 2.3).

  • Research problem: Predicting the internal poloidal flux distribution and magnetic geometry directly from external magnetic diagnostic signals (Section 2.1).

  • Evaluation tasks: Five train and test domain pairs designed to systematically evaluate same device generalization, simulation to real transfer, and cross-device transfer (Section 2.2).

Among these three elements, paired multi regime data serves as the essential foundation that enables the problem formulation and evaluation tasks. By contributing configuration aligned simulation and experimental datasets for EXL-50U and pairing them with MAST experimental data and ITER like simulation data, TokaReconBench supports all five evaluation tasks, including same device simulation to real transfer and cross-device transfer under limited target supervision. Conversely, the absence of certain data modalities in earlier benchmarks imposes fundamental limitations that cannot be resolved by simple code modifications or model development:

  • TokaMark [Rousseau et al., 2026] contains only experimental data from a single device (MAST). Because it covers only one device, cross-device equilibrium reconstruction cannot be evaluated. Moreover, even if combined with external ITER-like simulation data, it lacks paired simulation data on MAST and therefore cannot support same device simulation to real evaluation (task T3\mathbb{T}_3 in our benchmark).

  • FusionEQ [Nakkina et al., 2026] includes experimental data only for DIII-D and MAST without corresponding simulation datasets, preventing simulation to real evaluation. Furthermore, FusionEQ uses non magnetic signals, such as coil currents and Thomson scattering profiles, as inputs to infer equilibria for future reactor scenarios. Because of these differences in input modalities and problem formulation, FusionEQ addresses a fundamentally different conditional inference task. In contrast, TokaReconBench targets the primary operational requirement in existing tokamak facilities: real time equilibrium reconstruction driven by external magnetic diagnostics.

  • CrossGS [Yoo et al., 2026] differs from our work even more fundamentally. First, its inputs are not magnetic diagnostics or experimental measurements, but rather four equilibrium parameters together with the two spatial coordinates (R,Z)(R,Z), forming six input channels. The model learns to evaluate the forward mapping from known parameters to ψ(R,Z)\psi(R, Z), rather than solving the inverse equilibrium reconstruction problem from external sensors. Second, CrossGS relies exclusively on analytic equilibria without experimental discharges, making simulation to real evaluation impossible. Third, all devices in CrossGS share the exact same six dimensional parameter space, reducing cross-device transfer to degenerated covariate shift problem within a homogeneous input space. In contrast, cross-device tasks in TokaReconBench reflect realistic deployment conditions where sensor counts, spatial locations, orientations, coil layouts, and output grids differ substantially across devices, requiring transfer across heterogeneous domains.

In summary, TokaReconBench differs fundamentally from these three earlier benchmarks in data availability, inverse problem formulation, and cross-domain evaluation scope. Because existing benchmarks lack paired simulation and experimental datasets on the same device (such as EXL-50U) along with standardized external magnetic diagnostics across multiple machines, they cannot support this comprehensive evaluation of equilibrium reconstruction generalization. This is an intrinsic data and problem formulation barrier that cannot be overcome simply by using an AI agent.

The detailed comparative analysis is located in Appendix A.

(Continued in Part 3.)

AuthorsOpenReview ↗

Response to Reviewer 77kf (3/3)

(Continued from Part 2.)

Concern 3

Weakness3: This work relies on existing models and does not include model design specifically for the task. Despite the loss functions are designed for the task, they do not show convincing improvement in Table 5.

Reply:

We thank the reviewer for sharing this perspective. Consistent with established benchmark papers in AI for Science, the primary goal of TokaReconBench is not to propose a new model architecture or training loss, but to release standardized, configuration aligned datasets (including paired simulation and experimental data on EXL-50U), define realistic multi regime evaluation protocols, and establish rigorous baselines across representative architectures and losses. Through this systematic evaluation, we uncover critical bottlenecks, such as the lack of consistent cross-device pretraining gains and the task- and architecture-dependent effects of physics-informed auxiliary losses, that would otherwise remain hidden. We believe exposing these gaps provides essential guidance for the field, and designing specialized architectures and transfer algorithms to overcome them is an important direction for future work.

References

[Casper et al., 2014] Development of the ITER baseline inductive scenario. Nuclear Fusion 54, 013005. https://doi.org/10.1088/0029-5515/54/1/013005

[Shi et al., 2025] Strategy and experimental progress of the EXL-50U spherical torus in support of the EHL-2 project. Plasma Science and Technology 27, 024003. https://doi.org/10.1088/2058-6272/ad9e8f

[Rousseau et al., 2026] TokaMark: A Comprehensive Benchmark for MAST Tokamak Plasma Models. arXiv:2602.10132. https://arxiv.org/abs/2602.10132

[Nakkina et al., 2026] The Fusion Equilibrium Challenge: Inferring Magnetic Geometry Without Magnetic Diagnostics. arXiv:2609.01750. https://arxiv.org/abs/2609.01750

[Yoo et al., 2026] Towards Data-Efficient Cross-Device Generalization of Grad-Shafranov Equilibria via Transfer Learning Neural Operator. arXiv:2606.15512. https://arxiv.org/abs/2606.15512

Reviewer_BQALOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces TokaReconBench, a benchmark for magnetics-based tokamak plasma equilibrium reconstruction with an explicit focus on transferability. The task is to predict the poloidal flux field on each device’s native grid from magnetic diagnostics, coil currents, and optional device geometry.

The benchmark defines five tasks along a device and data-domain grid: in-domain sim2sim on ITER-like and EXL-50U simulation, zero-shot same-device sim2real on EXL-50U, and two cross-device transfers with 5% target fine-tuning.

Validation consists of two studies: eight architectures (U-Net, a geometry-attention Transformer, PlaNet, DeepONet, FNO, SetONet, GINO, GNOT) trained with a default field+gradient loss across all five tasks, and a loss-combination study (fixed weight) on three architectures adding probe-consistency and GS-residual terms.

The paper has valuable data contribution and genuinely useful task design. But the empirical layer does not currently support some of its conclusions, and there are hard inconsistencies.

Critical strengths

The paper provides a good task and metrics design for the Tokamak equilibrium reconstruction. Several discussions for loss term are thoughtful in a way that benchmark papers in scientific ML usually are not. For the main result table (Table.4), the study also lists comprehensive comparison across 8 different model architectures, each with different performance evaluations.

The configuration-aligned EXL-50U simulation/experiment pair is a genuinely scarce artifact. Most sim2real claims in this literature are confounded because the simulated and experimental sides differ in sensor layout, target definition, or label pipeline simultaneously. It is indeed very important to have a cross-task benchmark in this regime.

Critical weaknesses

There are several confusing and inconsistent part existing in this manuscript which I think requiring additional details and explanations. For example, the same configurations are reported twice in Table.4/5/12 with irreconcilable numbers, and the discrepancies are larger than many affects the paper interprets. Table 4 (architecture study, default loss) and Tables 5/12 (loss study, L_D, seed 1101) nominally report the same model/task/loss cells. They do not agree, especially for U-Net, L_D, and the difference are even larger than the architecture difference, so the uncertainty/error analysis would be necessary to determine how much one can trust the number on the table, either using multi-seed/bootstrap way to evaluate and provide a range (+/-) for most metric value on the table.

Also, there is no numerical-solver baseline and no label-noise floor, so the benchmark provides no reference scale and the abstract claims otherwise. Especially for the multi-task with different device design/geometry, there should exists an absolute limit by comparing among reference dataset.

The same questions for loss design, full loss includes 4-5 term, but only with fixed weight (0.05), full ablation for different loss weight range, correlation between different loss component is barely discussed in the paper.

Questions for authors

Clarifications that could change my assessment:

  1. What exactly differs between the Table 4 runs and the Tables 5/12 runs? Please state the protocol difference explicitly, and reconcile the cells in the table.4/5/12 in the paper.

  2. Can you provide multi-seed results with intervals for at least T2, T3, and the loss study? The uncertainty completeness is very important to quantify how much we can trust the improvement.

  3. Can you add at least one numerical-solver row (PTEFIT or an EFIT-class solver) with both accuracy and wall-clock latency on the EXL-50U test set, alongside measured inference latency for the learned models? If not, please remove “physics-based baselines” from the abstract.

  4. What is the 5% fine-tuning budget in shots and in slices, and is the 5% sampled at slice level or shot level?

  5. Were the λ weights (all fixed at 0.05) tuned at all? It will be better to provide a table/plot with different component with loss weight ranging from 1e-4 to 10.

13 comments
AuthorsOpenReview ↗

Response to Reviewer BQAL (1/6)

We thank the reviewer for the insightful review. We appreciate the reviewer's recognition of the value of our configuration aligned EXL-50U simulation and experimental datasets, the well designed transfer tasks and complementary evaluation metrics, and the breadth of our comparisons across representative model architectures. We are also encouraged by the reviewer's positive assessment of our discussion of task specific loss components.

Concern 1

Weakness1: There are several confusing and inconsistent part existing in this manuscript which I think requiring additional details and explanations. For example, the same configurations are reported twice in Table.4/5/12 with irreconcilable numbers, and the discrepancies are larger than many affects the paper interprets. Table 4 (architecture study, default loss) and Tables 5/12 (loss study, L_D, seed 1101) nominally report the same model/task/loss cells. They do not agree, especially for U-Net, L_D, and the difference are even larger than the architecture difference, so the uncertainty/error analysis would be necessary to determine how much one can trust the number on the table, either using multi-seed/bootstrap way to evaluate and provide a range (+/-) for most metric value on the table.

Question1: What exactly differs between the Table 4 runs and the Tables 5/12 runs? Please state the protocol difference explicitly, and reconcile the cells in the table.4/5/12 in the paper.

Reply:

We thank the reviewer for pointing out the inconsistency among the overlapping configurations in Tables 4 and 5. In the original manuscript, the architecture comparison and the loss comparison used different numbers of training steps, and we did not clearly state this protocol difference, which caused confusion when comparing the numbers.

We clarify this point here. Specifically, Table 4 compares reconstruction performance across different model architectures and used a training budget of 30k iteration steps so that each architecture could converge sufficiently. In contrast, Tables 5 hold the model architecture fixed and compare different loss functions. To manage the computational cost across multiple loss configurations, those runs used 20k steps.

Thus, they corresponded to different training budgets.

During the rebuttal period, we extended all loss comparison experiments to 30k training steps so that both studies share the exact same training budget.

We have updated Tables 4 and 5 in the revised manuscript using the exact same training and evaluation runs for all overlapping default loss (LD\mathcal{L}_D) configurations.

Consequently, all overlapping cells across Tables 4 and 5 are now completely consistent.

Concern 2

Question2: Can you provide multi-seed results with intervals for at least task T2\mathbb{T}_2, task T3\mathbb{T}_3, and the loss study? The uncertainty completeness is very important to quantify how much we can trust the improvement.

Reply:

We thank the reviewer for suggesting random seed experiments to assess ranking stability.

During the rebuttal period, to evaluate sensitivity to random initialization, we trained all models in Tables 4 and 5 across three random seeds for 30k steps each, and report the mean and standard deviation across the three runs.

The updated results are presented in following Tables 6(part 1), 6(part 2), and 7.

Table 6(part 1). Main results across eight models. Within each scenario and metric, boldface marks minima. Geometry errors are conditional means over successful extractions. Flux RMSE is in mWb/rad. Relative errors are dimensionless.

TaskModelParams(M)EψRMSE↓Eψrel↓E∇ψrel↓Eaxis↓ELCFS95↓EX−pt↓Ephys↓mWb/radcmcmcmtask T1U-Net1.9541.7±0.210.0102±0.000380.0200±0.00130.903±0.0980.917±0.0534.47±1.40.0839±0.016Transformer4.4737.8±1.60.00899±0.00120.0168±0.00200.824±0.0821.42±0.145.51±0.830.0688±0.0039PlaNet1.5659.4±1.90.0145±0.00180.0237±0.00141.51±0.161.20±0.174.63±1.30.179±0.059DeepONet0.59678.0±0.450.0173±0.000910.0308±0.00151.90±0.152.07±0.706.99±0.520.0993±0.0092FNO2.0635.2±0.360.00808±0.0000260.0182±0.00105.43±0.241.16±0.0834.16±0.130.143±0.029\begin{array}{llrrrrrrrr}\hline\text{Task}&\text{Model}&\text{Params(M)}&E_{\psi}^{\mathrm{RMSE}}↓&E_{\psi}^{\mathrm{rel}}↓&E_{\nabla\psi}^{\mathrm{rel}}↓&E_{\mathrm{axis}}↓&E_{\mathrm{LCFS}}^{95}↓&E_{\mathrm{X-pt}}↓&E_{\mathrm{phys}}↓\\\\&&&\text{mWb/rad}&&&\text{cm}&\text{cm}&\text{cm}&\\\\\hline\text{task }\mathbb{T}_1&\text{U-Net}&1.95&41.7±0.21&0.0102±0.00038&0.0200±0.0013&0.903±0.098&\mathbf{0.917}±0.053&4.47±1.4&0.0839±0.016\\\\&\text{Transformer}&4.47&37.8±1.6&0.00899±0.0012&\mathbf{0.0168}±0.0020&\mathbf{0.824}±0.082&1.42±0.14&5.51±0.83&\mathbf{0.0688}±0.0039\\\\&\text{PlaNet}&1.56&59.4±1.9&0.0145±0.0018&0.0237±0.0014&1.51±0.16&1.20±0.17&4.63±1.3&0.179±0.059\\\\&\text{DeepONet}&0.596&78.0±0.45&0.0173±0.00091&0.0308±0.0015&1.90±0.15&2.07±0.70&6.99±0.52&0.0993±0.0092\\\\&\text{FNO}&2.06&\mathbf{35.2}±0.36&\mathbf{0.00808}±0.000026&0.0182±0.0010&5.43±0.24&1.16±0.083&\mathbf{4.16}±0.13&0.143±0.029\\\\\hline\end{array}

(Continued in Part 2.)

AuthorsOpenReview ↗

Response to Reviewer BQAL (2/6)

(Continued from Part 1.)

Table 6(part 1) (continued).

TaskModelParams(M)EψRMSE↓Eψrel↓E∇ψrel↓Eaxis↓ELCFS95↓EX−pt↓Ephys↓mWb/radcmcmcmtask T1SetONet0.10398.9±2.00.0266±0.00120.0442±0.00474.29±0.552.41±0.137.60±1.80.315±0.042GINO0.18462.9±3.90.0144±0.000740.0301±0.00329.95±1.01.63±0.0756.98±1.60.434±0.011GNOT0.62954.9±3.50.0123±0.000710.0206±0.000751.21±0.0681.14±0.0846.59±1.10.0731±0.0055task T2U-Net1.950.270±0.0260.00884±0.000290.0110±0.000380.377±0.0450.276±0.0191.92±0.350.231±0.023Transformer4.470.257±0.0350.00858±0.000380.0133±0.00170.380±0.0140.331±0.0250.892±0.170.167±0.020PlaNet1.560.512±0.00590.0177±0.00130.0222±0.000971.30±0.320.453±0.0962.63±0.380.798±0.059DeepONet0.5960.597±0.0260.0191±0.000880.0360±0.00281.13±0.0910.542±0.0595.75±1.30.231±0.020FNO2.060.275±0.00830.00882±0.000440.0106±0.000260.520±0.0830.276±0.0100.755±0.370.184±0.0060SetONet0.1030.538±0.0130.0205±0.000860.0293±0.00110.996±0.0920.677±0.0443.23±0.490.948±0.085GINO0.1840.300±0.0120.0117±0.000680.0271±0.000850.852±0.0250.417±0.0171.49±0.0581.13±0.12GNOT0.6290.270±0.0160.00900±0.000810.0119±0.000170.260±0.0450.389±0.0480.988±0.250.193±0.063task T3U-Net1.952.05±0.0480.106±0.00320.123±0.00742.15±0.542.52±0.231.54±0.140.531±0.13Transformer4.474.56±0.220.192±0.0110.185±0.0185.84±0.633.44±0.273.86±0.760.429±0.048PlaNet1.563.20±0.180.143±0.00560.164±0.0104.59±0.383.43±0.222.34±0.490.978±0.096DeepONet0.59610.4±0.420.316±0.00330.326±0.0269.68±1.55.58±0.723.77±1.40.958±0.062FNO2.063.13±0.300.132±0.00250.173±0.00394.40±0.333.11±0.241.50±0.230.988±0.21SetONet0.1034.39±0.170.178±0.00760.185±0.0133.98±0.213.61±0.163.46±0.921.01±0.14GINO0.1843.42±0.110.124±0.0110.179±0.003313.9±1.42.11±0.385.11±1.17.39±2.9GNOT0.6294.52±0.110.190±0.0120.195±0.00855.24±0.591.87±0.271.68±0.460.444±0.0076\begin{array}{llrrrrrrrr}\hline\text{Task}&\text{Model}&\text{Params(M)}&E_{\psi}^{\mathrm{RMSE}}↓&E_{\psi}^{\mathrm{rel}}↓&E_{\nabla\psi}^{\mathrm{rel}}↓&E_{\mathrm{axis}}↓&E_{\mathrm{LCFS}}^{95}↓&E_{\mathrm{X-pt}}↓&E_{\mathrm{phys}}↓\\\\&&&\text{mWb/rad}&&&\text{cm}&\text{cm}&\text{cm}&\\\\\hline\text{task }\mathbb{T}_1&\text{SetONet}&0.103&98.9±2.0&0.0266±0.0012&0.0442±0.0047&4.29±0.55&2.41±0.13&7.60±1.8&0.315±0.042\\\\&\text{GINO}&0.184&62.9±3.9&0.0144±0.00074&0.0301±0.0032&9.95±1.0&1.63±0.075&6.98±1.6&0.434±0.011\\\\&\text{GNOT}&0.629&54.9±3.5&0.0123±0.00071&0.0206±0.00075&1.21±0.068&1.14±0.084&6.59±1.1&0.0731±0.0055\\\\\hline\text{task }\mathbb{T}_2&\text{U-Net}&1.95&0.270±0.026&0.00884±0.00029&0.0110±0.00038&0.377±0.045&\mathbf{0.276}±0.019&1.92±0.35&0.231±0.023\\\\&\text{Transformer}&4.47&\mathbf{0.257}±0.035&\mathbf{0.00858}±0.00038&0.0133±0.0017&0.380±0.014&0.331±0.025&0.892±0.17&\mathbf{0.167}±0.020\\\\&\text{PlaNet}&1.56&0.512±0.0059&0.0177±0.0013&0.0222±0.00097&1.30±0.32&0.453±0.096&2.63±0.38&0.798±0.059\\\\&\text{DeepONet}&0.596&0.597±0.026&0.0191±0.00088&0.0360±0.0028&1.13±0.091&0.542±0.059&5.75±1.3&0.231±0.020\\\\&\text{FNO}&2.06&0.275±0.0083&0.00882±0.00044&\mathbf{0.0106}±0.00026&0.520±0.083&0.276±0.010&\mathbf{0.755}±0.37&0.184±0.0060\\\\&\text{SetONet}&0.103&0.538±0.013&0.0205±0.00086&0.0293±0.0011&0.996±0.092&0.677±0.044&3.23±0.49&0.948±0.085\\\\&\text{GINO}&0.184&0.300±0.012&0.0117±0.00068&0.0271±0.00085&0.852±0.025&0.417±0.017&1.49±0.058&1.13±0.12\\\\&\text{GNOT}&0.629&0.270±0.016&0.00900±0.00081&0.0119±0.00017&\mathbf{0.260}±0.045&0.389±0.048&0.988±0.25&0.193±0.063\\\\\hline\text{task }\mathbb{T}_3&\text{U-Net}&1.95&\mathbf{2.05}±0.048&\mathbf{0.106}±0.0032&\mathbf{0.123}±0.0074&\mathbf{2.15}±0.54&2.52±0.23&1.54±0.14&0.531±0.13\\\\&\text{Transformer}&4.47&4.56±0.22&0.192±0.011&0.185±0.018&5.84±0.63&3.44±0.27&3.86±0.76&\mathbf{0.429}±0.048\\\\&\text{PlaNet}&1.56&3.20±0.18&0.143±0.0056&0.164±0.010&4.59±0.38&3.43±0.22&2.34±0.49&0.978±0.096\\\\&\text{DeepONet}&0.596&10.4±0.42&0.316±0.0033&0.326±0.026&9.68±1.5&5.58±0.72&3.77±1.4&0.958±0.062\\\\&\text{FNO}&2.06&3.13±0.30&0.132±0.0025&0.173±0.0039&4.40±0.33&3.11±0.24&\mathbf{1.50}±0.23&0.988±0.21\\\\&\text{SetONet}&0.103&4.39±0.17&0.178±0.0076&0.185±0.013&3.98±0.21&3.61±0.16&3.46±0.92&1.01±0.14\\\\&\text{GINO}&0.184&3.42±0.11&0.124±0.011&0.179±0.0033&13.9±1.4&2.11±0.38&5.11±1.1&7.39±2.9\\\\&\text{GNOT}&0.629&4.52±0.11&0.190±0.012&0.195±0.0085&5.24±0.59&\mathbf{1.87}±0.27&1.68±0.46&0.444±0.0076\\\\\hline\end{array}

Table 6(part 2). Main results across eight models (continued from Table 6 (part 1)). Within each scenario and metric, boldface marks minima. Geometry errors are conditional means over successful extractions. Flux RMSE is in mWb/rad. Relative errors are dimensionless.

TaskModelParams(M)EψRMSE↓Eψrel↓E∇ψrel↓Eaxis↓ELCFS95↓EX−pt↓Ephys↓mWb/radcmcmcmtask T4U-Net1.951.44±0.0210.0783±0.00130.0823±0.00521.99±0.295.78±0.452.64±0.121.83±0.20Transformer4.471.28±0.0970.0620±0.00610.100±0.00691.94±0.264.23±0.532.37±0.180.352±0.032PlaNet1.561.94±0.0590.0892±0.00710.103±0.000253.80±0.504.90±0.632.38±0.371.24±0.16DeepONet0.5962.74±0.0660.133±0.00210.213±0.0132.83±0.186.49±0.513.36±0.510.783±0.16FNO2.063.92±0.120.179±0.00820.153±0.00853.11±0.228.47±0.915.07±1.41.14±0.052SetONet0.1033.83±0.360.183±0.00690.188±0.00764.24±0.138.77±0.483.72±0.272.04±0.17GINO0.1843.96±0.160.202±0.00360.362±0.0196.73±1.229.6±3.35.89±1.015.9±1.6GNOT0.6292.36±0.0660.115±0.0170.140±0.0182.63±0.0806.18±0.212.66±0.240.311±0.030task T5U-Net1.953.38±0.120.231±0.0150.303±0.01313.4±1.614.1±0.974.06±0.691.05±0.15Transformer4.471.76±0.0990.121±0.000560.185±0.00637.78±0.3210.9±1.41.03±0.380.493±0.040PlaNet1.565.34±0.200.350±0.0170.425±0.008514.5±1.324.4±2.58.00±1.21.73±0.21DeepONet0.5964.22±0.390.276±0.00460.380±0.0228.63±0.6315.6±2.96.00±0.921.08±0.058FNO2.063.84±0.0750.228±0.00810.329±0.02214.7±2.027.0±4.95.65±2.21.19±0.088\begin{array}{llrrrrrrrr}\hline\text{Task}&\text{Model}&\text{Params(M)}&E_{\psi}^{\mathrm{RMSE}}↓&E_{\psi}^{\mathrm{rel}}↓&E_{\nabla\psi}^{\mathrm{rel}}↓&E_{\mathrm{axis}}↓&E_{\mathrm{LCFS}}^{95}↓&E_{\mathrm{X-pt}}↓&E_{\mathrm{phys}}↓\\\\&&&\text{mWb/rad}&&&\text{cm}&\text{cm}&\text{cm}&\\\\\hline\text{task }\mathbb{T}_4&\text{U-Net}&1.95&1.44±0.021&0.0783±0.0013&\mathbf{0.0823}±0.0052&1.99±0.29&5.78±0.45&2.64±0.12&1.83±0.20\\\\&\text{Transformer}&4.47&\mathbf{1.28}±0.097&\mathbf{0.0620}±0.0061&0.100±0.0069&\mathbf{1.94}±0.26&\mathbf{4.23}±0.53&\mathbf{2.37}±0.18&0.352±0.032\\\\&\text{PlaNet}&1.56&1.94±0.059&0.0892±0.0071&0.103±0.00025&3.80±0.50&4.90±0.63&2.38±0.37&1.24±0.16\\\\&\text{DeepONet}&0.596&2.74±0.066&0.133±0.0021&0.213±0.013&2.83±0.18&6.49±0.51&3.36±0.51&0.783±0.16\\\\&\text{FNO}&2.06&3.92±0.12&0.179±0.0082&0.153±0.0085&3.11±0.22&8.47±0.91&5.07±1.4&1.14±0.052\\\\&\text{SetONet}&0.103&3.83±0.36&0.183±0.0069&0.188±0.0076&4.24±0.13&8.77±0.48&3.72±0.27&2.04±0.17\\\\&\text{GINO}&0.184&3.96±0.16&0.202±0.0036&0.362±0.019&6.73±1.2&29.6±3.3&5.89±1.0&15.9±1.6\\\\&\text{GNOT}&0.629&2.36±0.066&0.115±0.017&0.140±0.018&2.63±0.080&6.18±0.21&2.66±0.24&\mathbf{0.311}±0.030\\\\\hline\text{task }\mathbb{T}_5&\text{U-Net}&1.95&3.38±0.12&0.231±0.015&0.303±0.013&13.4±1.6&14.1±0.97&4.06±0.69&1.05±0.15\\\\&\text{Transformer}&4.47&\mathbf{1.76}±0.099&\mathbf{0.121}±0.00056&\mathbf{0.185}±0.0063&\mathbf{7.78}±0.32&\mathbf{10.9}±1.4&\mathbf{1.03}±0.38&0.493±0.040\\\\&\text{PlaNet}&1.56&5.34±0.20&0.350±0.017&0.425±0.0085&14.5±1.3&24.4±2.5&8.00±1.2&1.73±0.21\\\\&\text{DeepONet}&0.596&4.22±0.39&0.276±0.0046&0.380±0.022&8.63±0.63&15.6±2.9&6.00±0.92&1.08±0.058\\\\&\text{FNO}&2.06&3.84±0.075&0.228±0.0081&0.329±0.022&14.7±2.0&27.0±4.9&5.65±2.2&1.19±0.088\\\\\hline\end{array}

(Continued in Part 3.)

AuthorsOpenReview ↗

Response to Reviewer BQAL (3/6)

(Continued from Part 2.)

Table 6(part 2) (continued).

TaskModelParams(M)EψRMSE↓Eψrel↓E∇ψrel↓Eaxis↓ELCFS95↓EX−pt↓Ephys↓mWb/radcmcmcmtask T5SetONet0.1034.43±0.240.287±0.0160.401±0.02310.0±0.7519.9±2.76.06±0.601.28±0.10GINO0.1844.21±0.340.317±0.0120.452±0.03278.3±7.551.1±4.220.3±2.117.8±3.9GNOT0.6293.11±0.200.211±0.0110.293±0.0148.84±0.7612.1±1.82.48±0.310.417±0.066\begin{array}{llrrrrrrrr}\hline\text{Task}&\text{Model}&\text{Params(M)}&E_{\psi}^{\mathrm{RMSE}}↓&E_{\psi}^{\mathrm{rel}}↓&E_{\nabla\psi}^{\mathrm{rel}}↓&E_{\mathrm{axis}}↓&E_{\mathrm{LCFS}}^{95}↓&E_{\mathrm{X-pt}}↓&E_{\mathrm{phys}}↓\\\\&&&\text{mWb/rad}&&&\text{cm}&\text{cm}&\text{cm}&\\\\\hline\text{task }\mathbb{T}_5&\text{SetONet}&0.103&4.43±0.24&0.287±0.016&0.401±0.023&10.0±0.75&19.9±2.7&6.06±0.60&1.28±0.10\\\\&\text{GINO}&0.184&4.21±0.34&0.317±0.012&0.452±0.032&78.3±7.5&51.1±4.2&20.3±2.1&17.8±3.9\\\\&\text{GNOT}&0.629&3.11±0.20&0.211±0.011&0.293±0.014&8.84±0.76&12.1±1.8&2.48±0.31&\mathbf{0.417}±0.066\\\\\hline\end{array}

Table 7. Training loss comparison on selected reconstruction metrics. Within each scenario and metric, boldface marks minima. Geometry errors are conditional means over successful extractions. More results are in Appendix G.1, "Main Results".

TransferscenarioLossFieldFieldFieldLCFSLCFSLCFSX-pointX-pointX-pointEψRMSE↓EψRMSE↓EψRMSE↓ELCFS95↓ELCFS95↓ELCFS95↓EX−pt↓EX−pt↓EX−pt↓U-NetFNOGNOTU-NetFNOGNOTU-NetFNOGNOT(mWb/rad)(mWb/rad)(mWb/rad)(cm)(cm)(cm)(cm)(cm)(cm)task T1LD41.7±0.2135.2±0.3654.9±3.50.917±0.0531.16±0.0831.14±0.0844.47±1.44.16±0.136.59±1.1LD+λprobeLprobe40.4±2.133.9±1.858.9±3.60.994±0.0731.17±0.111.17±0.243.80±1.54.35±0.285.47±1.9LD+λphysLphys47.0±1.534.7±0.5361.0±1.61.02±0.0961.16±0.0471.21±0.184.17±0.373.79±0.375.66±0.35LF44.4±0.9633.5±2.060.7±2.11.04±0.0191.13±0.0521.15±0.124.81±1.43.86±0.285.11±1.0task T2LD0.270±0.0260.275±0.00830.270±0.0160.276±0.0190.276±0.0100.389±0.0481.92±0.350.755±0.370.988±0.25LD+λprobeLprobe0.266±0.0140.254±0.00230.284±0.0120.284±0.0280.275±0.0210.380±0.0562.02±0.590.808±0.190.929±0.32LD+λphysLphys0.288±0.00810.291±0.00830.296±0.0210.306±0.0170.262±0.0380.426±0.0241.97±0.150.772±0.341.23±0.59LF0.289±0.00400.290±0.00760.240±0.0170.306±0.00760.267±0.0120.424±0.0502.08±0.400.844±0.291.07±0.40task T3LD2.05±0.0483.13±0.304.52±0.112.52±0.233.11±0.241.87±0.271.54±0.141.50±0.231.68±0.46LD+λprobeLprobe1.94±0.0802.94±0.114.65±0.262.18±0.152.94±0.161.58±0.231.64±0.111.47±0.292.00±0.24LD+λphysLphys2.23±0.0523.07±0.154.90±0.272.24±0.0602.96±0.0661.78±0.461.47±0.0331.66±0.261.75±0.33LF2.23±0.0693.08±0.284.31±0.262.68±0.173.27±0.271.67±0.111.64±0.161.23±0.352.13±0.44task T4LD1.44±0.0213.92±0.122.36±0.0665.78±0.458.47±0.916.18±0.212.64±0.125.07±1.42.66±0.24LD+λprobeLprobe1.36±0.0173.84±0.212.55±0.0215.49±0.368.67±1.46.32±0.772.98±0.355.63±1.32.76±0.18LD+λphysLphys1.42±0.0803.72±0.0772.61±0.0995.71±0.156.98±0.456.21±1.02.88±0.193.71±0.462.73±0.49LF1.28±0.0613.69±0.122.16±0.0596.12±1.09.19±0.866.00±0.662.81±0.774.07±0.832.64±0.31task T5LD3.38±0.123.84±0.0753.11±0.2014.1±0.9727.0±4.912.1±1.84.06±0.695.65±2.22.48±0.31LD+λprobeLprobe3.11±0.293.72±0.132.84±0.07314.8±1.426.1±3.510.7±2.23.82±0.395.53±0.312.96±1.2LD+λphysLphys3.46±0.0463.55±0.152.96±0.06816.7±2.123.3±6.111.2±0.953.34±0.334.47±1.82.42±0.57LF3.42±0.163.74±0.103.21±0.08316.2±0.8726.5±4.010.5±2.03.90±0.564.61±0.222.51±0.28\begin{array}{llrrrrrrrrr}\hline\substack{\text{Transfer}\\\\\text{scenario}}&\text{Loss}&\text{Field}&\text{Field}&\text{Field}&\text{LCFS}&\text{LCFS}&\text{LCFS}&\text{X-point}&\text{X-point}&\text{X-point}\\\\&&E_{\psi}^{\mathrm{RMSE}}\downarrow&E_{\psi}^{\mathrm{RMSE}}\downarrow&E_{\psi}^{\mathrm{RMSE}}\downarrow&E_{\mathrm{LCFS}}^{95}\downarrow&E_{\mathrm{LCFS}}^{95}\downarrow&E_{\mathrm{LCFS}}^{95}\downarrow&E_{\mathrm{X-pt}}\downarrow&E_{\mathrm{X-pt}}\downarrow&E_{\mathrm{X-pt}}\downarrow\\\\&&\text{U-Net}&\text{FNO}&\text{GNOT}&\text{U-Net}&\text{FNO}&\text{GNOT}&\text{U-Net}&\text{FNO}&\text{GNOT}\\\\&&\text{(mWb/rad)}&\text{(mWb/rad)}&\text{(mWb/rad)}&\text{(cm)}&\text{(cm)}&\text{(cm)}&\text{(cm)}&\text{(cm)}&\text{(cm)}\\\\\hline\text{task }\mathbb{T}_1&\mathcal{L}_D&41.7±0.21&35.2±0.36&54.9±3.5&\mathbf{0.917}±0.053&1.16±0.083&1.14±0.084&4.47±1.4&4.16±0.13&6.59±1.1\\\\&\mathcal{L}_D+\lambda_{\mathrm{probe}}\mathcal{L}_{\mathrm{probe}}&40.4±2.1&33.9±1.8&58.9±3.6&0.994±0.073&1.17±0.11&1.17±0.24&3.80±1.5&4.35±0.28&5.47±1.9\\\\&\mathcal{L}_D+\lambda_{\mathrm{phys}}\mathcal{L}_{\mathrm{phys}}&47.0±1.5&34.7±0.53&61.0±1.6&1.02±0.096&1.16±0.047&1.21±0.18&4.17±0.37&\mathbf{3.79}±0.37&5.66±0.35\\\\&\mathcal{L}_F&44.4±0.96&\mathbf{33.5}±2.0&60.7±2.1&1.04±0.019&1.13±0.052&1.15±0.12&4.81±1.4&3.86±0.28&5.11±1.0\\\\\hline\text{task }\mathbb{T}_2&\mathcal{L}_D&0.270±0.026&0.275±0.0083&0.270±0.016&0.276±0.019&0.276±0.010&0.389±0.048&1.92±0.35&\mathbf{0.755}±0.37&0.988±0.25\\\\&\mathcal{L}_D+\lambda_{\mathrm{probe}}\mathcal{L}_{\mathrm{probe}}&0.266±0.014&0.254±0.0023&0.284±0.012&0.284±0.028&0.275±0.021&0.380±0.056&2.02±0.59&0.808±0.19&0.929±0.32\\\\&\mathcal{L}_D+\lambda_{\mathrm{phys}}\mathcal{L}_{\mathrm{phys}}&0.288±0.0081&0.291±0.0083&0.296±0.021&0.306±0.017&\mathbf{0.262}±0.038&0.426±0.024&1.97±0.15&0.772±0.34&1.23±0.59\\\\&\mathcal{L}_F&0.289±0.0040&0.290±0.0076&\mathbf{0.240}±0.017&0.306±0.0076&0.267±0.012&0.424±0.050&2.08±0.40&0.844±0.29&1.07±0.40\\\\\hline\text{task }\mathbb{T}_3&\mathcal{L}_D&2.05±0.048&3.13±0.30&4.52±0.11&2.52±0.23&3.11±0.24&1.87±0.27&1.54±0.14&1.50±0.23&1.68±0.46\\\\&\mathcal{L}_D+\lambda_{\mathrm{probe}}\mathcal{L}_{\mathrm{probe}}&\mathbf{1.94}±0.080&2.94±0.11&4.65±0.26&2.18±0.15&2.94±0.16&\mathbf{1.58}±0.23&1.64±0.11&1.47±0.29&2.00±0.24\\\\&\mathcal{L}_D+\lambda_{\mathrm{phys}}\mathcal{L}_{\mathrm{phys}}&2.23±0.052&3.07±0.15&4.90±0.27&2.24±0.060&2.96±0.066&1.78±0.46&1.47±0.033&1.66±0.26&1.75±0.33\\\\&\mathcal{L}_F&2.23±0.069&3.08±0.28&4.31±0.26&2.68±0.17&3.27±0.27&1.67±0.11&1.64±0.16&\mathbf{1.23}±0.35&2.13±0.44\\\\\hline\text{task }\mathbb{T}_4&\mathcal{L}_D&1.44±0.021&3.92±0.12&2.36±0.066&5.78±0.45&8.47±0.91&6.18±0.21&\mathbf{2.64}±0.12&5.07±1.4&2.66±0.24\\\\&\mathcal{L}_D+\lambda_{\mathrm{probe}}\mathcal{L}_{\mathrm{probe}}&1.36±0.017&3.84±0.21&2.55±0.021&\mathbf{5.49}±0.36&8.67±1.4&6.32±0.77&2.98±0.35&5.63±1.3&2.76±0.18\\\\&\mathcal{L}_D+\lambda_{\mathrm{phys}}\mathcal{L}_{\mathrm{phys}}&1.42±0.080&3.72±0.077&2.61±0.099&5.71±0.15&6.98±0.45&6.21±1.0&2.88±0.19&3.71±0.46&2.73±0.49\\\\&\mathcal{L}_F&\mathbf{1.28}±0.061&3.69±0.12&2.16±0.059&6.12±1.0&9.19±0.86&6.00±0.66&2.81±0.77&4.07±0.83&2.64±0.31\\\\\hline\text{task }\mathbb{T}_5&\mathcal{L}_D&3.38±0.12&3.84±0.075&3.11±0.20&14.1±0.97&27.0±4.9&12.1±1.8&4.06±0.69&5.65±2.2&2.48±0.31\\\\&\mathcal{L}_D+\lambda_{\mathrm{probe}}\mathcal{L}_{\mathrm{probe}}&3.11±0.29&3.72±0.13&\mathbf{2.84}±0.073&14.8±1.4&26.1±3.5&10.7±2.2&3.82±0.39&5.53±0.31&2.96±1.2\\\\&\mathcal{L}_D+\lambda_{\mathrm{phys}}\mathcal{L}_{\mathrm{phys}}&3.46±0.046&3.55±0.15&2.96±0.068&16.7±2.1&23.3±6.1&11.2±0.95&3.34±0.33&4.47±1.8&\mathbf{2.42}±0.57\\\\&\mathcal{L}_F&3.42±0.16&3.74±0.10&3.21±0.083&16.2±0.87&26.5±4.0&\mathbf{10.5}±2.0&3.90±0.56&4.61±0.22&2.51±0.28\\\\\hline\end{array}

(Continued in Part 4.)

Reviewer_wdKuOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TokaReconBench, a benchmark for evaluating the transferability of AI models in tokamak magnetic equilibrium reconstruction. It integrates experimental and simulated data from multiple tokamak devices, including newly contributed EXL-50U datasets, and defines five evaluation tasks covering in-domain reconstruction, simulation-to-real transfer, and cross-device adaptation. The benchmark evaluates field reconstruction accuracy, geometric fidelity, and physics consistency, with experiments involving eight representative neural surrogate and operator-learning models. The results highlight substantial generalization gaps and inconsistencies between field accuracy, geometric reconstruction, and physics-based evaluation metrics.

Critical strengths

Valuable benchmark and dataset contribution. The paper addresses an important gap in evaluating the transferability of AI-based equilibrium reconstruction across different tokamak devices and data domains. The newly contributed EXL-50U experimental and simulation datasets provide a useful foundation for studying simulation-to-real generalization. The benchmark considers multiple transfer scenarios, representative neural architectures, and complementary evaluation metrics covering field accuracy, equilibrium geometry, and physical consistency. The results provide useful insights into the limitations of existing learning-based reconstruction approaches.

Critical weaknesses

Insufficient evaluation of the actual benefits of cross-device transfer. The cross-device experiments primarily evaluate source-pretrained models after fine-tuning on 5% of the target-domain training data. However, the paper does not clearly establish how much benefit comes from source-domain pretraining compared with training from scratch using the same target-domain data. Without such comparisons, it is difficult to quantify positive or negative transfer. This requirement is central to the benchmark's stated objective.

Limited independent validation of physical reconstruction accuracy. The experimental reference equilibria are derived from the PTEFIT reconstruction pipeline rather than direct measurements of the complete magnetic flux field. Consequently, agreement with reference flux fields and associated GS-based diagnostics does not necessarily establish physical accuracy independently of the reference reconstruction procedure. Further clarification of label reliability and independent diagnostic validation would strengthen the benchmark's credibility.

Questions for authors

a. For the cross-device tasks T4 and T5, how do source-pretrained models compare with models trained from scratch using the same 5% target-domain data? Can the authors provide transfer gains across different target-label budgets?

b. How is the reliability of the PTEFIT-generated reference equilibrium assessed? Are independent diagnostic consistency measures available to complement the reconstruction-based field and physics metrics?

c. For geometric metrics computed only on samples with successful structure extraction, can the authors report extraction success rates alongside conditional errors? Additionally, how stable are the main model rankings across different training seeds?

6 comments
AuthorsOpenReview ↗

Response to Reviewer wdKu (1/6)

We thank the reviewer for the thoughtful and constructive review. We appreciate the reviewer’s recognition of TokaReconBench as a valuable benchmark addressing an important gap in transferability evaluation, the contribution of our new EXL-50U simulation and experimental datasets, and the comprehensive evaluation framework and informative empirical findings.

Concern 1

Weakness1: Insufficient evaluation of the actual benefits of cross-device transfer. The cross-device experiments primarily evaluate source-pretrained models after fine-tuning on 5% of the target-domain training data. However, the paper does not clearly establish how much benefit comes from source-domain pretraining compared with training from scratch using the same target-domain data. Without such comparisons, it is difficult to quantify positive or negative transfer. This requirement is central to the benchmark's stated objective.

Question1: For the cross-device task T4\mathbb{T}_4 and task T5\mathbb{T}_5, how do source-pretrained models compare with models trained from scratch using the same 5% target-domain data? Can the authors provide transfer gains across different target-label budgets?

Reply:

We agree that directly comparing source-pretrained fine-tuning with training from scratch on the same target-domain data is essential for determining whether the benefits of cross-device transfer originate from the source-pretrained models or solely from the new data.

During the rebuttal period, we have supplemented such a new control experiment, covering task T4\mathbb{T}_4 (ITER simulation to EXL-50U simulation) and task T5\mathbb{T}_5 (MAST experiment to EXL-50U experiment).

Taking the U-Net network described in Appendix E.1 as an example, we compared the performance of source-pretrained models fine-tuned with 5% of the target-domain data against models trained from scratch using the same target-domain training subset. The results are presented in the following Table 1.

Table 1. Comparison between source-pretrained fine-tuning and training from scratch both using 5% target-domain data on cross-device tasks (task T4\mathbb{T}_4 and task T5\mathbb{T}_5) with U-Net. Bold values indicate the better result within each task.

TaskTraining strategyEψRMSEE_{\psi}^{\mathrm{RMSE}} (Wb/rad) ↓EψrelE_{\psi}^{\mathrm{rel}} ↓Conditional mean LCFS HD95 (cm) ↓LCFS extraction success rate ↑
task T4\mathbb{T}_4: ITER Sim. → EXL-50U Sim.Source-pretrained + 5% target fine-tuning0.0015840.076675.2960,206 / 60,218 (99.98%)
task T4\mathbb{T}_4: ITER Sim. → EXL-50U Sim.Random init. + 5% target from scratch0.0015970.077225.1259,781 / 60,218 (99.27%)
task T5\mathbb{T}_5: MAST Exp. → EXL-50U Exp.Source-pretrained + 5% target fine-tuning0.0087020.4463723.8530,041 / 31,849 (94.32%)
task T5\mathbb{T}_5: MAST Exp. → EXL-50U Exp.Random init. + 5% target from scratch0.0078090.4211319.7329,987 / 31,849 (94.15%)

The experimental results indicate that in task T4\mathbb{T}_4, the field reconstruction errors of the two strategies are close, with a difference of less than 1%. In task T5\mathbb{T}_5, the relative flux error (EψrelE_{\psi}^{\mathrm{rel}}) for training from scratch decreased from 0.44637 to 0.42113, a reduction of 5.65%; the conditional mean of LCFS HD95 decreased from 23.85 cm to 19.73 cm, a reduction of 17.27%. Therefore, in this U-Net control experiment, pretraining did not provide consistent additional benefits across the two tasks.

A possible explanation for this phenomenon is the significant physical scaling differences and variations in internal electromagnetic environments between different tokamak devices [Casper et al., 2014; Shi et al., 2025]. For example, in sampled training data for task T4\mathbb{T}_4, the median reference plasma current is 12.41 MA for ITER-like simulation and 0.417 MA for EXL-50U simulation; the corresponding median centered flux RMS values are approximately 4.25 and 0.02149 Wb/rad, respectively, differing by more than two orders of magnitude.

Although we introduced dimensionless normalization and data standardization in the preprocessing stage to eliminate inconsistencies in physical dimensions, the essential differences in geometric configuration, coil layout, and magnetic field topology distribution among different devices make it difficult for the prior knowledge learned by pretraining models to generalize directly.

This finding is also consistent with our discussion in the original manuscript (Section 3.3, lines 430-431 and 463-466) regarding same-device simulation-to-real (sim2real) transfer and cross-device transfer: the tested two-stage pretraining and fine-tuning recipe remains insufficient to bridge the physical distribution gap across devices.

(Continued in Part 2.)

AuthorsOpenReview ↗

Response to Reviewer wdKu (2/6)

(Continued from Part 1.)

We have included the above experimental results and physical mechanism analysis in Table 15 of Appendix G.

Subsequently, we will extend the results to FNO and add them to the final version of the paper.

Concern 2

Weakness2: Limited independent validation of physical reconstruction accuracy. The experimental reference equilibria are derived from the PTEFIT reconstruction pipeline rather than direct measurements of the complete magnetic flux field. Consequently, agreement with reference flux fields and associated GS-based diagnostics does not necessarily establish physical accuracy independently of the reference reconstruction procedure. Further clarification of label reliability and independent diagnostic validation would strengthen the benchmark's credibility.

Question1: How is the reliability of the PTEFIT-generated reference equilibrium assessed? Are independent diagnostic consistency measures available to complement the reconstruction-based field and physics metrics?

Reply:

We thank the reviewer for raising this important question regarding the physical reliability of the experimental reference labels. The inverse-reconstruction problem and reference-label sources are described in Sections 2.1 and 2.3 of the original manuscript. Because the core temperature of a tokamak plasma reaches tens of millions to hundreds of millions of degrees, any sensor inserted into the plasma would be destroyed and would disturb or even interrupt the discharge; consequently, magnetic sensors can only be mounted on the vacuum vessel wall or outside the vacuum vessel [Wesson, 2011; Hutchinson, 2002; Strait, 2006]. Therefore, to obtain the internal magnetic flux field, one must start from external measurements and solve an inverse problem constrained by the Grad–Shafranov equation [Lao et al., 1985; Zakharov et al., 2008]. In academia and industry, the EFIT equilibrium reconstruction code is widely adopted as the reference for equilibrium results.

In practice, different tokamak devices often employ their own fine-tuned versions of the EFIT code for equilibrium inversion. For example, EAST uses the GPU version of the P-EFIT algorithm [Yue et al., 2013; Huang et al., 2017; Huang et al., 2020], while MAST-U uses EFIT++ for offline reconstruction [Kogan et al., 2022] and LEMUR for real-time reconstruction and shape control [Anand et al., 2024]. PTEFIT differs from offline EFIT only in the numerical simplifications made for real-time operation: a lower polynomial order in the current-profile parameterization and a single Picard iteration per time slice.

The PTEFIT code selected in this paper is the official equilibrium solver used by the EXL-50U device during discharges. From June 2025, PTEFIT has been continuously running on EXL-50U, covering about 10,900 shots. In published results, the target deviation for RmaxR_{\mathrm{max}} control during the flat-top phase driven by PTEFIT is approximately 2 cm; in isoflux control (shot #14488), the average distance from the reference point to the plasma boundary is approximately 2 cm [Zheng et al., 2026]. These facts indicate that the stability and reliability of PTEFIT have been continuously verified. Appendix C.3 reports the PTEFIT–EFIT comparison (Table 8) and cites independent validation of offline EFIT against camera imaging and Langmuir probes [Gu et al., 2026].

To investigate the reliability of PTEFIT further, we added an experimental evaluation of the deviation between PTEFIT and EFIT inversions during the rebuttal period.

We conducted a slice by slice comparison across 201,340 common time slices from 520 shots during the rebuttal period. The results are reported in following Table 2.

Table 2. Geometric differences between PTEFIT and offline EFIT (520 shots; 201,340 common time slices).

MetricMedianMean
∣ΔRmin⁡∣\lvert \Delta R_{\min}\rvert (cm)0.370.54
∣ΔRmax⁡∣\lvert \Delta R_{\max}\rvert (cm)1.281.47
∣ΔZmin⁡∣\lvert \Delta Z_{\min}\rvert (cm)1.041.81
∣ΔZmax⁡∣\lvert \Delta Z_{\max}\rvert (cm)1.141.75

As shown in Table 2, the geometric boundary deviations (∣ΔRmin⁡∣\lvert \Delta R_{\min}\rvert, ∣ΔRmax⁡∣\lvert \Delta R_{\max}\rvert, ∣ΔZmin⁡∣\lvert \Delta Z_{\min}\rvert, and ∣ΔZmax⁡∣\lvert \Delta Z_{\max}\rvert) between PTEFIT and offline EFIT have median values ranging from 0.37 cm to 1.28 cm, with typical deviations of approximately 1 cm. This indicates close agreement in these plasma boundary positions between PTEFIT and offline EFIT across discharges.

We have added this quantitative evaluation and analysis to Table 8 of Appendix C.3 in the revised manuscript, together with an offline-EFIT reference comparison on matched EXL-50U slices in Table 4 to provide an inter-solver discrepancy reference.

(Continued in Part 3.)

AuthorsOpenReview ↗

Response to Reviewer wdKu (3/6)

(Continued from Part 2.)

Concern 3

For geometric metrics computed only on samples with successful structure extraction, can the authors report extraction success rates alongside conditional errors?

Reply:

We thank the reviewer for the suggestion to supplement the extraction success rates for the LCFS and X-point.

During the rebuttal period, we retrieved the extraction success rates from the available evaluation records for the configurations in Table 4 and Table 5.

The extraction success rate is calculated as the proportion of successful samples among the corresponding valid evaluation samples. For the X-point, statistics are computed only on samples where the reference equilibrium possesses an X-point; cases where the predicted number of X-points does not match the reference are counted as detection failures.

The results are shown in the Table 3 below.

Table 3. LCFS / X-point extraction success rates (%) for the Table 4 and Table 5 configurations. Each task cell reports LCFS / X-point extraction success rates.

SourceModelLosstask T1\mathbb{T}_1task T2\mathbb{T}_2task T3\mathbb{T}_3task T4\mathbb{T}_4task T5\mathbb{T}_5
Table 4U-NetDefault97.69 / 99.8199.04 / 100.00100.00 / 100.0099.98 / 99.9988.84 / 100.00
Table 4TransformerDefault97.84 / 62.8399.04 / 99.32100.00 / 100.00100.00 / 88.2192.02 / 99.99
Table 4PlaNetDefault97.84 / 94.5199.04 / 88.47100.00 / 100.00100.00 / 99.8092.02 / 99.99
Table 4DeepONetDefault97.84 / 95.2599.04 / 94.58100.00 / 100.00100.00 / 98.4392.02 / 99.99
Table 4FNODefault97.84 / 99.8398.13 / 100.00100.00 / 100.0099.02 / 98.6292.02 / 99.95
Table 4SetONetDefault97.84 / 87.6899.04 / 96.27100.00 / 100.0098.44 / 88.2192.02 / 99.99
Table 4GINODefault97.84 / 72.6299.04 / 96.9593.65 / 96.8390.62 / 53.8392.02 / 99.99
Table 4GNOTDefault98.85 / 99.8599.83 / 100.00100.00 / 100.00100.00 / 98.8298.95 / 99.99
Table 5U-NetDefault97.69 / 99.8199.04 / 100.0098.64 / 100.0099.99 / 100.0088.84 / 100.00
Table 5U-Net+Probe97.94 / 99.8098.90 / 100.0098.89 / 100.0099.57 / 99.9693.08 / 99.98
Table 5U-Net+GS98.68 / 99.8399.37 / 100.0096.47 / 100.0099.86 / 100.0097.50 / 100.00
Table 5U-NetFull99.08 / 99.7799.10 / 100.0096.76 / 100.0099.69 / 100.0098.53 / 100.00
Table 5FNODefault97.84 / 99.8398.13 / 100.0094.24 / 99.9898.63 / 99.5892.02 / 99.95
Table 5FNO+Probe98.19 / 99.7899.40 / 100.0094.39 / 100.0098.45 / 100.0090.25 / 99.92
Table 5FNO+GS99.21 / 99.7799.24 / 100.0091.91 / 100.0096.68 / 99.9992.07 / 99.93
Table 5FNOFull99.05 / 99.7799.86 / 100.0093.58 / 99.9797.53 / 99.9894.77 / 99.98
Table 5GNOTDefault98.85 / 99.8599.83 / 100.0099.18 / 100.0099.99 / 100.0098.95 / 99.99
Table 5GNOT+Probe98.78 / 99.8199.95 / 100.0099.46 / 100.00100.00 / 100.0099.10 / 100.00
Table 5GNOT+GS98.81 / 99.8299.85 / 100.0099.60 / 100.00100.00 / 100.0098.56 / 99.99
Table 5GNOTFull99.07 / 99.6699.90 / 100.0099.44 / 100.0099.95 / 100.0098.78 / 100.00

The architecture and loss-study records are listed separately because some evaluation subsets differ, as detailed in Appendix G.1.

We have incorporated these geometric extraction success rates into Table 14 in Appendix G.1 of the revised manuscript, with the corresponding metric definitions provided in Appendices D.2 and D.3.

Concern 4

Question3: How stable are the main model rankings across different training seeds?

Reply:

We thank the reviewer for suggesting random seed experiments to assess ranking stability.

During the rebuttal period, to evaluate sensitivity to random initialization, we trained all models in Tables 4 and 5 across three training random seeds for 30k steps each, and report the mean and standard deviation across the three runs. We will extend these evaluations to additional random seeds in future work.

The updated results are presented in the following Table 4(part 1), Table 4(part 2), and Table 5.

Table 4(part 1). Main results across eight models. Within each scenario and metric, boldface marks minima. Results are means and standard deviations over three training seeds. Geometry errors are conditional means over successful extractions. Flux RMSE is in mWb/rad. Relative errors are dimensionless.

TaskModelParams(M)EψRMSE↓Eψrel↓E∇ψrel↓Eaxis↓ELCFS95↓EX−pt↓Ephys↓mWb/radcmcmcmtask T1U-Net1.9541.7±0.210.0102±0.000380.0200±0.00130.903±0.0980.917±0.0534.47±1.40.0839±0.016\begin{array}{llrrrrrrrr}\hline\text{Task}&\text{Model}&\text{Params(M)}&E_ψ^{\rm RMSE}↓&E_ψ^{\rm rel}↓&E_{∇ψ}^{\rm rel}↓&E_{\rm axis}↓&E_{\rm LCFS}^{95}↓&E_{\rm X-pt}↓&E_{\rm phys}↓\\\\&&&\text{mWb/rad}&&&\text{cm}&\text{cm}&\text{cm}&\\\\\hline\text{task }\mathbb{T}_1&\text{U-Net}&1.95&41.7±0.21&0.0102±0.00038&0.0200±0.0013&0.903±0.098&\mathbf{0.917}±0.053&4.47±1.4&0.0839±0.016\\\\\hline\end{array}

(Continued in Part 4.)

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response (1/1)

We thank all three reviewers for their time and constructive feedback. We are encouraged that all reviewers recognize the potential value of our datasets (Reviewers wdKu, BQAL, and 77kf), and that Reviewers wdKu and BQAL also highlight our task and metric design. Specifically, Reviewer wdKu notes our insights into the limitations of existing methods; Reviewer BQAL emphasizes the scarcity of configuration aligned EXL-50U simulation and experimental pairs as well as our discussion of loss terms; and Reviewer 77kf appreciates the paper's clear and accessible presentation.

Based on the reviewers' comments, we conducted additional experiments and clarified our experimental protocols and conclusions. Revisions span both the main text and appendices, with first-round changes highlighted in blue and second-round changes in red. Our main updates are summarized below:

  1. Training from scratch versus our original two stage source-domain pretraining and target-domain fine-tuning strategy: Following Reviewers wdKu and 77kf, we added a training from scratch control experiment using the same 5% target domain data to compare against our original source pretrained fine-tuning results on cross-device tasks (T4 and T5). The new results show that training from scratch is not separated from source-domain pretraining followed by target-domain fine-tuning in this single-seed control, indicating no consistent transfer gain from pretraining across these two cross-device tasks. These results and analysis are included in Appendix G (see our responses to Reviewers wdKu and 77kf).

  2. Loss weight sensitivity: Following Reviewer BQAL, we varied the physics loss weight for FNO on task T2 to examine its effect on flux field accuracy and Grad-Shafranov (GS) residual consistency. In this single-seed scan, increasing the physics weight steadily reduces the GS residual without corresponding gains in flux reconstruction accuracy. This study is included in Appendix G.2 (see our response to Reviewer BQAL).

  3. Consistency of experimental protocols and tables: Following Reviewer BQAL, we unified the training budget of the loss comparison (Table 5) with that of the architecture comparison (Table 4) at 30k steps, and populated all overlapping default loss entries from the exact same runs so that Tables 4 and 5 are consistent. The unified training protocol is detailed in Appendix F.3 (see our response to Reviewer BQAL).

  4. Variation across random seeds: Following Reviewers wdKu and BQAL, we ran three random seeds for all configurations in Tables 4 and 5 and now report means and standard deviations. This allows minor performance differences across models and loss functions to be interpreted alongside training variance. The evaluation protocol is described in Appendix F.4 (see our responses to Reviewers wdKu and BQAL).

  5. Geometric extraction success rates: Following Reviewer wdKu, we report LCFS and X-point extraction success rates alongside conditional geometric errors, making metric coverage explicit when interpreting conditional averages. The complete success rate table is provided in Appendix G.1, Table 14 (see our response to Reviewer wdKu).

  6. Reference equilibrium validation and runtime: Following Reviewer BQAL, we compared PTEFIT against offline EFIT across 520 shots to quantify differences between the reference reconstructions, added an offline-EFIT reference comparison on matched EXL-50U slices, and reported single-slice inference runtimes across all eight neural models alongside offline EFIT and literature-reported PTEFIT. The equilibrium comparison is reported in Appendix C.3, and the runtime comparison is reported in Appendix F.5 (see our response to Reviewer BQAL).

  7. Differences from existing benchmarks: Following Reviewer 77kf, we clarified how TokaReconBench differs fundamentally from the three earlier benchmarks in paired data availability, magnetic diagnostic inputs, and cross-domain transfer tasks. Thus there is an intrinsic data and problem formulation barrier between our work and previous benchmarks that cannot be easily overcome by using a strong AI agent. These differences are discussed in Appendix A (see our response to Reviewer 77kf).

  8. Benchmark positioning: Following Reviewer 77kf, we clarified the benchmark positioning of our work: rather than proposing a new neural network architecture or training loss, our contribution is to provide standardized datasets, realistic transfer tasks, and systematic baselines that uncover key limitations for future research (see our response to Reviewer 77kf).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.