- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an open-source repository of parallizable and differentiable environments for tokamak control. They show that their approach can enable 24x to 1103x speedups over prior work and fast RL training. They also benchmark existing classical RL methods with some open-loop control techniques (e.g., CMA-ES) and some additional analyses.
Critical strengths
- The problem of tokamak control for nuclear fusion is a trending and important problem to consider.
- The proposed contribution to give a simulator benchmark makes sense that it can speed up RL research along this direction and be useful for researchers.
Critical weaknesses
- The performance evaluation largely demonstrates throughput gains enabled by parallelization and jax compilation, rather than isolating any novel algorithmic or computational improvements. To my understanding, the speedup comes from comparing the environment steps per second of a CPU-based single-environment (nonparallelized) TORAX baseline, against their GPU-based (H100), 16384-environment (highly parallelized) same TORAX backend, using jax jitting. This kind of improvement is expected given the type of comparison made and jax's capabilities, and it is hard to understand what the scientific contribution this paper brings over the engineering aspect. In addition, the primary statistics are reported without considering the amortized compilation compute costs that come with jitting environment physics code. If the code is indeed compiled end-to-end, which could include loops etc., then this can be substantial for a complicated enough task.
- A significant portion of the results section and figures analyze performance of open and closed loop algorithms (e.g., CMA-ES, PPO, SAC) on the simulator, but doesn't actually tell me anything new about why the proposed contribution (i.e., the simulator) is superior over existing simulator benchmarks. To my understanding, algorithms such as PPO and SAC are well-known standard algorithms, and the design of specific environments whose reward functions are heuristically tuned (Line 261). Thus, I feel I don't understand how figures such as 7 and 8 should be convincing me why Plasmax gives a good novel contribution.
Questions for authors
- What does ``compiled end-to-end''? Is it with respect to the physics code in the environment, or also touches the RL training? And what were the statistics for the compile time costs?
3 comments
Thank you for the detailed response. It is helpful to see that this benchmark provides a 4.7x speedup on same envs and device. Compilation also seems reasonable when amortized over longer runs. I agree that having a faster benchmark allowing researchers to set up new experiments, can be a useful tool for the fusion community.
It seems that the baseline results are fairly sensitive to hyperparameter tuning, which is expected of the algorithms evaluated, and it will be helpful to explicitly acknowledge this limitation when drawing conclusions about the relative performance of different control methods.
I have increased my score to 3.
Thank you very much.
We've caveated our findings by mentioning that we compare untuned algorithms in the main text. We highlight the preliminary hyperparameter tuning results in Appendix H.
W1 Our implementation is faster than prior work while preserving TORAX’s differentiability, which is a strength of our work. We show that even for a single environment on the same device, our implementation is 4.7x faster than Gym-TORAX (L302 of the main text).
We add compilation costs below and in the paper (Appendix C1): compilation is a small upfront cost (~70-80s) that is quickly amortized when running the simulator longer. We also note that JAX-compiled functions are cached locally after first compilation; loading time is consistently about 5 seconds in that case. The values in the table below are reported without caching, so they present the worst-case speeds. We measured SPS for one environment (no parallelization).
W2: Plasmax is a benchmark for tokamak control. No such benchmark existed previously, since prior works (PopDownGym, Gym-TORAX) implement only a single environment with a single physics model. Additionally, we probe sim-to-sim transfer for different control methods across four physics simulators. Prior work only probes transfer within a selected simulator pair (Wang et al. 2025) or not at all. We found that, when given equal step budgets, untuned open-loop methods optimized with CMA-ES outperform other untuned baselines and generalize better. We felt this surprising finding merited explanation. Especially since the open-loop controllers are simple.
Q1: By “end-to-end” we mean compiling the entire policy training loop, including environment simulation, meaning it can run on the GPU without the additional overhead of communicating with the CPU. The latter is the usual case in RL, since environments are often simulated on the CPU instead. We highlight Figure 12 from the Appendix, which shows the times to compile and run the entire RL training loop across different backends. We give a code example of end-to-end compilation for training in the new Appendix F.3.