acceptodds
Under review as a conference paper at ICLR 2027

Plasmax: Differentiable & Parallelizable Environments for Tokamak Control

16 views

Abstract

Controlling unstable plasma in tokamaks to improve nuclear fusion performance remains an open problem, and any progress in this area brings us closer to efficient nuclear fusion power for carbon-free energy. While Reinforcement Learning (RL) has great potential to tackle such complex dynamical control problems, research on its application to fusion remains sparse. One significant barrier is the lack of fast, configurable RL environments tailored to fusion applications. In this work, we introduce _Plasmax_: 17 differentiable, parallelizable environments for tokamak kinetic control with 5 physics models for dynamics simulation. Plasmax enables benchmarking RL methods against other open- and closed-loop strategies, evaluating robustness to real-world-inspired observation corruption (e.g., noise or delays), and studying transfer between simulators with different fidelities. We show that Plasmax is significantly faster than prior work, enabling up to a speedup and training an RL agent in 7 minutes on hardware accelerators. We benchmark a wide array of trajectory and policy optimization methods, including RL, Evolution Strategies (ES), and differentiating through the environment. We find that open-loop control optimized with CMA-ES performs 32% better than the best RL method we evaluate (SAC). We also investigate how trained policies generalize to different backends, how real-world-inspired observation corruptions affect training, and how exploration can be used as a bug-hunting tool to improve simulators. We hope Plasmax will foster collaboration between machine learning and fusion researchers to develop practical fusion energy.

Primary area:
datasets and benchmarks
Keywords:
Reinforcement Learning, Nuclear Fusion, Control, Generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kSegOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an open-source repository of parallizable and differentiable environments for tokamak control. They show that their approach can enable 24x to 1103x speedups over prior work and fast RL training. They also benchmark existing classical RL methods with some open-loop control techniques (e.g., CMA-ES) and some additional analyses.

Critical strengths
  • The problem of tokamak control for nuclear fusion is a trending and important problem to consider.
  • The proposed contribution to give a simulator benchmark makes sense that it can speed up RL research along this direction and be useful for researchers.
Critical weaknesses
  • The performance evaluation largely demonstrates throughput gains enabled by parallelization and jax compilation, rather than isolating any novel algorithmic or computational improvements. To my understanding, the 1103×1103\times speedup comes from comparing the environment steps per second of a CPU-based single-environment (nonparallelized) TORAX baseline, against their GPU-based (H100), 16384-environment (highly parallelized) same TORAX backend, using jax jitting. This kind of improvement is expected given the type of comparison made and jax's capabilities, and it is hard to understand what the scientific contribution this paper brings over the engineering aspect. In addition, the primary statistics are reported without considering the amortized compilation compute costs that come with jitting environment physics code. If the code is indeed compiled end-to-end, which could include loops etc., then this can be substantial for a complicated enough task.
  • A significant portion of the results section and figures analyze performance of open and closed loop algorithms (e.g., CMA-ES, PPO, SAC) on the simulator, but doesn't actually tell me anything new about why the proposed contribution (i.e., the simulator) is superior over existing simulator benchmarks. To my understanding, algorithms such as PPO and SAC are well-known standard algorithms, and the design of specific environments whose reward functions are heuristically tuned (Line 261). Thus, I feel I don't understand how figures such as 7 and 8 should be convincing me why Plasmax gives a good novel contribution.
Questions for authors
  1. What does ``compiled end-to-end''? Is it with respect to the physics code in the environment, or also touches the RL training? And what were the statistics for the compile time costs?
3 comments

W1 Our implementation is faster than prior work while preserving TORAX’s differentiability, which is a strength of our work. We show that even for a single environment on the same device, our implementation is 4.7x faster than Gym-TORAX (L302 of the main text).

We add compilation costs below and in the paper (Appendix C1): compilation is a small upfront cost (~70-80s) that is quickly amortized when running the simulator longer. We also note that JAX-compiled functions are cached locally after first compilation; loading time is consistently about 5 seconds in that case. The values in the table below are reported without caching, so they present the worst-case speeds. We measured SPS for one environment (no parallelization).

MetricGymTORAX-CPUQLKNN-CPUQLKNNTGLFNNBohm-GyroBohmCGMWorld Model
Compilation time (s)7.1621.1486.0177.8072.9572.431.55
SPS @ 1 step0.13880.04730.01160.01280.01370.01380.6428
SPS @ 10k steps25.7282.1713.0213.0613.7035.69276.46

W2: Plasmax is a benchmark for tokamak control. No such benchmark existed previously, since prior works (PopDownGym, Gym-TORAX) implement only a single environment with a single physics model. Additionally, we probe sim-to-sim transfer for different control methods across four physics simulators. Prior work only probes transfer within a selected simulator pair (Wang et al. 2025) or not at all. We found that, when given equal step budgets, untuned open-loop methods optimized with CMA-ES outperform other untuned baselines and generalize better. We felt this surprising finding merited explanation. Especially since the open-loop controllers are simple.

Q1: By “end-to-end” we mean compiling the entire policy training loop, including environment simulation, meaning it can run on the GPU without the additional overhead of communicating with the CPU. The latter is the usual case in RL, since environments are often simulated on the CPU instead. We highlight Figure 12 from the Appendix, which shows the times to compile and run the entire RL training loop across different backends. We give a code example of end-to-end compilation for training in the new Appendix F.3.

Thank you for the detailed response. It is helpful to see that this benchmark provides a 4.7x speedup on same envs and device. Compilation also seems reasonable when amortized over longer runs. I agree that having a faster benchmark allowing researchers to set up new experiments, can be a useful tool for the fusion community.

It seems that the baseline results are fairly sensitive to hyperparameter tuning, which is expected of the algorithms evaluated, and it will be helpful to explicitly acknowledge this limitation when drawing conclusions about the relative performance of different control methods.

I have increased my score to 3.

Thank you very much.

We've caveated our findings by mentioning that we compare untuned algorithms in the main text. We highlight the preliminary hyperparameter tuning results in Appendix H.

Reviewer_xT5eOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces Plasmax, a set of 17 parallelized and differentiable simulators for Tokamak control used in the fusion process built on top of the TORAX sim. It supports different dynamics for stepping through the different environments called backends and each task consists of hand crafted reward functions based on past knowledge. They also further compare performance of various different controllers in these environments i.e closed loop and open loop ones, and what is seen is that the open loop controllers perform the best in 11 out of the 17 environments

Critical strengths
  1. It is a very fast and broad testbed which runs thousands of environments in parallel on one GPU, 1103× faster than Gym-TORAX, across 17 tasks, 4 devices and 5 backends
  2. It has Transparent benchmarking with Full settings, 10 seeds and per-task results make the claims checkable, and the testing surfaced real TORAX bugs
Critical weaknesses
  1. The headline result is unsupported. 3 things favor a fixed schedule - reward is dominated by survival (barely separates good from mediocre control), most tasks have fixed goals and little randomness and the RL baselines are untuned and short-horizon. Also, the 11/17 count includes a tie, tuned PPO matches CMA-ES on ITER hybrid flat-top, and SAC wins on KSTAR, the one task where goals vary.
  2. The differentiability claim is unsupported. The gradient-based methods don’t learn at all in several tasks. And PopDownGym, which the paper mis-describes, is already a differentiable, GPU-parallel JAX environment.
Questions for authors
  1. Which wrappers and observations were active in section 5.2 and 5.3? And did closed-loop policies observe time ?
  2. How do the reference trajectory and a very simple PI controller score? Does tuned PPO or SAC with Gamma = 1 still trail open-loop CMA-ES on a few ramp-up or ramp-down tasks ?
  3. On the 5 tasks where closed-loop backprop ends at its untrained score, did it update at all or were its gradients masked as non-finite ?
  4. Non-finite states get a Reward of 0, but disruptions get a negative reward in ramp-down. Doesn’t Eq. (2) favor hitting NaNs over disrupting?
4 comments

W1 The reviewer claims that the experimental setup favors a fixed actuator schedule (open-loop controllers). We disagree; we were surprised that open-loop controllers performed so well.

  • “reward is dominated by survival, [favoring a fixed schedule]” - This is by design: survival is important. Disruptions in a tokamak cause material damage and need to be avoided, even if it means mediocre control. To us, it is not obvious how optimizing for survival would systematically benefit open-loop methods. If anything, longer episodes would make open-loop actions increasingly stale, which would be a disadvantage. We invite the reviewer to further justify this claim.
  • “[The reward being dominated by survival] barely separates good control from mediocre control.” We have added plots to the paper that show episode lengths for all environments and baselines (Figure 17). Interestingly, we find that methods with longer episode lengths (Figure 17) do not consistently achieve higher returns (Figure 16), which weakens the reviewer’s claim. We are open to the reviewers' interpretation of these results. In SPARC, we find that most algorithms perform very similarly on the two flat-top tasks (P-diff reward). We analyze and address this in the new Appendix G.
  • “RL baselines are untuned, [favoring a fixed schedule]”--as mentioned in the paper, we use existing hyperparameters that were tuned for continuous control in robotics. Our results simply show that RL algorithms underperform when untuned relative to other untuned methods. We do not make broad claims about their best ceiling performance, and we caveat our findings with tuning results in Appendix C3. For completeness, we have started tuning all baselines for all environments, and we will report these results as soon as the sweeps are completed.
  • “Most tasks have fixed goals and little randomness, [favoring a fixed schedule]” - fixed reward functions (goals) are standard in RL/control literature, and we find “little randomness” ambiguous. We interpret this as criticism of little variation in initial states. We sweep through different initial-state distributions and report the findings in Appendix G. Open-loop controllers remain robust to high variance in the initial state and action corruption (suggested by reviewer i6St). Prior work finds similar results in locomotion and manipulation tasks (see Appendix G).
  • “RL methods are short-horizon, [favoring a fixed schedule]” - this claim needs clarification; for SPARC ramp-down environments, which last 50 steps (see Table 1), a gamma of 0.99 covers the entire episode. In these environments, SAC and PPO underperform other baselines. Additionally, the tuned RL results in Appendix C3 suggest discount factors of 0.97 and 0.95 for ITER/hybrid/flattop, which has a horizon of 4400 steps, suggesting that a smaller discount factor is preferred in these environments. The authors of Brax find similar gammas for their 1000-step tasks, so this is not unique to our setting.
  • We corrected the 11/17 environment statements to account for the tie; we thank the reviewer for pointing this out.

W2 We’ve clarified that PopDownGym is a JAX environment that is also parallelizable and differentiable in the Related Work section. However, we note that Plasmax is a benchmark that supports multiple backends, scenarios, and wrappers, while PopDownGym implements a single RL environment. We also ran a new experiment (new Appendix E) inspired by the methodology in PopDownGym: we tune a backend to match another backend by tuning its parameters via backpropagation. We collect trajectories from the most computationally expensive backend and tune the computationally cheapest backend to fit this data. We show that the validation loss decreases by 30.9% during training, indicating that backpropagation is working as expected. Additionally, policies trained on this tuned backend transfer better to the target backend than those trained on the untuned one.

Q1: We added clarifications in the paper about the wrappers; all wrappers except for action quantization were active for all main experiments. This includes a wrapper that adds elapsed time to the observation.

Q2: We added reference-trajectory returns: holding the actuator values fixed at the actuator midpoint=(max-min)/2 for the episode. See Table 15, Appendix D.

Q3: Yes, when gradients are not finite, we mask these with 0. As noted in the paper, some of these gradients have high variance, which we attribute to repeated backpropagation through the simulator. The environments use the same backend, so performance differences across environments stem from stochasticity rather than the autodiff implementation.

Q4 The effect that reaching a non-finite state might be preferable to a disruption was indeed observed. Our rationale was that a non-finite state reflects a solver failure, not a controller failure, and is therefore preferable to a disruption. We have added clarifications in Appendix A3.

Thanks for the response. I'll keep the score and may revisit with the following questions answered:-

  1. Eq. (2) now gives solver failures a reward of 0, but the original version gave them the same reward as a disruption. Was the original a typo ?
  2. On KSTAR, all three backprop variants return exactly −34.19. How many of their updates were zeroed out because the gradients weren’t finite?
  3. In Fig. 24, closed-loop seems to end higher than open-loop at reset noise 0.3 and 0.5, which doesn’t match the G.1 text. Am I reading this wrong
  4. Open-loop CMA-ES only wins the ramp-up and ramp-down tasks, and on several flat-top tasks, fixed actuators do almost as well as the best method. Will you update the abstract and conclusion to say this?
  1. We proactively identified and addressed this issue after the submission, and we apologize for not explicitly informing the reviewer. It affected the appendix text and code. The original code gave the log penalty to TORAX-flagged solver failure, but the 0 reward to the Plasmax-flagged solver failure. We have since fixed this bug and updated the appendix accordingly, but some results in the paper were generated while the bug was still in place. We analyzed if this bug affected the results in the paper, and found this to be unlikely: the solver error affects 2.09% of logged episodes in 4 environments, with 0% in other environments. We estimated the impact of this bug on the results by recalculating the returns as if the bug was not in place. In practice, this means replacing the final reward on TORAX-flagged solver failure with 0 for the most affected environment and algorithm (ITER advanced ramp-up with PPO), we were able to do this as we had the trajectories saved locally. The final return is almost unchanged: it is reduced by 0.018%. We believe this to be strong evidence that this bug does not materially change our results.

  2. These 3 entries belong to an earlier set of runs during development where we had a bug in the KSTAR auto-diff (use of jnp.trunc rather than the differentiable jnp.clip). We’ve updated the table and Figures 7, 16, 17 in the paper. The original runs had 0 gradients, the updated runs all have non-zero and finite gradients. These updated results do not affect the aggregate ranking of the algorithms in the main text.

  3. Yes, closed-loop achieves higher final mean return at reset-noise scales 0.3 and 0.5; our previous wording was too broad. We’ve clarified the wording of L1931 to be more precise: “We would traditionally expect to see closed-loop methods perform better, as they can adapt to different initial states, which open-loop controllers cannot. Surprisingly, we see in Figure 24 that the relative gap between closed-loop and open-loop controllers is small, with a large performance gap in favor of closed-loop only appearing at the high reset-noise scales (noise scale=0.5).”

  4. We have updated the abstract, results, and conclusion to mention that holding actuator values fixed is a strong baseline. We additionally added these to the relevant figures (7, 8, 16, 17). We have recomputed the percentage improvements and updated accordingly. We find that hyperparameter tuning on individual tasks can help methods to significantly improve on this baseline (New Appendix H).

    • (Also related to Q4) Update on Hyperparameter tuning: the runs are not finished due to compute limitations, so we show early visualizations and results in Appendix H. We find that hyperparameter tuning makes a large difference for individual environments, and can significantly improve closed-loop methods. We caveat our findings that open-loop is most performant in the main text; we highlight that we compare untuned configurations. As they stand, the results suggest that no single hyperparameter setting seems preferred across the suite for each algorithm.

We thank the reviewer for their thoroughness. Regardless of their decision, we believe the paper (and code) is materially better thanks to their feedback. If the reviewer feels that their concerns were adequately addressed, we would appreciate it if they could reconsider their score.

Thanks, those answer my remaining questions. I'll update the score to a 3

Reviewer_i6STOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Plasmax, a differentiable, parallelizable simulation platform for tokamak kinetic control. It includes 17 tasks spanning four tokamaks and five simulation backends, with a unified control interface. The authors benchmark six RL and trajectory optimization approaches, and showcase cross-backend generalization, noisy-observation evaluation, and policy-driven simulator debugging.

Critical strengths
  1. This infrastructure is useful for fusion control research. Its unified API, ready-made tasks and multi-backend design lower the domain barrier for RL researchers setting up fusion simulation environments.
  2. JAX compilation, GPU parallelism and autograd accelerate simulation and optimization, speeding up algorithm prototyping.
  3. Baselines span RL, open-loop trajectory optimization and direct backpropagation. Coupled with cross-backend tests, the platform supports evaluating diverse control algorithms and generalization.
Critical weaknesses
  1. As an environment paper, usability and extensibility validation is limited. It remains unclear how much work is needed to plug in standard RL code, add new tasks, or swap physics models.
  2. Experiments lack convincing closed-loop challenges like abrupt disturbances or actuator faults. Stronger evidence is needed to validate the platform for feedback control and to link cross-backend transfer to real-control robustness.
Questions for authors
  1. Please supply a minimal example for integrating standard RL implementations and adding new tasks or backends.
  2. Can the platform model sudden disturbances or actuator degradation? Are constraint-violation or disturbance-recovery results available to illustrate closed-loop feedback benefits?
2 comments

W1: We show that extending environments and integrating them into standard RL training is straightforward by adding minimal examples to our paper (New Appendix F).

  • (Environments and backends). We create a new environment with higher state reset variance simply by extending the flattop.yaml file. Additionally, we provide a new backend (CGM-tuned), following suggestions from xT5e. The new backend and environments are registered in registry.py, and these changes add 22 LOC in total (most of which are YAML definitions). See Appendix F.
  • (Integration into standard RL training) We added two minimal training examples in Plasmax in an example/ folder with cleanRL and Rejax, two popular RL libraries. See the anonymous repository.

W2: We thank the reviewer for suggesting further ways to improve realism.

  • We added a CorruptedActionsWrapper that adds Gaussian noise to actuator values chosen by the agent. We show the results in the new Appendix G. The results show that as actuator noise increases, performance drops for both open- and closed-loop policies. Surprisingly, open-loop policies still seem to perform well under high action corruption. This was a surprising result to us, and upon further investigation, we found that prior work found similar results. We qualify this finding in more detail in the new Appendix G.

  • Disruptions and instabilities need to be modeled by the simulation backend. Unfortunately, TORAX does not support this, but we agree with the reviewer that these are crucial for improving control that transfers to the real world, and we hope future simulators implement them. We mentioned this in the limitations, and we have added further clarification.

We have not yet received a reply to our rebuttal. If the reviewer still has concerns, we would appreciate it if they could let us know. Otherwise, if their concerns have been addressed, we would invite them to increase their score.

Thank you.

Comments on the paper

3 comments
AuthorsOpenReview ↗

To all reviewers

We thank all reviewers for the time they spent reviewing the paper. We aimed to keep the rebuttals short for the benefit of the reviewers and the AC, and we did not use LLMs to generate any text. We did use LLMs to run experiments and plot results. We included additional details in the paper, but if the reviewers would like more detail, we are happy to provide it. Each reviewer's requests are different, and we focused on the ones we found to be most relevant. If a reviewer believes a specific experiment would help the paper, we invite them to suggest it explicitly. We added all new content in the paper in red.

AuthorsOpenReview ↗

Request for Engagement

After 5 days, we have received no reply to our rebuttals. Our rebuttals include:

  • Experiments to understand how noise in the initial-state distribution and the selected action affects the performance of closed- and open-loop controllers.
  • Clarifications on results and task definitions.
  • Throughput measurements that include compilation costs.
  • Clarifications and examples in the paper. This includes elaborating on prior work, details about extensibility of our API (environments and backends), and code examples.
  • The hyperparameter sweeps are still running; we are updating the paper with a preliminary version of the results.

We believe that the rebuttal addresses most points of criticisms the reviewers have brought to our attention, and should resolve any open questions. If concerns remain, we invite the reviewers to come forward and express them. Otherwise, if the concerns were addressed, we invite the reviewers to increase their score appropriately.

AuthorsOpenReview ↗

Summary of Rebuttal Period

We summarize the reviewers' requests and how we addressed them below.

  • Contribution and usability: What does Plasmax add, and how easily can researchers use or extend it? [i6ST, kSeg, xT5e]

    • Clarified Plasmax’s broader task and physics-model coverage relative to prior work, corrected the description of prior work, and added minimal training and environment/backend extension examples.
  • Speed and compilation: Are the speedups meaningful after accounting for hardware, parallelism, and compilation? [kSeg]

    • Clarified a 4.7× speedup on the same device, added compilation-inclusive timings, and clarified what is compiled.
  • Tuning: Do simple baselines and tuning change the conclusions? [xT5e, kSeg]

    • Added fixed-actuator comparisons, corrected the win count, and limited claims to untuned methods. Tuning results suggest it can substantially improves feedback controllers.
  • Open- vs closed-loop controllers: When does adapting to observations help? [i6ST, xT5e]

    • Added experiments with varied initial state distributions and noisy actions, clarified where feedback improves performance, and acknowledged limits in modeling sudden disturbances.
  • Differentiability and numerical correctness: Verification of the auto-differentiable property, and clarification on termination penalties [xT5e]

    • Added experiments highlighting auto-differentiability.
    • Corrected stale data in the results table in the Appendix and associated plots.
    • Corrected inconsistent termination penalties; a check found a negligible effect on returns.
  • Experimental setup: Which observations and wrappers were used? [xT5e]

    • Clarified the active wrappers and confirmed that feedback controllers observed elapsed time.

Reviewers xT5e and kSeg raised their ratings to 3. xT5e explicitly confirmed that their remaining questions were answered. i6ST has not yet replied.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.