acceptodds
Under review as a conference paper at ICLR 2027

Jaxnasium: Embracing PyTrees For Any-Agent, Cross-Suite Reinforcement Learning

Abstract

Hardware-accelerated reinforcement learning in JAX has gained rapid popularity in recent years and has proven to be a successful method of scaling online reinforcement learning. The resulting ecosystem, however, is fragmented: environment suites define their own interfaces, and algorithm implementations are typically written for a single suite, a single type of space, and either JAX-native or host-based environments. Here we present Jaxnasium, a library for JAX-based reinforcement learning, aiming to make the field more accessible. Jaxnasium represents observation and action spaces as PyTrees, and uses this structure to build networks and support multi-agent processing. Through this, its near-single-file implementations run on discrete, continuous, image, and composite spaces, as well as in multi-agent environments, without any changes to the algorithm code. The same files serve as readable, research-friendly algorithms, as well as importable algorithms for more applied workloads. Through a minimal Gymnasium-style interface and wrappers for 11 suites, over 300 environments are available on which a small PPO implementation with default hyperparameters is able to run. Additionally, through a simple wrapper, we can run Gymnasium and EnvPool environments with the same algorithm and training loop, with no performance loss compared to architectures specifically built for training JAX algorithms on host-based environments. Jaxnasium further provides tools for multi-seed evaluation, hyperparameter sweeps, and project scaffolding. Jaxnasium is open-sourced at https://anonymous.4open.science/r/jaxnasium-2909.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Reinforcement Learning, Jax, Software Library, End-to-end RL in Jax

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DCZ4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

They introduced Jaxnasium, a JAX-based reinforcement learning library that supports most benchmarks with diverse observation and action spaces, as well as multi-agent settings, using PyTrees, which represent nested data structures in Python. The library also supports non-JAX environments, such as Gymnasium and EnvPool. On the algorithmic side, they implemented PPO, SAC, DQN, and PQN, covering widely used algorithms for both continuous and discrete action spaces. They demonstrated that their PPO implementation can run on more than 300 registered environments. They also compared the execution speed of Jaxnasium with that of the original environments and observed no performance degradation.

Critical strengths

As someone who regularly works with JAX-based RL codebases, I found the following two aspects particularly appealing.

  • Unified API for diverse environments: As the authors point out, different RL libraries tend to use different APIs (e.g., Gym-style, DM-style, and state-based APIs), which makes it cumbersome to transfer RL code across libraries. This issue is not limited to the JAX RL community.
  • Extensive evaluation of execution speed: I also appreciated the broad evaluation of Jaxnasium's execution speed.
Critical weaknesses

That said, I identified the following two main weaknesses.

  • Limited novelty beyond software unification: While the unification is quite useful, the idea of integrating different benchmarks under a single API is not novel. Furthermore, the paper does not report the learning performance of the implemented algorithms, which makes it difficult to assess the validity of their implementations and places an additional burden on users to verify their correctness and tune the hyperparameters themselves.
  • Multi-agent integration as an overextension of the scope: While it is appealing that policies can be trained in multi-agent environments through a unified API, multi-agent RL introduces several fundamental differences from single-agent RL that require dedicated mechanisms, such as centralized training with decentralized execution (CTDE), parameter sharing, and joint value functions. Although the authors discuss some of these mechanisms, these mechanisms cannot be incorporated simply by extending single-agent algorithms through PyTrees.
Questions for authors

Can authors provide performance report of the algorithms on representative benchmarks to validate their implementations and default hyperparameters?

5 comments

We thank the reviewer for their constructive feedback. We are glad the unified interface appeals to someone who regularly works with JAX-based RL codebases, as this friction of moving code between suites and libraries is one of the reasons Jaxnasium exists. Below we answer the question and both weaknesses. All changes are in the uploaded revision.

Question 1: Can authors provide performance report of the algorithms on representative benchmarks to validate their implementations and default hyperparameters? (Weakness 1: no learning performance reported)

Yes, and we agree this was missing. The revision adds two sets of experiments:

  1. Validating the implementations. We compare PPO and DQN against PureJaxRL and PQN against PureJaxQL on 18 Gymnax environments (10 seeds, 10M steps; DQN and PQN on the 14 with discrete actions), and SAC against Brax on 15 MuJoCo Playground tasks (10 seeds, 3M steps), using the references' hyperparameters and networks. Jaxnasium closely tracks the references: see Section 4.3 (Figure 3), with per-environment curves in Appendix F (Figures 9-13).
  2. Validating the default hyperparameters. We train all four algorithms with their untuned defaults on every registered environment (430 by now; 8 seeds, 3M steps) and compare them with a random policy (one-sided Welch's t-test). PPO significantly outperforms random on 377 environments (88%), PQN, SAC and DQN on 77%, 70% and 64% (Appendix A, Figure 5).

We want to be clear about what this does and does not show: the defaults give a learning signal on most environments without tuning, but we do not maintain per-environment SOTA tuned hyperparameters.

(Weakness 1: Limited novelty beyond software unification)

We agree that integrating benchmarks under one API is not novel in itself, and the revised Section 3 now states this explicitly. The unification is a means to an end in our paper and primarily exists to give our algorithms a single target. What we do believe is novel when it comes to our API:

  • Spaces are plain PyTrees, and each algorithm builds its networks by mapping over them. To the best of our knowledge, no other JAX library has algorithms that natively operate on nested composite spaces, including hybrid discrete-continuous actions.
  • As a result there is one small file per algorithm, instead of one per space type, per suite, or per JAX-native and host-based setting, and we show the latter costs no throughput (Section 5).
  • The same files are importable as a library, which usually comes at the cost of readable single-file implementations.

Appendix D adds a comparison with nearby libraries.

(Weakness 2: Multi-agent integration as an overextension of the scope)

We hope to change the reviewer's mind here. Multi-agent support has been a core pillar since we started building Jaxnasium close to two years ago: one of the reasons for its inception was to manage several projects, some of them multi-agent, from shared/inherited codebases, and it has proven highly useful to us.

We have extended Section 4.2 to make clearer how the multi-agent support works. Briefly: on a multi-agent environment, an algorithm creates one agent per entry of the space PyTree and places them in a multi-agent wrapper. Per-agent methods are dispatched per agent by default (via jax.vmap where possible, else jax.tree.map), while methods marked @collective are called on the whole team. On the reviewer's examples:

  • CTDE works out of the box: when an environment returns a critic observation, it is routed to the value and Q-networks only. We now validate this on eight cooperative JaxMARL tasks (10M steps, 16 seeds; Appendix F.3): a centralized critic helps under partial observability and makes no difference where agents already observe the full state, in line with prior work.
  • Parameter sharing fits in a small subclass of the multi-agent wrapper, for which our documentation has an example.
  • Joint value functions are not provided out of the box, but do not require changing the training loop: an agent subclass can override the per-agent update with a team-level method (@collective). We do not ship or benchmark such a subclass in this submission, but hope this sheds some light on what the mechanism can do and look to provide more documantation examples later.

Overall, we do not aim to replace dedicated multi-agent libraries. What Jaxnasium offers is that the algorithm file someone modifies for a single-agent idea, is the same file that gives an IPPO or MAPPO baseline on, for example, JaxMARL, so an idea can be tested in both settings without a second codebase. And as each algorithm is a single file that can be copied into a project through our CLI (Section 6), it is a practical starting point for more specialized methods.

We hope these changes resolve your concerns and, if they do, that you would consider updating your rating. We are happy to answer any follow-up questions.

Thank you for the detailed rebuttal and additional experiments. I would like to clarify which of my concerns have been resolved and which remain.

For performance report, the comparisons with reference implementations, using matched hyperparameters and network architectures, provide evidence for the correctness of the implementations (and environments). This resolves my concern about the lack of performance validation. The comparisons against random policies are also useful.

For multi-agent integration, I now better understand how the proposed abstraction supports multi-agent settings, particularly CTDE, which is also validated experimentally. As done by works such as PQN, evaluating algorithms on both single-agent and multi-agent benchmarks can be a nice benchmark format to show the generality of the algorithm. I therefore recognize the certain practical benefits of sharing algorithm implementations across these settings, particularly for baselines such as IPPO and MAPPO.

However, at the same time, multi-agent RL still requires specialized mechanisms, such as value decomposition (e.g., QMIX), opponent shaping, to name a few. These go beyond simply applying single-agent algorithms to multiple agents. While I now understand that Jaxnasium allows such extensions, I believe that the benefit of applying the same algorithm implementations to both single-agent and multi-agent settings is thus limited.

About the novelty: I am not convinced by the novelty and practical benefit of the pytree-based abstraction. As the authors acknowledge, unifying environment interfaces and supporting nested spaces are not novel in themselves, with related approaches already proposed in libraries such as Stoa. While I recognize that Jaxnasium goes further by unifying algorithm implementations, I still find its conceptual novelty limited.

For practical benefit, observation-agnostic algorithms can already be implemented by delegating observation processing to separate networks, without requiring pytree-based space abstractions. Moreover, network architectures often need to be customized for individual tasks, meaning that researchers would need to implement task-specific networks anyway, which may limit the benefits of automatically constructing encoders for each observation leaf. While I appreciate the convenience of the unified design, I remain unconvinced that it offers advantages over conventional approaches without pytree.

Thank you for the follow-up, and for laying out which concerns are resolved and which remain. Glad to hear the new experiments were helpful. About the remaining points:

Shared implementation for multi-agent RL

We agree that multi-agent RL has more specialized mechanisms, and that not including these out of the box may feel limiting. At the same time, we are glad you see practical benefits for baselines such as IPPO and MAPPO. These are widely used and have been found competitive with more specialized methods (de Witt et al., 2020; Yu et al., 2022; Papoudakis et al., 2021). Specialized methods also typically start from such an algorithm.

While we considered integrating some mechanisms as algorithm options, we opted not to. Instead, we document advanced use cases in a similar fashion as the Equinox package, where advanced networks are not included in the package but deferred to documentation examples (e.g. for unet). We follow this approach for the following reasons:

  1. It keeps the overall API of our algorithms slim.
  2. It lessens the maintenance burden, especially for features that have not yet been requested.
  3. Libraries implementing advanced multi-agent methods already exist, including in JAX (Mava).
  4. As the above already exists (3), rather than providing another set of pre-built solutions, we aim to provide tools such that researchers can customize their algorithms for their use cases or research (e.g. scaffolding an algorithm, subclassing, overriding methods with decorators).

These choices may change with (community) feedback. For now, we believe this way we can bring value (it has for us) while keeping the project maintainable. We will add examples of more mechanisms to our documentation.

Novelty and practical benefits

We agree that the individual ideas are not new, and the paper states that neither a unified interface nor nested spaces are. Our contribution on the algorithm side is practical: small, readable implementations that run unchanged across space types, suites, single- and multi-agent settings, and JAX-native and host-based environments.

Whether PyTrees are required for this: observation-agnostic algorithms can indeed be created by delegating observation processing to separate networks. Jaxnasium does exactly this, but builds these networks automatically from the observation structure; they only have to be callable modules, and the defaults are optional. We agree that networks are often customized per task, but note that may often only be in part (e.g. an image encoder), and the remaining parts can stay at their automatic default. Furthermore, PyTrees are used for more than creating networks:

  1. Everything around the network. The replay buffer, running statistics for normalization, space wrappers, and multi-agent dispatch are all instantiated from and operate on leaves on the same PyTrees.
  2. Actions. For composite and hybrid action spaces, not only the network follows the pytree structure. Each action leaf gets its own head, and sampling, log-probabilities, entropies, Q-targets, ε-greedy exploration, and action masks (given as a PyTree matching the action space) are all computed per leaf and combined, as a mapping over the action PyTree rather than a special case per space type (We added new experiments for hybrid spaces in host-based environments to Appendix G).

Indeed, none of this is impossible without PyTrees. Our argument is that it takes considerably less code. PyTrees are what allow a single small file per algorithm to cover all of these settings, and that same file also covers host-based Gymnasium environments. For benchmarking, a reference implementation that runs on a new environment unmodified leaves fewer implicit choices to whoever runs the baseline. Furthermore, for research, a single small file is a convenient starting point: since we expect some users that will adapt algorithms to their task or research, we encourage copying our algorithms through a CLI tool.

We understand that you may still weigh the conceptual novelty differently, and appreciate that you recognize the convenience of the unified design. We hope these clarifications help show its broader practical value as a software library, and thank you again for the constructive discussion.


  • Yu, Chao, et al. "The surprising effectiveness of ppo in cooperative multi-agent games." Neurips Datasets and Benchmarks Track (2022)
  • Papoudakis, Georgios, et al. "Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks" Neurips Datasets and Benchmarks Track (2021)
  • De Witt, Christian Schroeder, et al. "Is independent learning all you need in the starcraft multi-agent challenge?." (2020).
Reviewer_mDsyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents Jaxnasium, a JAX RL library that represents observation and action spaces as pytrees of primitive spaces. Networks are built by mapping over these trees: each observation leaf gets an encoder and each action leaf gets an output head matching its type. This allows near-single-file algorithm implementations that handle discrete, continuous, image, composite, and multi-agent spaces without separate code paths. Multi-agent environments are treated as pytrees with one entry per agent, giving independent learners by default. The authors wrap 333 environments from 11 JAX suites, and PPO with default hyperparameters runs on over 300 of them. Gymnasium and EnvPool environments run inside the same compiled training loop via callbacks, with no throughput penalty relative to CleanRL, SB3, and Stoix on a single GPU. The library also includes tooling for project scaffolding, multi-seed evaluation, and hyperparameter sweeps.

Critical strengths
  • No throughput penalty relative to CleanRL, SB3, and Stoix on a single GPU
  • Single-file implementations of RL algorithms
  • Wrappers for existing RL libraries
  • Works with over 300 environments across JAX environment libraries
Critical weaknesses
  • Novelty relative to prior work. Nested spaces aren’t new. RLlib handles Dict/Tuple observation and action spaces, SB3 handles Dict observations, and Stoa already provides a JAX-native wrapper interface across suites. The contribution is combining these in JAX with a clean API, and the paper should position itself explicitly against these. Currently, Stoa gets one sentence.
Questions for authors
  • What does Jaxnasium offer beyond RLlib’s and SB3’s nested-space handling and Stoa’s interface?
  • Unsupported environments: PPO runs on over 300 of the 333 registered environments. Which environments does it not run on, and why (e.g., unsupported space types, interface limitations, or failures during training)?
1 comment

We would like to thank the reviewer for their constructive feedback. We answer both questions below and adress the criticism. All changes are in the uploaded revision.

Question 1: What does Jaxnasium offer beyond RLlib's and SB3's nested-space handling and Stoa's interface? (Weakness 1: Position Jaxnasium explicitly against prior work. Novelty is unclear)

We agree that neither nested spaces nor a common interface is new in itself, and have now made this explicit in Section 3 of the revised paper. A comparison table across libraries is added in Appendix D.

What PyTrees add. A space in Jaxnasium is a plain PyTree instead of a Dict or Tuple class, which is a more general representation: any nesting of standard Python containers is a valid space. Because JAX already operates on PyTrees, algorithms need little extra code to handle such a space, so each implementation stays a single small file that still runs on any space type. Additionally, we can use the same abstraction for multi-agent environments (one entry per agent), without a separate interface or code path.

RLlib and SB3. Neither is aimed at end-to-end reinforcement learning in JAX, and to the best of our knowledge no JAX library has algorithms that natively operate on nested spaces. Additionally, SB3 supports Dict observations but no composite actions, and RLlib flattens nested observations by default. Finally, both are large frameworks, whereas small single file implementations are often prefered by researchers for algorithmic changes, and those are usually written for one space type. We believe a small starting point that operates on any space benefit these algorithmic changes by making them more widely applicable and, in turn, ease their comparison across a wider set of tasks..

Stoa. Stoa is the closest work to our interface: both provide a common functional interface with adapters for existing JAX suites. What is new in Jaxnasium is that the interface is built on PyTrees. Spaces are plain PyTrees instead of dedicated Dict and Tuple classes, a multi-agent environment is the same interface with one entry per agent, and host-based Gymnasium and EnvPool environments run through it within the same compiled training loop. The interface is also designed together with the algorithms that use it: they consume composite spaces directly, whereas, as far as we are aware, existing JAX algorithm libraries operate on single-array observations and discrete or continuous actions.

The interface is also only one part of the paper. The same algorithm files serve JAX-native and host-based environments without loss of throughput (Section 5), train in multi-agent environments without code cahnges (Section 4.2, Appendix F.3), and Jaxnasium comes equiped with tooling for multi-seed evaluation, hyperparameter sweeps and project scaffolding (Section 6).

Question 2: Unsupported environments: which environments does PPO not run on, and why?

We realize the paper lacked a clear number and experiments to back it up. We have added an experiment in which we run all four algorithms on every registered environment (Appendix A, Figure 5, with a summarized image in Figure 1). The registry has grown from 333 to 430 environments since submission, and all four algorithms run on 427 of them (330 of the original 333). For some, this requires standard space wrappers (e.g. discretization for PQN/DQN on continuous environments, or flattening the action space for dependent action masks in Jumanji). The environments that do not run are the three JaxMARL storm tasks. These differ from the rest of JaxMARL in what they return, and each deviates in its own way. Rather than special-casing each of these, we prefer to address this upstream in the near future. They have remained in the registry so that anyone can still make them. Our CI test suite also includes a test for every registered environment to ensure each algorithm builds and runs on each of them, with a few exceptions due to versioning conflicts.

We hope these additions address your concerns and, if so, that you would consider updating your score. We are happy to answer any further questions.

Reviewer_C8dQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Jaxnasium which is a lightweight RL library which is built on JAX and Equinox. The main design idea is to use arbitrary PyTrees for observation and action spaces. The leaves of these PyTrees are primitive spaces. Observation leaves are mapped to encoders and action leaves are mapped to distribution or value heads. The benefit of such design is that the same implementations of PPO, SAC, DQN and PQN can be reused for discrete / continuous / image-based / composite spaces.

Authors extend their method to fixed structure multi-agent settings by placing one agent at the top of PyTree. Also Jaxnasium defines a small Gymnasium-ish functional environment interface. They wrap 11 JAX environment suites with 333 registered environments and use callbacks so that Gymnasium and EnvPool environments can enter the same training loop.

The empirical section is mostly about throughput with PPO compared agains Stable Baselines 3, CleanRL-JAX, Stoix across JAX environments in Gymnasium, EnvPool, EnvPool-XLA. They use RTX 5090 for their experiments.

Good addition is that the library allows multi-seed evals and hyperparameter sweeps.

Critical strengths
  1. The paper uses a neat abstraction that can reduce a lot of duplicated code. Reusable structure with composite spaces as ordinary PyTrees for encoders, action heads can allow researches get single file algorithms and iterate quickly on their experiments.
  2. One functional interface for Gymnasium and EnvPool environments in one training loop does not appear to reduce throughput, but allows simplification and readability.
Critical weaknesses
  1. The paper mainly shows speed figures and it would be good to show learning curves as well. Section 5 Figure 3 test PPO throughput on 6 simple environments (lines 372 - 412). Appendix C adds more speed results. I have not found any learning curve / RL returns / success rates. SAC, DQN, PQN are listed but not supported with eval figures in the paper (lines 270-279). So Figure 1 claim that the library can "train over 300" environments (lines 105-107) is not accurate.
  2. The limits of "arbitrary" spaces and multi-agent support are unclear. Figure 2 uses one output head per action leaf (lines 283-310), but the paper does not explain whether those action parts can depend on each other. Section 4.2 uses independent learners by default (lines 335-338). Section 7 says more complex multi-agent methods need algorithm changes (lines 462-465).
  3. The paper lacks a direct comparison in terms of supported spaces, algorithms, or API with nearby libraries likeStoa, Stoix, PureJAXRL, CleanRL.
Questions for authors
  1. Which tasks were tested for each algorithm? Please report learning curves or final RL returns.
  2. Is the joint action distribution factorized across leaves? How are leaf values are combined?
  3. Listing 1 reuses step_key for action sampling and env.step (lines 221-226). Is this intentional and what is the motivation of such a design choice?
1 comment

We thank the reviewer for the careful reading and constructive feedback. Below we answer each question and weakness. All changes are in the uploaded revision.


Question 1: Which tasks were tested for each algorithm? Please report learning curves or final RL returns.
(Weakness 1: Lack of learning curves & claim of the Figure 1 "train over 300")

We agree that this has been an omission from the original submission. In our revised draft, we have added multiple experiments to fill this gap:

  1. Reference comparison. We compare PPO and DQN against PureJaxRL and PQN against PureJaxQL (the original PQN repository) on 18 Gymnax environments (10 seeds, 10M steps; DQN and PQN on 14 environments with discrete actions), and SAC against Brax on 15 MuJoCo Playground tasks (10 seeds, 3M steps). All algorithms use the references' hyperparameters and networks. The results show that Jaxnasium algorithms closely track the references' performance. Summarized results are given in Section 4.3 (Figure 3), with extended details and per-environment curves in Appendix F (Figures 9-13).
  2. Full registry. To support the claim of Figure 1 (runs on 300+ environments), we trained all four algorithms with default hyperparameters on every registered environment (8 seeds, 3M steps) and compared them with a random policy (one-sided Welch's t-test) to test not only whether the algorithms run, but also whether their default hyperparameters yield a significant learning signal. Since submission the registry has grown from 333 to 430 environments. All four algorithms run on 427; the three exceptions, JaxMARL storm, tasks, have different return signatures compared to the rest of the stack, and each deviates in its own way.

PPO significantly outperforms random on 377 (88%), with a mean probability of improvement of 0.94; PQN, SAC and DQN do so on 77%, 70% and 64%. Because enlarging the registry may seem like we moved the goalposts: on the original 333 environments, PPO runs on 330 and outperforms random on 298 (89%, probability of improvement 0.95). Some combinations need a standard space wrapper: (e.g. eight Jumanji tasks with joint action masks use a flattened action space). Figure 1 is partly adjusted to contain a summary of these new results, with the full results appended to Appendix A (Figure 5). We additionally note that registered environments are also tested with each algorithm in our CI on every new version (with a few version-related exceptions)..

Question 2: Is the joint action distribution factorized across leaves? How are leaf values are combined? (Weakness 2: Unclear limits of "arbitrary" spaces and multi-agent support)

Yes. By default the joint policy is a product over the action leaves (and over the dimensions within a leaf), each head conditioned on a shared body. Leaf values are combined as follows:

Log-probabilities and entropies are summed over the leaves (PPO directly uses these). For Q-values (DQN, PQN, SAC), the value of a composite action is the mean of the selected leaf values [1]. Target values in this case are also computed per leaf and then averaged. This avoids exponentially growing joint action spaces and is common practice. The downside is that leaves do not condition on each other's samples. For dependent actions or joint action masks, Jaxnasium provides a wrapper that flattens the action space, and an autoregressive head can be supplied as a custom network. We have added clarification on this in short form to Section 4.1, and in extended form to Appendix E.

For multi-agent, independent learners are the default. Centralized critics (CTDE) are supported when the environment supplies a critic observation, which we now validate on eight JaxMARL tasks (Appendix F.3), and parameter sharing fits in a thin wrapper. More elaborate methods, such as QMIX-style value mixing, require subclassing agents or scaffolding and editing the algorithm file. Sections 4.2 and 7 now state this explicitly.

Question 3: Listing 1 reuses step_key.

Thank you for spotting this. This was a mistake in the hand-written listing and is addressed in the revised version.

Weakness3: lacks a direct comparison ... with nearby libraries.

Section 3 now positions the interface against Stoa, RLlib and Stable-Baselines3, and Appendix D adds a table comparing nearby libraries in terms of spaces, algorithms and use. To our knowledge, no JAX library has algorithms that natively handle nested composite (hybrid) spaces, nor do existing libraries provide small (copy-able) single-file implementations that can additionally be imported and used as SB3-like classes.

We hope this adresses all your questions and concerns, and in that case, that this may positively affect your score. Please let us know if you have any follow up questions.

[1] Tavakoli et al. Action branching architectures for deep reinforcement learning. AAAI, 2018.

Comments on the paper

1 comment

As the discussion phase is coming to a close, we would like to thank all those involved again for their time spent on this reviewing period, as well as providing constructive criticism for us to improve the paper.

As an overview, we would like to briefly summarize the changes between the submitted and current paper below, followed by a short note on the remaining discussion.

  • Section 4.3 and Appendix F add learning curves against reference implementations, using their hyperparameters and networks: Gymnax environments are evaluated in F.1 and MuJoCo Playground in F.2. F.3 compares independent learners versus CTDE on eight cooperative multi-agent environments, without changes to the algorithms. (R1, R3)
  • Four suites have been added, raising the registry from 333 environments in 11 suites to 430 in 15. Numbers in the abstract and throughout the text are updated to match.
  • Figure 1 is updated, and an extended version is given in Appendix A. Experiments shown here now back the claim that all algorithms run on 427 of the registered environments and show on how many environments the default hyperparameters provide a learning signal by significantly beating random (PPO on 88%). The three environments that do not run are now explicitly listed, along with the reason why. (R1, R2, R3)
  • Section 5, Appendix A and the Limitations explicitly state that beating random should not be interpreted as solving the task. (R1)
  • New Appendix G runs similar evaluations (to Appendix A) for host-based environments with hybrid discrete-continuous action spaces. These show that the algorithms natively operate on these spaces, and that they also operate on host-based (Gymnasium) environments. (R1)
  • New Appendix H repeats these experiments on the environments with spatial observations, using a CNN instead of flattening, and compares both. (R3) (Currently, this is done for PPO and PQN due to the remaining time available. We will add the remaining two algorithms to the camera ready version, but expect the same trend to hold).
  • Section 4.1 and Appendix E now describe how different action spaces are combined by default, what the limitations of this are, and how to work around them. The limitations imposed by the default factorization have also been added to the Limitations. (R1)
  • Section 4.2 and the Limitations describe the multi-agent transformation more clearly, together with what works out of the box or requires further modifications (we provide examples for this). (R1, R3)
  • Section 2 and Section 3 have been expanded to better position the work in the current landscape and to state more explicitly which parts existed before. Appendix D adds a comparison table for nearby libraries in terms of library API, algorithms and supported spaces. (R1, R2, R3)
  • Small fixes (typos, PRNG key use in Listing 1 (R1)), reproducibility statement added, and the text has been tightened throughout.

A remaining concern is that complex settings often require custom modifications, reducing the practical value of our unification there. We agree that such settings require customizations, and (as a general RL library) we cannot (and would rather not) anticipate each of them. In our view, this is where our small implementations help: because a single implementation covers a lot of environment suites, while remaining compact and readable, it serves as a convenient starting point that researchers can copy and modify (e.g. via our CLI). Furthermore, as the same algorithms run on most well-known JAX suites as well as host-based Gymnasium and EnvPool environments, we believe they are a useful tool for benchmarking and reduce friction to those entering the field.

Finally, while Reviewer mDsy did not respond to our reply, we hope their concerns are addressed either through our reply or changes to our revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.