- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
They introduced Jaxnasium, a JAX-based reinforcement learning library that supports most benchmarks with diverse observation and action spaces, as well as multi-agent settings, using PyTrees, which represent nested data structures in Python. The library also supports non-JAX environments, such as Gymnasium and EnvPool. On the algorithmic side, they implemented PPO, SAC, DQN, and PQN, covering widely used algorithms for both continuous and discrete action spaces. They demonstrated that their PPO implementation can run on more than 300 registered environments. They also compared the execution speed of Jaxnasium with that of the original environments and observed no performance degradation.
Critical strengths
As someone who regularly works with JAX-based RL codebases, I found the following two aspects particularly appealing.
- Unified API for diverse environments: As the authors point out, different RL libraries tend to use different APIs (e.g., Gym-style, DM-style, and state-based APIs), which makes it cumbersome to transfer RL code across libraries. This issue is not limited to the JAX RL community.
- Extensive evaluation of execution speed: I also appreciated the broad evaluation of Jaxnasium's execution speed.
Critical weaknesses
That said, I identified the following two main weaknesses.
- Limited novelty beyond software unification: While the unification is quite useful, the idea of integrating different benchmarks under a single API is not novel. Furthermore, the paper does not report the learning performance of the implemented algorithms, which makes it difficult to assess the validity of their implementations and places an additional burden on users to verify their correctness and tune the hyperparameters themselves.
- Multi-agent integration as an overextension of the scope: While it is appealing that policies can be trained in multi-agent environments through a unified API, multi-agent RL introduces several fundamental differences from single-agent RL that require dedicated mechanisms, such as centralized training with decentralized execution (CTDE), parameter sharing, and joint value functions. Although the authors discuss some of these mechanisms, these mechanisms cannot be incorporated simply by extending single-agent algorithms through PyTrees.
Questions for authors
Can authors provide performance report of the algorithms on representative benchmarks to validate their implementations and default hyperparameters?
5 comments
Thank you for the detailed rebuttal and additional experiments. I would like to clarify which of my concerns have been resolved and which remain.
For performance report, the comparisons with reference implementations, using matched hyperparameters and network architectures, provide evidence for the correctness of the implementations (and environments). This resolves my concern about the lack of performance validation. The comparisons against random policies are also useful.
For multi-agent integration, I now better understand how the proposed abstraction supports multi-agent settings, particularly CTDE, which is also validated experimentally. As done by works such as PQN, evaluating algorithms on both single-agent and multi-agent benchmarks can be a nice benchmark format to show the generality of the algorithm. I therefore recognize the certain practical benefits of sharing algorithm implementations across these settings, particularly for baselines such as IPPO and MAPPO.
However, at the same time, multi-agent RL still requires specialized mechanisms, such as value decomposition (e.g., QMIX), opponent shaping, to name a few. These go beyond simply applying single-agent algorithms to multiple agents. While I now understand that Jaxnasium allows such extensions, I believe that the benefit of applying the same algorithm implementations to both single-agent and multi-agent settings is thus limited.
About the novelty: I am not convinced by the novelty and practical benefit of the pytree-based abstraction. As the authors acknowledge, unifying environment interfaces and supporting nested spaces are not novel in themselves, with related approaches already proposed in libraries such as Stoa. While I recognize that Jaxnasium goes further by unifying algorithm implementations, I still find its conceptual novelty limited.
For practical benefit, observation-agnostic algorithms can already be implemented by delegating observation processing to separate networks, without requiring pytree-based space abstractions. Moreover, network architectures often need to be customized for individual tasks, meaning that researchers would need to implement task-specific networks anyway, which may limit the benefits of automatically constructing encoders for each observation leaf. While I appreciate the convenience of the unified design, I remain unconvinced that it offers advantages over conventional approaches without pytree.
Thank you for the follow-up, and for laying out which concerns are resolved and which remain. Glad to hear the new experiments were helpful. About the remaining points:
Shared implementation for multi-agent RL
We agree that multi-agent RL has more specialized mechanisms, and that not including these out of the box may feel limiting. At the same time, we are glad you see practical benefits for baselines such as IPPO and MAPPO. These are widely used and have been found competitive with more specialized methods (de Witt et al., 2020; Yu et al., 2022; Papoudakis et al., 2021). Specialized methods also typically start from such an algorithm.
While we considered integrating some mechanisms as algorithm options, we opted not to. Instead, we document advanced use cases in a similar fashion as the Equinox package, where advanced networks are not included in the package but deferred to documentation examples (e.g. for unet). We follow this approach for the following reasons:
- It keeps the overall API of our algorithms slim.
- It lessens the maintenance burden, especially for features that have not yet been requested.
- Libraries implementing advanced multi-agent methods already exist, including in JAX (Mava).
- As the above already exists (3), rather than providing another set of pre-built solutions, we aim to provide tools such that researchers can customize their algorithms for their use cases or research (e.g. scaffolding an algorithm, subclassing, overriding methods with decorators).
These choices may change with (community) feedback. For now, we believe this way we can bring value (it has for us) while keeping the project maintainable. We will add examples of more mechanisms to our documentation.
Novelty and practical benefits
We agree that the individual ideas are not new, and the paper states that neither a unified interface nor nested spaces are. Our contribution on the algorithm side is practical: small, readable implementations that run unchanged across space types, suites, single- and multi-agent settings, and JAX-native and host-based environments.
Whether PyTrees are required for this: observation-agnostic algorithms can indeed be created by delegating observation processing to separate networks. Jaxnasium does exactly this, but builds these networks automatically from the observation structure; they only have to be callable modules, and the defaults are optional. We agree that networks are often customized per task, but note that may often only be in part (e.g. an image encoder), and the remaining parts can stay at their automatic default. Furthermore, PyTrees are used for more than creating networks:
- Everything around the network. The replay buffer, running statistics for normalization, space wrappers, and multi-agent dispatch are all instantiated from and operate on leaves on the same PyTrees.
- Actions. For composite and hybrid action spaces, not only the network follows the pytree structure. Each action leaf gets its own head, and sampling, log-probabilities, entropies, Q-targets, ε-greedy exploration, and action masks (given as a PyTree matching the action space) are all computed per leaf and combined, as a mapping over the action PyTree rather than a special case per space type (We added new experiments for hybrid spaces in host-based environments to Appendix G).
Indeed, none of this is impossible without PyTrees. Our argument is that it takes considerably less code. PyTrees are what allow a single small file per algorithm to cover all of these settings, and that same file also covers host-based Gymnasium environments. For benchmarking, a reference implementation that runs on a new environment unmodified leaves fewer implicit choices to whoever runs the baseline. Furthermore, for research, a single small file is a convenient starting point: since we expect some users that will adapt algorithms to their task or research, we encourage copying our algorithms through a CLI tool.
We understand that you may still weigh the conceptual novelty differently, and appreciate that you recognize the convenience of the unified design. We hope these clarifications help show its broader practical value as a software library, and thank you again for the constructive discussion.
- Yu, Chao, et al. "The surprising effectiveness of ppo in cooperative multi-agent games." Neurips Datasets and Benchmarks Track (2022)
- Papoudakis, Georgios, et al. "Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks" Neurips Datasets and Benchmarks Track (2021)
- De Witt, Christian Schroeder, et al. "Is independent learning all you need in the starcraft multi-agent challenge?." (2020).
We thank the reviewer for their constructive feedback. We are glad the unified interface appeals to someone who regularly works with JAX-based RL codebases, as this friction of moving code between suites and libraries is one of the reasons Jaxnasium exists. Below we answer the question and both weaknesses. All changes are in the uploaded revision.
Yes, and we agree this was missing. The revision adds two sets of experiments:
We want to be clear about what this does and does not show: the defaults give a learning signal on most environments without tuning, but we do not maintain per-environment SOTA tuned hyperparameters.
We agree that integrating benchmarks under one API is not novel in itself, and the revised Section 3 now states this explicitly. The unification is a means to an end in our paper and primarily exists to give our algorithms a single target. What we do believe is novel when it comes to our API:
Appendix D adds a comparison with nearby libraries.
We hope to change the reviewer's mind here. Multi-agent support has been a core pillar since we started building Jaxnasium close to two years ago: one of the reasons for its inception was to manage several projects, some of them multi-agent, from shared/inherited codebases, and it has proven highly useful to us.
We have extended Section 4.2 to make clearer how the multi-agent support works. Briefly: on a multi-agent environment, an algorithm creates one agent per entry of the space PyTree and places them in a multi-agent wrapper. Per-agent methods are dispatched per agent by default (via
jax.vmapwhere possible, elsejax.tree.map), while methods marked@collectiveare called on the whole team. On the reviewer's examples:@collective). We do not ship or benchmark such a subclass in this submission, but hope this sheds some light on what the mechanism can do and look to provide more documantation examples later.Overall, we do not aim to replace dedicated multi-agent libraries. What Jaxnasium offers is that the algorithm file someone modifies for a single-agent idea, is the same file that gives an IPPO or MAPPO baseline on, for example, JaxMARL, so an idea can be tested in both settings without a second codebase. And as each algorithm is a single file that can be copied into a project through our CLI (Section 6), it is a practical starting point for more specialized methods.
We hope these changes resolve your concerns and, if they do, that you would consider updating your rating. We are happy to answer any follow-up questions.