- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
KestRL is a Flax NNX library for reinforcement learning that aims to let researchers write learning components once and reuse them across host-driven and fully compiled training. It splits all networks and optimizers into a fixed graph definition and a single learner-state PyTree, which turns updates into pure state-transition functions. This interface supports two routes: a Standard route that pairs Gymnasium environments with compiled updates, and a Compiled route that scans Brax rollouts and updates on the accelerator and vmaps independent seeds. The authors use SAC and PPO as case studies of how much code the two routes can share. Experiments show reduced dispatch overhead compared with naive NNX; KestRL is faster than cached NNX on PPO but not on SAC, and reaches 72–85% of hand-written JAX throughput. They also show faster end-to-end PPO than CleanRL at a fixed step budget, at the cost of a lower final return, and seed-batching scaling for SAC comparable to Rejax.
Critical strengths
- Clean, useful abstraction: a single NNX graph-and-state split yields pure update functions that work in a host loop, under
scanand undervmap, so learning code can be shared between host-driven and fully compiled training. - Honest reporting: the paper openly reports where KestRL is slower, the lower PPO returns and the compiled PPO failure.
- Strong reproducibility: results are checked for agreement before timing, and the paper provides detailed configurations, phase-level timing and run-level results.
Critical weaknesses
- The PPO speedup may not reflect the library design. As the paper notes (Section 4.2), KestRL compiles minibatch scheduling and optimizer updates, while the Inductor configuration leaves these outside the compiled graph. The speedup over CleanRL may therefore reflect how much is compiled rather than KestRL's design. A comparison to a JAX-based PPO baseline would make this clearer.
- Limited evidence that compiled PPO learns. As the paper acknowledges, compiled PPO on Brax Ant remains in a low-return regime for both KestRL and Rejax (Table 11), so these results offer limited evidence of learning performance in the compiled PPO setting.
Questions for authors
- On code reuse: Can you quantify how much code SAC and PPO actually share between the two routes, for example the fraction of algorithm code that is shared versus route-specific?
- On Brax Ant, compiled PPO reaches returns of only about 100–150 for KestRL and Rejax after 1M steps, which seems low for Brax PPO. What causes this? Is it the configuration (e.g., number of parallel environments or observation normalization), or something in the implementation?
We thank the reviewer for recognising the practical value of the shared update interface and our effort to report its strengths and limitations clearly. We invite the reviewer to consult the accompanying two-page
kestrl_before_after.pdf: R1(d) shows the compiled PPO correction results, and R2 presents the comparisons run for this response.PPO runtime and compilation scope (Weakness 1)
As the reviewer notes, Section 4.2 attributes the PPO timing difference to the implementations' update and compilation choices. We do not attribute that gain to NNX modularity itself.
We give the full explanation and the new functional-JAX update-phase control in our response to reviewer 9hqp, under "Matched comparisons and fusion" (Weakness 1; Questions 1–2).
How much code is shared? (Question 1)
In the submitted code, PPO shares its policy/value losses and complete minibatch update, including gradients and optimiser steps. SAC shares actor/critic losses, action helpers and target-network updates, with separate gradient loops.
A loss change using the existing batch and learner state can therefore be implemented once for both routes. The PPO critic-weight correction illustrates this directly: one added line in the shared update builder reached both routes. The revised Section 3.4 and Appendix C also describe the update functions now shared by continuous SAC.
Across the submitted algorithm packages, these shared definitions occupy 122/742 code lines in PPO (16.4%) and 114/1,032 in SAC (11.0%). The denominator includes both routes' setup, collection, evaluation and scheduling. Comments, docstrings, blank lines and generic infrastructure, including the network modules both routes reuse, are excluded. The counting script and definition-level breakdown accompany the response.
Compiled PPO learning (Weakness 2; Question 2)
We share the reviewer's reading of Table 11: both KestRL and Rejax obtain low returns, as Section 4.2 discusses. Here, each policy is trained for one million steps with one Brax environment and no observation or reward normalisation. Batching independent seeds does not give each policy experience from multiple environments.
It helps to put this setting alongside Brax's reference Ant PPO example, which uses 50 million steps, thousands of environments and observation normalisation. The public CleanRL MuJoCo curves also show PPO remaining well below off-policy methods within the first million steps on HalfCheetah, even with observation and reward normalisation. We think this budget and configuration context is important when interpreting Table 11.
Figure R1(d) in the accompanying
kestrl_before_after.pdfadds the corrected KestRL outcomes.The accompanying
kestrl-response-artifacts.zipcontains the control outputs, per-seed results, source snapshots and reuse audit. We hope this additional evidence clarifies the practical contribution the reviewer recognised and helps them reconsider their assessment.