acceptodds
Under review as a conference paper at ICLR 2027

KestRL: A Dual-Route JAX Library for Modular Reinforcement Learning

Abstract

We present KestRL, a dual-route reinforcement learning library built on Flax NNX, designed to let researchers build algorithms from reusable components and run them with both host-driven and JAX-native environments. KestRL builds on NNX graph/state separation to organize learning updates around explicit training state. It provides two execution routes: a Standard route, which combines Gymnasium environments with compiled learner updates, and a Compiled route, which brings Brax environments and learning updates into compiled training loops and supports batched execution of independent runs. SAC and PPO illustrate how learning code can be shared across these routes as collection and scheduling adapt to each environment. Controlled update comparisons and complete training experiments examine the resulting execution costs. For host-dispatched updates, the controlled comparisons show improvements over naive NNX dispatch, alongside workloads where cached NNX or hand-written functional JAX remains faster. In a five-run Ant PPO study at a fixed environment-step budget, KestRL Standard reduces end-to-end time by 32.8% relative to eager CleanRL and 25.7% relative to the evaluated Inductor-enabled configuration. In the reported Brax SAC configurations, KestRL Compiled shows similar independent-run scaling to Rejax, with lower observed amortized runtime on Ant and Hopper across the tested batch sizes.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
reinforcement learning, software libraries, JAX, modularity, compiled execution, parallel experimentation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9Zq2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

KestRL is a Flax NNX library for reinforcement learning that aims to let researchers write learning components once and reuse them across host-driven and fully compiled training. It splits all networks and optimizers into a fixed graph definition and a single learner-state PyTree, which turns updates into pure state-transition functions. This interface supports two routes: a Standard route that pairs Gymnasium environments with compiled updates, and a Compiled route that scans Brax rollouts and updates on the accelerator and vmaps independent seeds. The authors use SAC and PPO as case studies of how much code the two routes can share. Experiments show reduced dispatch overhead compared with naive NNX; KestRL is faster than cached NNX on PPO but not on SAC, and reaches 72–85% of hand-written JAX throughput. They also show faster end-to-end PPO than CleanRL at a fixed step budget, at the cost of a lower final return, and seed-batching scaling for SAC comparable to Rejax.

Critical strengths
  • Clean, useful abstraction: a single NNX graph-and-state split yields pure update functions that work in a host loop, under scan and under vmap, so learning code can be shared between host-driven and fully compiled training.
  • Honest reporting: the paper openly reports where KestRL is slower, the lower PPO returns and the compiled PPO failure.
  • Strong reproducibility: results are checked for agreement before timing, and the paper provides detailed configurations, phase-level timing and run-level results.
Critical weaknesses
  • The PPO speedup may not reflect the library design. As the paper notes (Section 4.2), KestRL compiles minibatch scheduling and optimizer updates, while the Inductor configuration leaves these outside the compiled graph. The speedup over CleanRL may therefore reflect how much is compiled rather than KestRL's design. A comparison to a JAX-based PPO baseline would make this clearer.
  • Limited evidence that compiled PPO learns. As the paper acknowledges, compiled PPO on Brax Ant remains in a low-return regime for both KestRL and Rejax (Table 11), so these results offer limited evidence of learning performance in the compiled PPO setting.
Questions for authors
  • On code reuse: Can you quantify how much code SAC and PPO actually share between the two routes, for example the fraction of algorithm code that is shared versus route-specific?
  • On Brax Ant, compiled PPO reaches returns of only about 100–150 for KestRL and Rejax after 1M steps, which seems low for Brax PPO. What causes this? Is it the configuration (e.g., number of parallel environments or observation normalization), or something in the implementation?
1 comment

We thank the reviewer for recognising the practical value of the shared update interface and our effort to report its strengths and limitations clearly. We invite the reviewer to consult the accompanying two-page kestrl_before_after.pdf: R1(d) shows the compiled PPO correction results, and R2 presents the comparisons run for this response.

PPO runtime and compilation scope (Weakness 1)

As the reviewer notes, Section 4.2 attributes the PPO timing difference to the implementations' update and compilation choices. We do not attribute that gain to NNX modularity itself.

We give the full explanation and the new functional-JAX update-phase control in our response to reviewer 9hqp, under "Matched comparisons and fusion" (Weakness 1; Questions 1–2).

How much code is shared? (Question 1)

In the submitted code, PPO shares its policy/value losses and complete minibatch update, including gradients and optimiser steps. SAC shares actor/critic losses, action helpers and target-network updates, with separate gradient loops.

A loss change using the existing batch and learner state can therefore be implemented once for both routes. The PPO critic-weight correction illustrates this directly: one added line in the shared update builder reached both routes. The revised Section 3.4 and Appendix C also describe the update functions now shared by continuous SAC.

Across the submitted algorithm packages, these shared definitions occupy 122/742 code lines in PPO (16.4%) and 114/1,032 in SAC (11.0%). The denominator includes both routes' setup, collection, evaluation and scheduling. Comments, docstrings, blank lines and generic infrastructure, including the network modules both routes reuse, are excluded. The counting script and definition-level breakdown accompany the response.

Compiled PPO learning (Weakness 2; Question 2)

We share the reviewer's reading of Table 11: both KestRL and Rejax obtain low returns, as Section 4.2 discusses. Here, each policy is trained for one million steps with one Brax environment and no observation or reward normalisation. Batching independent seeds does not give each policy experience from multiple environments.

It helps to put this setting alongside Brax's reference Ant PPO example, which uses 50 million steps, thousands of environments and observation normalisation. The public CleanRL MuJoCo curves also show PPO remaining well below off-policy methods within the first million steps on HalfCheetah, even with observation and reward normalisation. We think this budget and configuration context is important when interpreting Table 11.

Figure R1(d) in the accompanying kestrl_before_after.pdf adds the corrected KestRL outcomes.

The accompanying kestrl-response-artifacts.zip contains the control outputs, per-seed results, source snapshots and reuse audit. We hope this additional evidence clarifies the practical contribution the reviewer recognised and helps them reconsider their assessment.

Reviewer_Pwe4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

KestRL is a Flax NNX RL library with two execution modes. The Standard flow keeps environments interaction in Python and compiles learner updates. The Compiled flow places environments interaction, storage and learning inside JAX control flow and can batch independent runs. SAC and PPO show component reuse. The evaluations include update microbenchmarks, a PPO comparison with CleanRL and SAC comparisons with several libraries.

Critical strengths
  1. The dual-route design is practical and addresses an actual researcher's problem of reusing learning components across host and fully compiled environments.
  2. The paper reports unfavorable results as well as favorable ones. It also includes cases where cached NNX or functional JAX is faster.
Critical weaknesses
  1. KestRL and Rejax differ in evaluation workload, learning rates, and return collection. Some figures also mix MuJoCo and Brax. The results, thus, do not isolate library efficiency.
  2. The submitted artifact appears inconsistent with the protocol described in the paper. Table 8 specifies a shared SAC configuration with a 5,000-step warm-up and policy-update interval of 2, but the Compiled SAC implementation never reads policy_frequency and updates the actor and temperature on every gradient step. Its initial replay data are also collected from the randomly initialized policy, whereas Standard SAC uses uniformly random actions. More seriously, Standard SAC’s deterministic action helper returns a triple that SAC.get_action wraps inside another triple; the inherited evaluator then passes this un-unpacked value to np.asarray and env.step. Since the stated Standard entry point always constructs an evaluation environment and the trainer always performs final evaluation, the submitted path appears unable to complete the reported runs. In PPO, vf_coef affects only a logged metric, not the critic gradient, and the Standard learning-rate schedule is sized using floor division even though training executes the rollout crossing the requested budget. Finally, the archive omits misc/rebuttal_artifacts/, misc/appendix/build_tables.py, and the 15 audit logs explicitly named in the reproducibility appendix.
Questions for authors
  1. Did all reported experiments use the submitted code? If not, please provide the exact commits.
  2. Can the authors release the raw logs and rerun the main comparisons with matched workloads and settings?
  3. How do the results change after correcting the implementation issues?
1 comment

We appreciate the reviewer's recognition of the paper's clarity and its reporting of unfavourable as well as favourable results. To make the changes easy to check, we invite the reviewer to consult the accompanying two-page kestrl_before_after.pdf: R1 shows the corrections' effects; R2 presents the matched comparisons.

Below, we address each of the reviewer’s concerns in turn.

Comparison protocols (Weakness 1; Question 2)

We would like to clarify that KestRL Compiled and Rejax both used Brax, whereas KestRL Standard, CleanRL and SB3 used MuJoCo. Section 4.3 and the SAC runtime caption distinguish these groups, with the additional compiled curve in the left panel explicitly included as a reference.

The learning-rate, evaluation-workload and return-collection differences were also described in Section 4.3 and Appendix A.3. Section 4.1 holds computation, state and inputs fixed. We believe these controlled experiments provide a fair assessment of update execution cost.

At the same time, we agree that the reviewer's requested rerun makes the complete-run comparison easier to interpret. For this response, we matched SAC learning rates and warm-up, used a common deterministic evaluator, and timed evaluation separately from training.

Figure R2 presents this Brax comparison and corrected MuJoCo PPO, with five seeds per method. At the tested step budget, PPO remains faster than both CleanRL configurations; SAC runtimes are close. The learning curves show the corresponding returns.

Historical code and SAC evaluation (Weakness 2; Question 1)

The historical Standard SAC run we traced could complete because it did not invoke the faulty evaluator. In that historical version, the entry point supplied no evaluation environment, and final evaluation required one. The recorded local changes left this path unchanged. We have also corrected and regression-tested the submitted evaluation path.

This also clarifies the reviewer's source question: the submitted snapshot differs from that historical execution. PPO was added later in the library's development and is included in the submitted archive.

The relevant historical Standard SAC source files are included under code/historical-standard-sac/, with the recorded commit and available attribution in source_provenance.json.

Corrections and their effects (Weakness 2; Question 3)

We thank the reviewer for spotting these implementation bugs and helping us improve the library. The fixes preserve the paper's central thesis: reusable learning components can support both host-driven and compiled execution.

We have aligned Compiled SAC with the stated uniform warm-up and actor/temperature update schedule. PPO now applies vf_coef to critic gradients, and Standard PPO sizes annealing to all executed rollouts, following the convention already specified in Section 4.2 and Appendix A.2.

We compare submitted and corrected code on the same H200 with five training seeds in Figure R1, so the effects can be inspected directly. Main-study runtimes remain close; Figure R1 also shows the learning outcomes and variation across seeds for each route, including compiled PPO. The corrected PPO controls also retain their timing order.

We also commit to rerunning the paper's full experimental suite on both H200 and A6000, under common protocols and with complete logs. These reruns will build on the paper's existing evaluation across hardware, extending the assessment to the corrected implementations.

Files and reproducibility (Weakness 2; Question 2)

The original raw logs remain in our private Weights & Biases project. Sharing its link would reveal our identity. An anonymous export with a verified mapping of its hundreds of runs to the paper's experiments is not feasible within the remaining discussion period.

We have therefore focused kestrl-response-artifacts.zip on the reviewer's requested reruns, with their logs and frozen source snapshots. Because these include the "before" runs of the submitted implementation on H200, the reviewer can inspect both versions and check the corrections' effects directly. The archive also supplies all 15 historical timing-audit logs and their table-reproduction script; README.txt locates the files.

The paths misc/rebuttal_artifacts/ and misc/appendix/build_tables.py came from our internal workspace. We have removed those references from Appendix A's opening reproducibility paragraph, which now states that our Weights & Biases experiments, reports and logs will be made public.

We hope these clarifications and results address the reviewer's concerns and help them reconsider the soundness assessment.

Reviewer_9hqpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents KestRL, a JAX library for reinforcement learning built on Flax NNX. The main idea is to split each learner into a fixed graph definition and an explicit state, so the same update function can be used in two ways. In the Standard route, Python controls the loop with Gymnasium environments and only the learner update is compiled. In the Compiled route, Brax environments, storage and learning are all carried in one state and run through scan, and many independent runs are batched with vmap. SAC and PPO show the code sharing. The authors test the cost of the design in three ways: controlled update timing, a five-run Ant PPO study against CleanRL, and a SAC batching study against Rejax and Stoix.

Critical strengths
  1. A practical design that is shown clearly. The explicit-state update interface is simple. The paper shows it works both in a host loop and inside scan/vmap, which cached NNX cannot do unchanged. For PPO, the same loss and minibatch builder are used in both routes, which is a real benefit to researchers.

  2. Honest and careful measurement. The controlled comparisons check that the outputs agree before timing, use warm-ups, and report medians and IQRs. The authors openly show where cached NNX is faster (SAC) and where KestRL reaches only about 70–85% of hand-written JAX. The appendix also gives a full phase-by-phase time accounting. This makes the results trustworthy and the paper easy to reproduce.

Critical weaknesses
  1. Speed comparisons are not fully fair. In SAC, Rejax uses a different critic learning rate and its timed runs include evaluation every 5,000 steps, so the “lower amortized runtime” claim may not come from KestRL’s design. In PPO, KestRL fuses minibatch updates into one compiled phase but CleanRL does not, so the 32.8% gain shows the effect of fusion, not necessarily NNX modularity. KestRL’s final return is also lower (1006 vs 1196 and 1120), over only 5 seeds.

  2. The main goal, easy reuse, is not measured. The paper gives no numbers such as shared lines of code or effort to port a new loss. For SAC, only a narrow part is shared. Compiled PPO also gives very low returns on Brax Ant (about 150 and 100), and the novelty is modest, as split/merge comes from the Flax guide and batched compiled training already exists in Rejax, Stoix and PureJaxRL.

Questions for authors
  1. Can you rerun the Rejax comparison with matched learning rates and with evaluation removed from the timing? Does the result stay the same?

  2. Can you add a CleanRL or functional-JAX PPO with the same fused minibatch phase, to separate the effect of fusion from the effect of the library design?

  3. Why is compiled PPO so low on Brax Ant? Does it match Rejax or other references on learning curves under identical settings?

  4. Appendix B.2 says one Standard Ant seed was retried after a failed attempt, and earlier unfused or checkpointed runs were left out. How many runs were discarded, and what was the rule for this?

  5. Can you give some numbers on reuse, such as lines of code shared between routes for PPO and SAC?

1 comment

We appreciate the reviewer's positive comments of our work and their recognition of the practical design and careful measurement. We invite the reviewer to consult the accompanying two-page kestrl_before_after.pdf; the supporting measurements and code audit are in kestrl-response-artifacts.zip.

Matched comparisons and fusion (Weakness 1; Questions 1–2)

We agree that the reviewer's requested rerun makes the SAC comparison easier to interpret. Section 4.3 and Appendix A.3 describe the original differences; the five-seed Brax Ant experiment accompanying this response matches learning rates and warm-up and uses a common evaluator.

With evaluation excluded, mean training times are 80.0 minutes for KestRL and 81.7 for Rejax. The matched single-learner result is therefore close in runtime. Figure R2 shows the learning curves and end-to-end times, which include evaluation.

As the reviewer notes, KestRL Standard fuses learner updates while retaining host-driven environment interaction. Section 4.2 attributes the 32.8% end-to-end reduction to the implementations' update and compilation choices, not to NNX modularity itself. It also reports the lower mean return at the fixed training budget.

This is why we built KestRL. Researchers can change a shared loss once and use it in either route, with fused learner updates supported in the same library.

Section 4.1 separately measures update execution cost with computation and inputs held fixed. For this response, we also compared explicit NNX state and functional JAX under both host dispatch and a fused scan, using matched inputs and optimiser settings.

With the same fused 320-minibatch phase, median times are 28.51 ms for explicit NNX state and 28.10 ms for functional JAX on H200, over 20 repetitions. Parameters, optimiser state and metrics pass numerical agreement checks. This measures update-phase cost; Figure R2 separately reports complete-run runtime and learning.

Reuse and contribution (Weakness 2; Question 5)

We detail the shared learning operations, the PPO correction illustrating this reuse, and the code counts and their scope in our response to reviewer 9Zq2, under "How much code is shared?" (Question 1).

Sections 2 and 3.4 already acknowledge NNX split/merge and the compiled execution models of prior libraries. Our contribution is their organisation into reusable NNX learning components across the two routes, which the reviewer also identifies as useful.

Compiled PPO learning (Weakness 2; Question 3)

We understand the reviewer's concern about the low returns reported in Section 4.2 and Table 11. We discuss the training budget, configuration and relevant reference results in our response to reviewer 9Zq2, under "Compiled PPO learning" (Weakness 2; Question 2).

On the reviewer's question about learning curves, our comparison with Rejax currently covers final policy evaluations. Our planned full-suite reruns on H200 and A6000 will extend this learning evidence.

Run inclusion and the retry (Question 4)

We kept the earlier runs separate because checkpointing and unfused updates change what is being timed. For the final configuration, we report seeds 1–5 for each method and provide all 15 logs so the reviewer can inspect the comparison directly. Our retained records do not give a complete count of the earlier runs.

The seed-2 retry in Appendix B.2 concerns the separate Standard Ant learning suite. It followed an execution problem and kept the same seed; the learning summary contains one completed result per seed. All prescribed seeds in the new comparisons are also retained, with their logs.

We hope these clarifications and additional controls address the reviewer's concerns and strengthen their confidence in the contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.