acceptodds
Under review as a conference paper at ICLR 2027

Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RL

Abstract

Training agents to follow arbitrary instructions is an important goal of multi-task reinforcement learning (RL). Linear temporal logic (LTL) provides a precise and structured formalism for specifying instructions to agents, and has been successfully adopted for training generalist multi-task policies. However, differences in implementations, task distributions, and evaluation protocols make existing methods difficult to compare, while high computational costs limit the scale and statistical reliability of experiments. We introduce Jaxolotl, a unified high-performance benchmark suite for multi-task LTL-RL to address these concerns. Jaxolotl provides a modular, end-to-end JAX implementation of six representative algorithms and four environments, together with newly curated task suites and a standardised, statistically robust evaluation protocol. By precompiling symbolic task representations into static arrays, Jaxolotl enables fully JIT-compiled training and evaluation, achieving end-to-end speedups of up to and supporting controlled comparisons at substantially greater experimental scale. We use this framework to systematically evaluate existing approaches, revealing complementary limitations: general methods capable of non-myopic reasoning struggle as the number of propositions grows, while methods with stronger scaling rely on environment-specific assumptions and suffer from myopia.

Primary area:
datasets and benchmarks
Keywords:
reinforcement learning, ltl, multi-task, instruction-following, benchmark, jax

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yX3oOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduced the JAXOLOTL a benchmarking tool can be use in multitask RL where tasks expressed in LTL. They have implemented their tool using JAX, and included four multitask environments. Finally they have shown their efficiency of their tool comparing with the original implementations and also perform a benchmarking on the popular RL-LTL methods.

Critical strengths

1- The motivation behind the reason why we need a unified tool for RL-LTL is presented well.

2- The engineering side of the project is valuable. Using JAX gives JAXOLOTL an significant advantage over original implementation.

3- Experiments are thorough and well prepared. I like the stress test had been done in sec.5.2.2 and 5.2.3. The observations from these two experiments are valuable for RL-LTL community who want to develop their own method. Now they know what are the gaps and how they can fill this gap.

Critical weaknesses

I thank the authors for their hard work implementing JAXOLOTL, however there are some issues which I split them in two main categories: Formalism and Writing, and Implementation.

Formalism and Writing

1- Paper has few formalism errors such as: P:S×A×S→[0,1]P: S\times A\times S\rightarrow[0,1] should be P:S×A→Δ(S)P: S\times A\rightarrow \Delta(S), or you need to add the constraints that the sum of probabilities are not going to be bigger than 1. In addition, π\pi is defined as π:S→Δ(A)\pi: S \rightarrow \Delta(A), but in line 159 eq, it conditioned on the specification φ\varphi. Further, in eq (line 159), φ\varphi is sampled from the D\mathcal{D} on the right side, how we have φ\varphi on the left side as well. Are we looking for a fixed specification? Also D\mathcal{D} should be uniform distribution, since if it is an arbitrary distribution, the policy can get biased to optimized satisfying only some of specifications.

2- In my opinion the most important thing that a new benchmark should have, is how easy it can be used for new methods. The Modular Abstractions section, could not deliver that, I think more explanation, demo or part of a code could help to show this.

Implementation

3- Since we call JAXOLOTL as an unified benchmark it should have the environments to distinguish different methods with different abilities, for example in PlatoLTL (Cloete et al., 2026) they introduced new environments to show their advantage over parametrized settings, so I think JAXOLOTL should include these environments as well.

4- In Fig.5, we see that most (3/4) of settings have already been dominated by some of the methods, and there is absolutely no room for improvement for new methods. So new methods can not show their superiority over the RL-LTL baselines if they use this benchmark.

Questions for authors

My main questions are (1) what are the features proposed for the researchers want to work on RL-LTL field and develop their own method, (2) is there a room for improvement so they can work on, (3) How diverse are the environments in which different methods demonstrate their respective advantages?

5 comments

Many thanks for your review. We are happy to read your positive assessment, especially regarding the “valuable” engineering and “thorough and well prepared” experiments. Please see below for answers and comments to the points raised. Please note that we have updated the paper with new experiments and further discussion.

Formalism errors

Many thanks for pointing these out. We fixed these errors in the updated version of the paper: we corrected the transition kernel, introduced LTL-conditioned policies before the objective, and removed the specification from the left-hand side. We retained the wording of “arbitrary distribution” since we make no assumptions on uniformity (in fact, different methods employ different task distributions, e.g. via curriculum learning), but we’re happy to discuss this further.

In my opinion the most important thing that a new benchmark should have, is how easy it can be used for new methods.

We agree with this and think this is a valuable addition to the paper. We have hence added an overview of Jaxolotl’s code architecture in the new Appendix C, together with a brief description of how to extend the framework with new methods. As an example, we explain how StructLTL is implemented by reusing relevant components from DeepLTL. We will furthermore add a detailed developer guide to the final repository, with more detailed explanations and references to the code.

In Fig.5, we see that most (3/4) of settings have already been dominated by some of the methods, and there is absolutely no room for improvement for new methods.

This is a good point, which was also raised by reviewer RyrC. We hence updated the paper with a new challenge track of substantially harder environments. This includes QuadrotorEnv as a new environment, which is an adaptation of ZoneEnv with a 3D quadrotor instead of the point agent. The policy directly controls the thrusts of the four motors, forming a much harder nonlinear control challenge. The challenge track furthermore consists of a scaled up version of ZoneEnv with 8 propositions and non-myopic dynamics, the Franka environment with 12 propositions, and Warehouse-hard with six distinct objects, leading to 256 total valid proposition assignments.

We observe that existing methods struggle in these challenging domains: while GenZ-LTL still performs well in QuadrotorEnv and FrankaZoneEnv-12, its performance severely degrades without environment-specific processing. It also leaves significant headroom in ZoneEnv-NM-hard and does not apply to the hard Warehouse variant. The other methods similarly struggle, achieving maximum success rates of 79% (QuadrotorEnv), 69% (ZoneEnv-NM-hard), 29% (FrankaZoneEnv-12), and 31% (Warehouse-hard) on finite-horizon tasks.

We hope this challenge suite will serve as a useful testbed for future research, demonstrating the need for both non-myopic reasoning and scaling with atomic propositions.

PlatoLTL

PlatoLTL’s parameterised environments evaluate an important complementary capability: generalisation to previously unseen (parameterised) propositions. Our current benchmark focuses on compositional generalisation over fixed, finite proposition vocabularies, enabling controlled comparisons among the six included methods. Evaluating parameterised propositions would require adapting these methods and introducing a corresponding evaluation protocol. We therefore leave this extension to future work. The revised suite nevertheless distinguishes existing methods along several complementary dimensions, including non-myopic reasoning, proposition scaling, and simultaneous proposition satisfaction.

what are the features proposed for the researchers want to work on RL-LTL field and develop their own method

Researchers can reuse the training and evaluation loops, curriculum machinery, observation encoders, and LTL-processing infrastructure while implementing method-specific task representations, wrappers, policy models, and evaluation agents. Appendix C describes these extension points and their configuration, and illustrates how StructLTL reuses DeepLTL’s training/evaluation wrappers and evaluation agent while introducing a new task representation and encoder.

Regarding future research directions, we discuss this in the revised Section 5.3 and Conclusion (Section 6). Specifically, what is currently missing is a method that has all of (i) non-myopic reasoning, (ii) favourable scaling to large proposition vocabularies, and (iii) applicability without environment-specific preprocessing. Interesting potential directions include learning observation reduction functions via feature selection or attention mechanisms, and relaxing the assumptions of known propositions and accurate labelling functions.

AuthorsOpenReview ↗

Rebuttal Part 2

How diverse are the environments in which different methods demonstrate their respective advantages?

The suite exposes complementary strengths: GenZ-LTL with observation reduction handles large proposition vocabularies well, whereas StructLTL and DeepLTL perform better on the controlled ZoneEnv-NM test requiring non-myopic reasoning (92.7% and 83.6%, versus 70.0% for GenZ-LTL). StructLTL achieves the highest finite-horizon success rates on both Warehouse variants, which permit simultaneous propositions (Figure 6 and Table 1). For detailed descriptions of the differences between environments, see Appendix E.

We hope that the new results and discussion above are helpful and have clarified some of the points. Please let us know if you have any other questions or remarks!

Thanks for your detailed rebuttal.

formalism

Yes, I agree most of the issues are resolved but I the inconsistency regarding the policy is still there, π\pi defined as a mapping that only gets S\mathcal{S} as the input, then in ln159 same notation conditioned on the LTL spec. Either authors should change this notation chnage to something else or revise the current one. Also can you please specify a meaningful task that we need some distribution which consequently lead to some bias?

In my opinion the most ...

Resolved.

New challenging experiments

In the process of making the experiments more challenging, which feature of the envs targeted so now we have harder tasks? More feature, propositions, formulas ? This insights could be useful if added to paper so researchers can target to handle those aspects.

PlatoLTL

Unfortunately I am not convinced. In PlatoLTL they have all LTL-RL baselines, if JAXOLOTL is unified it should contain both PlatoLTL and it's envs.

Diversity

I think PlatoLTL and it's experiments is a good response to this concern. In my humble opinion adding that would strengthen the paper more than what it is right now.

Reviewer_RyrCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1(−1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for evaluating reinforcement learning agents that follow LTL instructions. In addition to defining a standardized evaluation protocol, the authors provide highly efficient implementations of six well-known methods. Their empirical comparison shows that no method strictly dominates the others across all evaluated tasks.

Overall, I find the paper valuable and believe that it addresses an important need in the LTL-based RL community. In particular, the efficient implementations and the effort to establish a standardized evaluation framework could have a meaningful impact on future research. However, my current assessment leans toward weak rejection because I am not yet convinced that the benchmark is sufficiently challenging or that the evaluation protocol supports fully fair and robust comparisons across methods.

My main concern is not that all methods must follow an identical training protocol, which may be unrealistic given their substantial differences. Rather, the paper should provide stronger evidence that its test tasks, or equivalent automata, were not encountered during training and that differences in training distributions do not unduly favor or disadvantage particular methods. I would also like to see a larger test set and at least one domain that more clearly exposes the current limitations of existing approaches.

I hope the authors can clarify these points, as their answers could materially affect my assessment. Even if all of them cannot be fully addressed in the current version, I believe that resolving these issues would substantially strengthen the benchmark and increase its chances of becoming a lasting standard for the community.

Critical strengths

The paper provides highly efficient implementations of existing methods. The improvements are remarkable. For example, the proposed implementation of DeepLTL reduces the training time on ZoneEnv from 14 hours to 8 minutes. These speedups could significantly accelerate research in this area by making broader experimentation, more extensive hyperparameter tuning, and the development of stronger LTL-based agents substantially more feasible.

The paper makes a valuable effort to compare six state-of-the-art methods within a unified benchmark. The benchmark covers four domains, each with approximately 20 non-trivial LTL tasks. Although the differences in training protocols and the limited test sets prevent the experiments from supporting definitive conclusions about the relative strengths and weaknesses of the methods, the comparison is a useful first step toward a standardized evaluation framework.

Critical weaknesses

The benchmark does not yet appear sufficiently challenging or forward-looking to serve as a long-term standard for the community. It contains only four domains, all of which, as far as I understand, were introduced in previous work. While consolidating existing domains is useful, a new benchmark should ideally also include novel and challenging environments that expose the current frontier of LTL-based RL methods and enable conclusions beyond those already established in the literature. For example, the distinction between myopic and non-myopic methods, together with their respective strengths, is already reasonably well understood.

More importantly, the proposed benchmark already appears close to being saturated by existing methods. This suggests that it may not provide enough room to distinguish future advances. Ideally, the benchmark would include at least one substantially more challenging domain in which current methods clearly struggle. Otherwise, future papers may still need to introduce additional environments or evaluation settings to demonstrate meaningful improvements, limiting the benchmark’s ability to become a lasting standard for the field.


The evaluation protocol may not support sufficiently robust or fair comparisons across methods. Each domain contains only 10 test formulas for finite tasks and 10 for infinite tasks. With so few test formulas, it is difficult to draw strong conclusions about whether one method consistently outperforms another. This also seems more limited than some previous evaluation protocols. For example, the original LetterWorld evaluation used hundreds of sampled formulas that were not observed during training. A benchmark intended to standardize evaluation should ideally include a much larger test set, potentially consisting of hundreds of formulas.

I am also concerned that the methods may have been trained using substantially different formula distributions or task-generation procedures. Because these methods follow different training protocols and do not necessarily operate directly on LTL formulas, I understand that providing exactly the same training set to every method may not be possible or even meaningful. Nevertheless, these differences make the results difficult to interpret unless the relationship between the training and test tasks is carefully controlled.

In particular, I believe LTL2Action may be disadvantaged because its training formulas appear to contain only the “until” and “eventually” operators, whereas the test formulas also contain operators such as “always” and "implications". If these operators were never observed during training, their learned representations have not been adequately trained, making poor performance on the selected test formulas difficult to interpret as a limitation of the method itself.

For the other methods, I suspect that exact overlap at the formula level is unlikely because formulas are transformed into DFAs before being used by the agent. However, this does not rule out overlap at the automaton level: syntactically different training and test formulas may induce identical or closely related DFAs. Therefore, even if the test formulas themselves were not observed during training, the corresponding automata, or behaviorally equivalent tasks, may have been. The benchmark should explicitly clarify and control for this possibility.

Relatedly, it is unclear how the hyperparameters were selected. The efficiency of the proposed implementations creates an excellent opportunity to compare all methods after meaningful and systematic hyperparameter optimization, something that may previously have been computationally prohibitive. Such a comparison would be both novel and valuable. Without a clear and consistent tuning protocol, however, it is difficult to determine whether the reported differences reflect the methods themselves or differences in their configurations.

More generally, I do not expect the benchmark to impose an identical training protocol on methods with fundamentally different learning mechanisms. However, it should standardize the evaluation setup as much as possible and provide safeguards against unintended train-test leakage. At a minimum, the authors should verify that the test formulas, their corresponding DFAs, or behaviorally equivalent automata were not encountered during training. Ideally, the benchmark would also characterize the structural similarity between training and test tasks, allowing the results to distinguish more clearly between in-distribution performance and out-of-distribution generalization.

Questions for authors

I would appreciate the authors’ clarification on the following points, as their responses could materially affect my assessment. I believe this is valuable work, and establishing a well-standardized benchmark for LTL-based RL is important for advancing the field. However, in its current form, I am not yet convinced that the proposed benchmark is sufficiently challenging or that its evaluation protocol supports fair and robust comparisons across methods.

  1. How do the authors ensure that the test tasks, or their corresponding DFAs, were not encountered during training? In particular, does the analysis account for cases in which syntactically different formulas produce identical or behaviorally equivalent automata?

  2. How were differences among the methods’ training distributions accounted for when interpreting their relative performance? For example, LTL2Action appears to be evaluated on operators that were not included in its training formulas, which may place it at a systematic disadvantage.

  3. Given that existing methods already achieve strong performance on much of the benchmark, what evidence suggests that it is sufficiently challenging to distinguish future methodological advances?

I would be happy to revise my assessment if I have misunderstood any aspect of the experimental setup or if the authors can clarify these concerns.

6 comments

Thank you for the review. We are glad that you find the paper “valuable” and “address[ing] an important need in the LTL-based RL community” with potentially “meaningful impact on future research”. We appreciate the detailed comments, which we agree with, and have thus updated the paper accordingly, including new results on a substantially more difficult challenge track, a discussion of the evaluation protocol, and generalisation results. We hope that these updates are able to resolve some of your concerns. Please find a detailed reply below (in two parts).

The benchmark does not yet appear sufficiently challenging or forward-looking to serve as a long-term standard for the community.

Thank you for pointing this out. We address this in the new revision (Section 5.2.1) by introducing a new challenge track, which is substantially more challenging for existing methods and reveals their failure modes. This includes QuadrotorEnv as a new environment, which is an adaptation of ZoneEnv with a 3D quadrotor instead of the point agent. The policy directly controls the thrusts of the four motors, forming a much harder nonlinear control challenge. The challenge track furthermore consists of a scaled up version of ZoneEnv with 8 propositions and non-myopic dynamics, the Franka environment with 12 propositions, and Warehouse-hard with six distinct objects, leading to 256 total valid proposition assignments.

We observe that existing methods struggle in these challenging domains: while GenZ-LTL still performs well in QuadrotorEnv and FrankaZoneEnv-12, its performance severely degrades without environment-specific processing. It also leaves significant headroom in ZoneEnv-NM-hard and does not apply to the hard Warehouse variant. The other methods similarly struggle, achieving maximum success rates of 79% (QuadrotorEnv), 69% (ZoneEnv-NM-hard), 29% (FrankaZoneEnv-12), and 31% (Warehouse-hard) on finite-horizon tasks.

We hope this challenge suite will serve as a useful testbed for future research, demonstrating the need for both non-myopic reasoning and scaling with atomic propositions. We discuss the strengths and weaknesses of existing methods in more detail in Section 5.3, and outline possible directions for future research to address these challenges in Section 6.

The evaluation protocol may not support sufficiently robust or fair comparisons across methods.

We think this is an important point to clarify, and discuss this in the updated Appendix E.4. As you note, the methods all employ different task representations: LTL2Action trains on formula syntax trees, SemLTL uses semantic automaton state embeddings, GCRL-LTL and GenZ-LTL are subgoal-conditioned, and DeepLTL and StructLTL use reach-avoid sequences. These representations are reflected in the training protocols (which randomly sample appropriate task structures), and generalisation therefore has different meanings across methods: LTL2Action and SemLTL must interpret unfamiliar formula structures and semantic embeddings, GenZ-LTL and GCRL-LTL must compose learned subgoal policies, and DeepLTL and StructLTL must handle unseen reach-avoid sequences and choose them at test time based on the automaton structure.

Nevertheless, we standardise training and evaluation as far as these differences permit. Specifically, we already employ syntactic rewrites for LTL2Action to eliminate always and $\rightarrow$ operators, replacing them with until, negation, and disjunction, which are all observed during training. For DeepLTL and StructLTL, we employ equivalent curricula (from their original implementations) in order to support a fair evaluation.

Due to these inherent differences in task representation, our main benchmark consists of curated, representative environment-specific tasks adapted from the literature. These tasks largely test in-distribution generalisation, but may be more or less similar to tasks seen during training for different methods. Concretely, for LTL2Action the evaluation tasks include novel combinations of operators and deeper formulae, for SemLTL they include unseen semantic embeddings, for GCRL-LTL and GenZ-LTL they test composition of learned subgoal policies, and for DeepLTL and StructLTL they include longer sequences, unseen reach-avoid combinations, and test test-time planning over automata structures.

Hence, the curated task suites test performance across meaningful tasks in the respective environments, while requiring generalisation across familiar objectives and novel compositions thereof, but the extent of required generalisation reflects the method-specific representation choices.

AuthorsOpenReview ↗

Rebuttal Part 2

How do the authors ensure that the test tasks, or their corresponding DFAs, were not encountered during training? In particular, does the analysis account for cases in which syntactically different formulas produce identical or behaviorally equivalent automata?

Following your suggestion, we compared the sampled training tasks of the methods to the evaluation tasks on ZoneEnv. We found no exact (syntactic) overlap for the formulae sampled for LTL2Action. For SemLTL, we found overlap in single-step reach-stay tasks, such as FG red (ξ1\xi_1 in ZoneEnv). DeepLTL and StructLTL sample generic reach-avoid sequences, some of which appear as subsequences of individual paths in the testing automata (we found 3 such cases in ZoneEnv). GCRL-LTL and GenZ-LTL instead sample uniformly from single propositions or reach-avoid subgoals; every test task hence consists of a sequence of subtasks seen during training. This is precisely the advantage of subgoal-conditioned (and, to an extent, sequence-conditioned) approaches: by training on simpler structures they are generally better at executing arbitrary formulae that map to similar structures at test time.

Out-of-distribution generalisation.

To add to the above, we introduce a controlled depth generalisation experiment in Appendix I.2. We sample 100 random sequential reach and reach-avoid formulae per depth level (up to 12 for reach and 8 for reach-avoid), yielding 1,800 new test formulae overall. Importantly, these are out-of-distribution for all methods (from depth 4 onwards), since they exceed the maximum depths seen during training. We find that all methods are able to successfully generalise, with LTL2Action and SemLTL struggling more than the explicit approaches. We admit that this is a limited controlled out-of-distribution generalisation experiment, but hope that it partially addresses the concerns together with our discussion in Appendix E.4.

Relatedly, it is unclear how the hyperparameters were selected.

We discuss this in the updated Appendix G.3. Our benchmark does not measure the maximum performance of each method attainable by tuning separately on each environment, but rather compares methods under a shared set of suitable hyperparameters. We standardise actor and critic architectures, learning rates, discount factors, PPO parameters, and policy architectures as much as possible across methods (following SpecRLBench), and adopt method-specific hyperparameters for each environment from the literature with no additional tuning.

We hope that the updates to the paper, including the new challenge track, out-of-distribution generalisation experiments, and the discussion above are helpful. Please let us know if these have addressed your concerns. We are more than happy to engage in further discussion!

Since the discussion period is coming to an end, we wanted to check whether we have been able to address your concerns with our response and the edits to the paper. Please let us know if you have any additional questions, we are happy to engage in further discussion!

Reviewer_RaxqOpenReview ↗
Rating
2(+1)
Confidence
1(−2)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

JAXOLOTL is a JAX-based benchmark suite for multi-task reinforcement learning with linear temporal logic (LTL) instructions. It unifies six representative methods and four main environments through shared task and policy abstractions. Its principal technical component is a precompilation strategy that converts method-specific symbolic task structures into static array representations, enabling end-to-end JIT-compiled training and evaluation. The framework also provides curated specification suites and a standardized evaluation protocol with uncertainty estimates over independently trained policies.

The experiments validate reimplementations against reference results on ZoneEnv, compare runtime against the official DeepLTL and GenZ-LTL implementations, and evaluate general task satisfaction, non-myopic behavior, and proposition scaling. The largest reported training speedup is 220× for completing ten GenZ-LTL training seeds on ZoneEnv. The benchmark identifies complementary limitations within the evaluated settings: several methods that use subsequent task information exhibit non-myopic behavior but degrade as the proposition set grows, whereas GenZ-LTL with observation reduction scales better but remains constrained by its current-subgoal policy interface.

Critical strengths

The main strength is:

The unified implementation and shared components provide useful infrastructure for reproducing and comparing LTL-RL methods. The authors validate their implementations on ZoneEnv and report uncertainty across independently trained policies. The curated task suites, reduced/unreduced GenZ-LTL comparison, and complementary non-myopic reasoning environments provide useful controls for examining different aspects of existing methods.

Critical weaknesses

The main weaknesses are:

  1. The incremental research contribution remains limited. The framework builds on existing LTL-RL algorithms, JAX-based acceleration, and static representations of symbolic structures. As the authors acknowledge, SpecRLBench already standardizes several experimental settings, while XLand-MiniGrid uses fixed-size arrays to represent symbolic rules and goals. JAXOLOTL does provide concrete LTL-specific adaptations, including precomputing formula-progression closures for LTL2Action and supporting heterogeneous task representations through shared interfaces. These adaptations and the expanded evaluation infrastructure have practical value. Nevertheless, in my assessment, the principal advance remains the integration, standardization, and acceleration of existing approaches. The additional technical and empirical insights are less substantial, particularly given the limitations in performance attribution and diagnostic interpretation discussed below.

  2. The performance contribution of the proposed precompilation design is not sufficiently characterized. Figure 3 reports end-to-end gains and their variation with the number of training seeds, while Figure 7 provides an independent environment-throughput benchmark. Appendix G.1 also documents the use of the same hardware, inclusion of JIT compilation overhead, and testing of concurrent reference runs. The authors additionally acknowledge that replacing the MuJoCo point robot with lightweight point-mass dynamics contributes to the ZoneEnv speedup. These results establish substantial practical acceleration for the complete system. However, they do not show how environment execution, task-representation handling, and policy/PPO computation contribute to runtime within the complete pipelines. The connection between the headline speedups and the paper’s LTL-specific precompilation design therefore remains insufficiently characterized. Precompilation, full-loop JIT compilation, and vectorization are interdependent, so their contributions need not admit a clean additive decomposition. Nevertheless, a clearer account of their respective roles is needed to assess the significance of the proposed technical design.

  3. The diagnostic findings could provide clearer insights into algorithm design. The paper already offers useful explanations for several observed limitations. It identifies the constraints of current-subgoal interfaces, shows that full-task conditioning does not guarantee non-myopic behavior, and discusses exploration and credit-assignment difficulties in the original ConveyorWorld. These analyses provide meaningful diagnostic value. The remaining concern is the depth of the connection between the observed failures and specific architectural or learning choices. For example, the proposition-scaling experiments keep evaluation specifications fixed, show degradation among current-subgoal methods, and compare reduced and unreduced observations in GenZ-LTL. These results support grounding as an important factor. However, identifying grounding as the main bottleneck does not fully explain why different policy architectures and training designs struggle to learn the relevant proposition-to-feature associations. Observation reduction demonstrates that the way task-relevant information is presented to the policy can substantially affect performance. A more explicit analysis of which representation, policy-conditioning, and training choices drive the remaining differences would make the empirical contribution more informative and provide clearer guidance for improving existing methods.

Questions for authors

The main questions are:

  1. Sources of the speedup: What are the main sources of the reported speedups? What roles do LTL-specific precompilation, JAX compilation, GPU parallelization, and environment implementation changes play, and how do the results demonstrate the contribution of the technical design introduced in this paper?

  2. Algorithm-design insights: How do model architecture and RL algorithm design contribute to the observed limitations in non-myopic behavior and proposition scaling? Which design choices do the authors consider most important for addressing these limitations, and what do the experiments suggest about how they could be improved?

4 comments

Thank you for taking the time to review our paper! We are pleased to read that you think our implementation provides “useful infrastructure” for future work and advances the “integration, standardization, and acceleration of existing approaches”. We agree with the points regarding the characterisation of the performance contribution and providing clearer insights, and have hence updated the paper to address the feedback, with new performance results (Appendix H.1) and a dedicated discussion section (Section 5.3). We provide a detailed response to the weaknesses below.

The performance contribution of the proposed precompilation design is not sufficiently characterized. (Question 1)

Thank you for pointing this out. We have added a controlled comparison in Appendix H.1 that retains symbolic Python task representations while keeping the same JAX environment implementation and JIT-compiled environment execution, policy inference, and PPO updates. Our design enables full JIT-compilation, leading to a 13.1× speedup for training DeepLTL on ZoneEnv (over 1M environment steps). This directly measures the impact of our compiled representations.

The above is for a single seed. Importantly, full JIT-compilation further enables effortless parallelisation across independent training seeds with vmap, which is not possible with the symbolic implementation. As shown in Figure 3, this leads to a further order of magnitude speedup. Extrapolating from the numbers in the controlled experiments above, training DeepLTL on 10 seeds of ZoneEnv would take ~8 hours without our precompilation design, compared to 8.2 minutes with our design. Appendix H.1 further compares the runtime contributions of different parts of the training loop, showing that rollout collection contributes roughly 2/3 of the time and PPO updates around 1/3.

Finally, Figure 7 further demonstrates the effect of our environment implementations and vectorisation. Implementing common environments in JAX is not straightforward, since every operation must be jittable and implemented without Python control flow (e.g. the LiDAR computation in ZoneEnv). As before, this is a crucial component to enable JIT-compilation of the entire training loop. As shown in the figure, the JIT-compiled JAX implementations of the environments achieve significant speeds over standard Python implementation when considering multiple executions in parallel.

The diagnostic findings could provide clearer insights into algorithm design. (Question 2)

We agree with this and have accordingly added a more detailed discussion section (Section 5.3) to complement the findings of our experiments. The scaling experiment (Section 5.2.3) indeed demonstrates that grounding is a key challenge for current approaches, which can be seen in particular by comparing the reduced and unreduced variants of GenZ-LTL. This can be partly attributed to the sparse reward setting common to LTL-RL methods. We further identify non-myopic reasoning as a separate key challenge, with myopic methods such as GCRL-LTL and GenZ-LTL struggling in domains where this is required.

Our results furthermore support that explicit task conditioning (e.g. subgoals or reach-avoid sequences) generally outperforms implicit task conditioning (e.g. syntax trees or semantic embeddings). Among subgoal-conditioned methods, GenZ-LTL outperforms GCRL-LTL due to properly accounting for avoidance via its safe RL formulation. For sequence-conditioned methods, StructLTL outperforms DeepLTL across the board, supporting that Boolean formulae and temporal attention enable better representation learning than sequences of assignments.

In Section 6 we point out some possible direction for future work to address these limitations of current approaches. We hope that our benchmark can serve as a valuable testbed for such future work.

Limited research contribution.

To summarize our contributions, we (1) provide an end-to-end JAX implementation of six algorithms and five environments in a unified framework, (2) demonstrate that our approach leads to order-of-magnitude speedups, accelerating future research, and (3) systematically benchmark existing approaches across different dimensions, showing different strengths and weaknesses. Our technical contribution is adapting heterogeneous, dynamically updated LTL task representations to this fully compiled pipeline. As you point out, we thus contribute to the “integration, standardization, and acceleration of existing approaches”, which we believe is a valuable contribution and appropriate scope for a benchmark paper, and can serve as a useful testbed for future research.

We hope that the updates to the paper, additional experimental results, and our comments are helpful. Please let us know if these have addressed your concerns. We are more than happy to engage in further discussion!

Thank you for the detailed response and for the new experiments. your resolves my concern about attributing the speedup to the precompilation design.

But my first concern was specifically about the delta relative to SpecRLBench, which already standardizes several experimental settings, and XLand-MiniGrid, which already uses fixed-size arrays for symbolic rules and goals. Could you address that comparison directly?

I am happy to raise my score on the basis of the new performance results, and would raise it further if question are addressed.

AuthorsOpenReview ↗

Comparison to SpecRLBench and XLand-MiniGrid

Thanks for your reply! We’re glad we could resolve your concerns regarding the speedup attribution.

Regarding SpecRLBench and XLand-MiniGrid, Jaxolotl compares to these two prior works as follows:

SpecRLBench

We agree that SpecRLBench already makes an important contribution to standardisation in LTL-RL. It provides common environments and evaluation specifications, and standardises several architectural and training choices. As described in its Appendix B, it integrates these environments into the methods’ original training and evaluation pipelines. Jaxolotl extends this standardisation in three main respects.

First, we standardise the algorithm implementations themselves. Jaxolotl implements six representative LTL-RL methods within a common modular codebase, sharing components such as PPO optimisation, observation encoders, and training and evaluation infrastructure, while retaining method-specific task representations and learning objectives. This reduces implementation-level confounds (e.g. advantage normalisation and other PPO implementation details) and enables controlled comparisons. It also enables easily implementing new methods and evaluating them on the exact same experimental setup. See Appendix C for an example of how StructLTL introduces a different task representation and encoder while reusing DeepLTL’s training and evaluation wrappers, evaluation agent, and common planning infrastructure.

Second, the framework supports end-to-end accelerated training and evaluation. Both the environments and methods are implemented in JAX, with symbolic task structures precompiled into array representations that support task updates within the compiled loop. This enables substantial acceleration and parallelisation across independent training seeds, making larger experiments practical. Our benchmark uses ten independently trained policies and reports confidence intervals over these policies (§4.2).

Third, we provide additional controlled diagnostics of algorithmic limitations. SpecRLBench studies generalisation to unseen specifications and increasing specification and environment complexity. Our experiments complement these analyses by explicitly testing non-myopic reasoning and scaling with the proposition vocabulary (§5.2.2 - 5.2.3). Together, these experiments expose complementary limitations of LTL-RL methods, and we hence propose possible directions for future research in Section 6.

Finally, the revised benchmark includes a challenge track that combines more demanding control, non-myopic dynamics, larger proposition sets, and object interactions. This provides substantial headroom for future methods; for example, the best evaluated method achieves only 31% finite-horizon success on Warehouse-hard (Table 1). We hope this challenge track can serve as a useful testbed for future research. In contrast, SpecRLBench’s benchmark is mostly saturated (with the best methods achieving close to perfect accuracies), leaving little headroom for future methods.

XLand-MiniGrid

XLand-MiniGrid also pre-samples tasks and represents symbolic rules and goals as fixed-size arrays within JAX. However, it primarily targets meta-RL, where agents infer hidden rules and goals through interaction, whereas Jaxolotl targets zero-shot execution of given LTL specifications. Our contribution is adapting such array representations to the heterogeneous and dynamic task representations used in LTL-RL.

In XLand-MiniGrid, the rules and goal remain fixed throughout an episode. In LTL-RL, the task representation itself must be updated as the agent completes parts of the specification. For example, LTL2Action progresses the formula into a new syntax tree, and automaton-based methods update the automaton state and corresponding subgoals. We enable these updates within a fully compiled training loop by precompiling the possible task transitions as well as the initial representation.

These representations further depend on symbolic preprocessing that cannot run within JAX’s compiled loop, including external tools such as Rabinizer and SemML. We perform this preprocessing offline and convert the outputs into array representations, allowing training and evaluation to proceed within JIT.

Furthermore, Jaxolotl supports six methods with different task representations through shared interfaces, including syntax trees, reach–avoid sequences, and semantic automaton embeddings, whereas XLand-MiniGrid has a single task format. We apply the same infrastructure across different methods and environments. Thus, while we build on the general approach of representing symbolic structures as static arrays, we extend it to support the task representations and online updates required for a unified, end-to-end compiled LTL-RL benchmark.

We hope these additional clarifications are helpful. Would you consider adjusting your score accordingly? If you have any other questions or comments, we are happy to engage!

Dear Reviewer,

Thanks again for your comments, which we believe has significantly improved the quality of our paper. We hope our clarification regarding SpecRLBench and XLand-MiniGrid in the last response was helpful. With less than one day remaining, we would be grateful if you could clarify if this has resolved your remaining concerns; otherwise we're happy to engage in further discussion!

Best, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.