acceptodds
Under review as a conference paper at ICLR 2027

PlatoLTL: Scaling LTL-Guided Multi-Task RL

Abstract

Linear temporal logic (LTL) has emerged as a powerful formalism for specifying structured, temporally extended tasks in multi-task reinforcement learning (RL). However, while existing approaches in LTL-guided multi-task RL demonstrate success in simple environments, they suffer from challenges in *representation learning* and *exploration efficiency* when applied to high-dimensional environments with *parameterized* specifications. We present PlatoLTL, which elevates state-of-the-art methods to address both challenges. We model atomic propositions as instances of *atomic predicates* and inject task parameters directly into the goal embedding to enable efficient generalization. We also leverage *simple priors* on closeness to predicate satisfaction to enable and accelerate learning of complex tasks without biasing the optimal policy. We validate our approach on challenging environments including robotic manipulation and multi-drone navigation.

Primary area:
reinforcement learning
Keywords:
linear temporal logic, reinforcement learning, multi-task reinforcement learning, reward shaping, robotics, neurosymbolic AI

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_My4QOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses two bottlenecks in multi-task RL with LTL specifications: (1) Plato uses a small predicate-specific network to map predicate parameters directly into the proposition/goal embedding. The authors argue that this induces a low-dimensional geometry over propositions and enables generalization across parameter values. (2) P2BRS constructs a potential function from manually supplied, coarse per-predicate robustness metrics. Those metrics are bounded and adaptively scaled; Boolean subgoals are composed with a persistence-gating mechanism, and the resulting potential is used with standard potential-based reward shaping.

The paper evaluates these ideas on four environments, including quadruped navigation, two-drone navigation, Franka manipulation, and a discrete navigation domain. It compares Plato against discretization, parameter concatenation, and SIAMS/FiLM, primarily using StructLTL, and additionally shows compatibility with LTL2Action.

Critical strengths

The paper has a clear motivation and a well-structured experimental story. The Franka manipulation results are particularly strong, showing that P2BRS can turn near-failing policies into reliable ones. The ablations are useful and support the importance of key design choices such as persistence gating. The paper is also transparent about its assumptions and provides strong implementation and reproducibility details.

Critical weaknesses

The comparison does not fully isolate the benefit of the Plato embedding design. Plato injects predicate parameters directly into the goal embedding, while Concatenate and SIAMS place the parameters on the state/observation side, and Discretize uses independent lookup embeddings. A simple goal-side baseline—for example, concatenating a learned predicate embedding with its parameters and passing them through a shared MLP—would help determine whether Plato's gains come from its predicate-specific embedding structure or simply from conditioning the goal representation on the parameters. Without this comparison, the source of the representation improvement is not fully clear.

Questions for authors

Have you tested P2BRS together with the other representation baselines, especially Discretize, Concatenate, or SIAMS in FrankaCubesEnv? This would help determine how much of the improvement comes from the Plato representation versus the reward-shaping method.

3 comments

We thank the reviewer for their review, and are glad that they found "a clear motivation and a well-structured experimental story", that "the Franka manipulation results are particularly strong", and that the paper is "transparent about its assumptions".

Both of the reviewer's suggestions concern disentangling the contributions of the two components of PlatoLTL. In response, we have run both suggested experiments in FrankaCubesEnv:

  • the goal-side Shared MLP baseline (Appendix N.5);
  • P$^2$BRS applied to each baseline representation (Appendix N.5).

We have revised the uploaded manuscript PDF accordingly, and address each point in turn below.

W1: Isolating the benefit of the Plato embedding design

We have added the suggested goal-side baseline, "Shared MLP", which embeds each predicate instance f(xf)f(x_f) as ϕ([ef;xf])\phi([e_f; x_f]), using a learned predicate embedding efe_f and a single MLP ϕ\phi shared by all predicates. As we note in Appendix N.5, this is equivalent to the Plato embedding module with every weight tied across predicates except a predicate-specific first-layer bias.

With P$^2$BRS, Shared MLP matches Plato (84.4% against 84.3% average success on finite tasks, and 20.9 against 21.4 visits on infinite tasks; Tables 4 and 5), with neither consistently ahead per specification, although Shared MLP trails by up to 13 points mid-training before catching up (Figure 7); without P$^2$BRS, both fail. In FrankaCubesEnv, Plato's gains over the baselines therefore come from conditioning the proposition embeddings on the parameters, i.e., from injecting the parameters into the goal embedding, rather than from having separate networks per predicate. We view this as supporting the central claim of Section 5, with per-predicate MLPs as one simple instantiation of it.

Q1: P$^2$BRS with the other representation baselines

We have trained Discretize, Concatenate, and SIAMS with P$^2$BRS. P$^2$BRS lifts their average success rates from 0.0% to 1.9%, 12.9%, and 14.9%, respectively, against 84.3% for Plato with P$^2$BRS (Table 4). Their successes are concentrated on the two tasks that require carrying a cube to a target position (φ26\varphi_{26}, φ29\varphi_{29}), where Concatenate and SIAMS reach 66.4% and 63.1% on φ29\varphi_{29}, respectively, but with episodes 2-4 times as long as Plato's. On tasks involving orientation targets or placing a cube, no baseline exceeds 14%; none succeeds on the 9-step sequence φ25\varphi_{25} (97.1% for Plato), and none improves on infinite-horizon tasks (Table 5). Thus, P$^2$BRS supplies an exploration signal to every representation, but only goal-side parameter conditioning can harness it to achieve high success rates. Neither component suffices alone: Plato without P$^2$BRS averages 2.9%, the baselines with P$^2$BRS at most 14.9%, and the combination 84.3%.

We thank the reviewer again for these suggestions, which we believe have clarified the respective contributions of both components, and hope that they strengthen the reviewer's confidence in the paper.

Reviewer_My4QOpenReview ↗

Thank you for your response, i have no more concern.

raise my score to 4.

AuthorsOpenReview ↗

Rebuttal Response Reply

We thank the reviewer for their engagement and for raising their score. We are glad our additional experiments addressed their concerns. Their suggestions clarified the respective contributions of the Plato embedding and P$^2$BRS, and we believe they have strengthened the paper.

Reviewer_uRGiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes PlatoLTL, which addresses two challenges in LTL-guided high-dimensional multi-task RL: representing parameterized propositions and exploring under sparse LTL rewards. It proposes the Plato embedding to encode predicate parameters directly into goal representations, and a potential-based reward shaping method that provides dense guidance. Experiments show substantial improvements in several robotics environments.

Critical strengths
  1. The paper is clearly written and sound. The structure is clear: the paper introduces two challenges and then proposes two methods for each.
  2. Plato embedding proposes an inductive bias for parameterized propositions, and it learns smooth and low-dimensional predicate embeddings. P²BRS provides dense within-subgoal guidance while retaining the potential-based reward shaping formulation.
  3. The experiments are thorough and show solid results across several benchmarks.
Critical weaknesses
  1. The method assumes propositions are instances of known predicates and that the agent has access to a ground-truth labeling function. While I understand this is common in LTL-guided RL, it is restrictive for real robotics, where labels would typically be inferred from noisy perception. Label errors may affect automaton transitions as well as the reward signal.
Questions for authors
  1. How robust is PlatoLTL to noisy predicate labels? Could the authors add an experiment in which a controlled fraction of predicate labels are corrupted and measure the effect on automaton progression and final task performance?
4 comments

We thank the reviewer for their review, and are glad that they found the paper "clearly written and sound", and the experiments "thorough", showing "solid results across several benchmarks".

In response, we have:

  • added the requested experiments with controlled label corruption, measuring the effect on both automaton progression and final task performance (Appendix N.6);
  • evaluated and trained policies with a learned labeling function that predicts labels from the policy's own observations, as a proof of concept (Appendix N.6).

We have revised the uploaded manuscript PDF accordingly, and address the reviewer's points in detail below.

W1 & Q1: Ground-truth labeling functions and robustness to noisy labels

We agree that this assumption, while common in LTL-guided RL, is restrictive for real robotics. In a preliminary study (Appendix N.6), we evaluate the trained Plato + P$^2$BRS policy in Go1RGBZoneEnv under four sources of label corruption, each swept over a range of levels: false-positive and false-negative flips applied to a controlled fraction of labels, Gaussian noise on perceived zone colors (which corrupts the policy's observations as well as its labels), and per-episode zone color offsets. We do so with both an analytic labeler, which reads privileged state information but is subject to the same corruptions, and a learned, pre-trained labeler that predicts labels from the policy's LiDAR observation alone. This setup resembles a common sim-to-real pipeline, in which a robot is first trained on ground-truth observations and rewards in simulation, and then deployed on hardware with those observations predicted by a learned perception module (Andrychowicz et al., 2020). We additionally train Plato policies from scratch with each labeler (without P$^2$BRS, whose geometric prior requires privileged state information), both without corruption and under combined color noise and offsets, such that the perceived labels drive both the sparse reward and curriculum progression. To measure the effect on automaton progression, we run a perceived and a true automaton in parallel, and report false acceptances, spurious violations, and missed violations alongside final task performance (Figures 8 and 9, Table 6). In summary:

  • False negatives have little effect on success. For both labelers, missing half of all true labels leaves success unchanged, since a missed label leaves the automaton in place while the policy continues to pursue the same subgoal.
  • False positives mainly disrupt automaton progression. For both labelers, a per-step false-positive rate of 0.08% reduces success from 97.5% to 84.3%; however, the loss lies mostly in automaton progression (around 11% false acceptances) rather than in the policy's behavior, with true violations rising only from 0.8-0.9% to 1.8%.
  • The learned labeler remains viable, at some cost under perception noise. When evaluating the pre-trained policy, the learned labeler matches the analytic labeler under label flips, but loses up to 3 points more success under color noise and offsets, mainly owing to more false acceptances (e.g., 7.4% against 4.3% at σ=0.35\sigma = 0.35, with 89.7% against 92.4% success). Training a policy from scratch with the learned labeler, without any ground-truth label information, matches the analytic labeler without corruption (98.0% against 97.3%). Under heavy color noise and offsets during both training and evaluation, every seed of both labelers still completes the curriculum, but success falls further for the learned labeler (78.0% against 84.9%), again mostly due to automaton progression.

Since the label errors caused by false-positive and false-negative flips are independent of the state, no policy can prevent them, and their mitigation lies at the labeling and automaton layer, e.g., via beliefs over automaton states (Li et al., 2024). Such approaches are compatible in principle with PlatoLTL, whose contributions concern the goal embedding and reward shaping. We also note that perception errors in the robustness metrics, as opposed to the labels, degrade only the quality of guidance and never the optimal policy, since shaping with any bounded potential is unbiased (Appendix O); Appendix N.7 further shows, as a proof of concept, that these metrics can themselves be learned. We leave further study of learned and uncertain labeling functions to future work (Appendix P).

We hope that these results address the reviewer's concerns, and would be grateful if they would consider revisiting their score in light of them.

Dear Reviewer uRGi, as the discussion period closes in a few days, we wanted to check whether our rebuttal has addressed your concerns about noisy labels.

In brief, we ran the controlled label-corruption experiment you requested (Appendix N.6), measuring automaton progression and task performance. False negatives leave success essentially unchanged, while false positives mainly disrupt automaton progression rather than the policy's behavior. We also trained policies from scratch with a learned labeling function that uses only the policy's LiDAR observations, so that perceived labels drive both the reward and the curriculum; without corruption, performance matches that of the analytic labeler, which uses privileged state information (98.0% vs. 97.3% success).

If these results address your concerns, we would be grateful if you would consider updating your assessment, and we would be happy to clarify anything further while there is still time.

Thank the reviewer for the response; it solves my concerns, and I have raised my score.

AuthorsOpenReview ↗

Rebuttal Response Reply

We thank the reviewer for their engagement and for raising their score. We are glad our additional experiments addressed their concerns. Their suggestion to study label corruption has provided insight into PlatoLTL's robustness to noisy perception and to training on perceived rather than ground-truth labels, which we believe has strengthened the paper.

Reviewer_Na8rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PlatoLTL, which extends an existing LTL-guided multitask reinforcement learning method by embedding continuously parameterized predicates directly into task representations. It also introduces a potential-based reward shaping method that uses predicate-level robustness measures. Experiments across quadruped navigation, two-drone control, robotic manipulation, and grid-based navigation demonstrate improvements in generalization to parameterized LTL specifications, particularly in sparse-reward environments.

Critical strengths
  • Strong empirical improvements in several settings, particularly robotic manipulation, with useful ablations such as persistence gating and robustness calibration.
  • The potential-based shaping guarantee is standard, but the construction integrates predicate-level robustness with LTL subgoals while preserving the backbone's optimal policy.
  • Broad temporal evaluations covering reachability, avoidance, conjunctions, persistence, recurrence, and long sequences.
Critical weaknesses
  • Limited multi-agent scaling demonstrated (only 2 drones). The authors also report unsuccessful preliminary experiments with 3 drones.
  • Dependence on hand-designed robustness metrics and heuristics such as persistence gating, which introduces manually specified knowledge about task dependencies.
  • The experiments show improvements for structured tasks with informative robustness measures, but do not establish whether those improvements persist when geometric guidance becomes misleading.
Questions for authors
  • Although the environments include collision avoidance and other constraints, they do not seem to test difficult navigation around nonconvex obstacles. How would the approach perform in maze-like environments where the robustness measure conflicts with progress along a feasible trajectory? For example, if the robot starts in a U-shaped maze, reaching a goal outside the U may require initially moving away from it, which would be discouraged by direct distance-based robustness measures. I would like to see whether P²BRS remains effective in such settings, especially with multiple sequential goals in larger mazes, similar to MuJoCo AntMaze environments.
  • Could the authors provide a clearer technical comparison with RAD Embeddings and explain how the approaches are related? The paper could better distinguish its representation contribution from possible extensions of existing automata encoders with parameter-conditioned symbol embeddings. For example, could RAD's transition-symbol representations be adapted to incorporate learned predicate-parameter embeddings, similar to PlatoLTL?
5 comments

We thank the reviewer for their careful review, and are glad that they found "strong empirical improvements in several settings, particularly robotic manipulation, with useful ablations", alongside "broad temporal evaluations".

In response, we have:

  • added XYAntMazeEnv, a maze environment with non-convex obstacles and long sequential tasks (Sections 7 and 8);
  • shown that the robustness metrics can be learned, as a proof of concept (Appendix N.7);
  • added a technical comparison with RAD embeddings (Appendix A);
  • clarified the scope of our two-drone experiments (Section 1, Appendix K.2.3).

We have revised the uploaded manuscript PDF accordingly, and address each point in turn below.

W1: Multi-agent scaling

We agree that we do not demonstrate scaling beyond two drones, and have revised the abstract and contributions to refer to two-drone rather than multi-drone navigation. With three drones, none of the methods we evaluated learned a strong policy; we attribute this to the centralized policy lacking weight sharing or permutation equivariance across drones, so the drone-to-target assignment must be learned without structural support. Such a setting is more naturally addressed by multi-agent RL, and is beyond the scope of this paper; Appendix K.2.3 now points to recent LTL-guided MARL work (Guo et al., 2026a;b). Two drones already suffice for the homogeneous conjunctions that X2DronesEnv is designed to study.

W2: Hand-designed robustness metrics and persistence gating

Our robustness metrics act as coarse priors: any metric leaves the optimal policy unbiased, and a trivial metric is always available. To show that they need not be hand-designed, preliminary results in Appendix N.7 replace them with time-to-go metrics learned from a waypoint-reaching dataset (collected with a scripted policy), with no change to P$^2$BRS. These perform on par with or slightly better than the hand-specified metrics: in X2DronesEnv, success rises from 90.5% to 92.3%, while results in XYAntMazeEnv are comparable. As for persistence gating, persistence is a single Boolean annotation per predicate, shared across all specifications (e.g., only has\mathsf{has} is persistent in FrankaCubesEnv), rather than knowledge about task dependencies. Since predicates are typically few, we hypothesize that these annotations could be selected via a hyperparameter sweep; we leave further study of learning metrics and persistence annotations to future work (Appendices O and P).

W3 & Q1: Misleading guidance and maze-like environments

We thank the reviewer for this valuable suggestion; following it, we introduce XYAntMazeEnv, in which a MuJoCo Ant navigates between goal positions in the large AntMaze layout with dead ends and U-turns, on tasks of up to 8 sequential goals (φ14\varphi_{14}). Its metric is the Euclidean distance to the goal, which ignores the walls and thus misleads as the reviewer describes. Nonetheless, P$^2$BRS raises Plato's success rates from 20.5-94.2% to 67.5-96.8%, while the baselines remain below 13% on all tasks but one (Table 1); Figure 3 shows the policy navigating a U-turn during an 8-step task. This is because P$^2$BRS uses the metric only as a shaping potential, not as the optimization objective: by policy invariance, its dead ends are never optimal under the shaped return, and the critic need only learn the residual between the true value and the prior, which it does quickly over locally convex stretches before value spreads across the maze (Appendix N.1). Moreover, a learned metric that accounts for the walls performs on par with the misleading Euclidean one (85.6% against 86.4% average success; Appendix N.7), suggesting that locally misleading metrics need not hinder P$^2$BRS.

Q2: Comparison with RAD embeddings

RAD encodes each automaton transition by one-hot encodings of its triggering symbols, so its first linear layer sum-pools learned per-proposition embeddings over the transition's guard. RAD therefore satisfies Assumption 5.1, and the adaptation the reviewer describes is equivalent to applying the Plato embedding module to RAD as a backbone, replacing each per-proposition embedding with ϕf(xf)\phi_f(x_f); P$^2$BRS likewise applies unchanged. What does not transfer directly is RAD's task-agnostic pre-training in an abstract MDP whose actions are the symbols themselves: there, distinct predicate instances are unrelated symbols regardless of their parameters, so the per-predicate geometry that Plato learns (Appendix N.2) cannot be acquired. A natural combination would thus retain RAD's frozen automaton encoder while training ϕf\phi_f end-to-end. We view the two as complementary: RAD targets generalization over automaton structure, whereas PlatoLTL targets generalization over predicate parameters and accelerated exploration.

We thank the reviewer again, and believe that the maze experiment has substantially strengthened the paper.

Dear Reviewer Na8r, as the discussion period closes in a few days, we wanted to check whether our rebuttal has addressed your concerns.

Following your suggestion, we added XYAntMazeEnv, in which a MuJoCo Ant navigates the large AntMaze layout with dead ends and U-turns, including tasks that require reaching 8 goals in sequence. Its Euclidean-distance metric ignores the walls, and thus misleads as you described. Nonetheless, P$^2$BRS raises Plato's success rates from 20.5-94.2% to 67.5-96.8%, while the baselines remain below 13% on all tasks but one (Table 1). Since P$^2$BRS uses the metric only as a shaping potential, dead ends are never optimal under the shaped return; moreover, a learned metric that accounts for the walls performs on par with the Euclidean one (Appendix N.7).

We have also added a technical comparison with RAD embeddings (Appendix A), and revised our claims to two-drone rather than multi-drone navigation (Abstract, Section 1), pointing to recent work on LTL-guided MARL for scaling to more agents (Appendix K.2.3).

If these results address your concerns, we would be grateful if you would consider updating your assessment, and we would be happy to clarify anything further while there is still time.

AuthorsOpenReview ↗

Follow-Up (24h Remaining)

Dear Reviewer Na8r, as the discussion period closes in roughly 24 hours, we wanted to follow up once more and kindly ask if you have any additional comments or concerns. We would be happy to clarify anything further while there is still time.

Comments on the paper

1 comment

We thank the AC and all reviewers for their time and constructive feedback, and summarize the discussion and our revisions below. All changes are included in the revised manuscript PDF.

Reviewer assessment

Reviewers found the paper "clearly written and sound" (uRGi), with "a clear motivation and a well-structured experimental story" (My4Q), and noted strong empirical performance, particularly in robotic manipulation (Na8r, My4Q). Following the rebuttal, My4Q raised their score from 3 to 4 and uRGi from 2 to 3, both stating that their concerns had been resolved.

Revisions

In response to the reviewers' suggestions, we have added new experiments and analyses, and clarified the paper's scope:

  • Maze environment with misleading guidance (Na8r; Sections 7 and 8). In XYAntMazeEnv, a MuJoCo Ant navigates the large AntMaze layout with dead ends and U-turns, on tasks of up to 8 sequential goals, where the Euclidean-distance metric (used as a simple prior) ignores the walls. P$^2$BRS raises Plato's success rates from 20.5-94.2% to 67.5-96.8%, while the baselines remain below 13% on all tasks but one.
  • Learned robustness metrics (Na8r; Appendix N.7). Time-to-go metrics learned from data perform on par with the hand-specified metrics, showing that P$^2$BRS need not rely on hand-designed metrics.
  • Learned labeling functions and robustness to noisy labels (uRGi; Appendix N.6). Under controlled label corruption, false negatives leave success essentially unchanged, while false positives mainly affect automaton progression rather than the policy's behavior. We also train policies from scratch with a learned labeling function that perceives labels from LiDAR observations alone. Without corruption, these match policies trained with an analytic labeler that uses privileged state information (98.0% vs. 97.3% success); under heavy color noise and offsets, the learned labeler incurs some cost (78.0% vs. 84.9%), mostly due to spurious automaton transitions.
  • Clarifying the contributions of each component (My4Q; Appendix N.5). In FrankaCubesEnv, a goal-side "Shared MLP" baseline matches Plato, confirming that the gains come from conditioning the proposition embeddings on the parameters. Applying P$^2$BRS to the baselines raises their average success to at most 14.9%, against 2.9% for Plato alone and 84.3% for Plato with P$^2$BRS. Thus, neither component suffices on its own.
  • Comparison with RAD embeddings (Na8r; Appendix A). We argue that RAD is complementary to PlatoLTL, and propose how it could serve as a backbone, as StructLTL and LTL2Action do.
  • Clarified scope (Na8r; Abstract, Section 1, Appendix K.2.3). We now refer to two-drone rather than multi-drone navigation, and point to recent LTL-guided MARL work for scaling to more agents.

We believe these additions have substantially strengthened the paper, and thank the reviewers again for their suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.