acceptodds
Under review as a conference paper at ICLR 2027

Contrastive Reinforcement Learning for Symbolic Planning

Abstract

Symbolic planning is a type of sequential decision-making problem for goal reaching modeled in a symbolic language like Planning Domain Definition Language (PDDL). The mainstream approaches for solving symbolic planning problems are search-based methods. Recently, along with the development of machine learning techniques, there is a growing interest in exploiting learning-based approaches in symbolic planning. In spite of the similarity between symbolic planning and Markov Decision Processes, most learning-based approaches for symbolic planning are supervised (imitation) learning methods. Model-free reinforcement learning methods, which can explore the environment freely, are under-explored in this area. In this paper, we study how to customize contrastive reinforcement learning (CRL) for symbolic planning, which results in a model-free approach for symbolic planning with competitive performance.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Reinforcement Learning, Planning, Neurosymbolic Policy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LkkjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes to apply Contrastive Reinforcement Learning (CRL) to learn policies for Symbolic Planning (SP) tasks. More specifically, it uses CRL to learn a Q-function/critic f(s,a,sg)f(s,a,s_g) that estimates the (discounted) likelihood to reach a goal state sgs_g after applying some action aa at the current state ss and then following the current policy π\pi. The authors explain several difficulties of directly applying CRL to the SP setting. To solve these issues, they propose to condition the critic on a single goal atom pgp_g at a time instead of the whole state sgs_g, thus learning f(s,a,pg)f(s,a,p_g). Then, as shown in Algorithm 2, the SP task can be solved by decomposing its goal GG into a sequence of goal atoms pg1,⋯ ,pgn∈Gp^1_g, \cdots, p^n_g \in G and solving each pgip^i_g sequentially (this is called goal serialization in the SP literature). Finally, the authors carry out experiments in standard IPC benchmark domains, showing their serialized CRL policy can solve some domains, but fails at others (see Fig. 4).

The motivation behind this work is interesting: CRL learns the likelihood of reaching a goal state, whereas SP tasks revolve around finding plans to reach a given goal. Thus, it seems natural that CRL would provide a better fit for SP than standard RL, which focuses on maximizing reward. Unfortunately, the paper presents many issues (see below). Mainly, it makes the assumption that task goals GG can be attained by sequentially achieving their goal atoms pg∈Gp_g \in G one-by-one and in an arbitrary order, which is extremely unrealistic and immensely restricts the scope of the paper. As a result, I strongly believe the paper should be rejected.

I found three main issues with the paper.

First, the paper is phrased in a strange way, and contains many grammatical errors, typos and confusing expressions, as if it had been written without careful prior planning. Example typos are: "avaliable" (line 55), "modify->modifies" (line 56), "another few more actions" (line 189), "trainig noise" (line 268), "aciton's" (line 270). Some phrasing issues and confusing parts are: lines 55-58, lines 65-66, Fig.1 caption (what is meant by "atoms that are irrelevant to an actual state"?), lines 160-161 ("[...] be dominated by an action-irrelevant").

Second, the paper is not rigorous, sometimes providing unsupported and misleading claims. For example, Lines 40-41 state: "Recently, (Stahlberg and Geffner, 2026) have proposed an approach in the model-free setting which does not require any offline training, however only performs well on limited domains". What is meant by "limited domains"? The authors in (Stahlberg and Geffner, 2026) perform experiments on IPC tasks, using a larger set than the current paper. Additionally, they do not assume task goals are serializable, which the current paper does. Another clear example is Section 3. Equation 3 rewrites f(s,a,sg)f(s,a,s_g) in terms of action-relevant and action-irrelevant terms. Then, the paragraph below states that "the ratio between the action-relevant and action-irrelevant terms is O(M/2M)O(M/2^M)", where MM is the number of atoms and 2M2^M the number of possible states. What does this even mean? Both the set of atoms MM and states 2M2^M, as truth-valuations over MM, depend on the action, i.e., they are both "action-relevant in principle". The authors provide no proof for this O(M/2M)O(M/2^M) anywhere in the paper or Appendix. After this, the next paragraph (Proposition 1) jumps to an unrelated topic, i.e., the fact that Q-values of good and bad actions are similar (Lines 189-190) (also, this is not true in many domains, e.g., domains with non-reversible actions and dead-ends, such as sokoban).

Third, and most importantly, the proposed method makes unrealistic assumptions and provides no real contributions. Section 3 presents several issues of CRL when applied to the SP setting, mainly that learning f(s,a,sg)f(s,a,s_g) is difficult in practice. Then, in paragraph "Naive Adaption" (line 202), the authors propose a solution: instead of learning ff for a goal state sgs_g, learn it for a goal atom pg∈Gp_g \in G instead. Then, Refinements 1 (line 245) and 2 (line 268) tackle how to improve learning, without addressing a crucial limitation: learning ff for single atoms pg∈Gp_g \in G only works if the task goal GG is serializable, i.e., can be achieved one goal atom at a time. Furthermore, the authors do not even consider the order in which these goal atoms should be achieved: Algorithm 2 (line 378) states that pgp_g should be sampled from GG (line 382), but does not state how. Thus, the scope is even narrower, i.e., tasks with serializable goals and for which goal ordering is not important. Paragraph "Limitations" (line 447) does not fully explain this. Lines 448-449 state "[in some domains] the order of achieving goal atoms matters". This is an extreme understatement: some domains require the policy/planner to consider several goal atoms at the same time or, else, the task may not be solved. Clear examples are puzzle-like domains, such as sokoban, with strong interactions between goals. The experiment results show some signs of this: the learned CRL policy performs poorly on Childsnack (see Fig. 4), precisely because goal ordering matters (as it is admitted in the "Limitation" paragraph).

Critical strengths

None.

Critical weaknesses
  • The proposed CRL method unrealistically assumes that tasks are goal-serializable (their goal GG can be attained by achieving each goal atom pg∈Gp_g \in G one-by-one) and, furthermore, that such goal atoms can be achieved in an arbitrary order (line 382 of Algorithm 2 does not consider in which order to achieve each pgp_g). As a result, the scope of the method is very narrow and provides no meaningful contribution.

  • The paper is hard to read, confusing, and sometimes provides unsupported and misleading claims. Some examples are the lack of a proof for the O(M/2M)O(M/2^M) ratio in Lines 167-174, or how authors state that the method in (Stahlberg and Geffner, 2026) "only performs well on limited domains" (Lines 40-41), despite the fact that the current paper uses similar benchmark domains (IPC) and assumes tasks are goal-serializable (which (Stahlberg and Geffner, 2026) do not).

Questions for authors

Below, I provide some questions for the authors:

  • What is the meaning behind the O(M/2M)O(M/2^M) ratio? What is it supposed to mean? And can you provide a mathematical proof?

  • Can you provide more information about the goal-serialization assumptions? Mainly, 1) why you assumed goals are serializable and 2) what happens when goals are serializable but the order in which they must be achieved matters (e.g., in childsnack). Also, it would be interesting to see experiments in domains where goals are not serializable, such as sokoban.

7 comments

We sincerely appreciate your efforts in reviewing our paper.

  • We agree that Alg. 2 relies on achieving goal atoms one at a time. However, we disagree that it makes unrealistic assumptions:

    • We do not assume serializability in arbitrary order in the strict sense. Alg. 2 does not protect achieved goal atoms: If an action undoes a previously achieved goal atom, the atom is marked as unachieved and pursued again (lines 384–385). So, we tolerate goal interactions that require undoing earlier subgoals. One such case is Blocksworld where our method solves 68% of test instances with a random order, compared with 26% for HER (Fig 4). What Alg. 2 does require is that greedily achieving an atom does not lead into a dead end. This is exactly our limitation in Childsnack and has already been discussed in the Limitations section. Note that we still outperform Stahlberg and Geffner's model-free approach in Childsnack.
    • Most IPC domains (which are standard benchmarks in Symbolic Planning) are serializable. In fact, the work by Lipovetzky & Geffner (2012) shows a stronger claim. They studied planning domains broader than IPC benchmarks and found that an optimal solution can be obtained for 88% of 37921 instances from those domains without considering more than 2 goal atoms at a time. In particular, there are also many search-based algorithms that solve planning problems by considering one goal atom at a time, e.g., Serialized IW (Lipovetzky & Geffner, 2012). These search-based algorithms also show competitive performance against other heuristic search algorithms. (Note that all search-based algorithms cannot work in the model-free setting, which is the focus of our work.)
    • One major issue we want to resolve using our approach actually implies the serializability of a domain. If a domain's actions are reversible, then this domain is also serializable. However, as we have discussed in the paper and here, the existence of reversible actions is a major reason that causes the ratio between the variance of the action-relevant term and the variance of the action-irrelevant term to be exponentially small, which is the problem we want to resolve using our approach.
    • From a practical perspective, many real-world scenarios that can be modeled as a symbolic planning domain are also serializable. As strong supporting evidence, the Rover domain from our evaluation benchmark set, which abstracts NASA's actual Mars Exploration mission, is serializable, and the Transport domain (also in our evaluation benchmark set), which abstracts a logistic scenario, is serializable too. On top of these, the presence of the FAIL token in our approach also introduces some capability to deal with domains with non-serializable goal.
    • We employ the most simple strategy (Alg 2) to deploy the learned policy because we also assume the model-free setting in policy deployment. If we ignore this restriction and assume that we have access to actions' preconditions and effects, then there are many existing symbolic methods that can be exploited to compute a good order of goal atoms (Koehler & Hoffmann, 2000; Hoffmann, Porteous & Sebastia, 2004).
  • Blocksworld is a very good example about the case where a domain is serializable but a good goal order is important. See our discussion above about this domain.

  • Sokoban is an extremely challenging domain. By using the same training diet and the same policy deployment strategy as in our experiments, we can solve 12.2% (again averaged over 3 seeds) test instances in Sokoban. As a comparison, Stahlberg and Geffner's model-free approach can only solve 3.3% test instances (averaged over 3 seeds). To our best knowledge, all policy learning approaches that work in Sokoban require a set of offline trajectories to train the policy, and search algorithms are usually involved (for generating offline trajectories or used in conjunction with the trained policy). See, e.g., Czechowski et al., 2021, and Ziarko et al., 2025. In particular, those approaches work directly on image-based Sokoban (i.e., the domain is not represented as a symbolic planning problem). In other words, no policy learning approaches in the context of Symbolic Planning work for Sokoban.

  • O(M/2M)\mathcal{O}(M/2^M) bounds the ratio between the variance of the action-relevant term and the variance of the action-irrelevant term along a trajectory. When the ratio is small, critic training fits the action-irrelevant term and largely ignores the action-relevant term, which is the only part the policy uses.

    • The term 2M2^M comes from trajectory length. Reaching a full goal state can require an exponentially long action sequence (which is a common assumption in symbolic planning), and the action-irrelevant term changes along that whole sequence. The action-relevant term stays bounded by O(log⁡(1/γ))\mathcal{O}(\log(1/\gamma)) at every step (Prop. 1).
    • We agree the original justification was not rigorous enough. The updated PDF gives the full derivation. Thanks for pointing this out!
  • Prop. 1 is not unrelated. On the contrary, together with the O(M/2^M) bound, it forms the core argument for why using model-free RL methods to learn a policy is challenge.

    • Prop. 1 says that when a domain's actions are reversible, the advantage of picking an optimal action at a state against picking a non-optimal one is small.
    • This small advantage means that a learned critic can very easily incorrectly rank actions. If the true advantage of a good action is α\alpha and for a bad action is β\beta, the advantage of choosing the good action over the bad one is given by α−β\alpha-\beta. If the learned critic overestimates β\beta, then the learned critic will misrank these actions.
    • Since the training process already tends to ignore the action-relevant term, the advantage of each action is more likely to be incorrectly estimated, causing the learned critic misranking actions.
  • Both action-irrelevant and action-relevant terms are meaningful parts of the critic. They behave like a value function VV, a Q-function QQ, and an advantage function AA in standard RL language: Q(s,a)=V(s)+A(s,a)Q(s,a) = V(s) + A(s,a) for a state-action pair (s,a)(s,a):

    • Action-irrelevant term (analogous to VV): how likely we are to reach sgs_g from ss when following the policy, i.e., the expectation over a∼π(⋅∣s)a \sim \pi(\cdot \mid s). It depends on ss, but not on the choice of aa.
    • Action-relevant term (analogous to AA): how much taking a specific action aa in ss changes that likelihood.
    • As with QQ, ranking actions only needs the action-relevant term. The problem is that, for domains with reversible actions and long plans, the action-irrelevant term dominates the training target.
  • As our experiments show (Fig. 4), Stahlberg and Geffner's approach only works for a few IPC domains.

    • Many of their experimental domains are not from IPC benchmarks. They constructed their own evaluation set where many of the domains do not feature complexity like Rovers and Transport from IPC.
    • In their setting, when deploying their policy, they enabled loop elimination, that is, when the action the policy going to take will result in a loop, the policy will choose the second best action instead. This loop elimination is not applicable in our model-free setting. Our policy deployment strategy deliberately does not feature any loop elimination.

Thank you very much for your clarifications. I now understand the paper better. Also, I agree that your model-free setting assumes less prior knowledge (due to being model-free) than the model-based setting considered in (Stahlberg and Geffner).

Reviewer_ZEDjOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyzes the possibilities and limitations of using model-free RL to solve symbolic planning problems. The authors propose to use contrastive RL (CRL) since it shares characteristics with planning. However, the use of CRL is not straightforward since symbolic planning goals almost never map directly to one state. Symbolic goals are a set of predicates that need to be true at the end of a trajectory but can be achieved at different times. Therefore, the paper first presents a naive approach of reaching each goal predicate independently, theoretically and empirically analyzes the problems encountered, and presents two refinements based on those observations. Empirical evaluation on six domains shows that the proposed method outperforms another model-free approach to solve symbolic planning problems and has similar performance to supervised approaches.

Critical strengths

Using model-free RL approaches to solve symbolic planning problems is currently under explored. The paper's analysis of the challenges of using CRL to solve symbolic planning problems is novel and contributes to addressing an important existing gap in this area and is clearly presented. In particular, the narrative shows the authors' reasoning and their analysis (both theoretical and experimental) is connected to the porposed solution. Results are supported by the experiments and are useful and important to guide future research in this direction. The literature review also helps situate the work within existing research and supports the motivation for investigating this under-explored problem.

Critical weaknesses

None.

Questions for authors

The paper argues that the ratio between the action-relevant and action-irrelevant terms in Equation 3 is O(M/2^M). Could the authors clarify the assumptions underlying this bound? For example, consider a domain with M independent Boolean switches, giving (2^M) reachable states, where each action flips only one switch. Suppose that, from the all-OFF state, a policy uniformly selects one of the M flip actions and remains in the resulting state after that. For a goal state in which only switch 1 is ON, and a uniform goal distribution over all states, the action-relevant term for flipping switch 1 is log M, and the action-irrelevant term is M*log 2-log M. Their ratio therefore scales to \Theta((log M)/M). Am I making different assumptions?

3 comments

We sincerely appreciate your efforts in reviewing our paper.

  • The term 2M2^M in the ratio comes from the assumption that in order to reach a goal state, an exponentially long action sequence is needed.
    • This is a common assumption in Symbolic Planning (since it is PSPACE-hard).
    • In the example you constructed, the goal state can be reached in at most polynomial number of steps from any state. Hence, the exponential ratio is not obvious in this example.
    • We refer to our updated PDF for a rigorous justification about this ratio. Thanks for pointing it out!
Reviewer_ZEDjOpenReview ↗

Thank you

Than you for your clarification and the revised analysis. One minor mathematical clarification: the conclusion about exponential decay appears to require a lower bound on L, since L = O(2^M) also allows L to grow linearly with M.

Thank you so much for the reply! Your clarification is very true! LL is lower bounded exponentially since even optimal solutions can be exponentially long. We have clarified this in our updated PDF.

Reviewer_1k4jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to customize contrastive reinforcement learning (CRL) for symbolic planning by estimating the likelihood of completing a task specified in the goal. Detailed analysis is provided to show the non-triviality of adapting CRL to symbolic planning. Results show the proposed method achieves comparable performance on the Mars Rovers domain.

Critical strengths

The paper is easy to follow.

The motivating example is sound.

Critical weaknesses

The main motivation for introducing contrastive RL into symbolic planning is not sufficiently clear. Previous studies have already investigated learning planning heuristics or search guidance through RL. Moreover, ConPoLe has explored contrastive reinforcement learning in symbolic reasoning domains, which is not discussed here. What is the fundamental difference between the proposed method and these existing approaches? Is the main contribution a new contrastive learning objective, a different representation of symbolic states and goals, or a new mechanism for guiding symbolic search?

The literature review in this paper is insufficient. There are closely related works that are not discussed or compared.

[1] Reinforcement Learning for Classical Planning: Viewing Heuristics as Dense Reward Generators

[2] Meta Reinforcement Learning for Heuristic Planning

[3] Reinforcement Learning for Planning Heuristics

[4] Exploiting Symbolic Heuristics for the Synthesis of Domain-Specific Temporal Planning Guidance Using Reinforcement Learning

[5] Contrastive Reinforcement Learning of Symbolic Reasoning Domains

Questions for authors

Please see the pros and cons part.

3 comments

We sincerely appreciate your efforts in reviewing our paper.

  • ConPoLe is disjoint from our approach. ConPoLe deals with symbolic reasoning tasks concerned with mathematical arithmetic operations. Both the syntax and the semantics of this problem family are completely different from Symbolic Planning. Furthermore, our approach assumes a model-free setting, while strictly speaking, ConPoLe does not. More specifically:

    • A state in ConPoLe is a string of textual symbols representing a mathematical expression. In contrast, a state in our approach is a set of atoms encoded as a graph and processed by a graph neural network whose semantics vary across different domains.
    • ConPole assumes that every problem instance has one fixed goal state, while in our approach, every state that contains all goal atoms is a goal state.
    • ConPoLe assumes the access to the consequence of an action (a mathematical operation), which is a very common assumption in mathematical reasoning tasks. However, we do not assume that we can know the consequence of an action in our approach.
  • Our work focuses on learning and deploying policies for Symbolic Planning in a model-free setting. Hence, it is an independent line of research from heuristic learning for Symbolic Planning.

    • The objective of heuristic learning for planning is to incorporate the learned heuristic into a search algorithm (e.g., A*) and produce a solution via search, while policy learning is to use the learned policy to produce a solution directly (i.e., no search involved). These are two independent lines of research in the area of ML for Symbolic Planning.
    • In particular, due to the model-free setting, search algorithms are not applicable (since we don't know actions' preconditions and effects). Hence, our approach and heuristic-learning-based approaches are incomparable.
  • Due to the above reasons and the fact that we did not have sufficient space earlier, we did not include the discussion about related work. We have already updated our PDF to cover this line of discussion. Thanks for pointing it out!

  • One motivation for using contrastive reinforcement learning is that the additional training signal about which atom cannot be achieved by taking an action in a state could result in a stronger policy.

    • This is supported by one of our empirical evaluations where we compared CRL against Goal-conditioned behavior cloning (GCBC). Our implementation for the latter can be viewed as a variant of our approach without using contrastive learning.
Reviewer_1k4jOpenReview ↗

response

Thanks for the response. I would maintain my score because the paper’s central contribution remains unclear to me. For example, it is unclear how much contrastive learning contributes to solving complete planning instances. The comparison in Figs. 3 and 4 does not include GCBC with the same goal-selection policy. This leaves open how much of the gain on complete tasks comes from contrastive learning beyond the benefits of goal decomposition and action filtering.

Thank you so much for the reply!

  • The key insight of this paper is to command the policy on individual goal atoms and achieve them sequentially using our trained goal atom selection policy, as opposed to solving for the entire goal state at once. We demonstrate this approach with CRL. In principle, the same decomposition could also be used with GCBC. As Figure 3 shows, GCBC was less successful than CRL at achieving individual goal atoms. This motivated us to focus our evaluation of learned goal selection on CRL. We leave the evaluation of GCBC on complete instances to future work. We also want to emphasize that in order to make GCBC work, the two novel refinements we proposed are necessary (as shown by Figure 3).
  • We cannot complete this additional evaluation within the rebuttal period. We would be happy to include the comparison in any future version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.