- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes to apply Contrastive Reinforcement Learning (CRL) to learn policies for Symbolic Planning (SP) tasks. More specifically, it uses CRL to learn a Q-function/critic that estimates the (discounted) likelihood to reach a goal state after applying some action at the current state and then following the current policy . The authors explain several difficulties of directly applying CRL to the SP setting. To solve these issues, they propose to condition the critic on a single goal atom at a time instead of the whole state , thus learning . Then, as shown in Algorithm 2, the SP task can be solved by decomposing its goal into a sequence of goal atoms and solving each sequentially (this is called goal serialization in the SP literature). Finally, the authors carry out experiments in standard IPC benchmark domains, showing their serialized CRL policy can solve some domains, but fails at others (see Fig. 4).
The motivation behind this work is interesting: CRL learns the likelihood of reaching a goal state, whereas SP tasks revolve around finding plans to reach a given goal. Thus, it seems natural that CRL would provide a better fit for SP than standard RL, which focuses on maximizing reward. Unfortunately, the paper presents many issues (see below). Mainly, it makes the assumption that task goals can be attained by sequentially achieving their goal atoms one-by-one and in an arbitrary order, which is extremely unrealistic and immensely restricts the scope of the paper. As a result, I strongly believe the paper should be rejected.
I found three main issues with the paper.
First, the paper is phrased in a strange way, and contains many grammatical errors, typos and confusing expressions, as if it had been written without careful prior planning. Example typos are: "avaliable" (line 55), "modify->modifies" (line 56), "another few more actions" (line 189), "trainig noise" (line 268), "aciton's" (line 270). Some phrasing issues and confusing parts are: lines 55-58, lines 65-66, Fig.1 caption (what is meant by "atoms that are irrelevant to an actual state"?), lines 160-161 ("[...] be dominated by an action-irrelevant").
Second, the paper is not rigorous, sometimes providing unsupported and misleading claims. For example, Lines 40-41 state: "Recently, (Stahlberg and Geffner, 2026) have proposed an approach in the model-free setting which does not require any offline training, however only performs well on limited domains". What is meant by "limited domains"? The authors in (Stahlberg and Geffner, 2026) perform experiments on IPC tasks, using a larger set than the current paper. Additionally, they do not assume task goals are serializable, which the current paper does. Another clear example is Section 3. Equation 3 rewrites in terms of action-relevant and action-irrelevant terms. Then, the paragraph below states that "the ratio between the action-relevant and action-irrelevant terms is ", where is the number of atoms and the number of possible states. What does this even mean? Both the set of atoms and states , as truth-valuations over , depend on the action, i.e., they are both "action-relevant in principle". The authors provide no proof for this anywhere in the paper or Appendix. After this, the next paragraph (Proposition 1) jumps to an unrelated topic, i.e., the fact that Q-values of good and bad actions are similar (Lines 189-190) (also, this is not true in many domains, e.g., domains with non-reversible actions and dead-ends, such as sokoban).
Third, and most importantly, the proposed method makes unrealistic assumptions and provides no real contributions. Section 3 presents several issues of CRL when applied to the SP setting, mainly that learning is difficult in practice. Then, in paragraph "Naive Adaption" (line 202), the authors propose a solution: instead of learning for a goal state , learn it for a goal atom instead. Then, Refinements 1 (line 245) and 2 (line 268) tackle how to improve learning, without addressing a crucial limitation: learning for single atoms only works if the task goal is serializable, i.e., can be achieved one goal atom at a time. Furthermore, the authors do not even consider the order in which these goal atoms should be achieved: Algorithm 2 (line 378) states that should be sampled from (line 382), but does not state how. Thus, the scope is even narrower, i.e., tasks with serializable goals and for which goal ordering is not important. Paragraph "Limitations" (line 447) does not fully explain this. Lines 448-449 state "[in some domains] the order of achieving goal atoms matters". This is an extreme understatement: some domains require the policy/planner to consider several goal atoms at the same time or, else, the task may not be solved. Clear examples are puzzle-like domains, such as sokoban, with strong interactions between goals. The experiment results show some signs of this: the learned CRL policy performs poorly on Childsnack (see Fig. 4), precisely because goal ordering matters (as it is admitted in the "Limitation" paragraph).
Critical strengths
None.
Critical weaknesses
-
The proposed CRL method unrealistically assumes that tasks are goal-serializable (their goal can be attained by achieving each goal atom one-by-one) and, furthermore, that such goal atoms can be achieved in an arbitrary order (line 382 of Algorithm 2 does not consider in which order to achieve each ). As a result, the scope of the method is very narrow and provides no meaningful contribution.
-
The paper is hard to read, confusing, and sometimes provides unsupported and misleading claims. Some examples are the lack of a proof for the ratio in Lines 167-174, or how authors state that the method in (Stahlberg and Geffner, 2026) "only performs well on limited domains" (Lines 40-41), despite the fact that the current paper uses similar benchmark domains (IPC) and assumes tasks are goal-serializable (which (Stahlberg and Geffner, 2026) do not).
Questions for authors
Below, I provide some questions for the authors:
-
What is the meaning behind the ratio? What is it supposed to mean? And can you provide a mathematical proof?
-
Can you provide more information about the goal-serialization assumptions? Mainly, 1) why you assumed goals are serializable and 2) what happens when goals are serializable but the order in which they must be achieved matters (e.g., in childsnack). Also, it would be interesting to see experiments in domains where goals are not serializable, such as sokoban.
7 comments
-
bounds the ratio between the variance of the action-relevant term and the variance of the action-irrelevant term along a trajectory. When the ratio is small, critic training fits the action-irrelevant term and largely ignores the action-relevant term, which is the only part the policy uses.
- The term comes from trajectory length. Reaching a full goal state can require an exponentially long action sequence (which is a common assumption in symbolic planning), and the action-irrelevant term changes along that whole sequence. The action-relevant term stays bounded by at every step (Prop. 1).
- We agree the original justification was not rigorous enough. The updated PDF gives the full derivation. Thanks for pointing this out!
-
Prop. 1 is not unrelated. On the contrary, together with the O(M/2^M) bound, it forms the core argument for why using model-free RL methods to learn a policy is challenge.
- Prop. 1 says that when a domain's actions are reversible, the advantage of picking an optimal action at a state against picking a non-optimal one is small.
- This small advantage means that a learned critic can very easily incorrectly rank actions. If the true advantage of a good action is and for a bad action is , the advantage of choosing the good action over the bad one is given by . If the learned critic overestimates , then the learned critic will misrank these actions.
- Since the training process already tends to ignore the action-relevant term, the advantage of each action is more likely to be incorrectly estimated, causing the learned critic misranking actions.
-
Both action-irrelevant and action-relevant terms are meaningful parts of the critic. They behave like a value function , a Q-function , and an advantage function in standard RL language: for a state-action pair :
- Action-irrelevant term (analogous to ): how likely we are to reach from when following the policy, i.e., the expectation over . It depends on , but not on the choice of .
- Action-relevant term (analogous to ): how much taking a specific action in changes that likelihood.
- As with , ranking actions only needs the action-relevant term. The problem is that, for domains with reversible actions and long plans, the action-irrelevant term dominates the training target.
-
As our experiments show (Fig. 4), Stahlberg and Geffner's approach only works for a few IPC domains.
- Many of their experimental domains are not from IPC benchmarks. They constructed their own evaluation set where many of the domains do not feature complexity like Rovers and Transport from IPC.
- In their setting, when deploying their policy, they enabled loop elimination, that is, when the action the policy going to take will result in a loop, the policy will choose the second best action instead. This loop elimination is not applicable in our model-free setting. Our policy deployment strategy deliberately does not feature any loop elimination.
Thank you very much for your clarifications. I now understand the paper better. Also, I agree that your model-free setting assumes less prior knowledge (due to being model-free) than the model-based setting considered in (Stahlberg and Geffner).
We sincerely appreciate your efforts in reviewing our paper.
We agree that Alg. 2 relies on achieving goal atoms one at a time. However, we disagree that it makes unrealistic assumptions:
Blocksworld is a very good example about the case where a domain is serializable but a good goal order is important. See our discussion above about this domain.
Sokoban is an extremely challenging domain. By using the same training diet and the same policy deployment strategy as in our experiments, we can solve 12.2% (again averaged over 3 seeds) test instances in Sokoban. As a comparison, Stahlberg and Geffner's model-free approach can only solve 3.3% test instances (averaged over 3 seeds). To our best knowledge, all policy learning approaches that work in Sokoban require a set of offline trajectories to train the policy, and search algorithms are usually involved (for generating offline trajectories or used in conjunction with the trained policy). See, e.g., Czechowski et al., 2021, and Ziarko et al., 2025. In particular, those approaches work directly on image-based Sokoban (i.e., the domain is not represented as a symbolic planning problem). In other words, no policy learning approaches in the context of Symbolic Planning work for Sokoban.