- Rating
- 1
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
This paper introduces PlanBench Gym -- a tool call wrapper for PlanBench. The setup works as follows: an LLM-backed agent executes some actions on Blocksworld tasks, a classic planning domain defined in PPDL (Planning Domain Definition Language) where agents must rearrange blocks to achieve desired configurations, receives feedback in the form of natural language, and continues to interact with the environment until either it achieves the original goal or consume its step budget.
The authors center this paper around the idea of “ergodicity” (an agent’s property to reach any next state from any current state, implying reversibility in their actions), arguing that existing agentic benchmarks are more likely than not ergodic. They classify subgoal interactions into three categories based on how the achievement of a subgoal directly impacts the cost of achieving others: no interaction (no change), positive interaction (reduces costs), and negative interaction (increases costs). The authors evaluate models across different configurations along two different axes: the agent harness (i.e., prompted ReAct, hybrid ReAct, and native tool calling) and the context fetched to the agent (either just the current state, or the full history of states’ interactions).
The three key findings of the paper are that (1) agent scaffolding plays an important role, specifically prompted ReAct outperforms both hybrid and native, across different models/configurations; (2) GPT-5.2 struggles to deal with negative subgoal interactions, to the extent that it even fails the easier tasks that are easily completed by the other models; (3) context management is extremely model-dependent, with some models (e.g., Sonnet 5) benefiting from only the current state (quite counterintuitive) and others (e.g., Qwen 3.6) benefitting from the full-interaction history. Although these differences are clear and explicitly observed, the core reasoning behind them is not properly investigated.
Critical strengths
-
The work centers around the relevant question of whether benchmarks accurately assess planning failures since mistakes can be undone.
-
PDDL provides an explicit and inspectable causal structure, addressing limitations of benchmarks where dependencies are embedded in natural language or tool code.
-
The instance generator validates candidates using Fast Downward, and the plan-length padding scheme effectively scales difficulty without altering class definitions.
-
Separating scaffolding from action elicitation (Prompted vs. Hybrid vs. Native) provides useful experimental control, with prompts clearly documented.
-
The work centers around the relevant question of whether benchmarks accurately assess planning failures since mistakes can be undone.
-
PDDL is a good structured approach for inspecting this problem, because it isn’t subject to the limitations of other benchmarks that put dependencies in language or tool code.
-
The benchmark is generally designed well, especially in how instances were generated and difficulty was controlled:
- the instance generator relies on the Fast Downward algorithm in order to validate candidate moves/instances;
- the difficulty of the instance is able to be scaled effectively via the padding scheme, without needing to alter the class definitions.
-
Separating scaffolding from action elicitation (Prompted vs. Hybrid vs. Native) provides useful control and helps to isolate the causes of their results
Critical weaknesses
I generally find the paper evaluation not to be rigorous. I enumerate my soundness concerns first and then the general issues/nits.
1/ Mismatched definition for negative interactions: the paper’s cost definition C(AND g_i) > sum c(g_i) is not satisfied by the negative interaction class in unit cost Blocksworld (e.g., tower variant should be in the no interaction case, given that each subgoal takes two actions, pick up and stack, and their joint optimal plan takes four). A similar issue applies for the Sussman variant (but please correct me if I’m wrong).
1.1/ Why claim this is for planning and PDDL and then only test on Blocksworld, when there are other domains too that could strengthen evaluation
2/ In Section 4, all three classes cost exactly two steps per subgoal, which implies additivity for the negative class. However, verification only checks cost inequalities for positive instances and only solvability for negative instances. This is not effective because all instances are solvable. Instead, these instances test goal ordering or non-serializability (Korf; Barret & Weld). Overall, from 1-3, it seems like the formal definitions are not correct here.
3/ Confounding across classes: Performance drop in negative tasks may stem from tower depth or chain reasoning, not subgoal interaction, because no-interaction instances consist of disjoint pairs, whereas negative instances involve single chained towers, I think there are some papers discussing this issue recently in broader context.
4/ Harness bugs - there are some big score drops that seem anomalous to me - integration or harness issues, such as: Claude Sonnet 5 drops from 100% (current-state) to 0 to 10% (full-history) on positive and negative tasks, Gemma 4 Native current-state scores 2% on no-interaction, which is a trivial task, GPT-5.2 Hybrid full-history scores 0% on every class, including no-interaction.
Without trajectory error analysis, it is difficult to determine whether these results reflect model capabilities or harness artifacts.
5/ Significant overclaiming:
- "Prompted ReAct is the strongest scaffold across all models, interaction classes, and context management modes"
- no, i see Prompted scores 2/0/0 and is the worst scaffold for at least one mode
- "GPT-5.2 fails outright under most scaffolds" and "solves 0% of negative-interaction instances at any step budget"
- Varies by configuration, e.g. GPT-5.2 reaches 58% under Prompted full-history
- The headline here is therefore an artifact of configuration choice.
- "All complete negative-interaction tasks with reasonable consistency"
- Not true for Qwen 3.6 current-state native or hybrid
- Efficiency metric limitations: You measure subopt only over complete runs -- my instinct is this would cause survivorship bias, since the ratio is capped at 2 by design. The gap is only 1.09x at most and without confidence intervals and given small sample size, isn’t conclusive.
General paper problems:
1/ The paper is in a weird place, where it generally reads like significant portions were generated by artificial intelligence without a human in the loop, but is written poorly enough in places that I don’t think any existing frontier tool would write so poorly. The excessive repetition of the term ergodic also stands out to me as natural writing would not repeat it so much or define it far more clearly.
2/ Lots of prior work and terms are used without citation throughout the introduction. There is one cite in all of page 2. Hugging face incident is not cited, Sokoban is not cited, many more examples. Very small number of references in general. PDDLGm is not cited. Back-prompting as a field is generally not included. Logistics is not evaluated at all.
3/ Organization is odd. Why is related work before background? It is very hard to situate the related work before hearing about the background. Also, why not have any planning work in related work?
4/ The title references analyzing existing agentic benchmarks but there’s a new benchmark without good eval of preexisting ones.
Questions for authors
- Are tool call rejections accounted in the consumption of the step budget? And what about in the suboptimality ratio?
- Could you provide an explanation of Claude’s collapse under the full-history condition, and regarding Gemma’s 2% score on no-interaction under native current-state?
- Could you make explicit how many samples per instances were run, and their settings (e.g., temperature, reasoning-effort, …)?
6 comments
Evaluation on Non-ergodic domain
There are many other relevant PDDL domains that could have been tested.
We have now extended our evaluation to another PDDL domain, the Sokoban environment. Unlike Blocksworld, Sokoban is a non-ergodic environment in which the agent cannot reverse certain actions once it reaches a dead-end state. For example, pushing a box into a corner can make the problem permanently unsolvable. This environment therefore provides a more realistic assessment of the agent's actual solvability rate. In contrast, Blocksworld is an ergodic environment in which, given a sufficiently large step budget, even a random walk can eventually reach the goal state.
We incorporated the Sokoban environment into the PlanBenchGym benchmark. The Sokoban problem instance generator is built on top of DeepMind's Boxoban [2] and it generates problem instances using the reverse-play algorithm [3]. For our evaluation, we generated 20 problem instances on a 1010 grid, varying the number of boxes from 2 to 4. The agent's goal is to push all boxes to their corresponding goal positions. From the reverse-play procedure, a random walk is utilized to carve out a random map topology. Utilizing that topology, two cells are chosen to be the start and goal for each box. The reverse-play algorithm ensures that there is a solution by essentially pulling a box across the map to the start position. In our modified version of reverse play, we increase the number of traps along a solution to increase difficulty. A detailed description of the Sokoban instance generation procedure is provided in Appendix E. The LLM agent interacts with the environment through tool calls that provide access to three actions: walk, push, and give up. We introduced the give up action to allow the agent to explicitly terminate an episode if it believes that the problem instance is unsolvable from the current state.
The tables below present the results on 20 Sokoban problem instances for Claude Sonnet 5 and GPT-5.2 using Prompted-ReAct scaffolding under two context settings: current-state and full-history.
Table 1 reports the failure rates of the two models. The results show that both frontier models exhibit failure rates ranging from 20% to 45% across the 20 Sokoban problem instances.
Table 1: Failure rate
| Model | Current State | Full History |
|---|---|---|
| Claude Sonnet 5 | 35% (7/20) | 45% (9/20) |
| GPT-5.2 | 30% (6/20) | 20% (4/20) |
We further categorize the failure cases into three categories:
- (a) Dead-state failures: The agent pushes a box into a corner, resulting in an unsolvable state. The environment then terminates the episode.
- (b) Premature termination: The agent selects the "give up" action even though the goal remains reachable, voluntarily terminating the episode without solving the problem.
- (c) Step-limit failures: The agent reaches the maximum step limit (2x optimal plan length), resulting in episode termination without achieving the goal.
continuation..
Table 2 presents the number of failures in each of the three categories across the 20 Sokoban problem instances for both models, using Prompted-ReAct scaffolding under the current-state and full-history context settings. From the table's results, it is evident that under the current-state setting, Claude Sonnet 5 selected the "give up" action in 6 out of its 7 failure cases, incorrectly assuming that the problems were unsolvable even though solutions remained reachable. This highlights the LLM agent's difficulty in distinguishing between solvable and unsolvable problem states.
Table 2: Why the runs failed (dead state / gave up in a solvable state / hit the max step limit)
| Model | Current State | Full History |
|---|---|---|
| Claude Sonnet 5 | 0 / 6 / 1 | 6 / 1 / 2 |
| GPT-5.2 | 1 / 0 / 5 | 3 / 0 / 1 |
Under the full-history setting, Claude Sonnet 5's actions led the environment into dead states in 6 out of its 9 failure cases. This finding highlights a critical limitation of LLM-based agents in non-ergodic environments, where certain decisions can lead to irreversible states from which recovery is impossible. Since many real-world environments exhibit similar non-ergodic characteristics, such failures raise important concerns about the reliability and safety of deploying LLM agents in real-world settings.
Real-world incidents further illustrate the potential consequences of such failures. In one incident involving Hugging Face, an OpenAI evaluation agent reportedly escaped its intended network boundary and compromised Hugging Face's production systems while attempting to complete a cybersecurity benchmark. Similarly, an OpenAI agent researching public medicine spending reportedly bypassed access controls on Australia's Medicare statistics portal after being denied access, gaining access to non-public files and writing to an internal server. These incidents underscore the importance of evaluating not only whether an agent can achieve its intended goal, but also whether its actions can lead to irreversible and potentially harmful states along the way.
We additionally report in Table 3 the suboptimality ratio within the solved Sokoban problem instances.
Table 3: Suboptimality ratio within the solved instances (mean, with range in parentheses)
| Model | Current State | Full History |
|---|---|---|
| Claude Sonnet 5 | 1.28 (1.0–1.5) | 1.18 (1.0–1.6) |
| GPT-5.2 | 1.27 (1.0–1.64) | 1.19 (1.0–1.78) |
Modified the Negative Interaction definition
We thank the reviewer for their detailed and critical review. We would like to provide clarifications below.
The cost inequality definition is a sufficient, but not necessary, condition for negative interaction. The Tower and Sussman anomaly instances exhibit negative interaction under the ordering/serializability-based definition in Korf's taxonomy. Accordingly, we have revised the definitions of subgoal interaction in the paper based on Korf's taxonomy [1] (see Section 2).
Under these definitions, a set of subgoals is considered serializable if there exists an ordering of the subgoals such that they can be solved sequentially without violating any previously achieved subgoal. However, certain subgoal orderings may lead to increased solution costs compared to the optimal cost, even when an ideal ordering would yield an additive solution cost. The Tower problem instances fall into this category.
In contrast, a set of subgoals is considered non-serializable if achieving further progress toward the main goal necessarily requires violating at least one previously satisfied subgoal, regardless of the ordering in which the subgoals are attempted. The Sussman anomaly problem instances fall into this category.
Based on these definitions, we further categorize negative interaction problem instances into two subcategories: (1) serializable problem instances, represented by the Tower problems, and (2) non-serializable problem instances, represented by the Sussman anomaly problems.
The table below presents the results for these two subcategories of negative interaction problem instances using the Claude Sonnet 5 model with Prompted-ReAct scaffolding and the current-state context setting. The results show that, under this setting, the agent solves all Tower problems optimally. However, for the Sussman anomaly problems, although the model achieves a 100% solve rate, it generates suboptimal plans, with a mean suboptimality ratio of 1.18. This indicates that, on average, the agent performs 4 redundant actions compared to the optimal plan length. (Note: the values in parentheses show the range.)
We have included the same analysis for the other models, agent scaffolding approaches, and both context settings (current-state and full-history) in the paper (see Section 5).