acceptodds
Under review as a conference paper at ICLR 2027

PLANSPACE: SEPARATING REFERENCE REPRODUCTION, PLAN VALIDITY, AND INTERFACE COMPLIANCE IN SYMBOLIC PLANNING

Abstract

A household task can have several correct plans: groceries may go into different allowed cabinets, and independent placements may occur in either order. Scoring only against one recorded sequence rejects such alternatives. It also leaves a low score hard to interpret: did the model format its answer incorrectly, choose an action it could not execute, or fail to reach the goal? We introduce PLANSPACE, an evaluation protocol that answers these questions separately. It checks whether a submitted plan can be interpreted, whether it matches a reference or the dependencies of a known plan family, and whether its actions achieve the goal under an explicit symbolic model. Across six open-weight instruction-tuned language models on 171 supported BEHAVIOR-1K tasks, exact matching misses successful plans for every model. For Qwen3-8B, the discrepancy is especially large when tasks permit alternative goal choices or require longer plans. Other low scores arise before planning can be assessed: a small formatting correction exposes successful plans in one model’s existing outputs but leaves the other models unchanged. These findings show why reference agreement and task success should be reported separately, alongside the interface checks needed to interpret a plan.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Embodied Planning, Plan Evaluation, Goal-Conditioned Planning, Execution-Grounded Evaluation, Partial-Order Planning, Large Language Models, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8LRhOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PlanSpace, which evaluates generated plans using four checks: formatting, exact reference match, plan-family match, and goal completion. The experiments show that exact matching misses some valid plans and formatting errors can also hide model performance. The authors also use symbolic replay, control cases, search, and comparisons with VAL to validate the method.

Critical strengths
  • The four-part evaluation clearly separates formatting errors, reference matching, and actual goal completion, making model failures easier to interpret.
  • The reported results are supported by controls, symbolic replay, and comparisons with VAL.
Critical weaknesses
  • The main finding (goal-based evaluation accepts valid plans missed by exact matching) is already standard in literature [1-4].
  • The exact-match gap depends heavily on lexical reference construction and prompt ordering. The new DAG-family metric has no clear practical benefit over goal replay.

Reference

[1] Valmeekam, Karthik, et al. "Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change." Advances in Neural Information Processing Systems 36 (2023): 38975-38987.

[2] Li, Manling, et al. "Embodied agent interface: Benchmarking llms for embodied decision making." Advances in Neural Information Processing Systems 37 (2024): 100428-100534.

[3] Huang, Wenlong, et al. "Language models as zero-shot planners: Extracting actionable knowledge for embodied agents." International conference on machine learning. PMLR, 2022.

[4] Mantenoglou, Periklis, et al. "LexiCon: a Benchmark for Planning under Temporal Constraints in Natural Language." Advances in Neural Information Processing Systems 38 (2026).

Questions for authors
  1. How is the exact reference selected? Are the results robust to changing the reference or shuffling prompt order?
  2. How does PlanSpace differ from prior goal-based evaluations, especially Embodied Agent Interface [2]?
  3. What did the “source-goal repair” change? Was it performed before examining model outputs?
  4. Will the code, tasks, and raw outputs be released for verification?

Reference

[2] Li, Manling, et al. "Embodied agent interface: Benchmarking llms for embodied decision making." Advances in Neural Information Processing Systems 37 (2024): 100428-100534.

Reviewer_rtFAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that scoring an LLM plan by exact match to one recorded plan is wrong, since many different plans reach the same goal (other cabinet, different order of independent steps). PlanSpace gives every output four flags: Interface (parses into known actions), Exact (same as the reference), Family (follows the dependency DAG of a reference plan) and Goal (symbolic replay reaches the goal). Tasks are 171 BEHAVIOR-1K tasks converted to a symbolic domain with only 5 skills (open, close, transfer, toggle on, toggle off). Six small open models, 5 samples per task, 5,130 outputs. Goal validity is above exact match for all six models (largest gap 17.4 points, Qwen3-4B), and 20.8% of Qwen3-8B's successful plans are rejected by exact match. Phi-4-mini and OLMo-2 lose 79% and 84% of outputs at parsing, and dropping a literal "action_id=" prefix moves Phi-4-mini goal validity from 7.6% to 46.2%.

Critical strengths
  • The measurement is done carefully for what it is: controls, a reference-free search that finds other solutions on 145/171 tasks, and a cost bound, so the undercount is not simply longer plans. I believe the main number.
  • Splitting a low score into parse failure / failed action / goal miss (Fig. 3e) is good practice and the Phi-4-mini vs OLMo-2 contrast illustrates it.
Critical weaknesses
  • Too small a contribution for ICLR. "Don't use a single reference, replay instead" is known. The only new object, DAG families, is never shown to change a conclusion that Goal alone gives. Gaps are modest and in a simplified 5-skill domain, so I don't see why anyone would change how they evaluate planners after reading this.
  • The evidence is partly unverifiable and the planning claims rest on very little. Supporting experiments are missing from the submission. Only the Qwen models are really evaluated (two models are 80%+ parse failures, Mistral 14%), all are 4–14B, no strong model at all. The diagnosis may not carry over to anything people actually care about.
Questions for authors
  1. Show real model outputs where Family=1 and Goal=0 or the reverse. What does Family give that Goal doesn't?
  2. How much of the false rejection is incompleteness of your constructor vs genuinely different plans? How many search-found solutions are outside all known families?
  3. Does the picture hold for a strong model (Qwen3-32B or a frontier one)? If the gaps vanish the paper has little point.
Reviewer_hi6yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PlanSpace looks at how the choice of evaluation rule affects our judgment of language-model planning. It checks whether an output can be parsed, matches a reference plan, belongs to a known plan family, and achieves the goal. The authors test six language models on 171 BEHAVIOR-1K-derived tasks and find that exact matching misses valid alternative plans. They also show how parsing requirements can change the reported success rates. I find the problem worth studying, and the paper is clear. My main concern is what we gain from PlanSpace that existing planning evaluators do not already provide.

Critical strengths
  1. The paper makes a useful distinction between matching a reference sequence and actually solving the task.

  2. The experiments include controls beyond the model comparisons. The reference-blind search and cost-bounded analysis make the findings more convincing.

  3. I like that goal validity is checked independently of DAG-family membership. Successful plans can still receive credit even when they fall outside the constructed families.

  4. The Phi-4-mini experiment is particularly telling. Removing the action_id= prefix raises measured goal validity from 7.6% to 46.2%, without changing the generated plans. This is a clear example of how parsing rules affect evaluation.

Critical weaknesses
  1. I am not fully convinced by the novelty. PlanBench (2023) and LLM-Modulo (2024) already use formal plan verification. The paper shows why exact matching can be misleading, but what PlanSpace adds beyond these approaches is still unclear.

  2. The comparison with LexiCon (2025) needs more attention. LexiCon already handles formatting deviations, plan validity, and optimality. I would expect a clearer account of what PlanSpace can diagnose that LexiCon cannot.

  3. I have some reservations about DAG-family scoring. Section 6.2 acknowledges that family mismatches can have different causes, and Section 3.2 validates only a subset of orderings for larger families. It is difficult to tell how reliably this score adds information beyond checking goal validity.

  4. Despite constructing 1,403 plan representatives, the authors use one selected reference for exact scoring. The main results leave open how much the Goal–Exact gaps depend on that choice or whether allowing multiple references would change the findings.

  5. The interface-failure analysis combines formatting problems with unknown-action failures. These may indicate different problems with a model's output. Although the paper identifies some unknown-action errors, the aggregate results make it difficult to separate parsing issues from failures to select permitted actions.

  6. The experiments cover 171 tasks with only five simplified symbolic skills. I am unsure how well the findings would carry over to richer action models or other planning domains.

Questions for authors
  1. What does DAG-family membership tell us that a standard plan verifier with execution-failure reports would miss?

2.Could the authors compare PlanSpace against a LexiCon-style evaluation on the same outputs to show what additional diagnostic information it provides?

  1. Which benchmarks are materially affected by single-reference scoring, and how would using multiple references change the reported Goal–Exact gaps?

  2. Could the authors separate recoverable formatting errors from unknown-action failures to clarify what drives the reported interface failures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.