- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PlanSpace, which evaluates generated plans using four checks: formatting, exact reference match, plan-family match, and goal completion. The experiments show that exact matching misses some valid plans and formatting errors can also hide model performance. The authors also use symbolic replay, control cases, search, and comparisons with VAL to validate the method.
Critical strengths
- The four-part evaluation clearly separates formatting errors, reference matching, and actual goal completion, making model failures easier to interpret.
- The reported results are supported by controls, symbolic replay, and comparisons with VAL.
Critical weaknesses
- The main finding (goal-based evaluation accepts valid plans missed by exact matching) is already standard in literature [1-4].
- The exact-match gap depends heavily on lexical reference construction and prompt ordering. The new DAG-family metric has no clear practical benefit over goal replay.
Reference
[1] Valmeekam, Karthik, et al. "Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change." Advances in Neural Information Processing Systems 36 (2023): 38975-38987.
[2] Li, Manling, et al. "Embodied agent interface: Benchmarking llms for embodied decision making." Advances in Neural Information Processing Systems 37 (2024): 100428-100534.
[3] Huang, Wenlong, et al. "Language models as zero-shot planners: Extracting actionable knowledge for embodied agents." International conference on machine learning. PMLR, 2022.
[4] Mantenoglou, Periklis, et al. "LexiCon: a Benchmark for Planning under Temporal Constraints in Natural Language." Advances in Neural Information Processing Systems 38 (2026).
Questions for authors
- How is the exact reference selected? Are the results robust to changing the reference or shuffling prompt order?
- How does PlanSpace differ from prior goal-based evaluations, especially Embodied Agent Interface [2]?
- What did the “source-goal repair” change? Was it performed before examining model outputs?
- Will the code, tasks, and raw outputs be released for verification?
Reference
[2] Li, Manling, et al. "Embodied agent interface: Benchmarking llms for embodied decision making." Advances in Neural Information Processing Systems 37 (2024): 100428-100534.