- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to improve symbolic solving for constrained planning tasks with tightly coupled decisions under a limited computation budget. It proposes SkeletonPlan, which represents planning requirements as compositional constraint programs and uses an LLM agent to choose between joint solving and a two-stage strategy that first commits to selected decisions and then completes the full plan. The framework further learns reusable textual guidance from execution feedback to improve the selection of these solving strategies and decision interfaces. Experiments cover TravelPlanner and three NaturalPlan tasks. In the reported 180-instance TravelPlanner replay, SkeletonPlan achieves a final pass rate of 95.00%, compared with 90.56% for joint solving and 59.44% for its empty-memory variant. On NaturalPlan, it reports an average accuracy of 93.33% using disjoint task-specific memory-learning and evaluation subsets.
Critical strengths
- A flexible neuro-symbolic solving framework. The paper proposes a framework that adapts solver organization to individual planning instances. It can choose between joint and two-stage solving and select which decision families and constraints enter the first-stage skeleton. Combined with guidance learned from execution feedback, this offers greater flexibility than using a fixed solving strategy.
- Promising empirical performance on planning tasks. The reported experiments demonstrate strong performance on TravelPlanner and three NaturalPlan tasks. On TravelPlanner, SkeletonPlan reports a final pass rate of 95.00%, exceeding the reported results of the compared neuro-symbolic methods, including RigorFormal at 93.30%. These results provide empirical support for the framework’s potential effectiveness in constrained planning.
Critical weaknesses
- Comparability of results across different backbone models. The proposed method and several published baselines use different backbone models, which makes it difficult to separate the contribution of the framework from differences in model capabilities. While the within-framework ablations provide useful comparisons, the performance advantage over existing methods would be more convincing with comparisons using a shared backbone and evaluation protocol.
- Clarification of the TravelPlanner evaluation split. There appears to be an inconsistency between the main text and Appendix H regarding the TravelPlanner evaluation. The main text describes the results as held-out evaluation, whereas Appendix H states that the same 180 instances were used to provide memory-learning evidence and evaluate the learned memory. Clarifying this distinction would be helpful. If the protocol follows the appendix, the reported improvement would demonstrate effectiveness on previously encountered instances, while generalization to unseen TravelPlanner instances would remain unclear.
Questions for authors
- Could you provide a comparison with an existing neuro-symbolic method using the same backbone model and evaluation split?
- Were the 180 TravelPlanner evaluation instances also used for memory learning? Please clarify the apparent discrepancy between the main text and Appendix H.
- Could you report results on the official TravelPlanner test set, using memory learned from a disjoint split and frozen during evaluation?
- Could you provide a learning curve showing performance on a fixed held-out set after each memory update?
- Have you explored allowing the agent to choose the number of solving stages, including more than two, instead of using a fixed two-stage procedure?