- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes S-MCTS for planning under natural-language constraints, priorities, and preferences. Each search node stores its best discovered complete plan and a semantic summary. An LLM uses these summaries to select branches and propagate better plans, replacing numerical selection and backup. Experiments on TravelPlanner, Natural Plan–Trip Planning, and generated flexible travel-planning queries show competitive performance across three backbone models. Additional experiments examine priority satisfaction, component ablations, and token consumption.
Critical strengths
-
Retaining complete candidate plans and explanations of which requirements they satisfy or violate is a meaningful direction for flexible planning. This information directly guides search decisions instead of only explaining the final output, potentially helping the planner handle conflicting requirements.
-
The evaluation is relatively comprehensive. Beyond overall success rates, the supplementary material reports satisfaction rates at different priority levels and separately examines semantic selection and backpropagation. These results provide useful evidence for the proposed components. The additional token consumption is also reported.
Critical weaknesses
The experiments do not sufficiently establish whether the gains come from semantic evaluation itself.
-
In Appendix E, the scalar baseline simply counts the fraction of satisfied constraints, ignoring their priorities. It can therefore prefer satisfying several low-priority requirements while violating a high-priority one. Outperforming this baseline does not demonstrate that semantic evaluation is better than numerical evaluation that appropriately represents the user’s priorities. The authors do not include scalar baselines that account for priorities, arguing that the appropriate weights are unknown. However, once the priorities are identified from the query, a baseline can compare higher-priority requirements first, without tuning weights for each task.
-
In addition, Appendix F shows that S-MCTS consumes more tokens, without comparing performance under equal token or LLM-call budgets. The results support the method’s effectiveness, but do not clearly separate the benefits of semantic information from those of a better-aligned evaluation objective and additional computation.
Questions for authors
-
Would S-MCTS retain its advantage over a numerical evaluator that respects the query’s priorities? For example, could the same LLM extract the priority levels, followed by fixed priority rules or lexicographic comparison? Why would such a baseline necessarily require unknown or tuned weights?
-
How much of the improvement remains under equal token or LLM-call budgets? Could the authors provide this comparison and specify the stopping parameter B and the budgets used for each method?