- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies regressions during LLM plan repair, where fixing an existing violation introduces another one. It examines constraint coupling, feedback content, and option order through 4,800 synthetic tasks across four main models, an additional synthetic environment where requirements are expressed in natural language, and 900 TravelPlanner plans. The results show regressions in some settings and improvements from feasible-alternative guidance, tools, step-by-step prompting, and changes in option order. Additional controls and limitations are reported in the appendix.
Critical strengths
-
The paper examines whether repair breaks previously satisfied requirements instead of only reporting overall success. This helps identify cases where an apparent fix merely moves the failure elsewhere.
-
The experiments cover multiple models, feedback formats, tools, and option orders in both synthetic environments and a real benchmark. The authors also report failed predictions, model differences, and additional controls, making the limits of the findings relatively transparent.
Critical weaknesses
The main research contribution remains unclear. The paper reports many experiments on constraint coupling, feedback content, and option order, but I am unsure what mechanistic understanding they provide beyond the intuitive observations that repairing interacting constraints can introduce new errors and more helpful feedback can improve repair. The results vary across models, prompts, and task settings, but these observations are not brought together into a clear explanation of when regressions occur and which intervention should be used. An empirical study does not need to propose a new algorithm, but I would like the paper to distill its findings into generalizable conclusions or a framework that can guide planning beyond the budget-planning settings studied here.
Questions for authors
What is the main generalizable conclusion of these experiments? Could the authors distill the findings into clear conditions or principles for predicting repair regressions and choosing a repair strategy? Which existing results show that these principles apply beyond the synthetic budget-planning settings?