acceptodds
Under review as a conference paper at ICLR 2027

Regression in Repair: Understanding Plan Repair Dynamics in Large Language Models

Abstract

Recent advances have shown that adding feedback and tools enables large language model (LLM) agents to solve increasingly complex planning problems across diverse domains. One effective paradigm for plan repair involves LLM planners producing a candidate plan, with programs checking every constraint of this plan and feedback or tools fixing the constraints it violates. In this paper, we study plan repair under coupled constraints, which price every choice against one budget so that every infeasible option of a slot is cheaper than every feasible one. We systematically analyze repair dynamics across three dimensions, coupling, the content of the repair message, and the order of the options, through empirical studies on 4,800 synthetic tasks, a second environment with requirements in text, and 900 TravelPlanner plans using two open-source models (4B and 8B parameters), gpt-4.1 and gpt-4.1-mini. Our experiments reveal three key findings about repair effectiveness when models answer in one step, namely (1) Uncoupled tasks allow feedback to more reliably fix failed attempts without breaking another constraint; (2) Exact totals from a calculator tool produce repairs that break fewer constraints than messages that only name the failed check; (3) Option choice is generally correlated with price, but this relationship varies with coupling, under which the cheaper choice is the infeasible one. These findings reveal opportunities for optimizing basic repair strategies in planning applications. First, given the same model, a single call on options sorted by the shared resources can nearly match a calculator tool in pass rate (e.g., with price and duration sorted by their weighted sum, the gap to the calculator tool shrinks from 20.0 and 15.0 points to 1.0 and 4.5 points at a quarter of its cost). Second, we identify cases where four samples at temperature 1.0 offer limited advantages over one attempt, as all samples read the same option order, suggesting that sampling alone cannot overcome position bias.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM agents, planning, plan repair, self-correction, feedback, coupled constraints, position bias, TravelPlanner

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_5UsPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies regressions during LLM plan repair, where fixing an existing violation introduces another one. It examines constraint coupling, feedback content, and option order through 4,800 synthetic tasks across four main models, an additional synthetic environment where requirements are expressed in natural language, and 900 TravelPlanner plans. The results show regressions in some settings and improvements from feasible-alternative guidance, tools, step-by-step prompting, and changes in option order. Additional controls and limitations are reported in the appendix.

Critical strengths
  • The paper examines whether repair breaks previously satisfied requirements instead of only reporting overall success. This helps identify cases where an apparent fix merely moves the failure elsewhere.

  • The experiments cover multiple models, feedback formats, tools, and option orders in both synthetic environments and a real benchmark. The authors also report failed predictions, model differences, and additional controls, making the limits of the findings relatively transparent.

Critical weaknesses

The main research contribution remains unclear. The paper reports many experiments on constraint coupling, feedback content, and option order, but I am unsure what mechanistic understanding they provide beyond the intuitive observations that repairing interacting constraints can introduce new errors and more helpful feedback can improve repair. The results vary across models, prompts, and task settings, but these observations are not brought together into a clear explanation of when regressions occur and which intervention should be used. An empirical study does not need to propose a new algorithm, but I would like the paper to distill its findings into generalizable conclusions or a framework that can guide planning beyond the budget-planning settings studied here.

Questions for authors

What is the main generalizable conclusion of these experiments? Could the authors distill the findings into clear conditions or principles for predicting repair regressions and choosing a repair strategy? Which existing results show that these principles apply beyond the synthetic budget-planning settings?

Reviewer_otFLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows the "Regression in Repair" problem of agents in planning, specifically when constraints share a budget (coupling). They provide the task, dataset, and evaluation results for this problem.

Critical strengths

The regression issue in plan repair is interesting, and it matches a common real-world failure where a fix breaks something else. Splitting the metric into m and r is reasonable and useful for showing different regression behaviors. The data generation, which introduces different levels of coupling on the budget, also makes sense, and the experiments provide evidence that the degree of coupling affects regression.

Critical weaknesses

W1. m/r metrics and pass rate p0.: The m/r split is useful, but the paper does not show how m and r relate to the baseline pass rate p0. Since m is computed only on fixed failures, a different p0 can change which attempts enter S_b. Reporting m and r by task difficulty and failure type would show this directly.

W2. Does the analysis extend beyond budget?: Budget is a natural choice since it affects globally, but regression behavior may differ for other kinds of shared resources or constraints. Do the findings hold for other types? Also, how hard would the data extension be?

W3. Table presentation needs improvement (e.g., Table 2)

W4. How reliable is the generated data, and how realistic are the tasks compared to real human planning?

Questions for authors

Please see the Critical Weaknesses section.

Reviewer_D6huOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is trying to understand how repairing a problem will lead to a new problem, from the aspects of constrain coupling, feedback information, and option ordering. It shows that under some conditions using such strategies may be a good way to improve the models performance on corrections. The empirical results provides more insights on this topic.

Critical strengths

the paper systematically studied this problem with experiments and shows reasonable findings

Critical weaknesses

personally I feel like the just showing the finding is not enough for this paper to be accepted. need to demonstrate more impact. willing to reconsider during rebuttal.

motivation

1.1Do regression patterns and position biases persist in more capable frontier models? how frequently do these failure patterns occur in realistic agent workflows?

experiment setting

2.1 How is your synthetic data comparing with real world scenarios?

2.2 Is the finding stable under held-out/ out of distribution data as well? (it might be hard to show)

findings

3.1 Is your combination (I know it's more than combination but still) of existing findings reveals a fundamentally new mechanism beyond the known individual effects? Need interaction analysis

3.2 The insights seems to be a combination of some already known findings?

3.3 Better demonstration of the finding works and how solid are the findings are? Some findings appear sensitive to model choice and experimental conditions

3.4 The evaluation basically compares basic prompting, sorting, which is unclear how your insights provide stronger planning approaches comparing with others under same budget.

visualizations,

personally i think good visualizations are very important to show your findings and results, which is insufficient for now.

Questions for authors

Please focus on all mentioned weakness

Reviewer_2jtQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper empirically studies plan repair in large language models, examining how constraint coupling, feedback content, and option ordering affect the introduction of new violations during repair. Through experiments on synthetic budget-planning tasks, a textual-requirements environment, and TravelPlanner, the authors report three main observations: (1) under the evaluated single-step settings, feedback-based repair introduces fewer additional constraint violations in uncoupled tasks; (2) a calculator-based protocol that provides exact totals produces fewer regressions than feedback that only identifies failed checks; and (3) model choices are associated with option prices and positions, with these relationships changing under coupling, particularly when cheaper options fail to satisfy local requirements.

The authors further investigate two practical strategies motivated by these observations: ordering options by shared-resource consumption and shuffling option order across repeated samples. Both show benefits in some evaluated settings. The paper’s main contribution is a systematic characterization of plan-repair behavior and the resulting implications for repair strategy design.

Critical strengths

S1.Systematic and substantial empirical analysis of plan repair: The authors conduct controlled comparisons of constraint coupling, feedback content, and option ordering, examining both final pass rates and violations introduced during repair (§§3–4). Supplementary analyses involving difficulty controls, reversed price relationships, weaker coupling, and multi-round repair help characterize failure transitions under different conditions (Appendices C.2 and D.1; Table 20). This systematic investigation has clear empirical value and is the main strength supporting my positive assessment of the work.

S2.The analysis motivates strategies with measurable benefits in several settings: The paper evaluates resource-based ordering and sampling with shuffled option order (§5). In the textual-requirements environment, four samples with shuffled ordering improve pass rates by 9.0–17.5 percentage points over four temperature samples with fixed ordering across the two GPT models and two resource settings (Table 2). The corresponding TravelPlanner results are 27.8% versus 16.7% (Table 16). These findings provide actionable suggestions and demonstrate practical value in the empirical observations.

Critical weaknesses

W1.The substantive analytical contribution emerging from the experiments is not yet sufficiently clear: Empirical analysis can itself constitute a technical contribution, and I appreciate the paper’s detailed investigation of plan repair. My main concern is that the manuscript does not yet clearly establish what new technical understanding its findings on coupling, feedback, and option ordering provide beyond the related work on constraint-repair conflicts, external feedback, and positional bias discussed in §2.The presentation emphasizes individual observations and their synthesis, while giving less attention to which insights are fundamentally new, what previously unresolved questions they answer, and which system-design or evaluation decisions they support. Consequently, the work currently reads as a substantial empirical report whose broader analytical contribution remains insufficiently articulated. I would encourage the authors to identify one or two central insights, explain how they extend existing understanding, and connect them to the most informative controlled comparisons. A well-supported account of mechanisms, a diagnostic approach, or a design principle could each provide a convincing contribution.

W2.The incremental value of the insights for practical plan-repair improvement needs further clarification: The paper demonstrates gains from the two strategies over some baselines. However, their implications for designing more reliable or efficient planning systems remain uncertain. For example, in the two-resource setting of Table 2, having the model enumerate feasible options and an exact solver select the plan achieves pass rates of 88.0% and 96.0% for gpt-4.1-mini and gpt-4.1, respectively, compared with 44.0% and 51.5% for weighted sorting, at reported API costs only approximately 4%–5% higher. On TravelPlanner, price sorting alone does not improve the final pass rate: the default and sorted settings achieve 10.6% and 10.0%, respectively. The 41.7% result combines filtering, sorting, and feedback containing feasible alternatives (Table 16). These results make it important to clarify when the analysis helps select or improve a repair strategy and which observed gains can be attributed to the paper’s new insights. An explanation grounded in the existing experiments would help establish the connection between the empirical findings and their incremental value for planning-system design.

Questions for authors

Q1.Analytical contribution: What are the paper’s one or two most important new technical insights? How do they extend the existing understanding summarized in §2, and which experiments provide the strongest supporting evidence? In particular, what does the analysis establish beyond a systematic characterization of plan-repair phenomena?

Q2.Design implications: Which concrete repair or planning decisions should change as a result of these insights? Please explain how the existing evidence informs choices about feedback content, option presentation, or sampling budget, including the conditions and counterexamples that limit these recommendations.

Q3.Practical value: In light of the solver baseline in Table 2 and the TravelPlanner results in Table 16, how should the practical role of sorting and shuffled-order sampling be understood? Which results most clearly demonstrate that the analysis provides a meaningful basis for improving existing repair approaches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.