- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper tries to study how LLM agents can recover from mid-task failures without redoing everything, which could be good for efficiency. It proposes to keep the verified intermediate results and fixing only the failed step when possible, and only revising the plan when local repairs do not work to maintain the performance. Experiments show that, with GPT-4o-mini, the full system reports more task successes on several benchmarks than the tested baselines.
Critical strengths
-
The problem this paper is trying to address is a practical problem: how much of an agent workflow should change after a local failure to avoid repeating works? Preserving useful intermediate results while repairing the failed part could be a natural way to go
-
The results seem to be positive. ALFWorld success increases from 23.88% to 43.28%, and GAIA accuracy increases from 17.58% to 21.82% over the strongest evaluated baselines. The paper also reports token usage and call counts, which is helpful
Critical weaknesses
-
Some ablations do not clearly ablate the components they are meant to isolate. In particular, Appendix A.8 describes removing Desire filtering as removing answer-format constraints. The resulting drop could reflect less complete task instructions, rather than the value of filtering candidate subgoals.
-
It is unclear what the BDI formulation adds more than retaining intermediate results, verifying outputs, and applying local repairs. A simpler controller with the same tools, verifier, available information, and execution budget would help to understand the advantage of the proposed method.
Questions for authors
-
What verifier criteria and repair budgets are used to determine that plan adaptation is insufficient? Algorithm 1 describes an escalation rule, but how do you assess whether it abandons an intention too early or persists for too long? A sensitivity analysis of these choices could be useful.
-
A bit more clarification on implemented definitions of Desire and Intention would be helpful.
-
How does the method compare with a simple local-repair controller using the same verifier, tools, retained results, and execution budget?
2 comments
Brief Summary of Our Response and Gentle Reminder
Thank you again for your thoughtful and constructive feedback, and for recognizing the practical importance of determining how much an agent workflow should change after a local failure.
Regarding your concerns:
Clarification of the Desire ablation (W1): We clarified that the original Desire ablation removes environment descriptions rather than format constraints, and provided additional ablations showing that removing either the task goal or instructions independently reduces ALFWorld success by over 20 percentage points.
Comparison with a simple local-repair controller (W2, Q3): We conducted a controlled comparison demonstrating that our commitment rule improves ALFWorld success by 9.70pp over local repair without intention reconsideration, while also examining the limitations of frequent reconsideration.
Reconsideration criterion and sensitivity (Q1): We clarified that intention reconsideration is triggered by repair-budget exhaustion and conducted sensitivity analyses showing that performance depends on the repair budget but remains relatively stable across stall-detection windows (≤1.49pp variation).
Definitions of Desire and Intention (Q2): We clarified the definitions of Desire, Intention, and Plan across GAIA/QA and ALFWorld, distinguishing re-planning, which preserves committed objectives, from intention switching, which revises them after repair exhaustion.
As today is the final day of the rebuttal period, we hope that these additional experiments and clarifications address your concerns, and would sincerely appreciate it if you could reconsider your assessment of our manuscript in light of these updates.
Thank you for recognizing the
practical importanceof deciding how much of an agent workflow should change after a local failure. Below, we address your questions one by one.W1. Clarifying the Desire Ablation
The Desire ablation is not solely attributable to missing format instructions. Removing either the task goal or instructions independently reduces ALFWorld success by over 20pp.
The w/o Desire setting in Table 4 removes the environment description (rooms and receptacles) used to guide sub-goal selection, rather than answer-format constraints. We will clarify these settings and add the new ablations in Appendix A.8.
W2, Q3. Comparison with a Simple Local-Repair Controller
Our commitment rule improves ALFWorld success by 9.70pp over local repair under identical conditions. Unlike a controller that repeatedly repairs blocked sub-goals, our method reconsiders them after exhausting the repair ladder. Frequent reconsideration also reduces success, although the Table 4 comparison additionally changes the repair policy.
Q1. Reconsideration Criterion and Sensitivity
Intention reconsideration is triggered by repair-budget exhaustion, with performance sensitive to the budget but robust to the stall-detection window. GAIA allows 5 repairs per node and 20 per run, while ALFWorld allows 8 per sub-goal. The verifier accepts steps with completeness ≥ 0.8.
A smaller budget reduces success by 5.97pp, while a larger budget improves it by 2.24pp. LLM-based feasibility judgment underperforms the default by 2.24pp.
Stall-window changes affect success by at most 1.49pp. We will report these results in Table 5 of the revision.
Q2. Definitions of Desire and Intention
Desire defines task objectives and constraints, Intention represents committed sub-goals, and Plan specifies their execution. Re-planning preserves committed objectives, whereas intention switching revises them after repair exhaustion. We will align Eq. (1), Section 3.1.2, and Appendix A.1 accordingly.