- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper systematically investigates when retaining context from failed attempts helps or hurts LLM recovery across tool-use and coding benchmarks. The authors find that effect is highly nuanced: while hiding failed code often improves retry success and recovery training tends to overfit specific failure distributions, retaining conversation history after repository rollbacks boosts recovery from 11.8% to 35.8% on MettleBench. Overall, the work provides a controlled empirical characterization of how LLM recovery depends on retained information, failure provenance, and restart boundaries.
Critical strengths
None
Critical weaknesses
- The writing logic and organizational structure of this paper are very poor, making it difficult to follow.
- The current paper even has incomplete experimental results (for the ALFWorld benchmark), and does not meet the acceptance level of ICLR.
Questions for authors
See weeknesses.
3 comments
Thank you for your explanation; I stand by my score.
Dear Reviewer,
We hope you are doing well. Could we check for the final time on which part of the paper was hard to follow and what part of the alfworld results are missing? This would greatly help us in addressing those. Thank you!
Authors
Thank you for your review.
ALFWorld results. We would like to clarify the statement that our ALFWorld experiments are incomplete.
In the original submission, we completed all nine experimental settings for both Qwen3-8B and Gemma-4-12B-it across 60 ALFWorld states, totaling 8,640 recovery runs. Appendix H.3 (Table 33) reports the success rates for every setting and the effects of keeping versus removing the earlier memory.
Adding one correct object location to the old memory increased its benefit for recovery:
Both findings are statistically supported.
We also tested whether providing correct locations separately reduces the usefulness of the old memory. The estimates suggested a reduction, but the results were statistically inconclusive. We explicitly reported this limitation.
To further strengthen the evidence, we repeated the experiment on 60 new ALFWorld states across 39 additional floor plans. The positive effect of adding correct information to the old memory was reproduced for both models:
The effect of supplying correct information separately remained inconclusive in the new experiment.
The original ALFWorld experiments were completed and reported in full. Some results were inconclusive, but no experimental settings or results were missing.
Organization. In response to all reviewer feedback, we have improved the paper's readability by revising the introduction, related work, contribution statements, and experimental design. The revised manuscript now more explicitly explains the purpose of each experiment, how the four studies connect, and how our findings extend prior work.
We would appreciate more specific feedback on the organizational concern, as the review does not identify which sections or arguments were difficult to follow.
Could the reviewer clarify which ALFWorld experiments or results they consider incomplete, which sections were difficult to follow, and whether there are specific methodological concerns underlying the soundness rating?
We would be happy to address these points directly.
Thank you for your consideration.