- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SETA, a failure-aware memory framework for tool-using agents. It converts externally verified false-success trajectories into explicit-failure memories through deterministic rewriting and reuses them during single-attempt inference. Experiments on two AgentArch workflows show improvements in final-outcome correctness under Think mode and examine the effect of memory placement in long-horizon tasks.
Critical strengths
- Practical Motivation. The paper addresses false-success cases in tool-using agents, where agents incorrectly report successful task completion. The ablation studies suggest that task-specific failure diagnostics can improve final-outcome correctness in the evaluated settings.
- Memory Placement Analysis. The paper examines different memory injection strategies for long-horizon workflows. The results show that repeated memory injection can be more effective than single injection in the evaluated setting.
Critical weaknesses
- Limited Novelty and Generalization. The methodological distinction from existing failure-aware memory approaches is not sufficiently established. Moreover, the main evaluation allows retrieving previously verified correct outcomes for the same tasks, making it difficult to distinguish the benefits of failure-memory construction from direct answer reuse. The evaluation is also limited to two workflows, with weak cross-family and cross-benchmark results. (Presentation issues: The abstract is divided into two paragraphs, which does not comply with the ICLR formatting requirements, and Fig. 1 extends beyond the prescribed text width. The authors are encouraged to further improve the overall presentation of the manuscript.)
- Concerns about Evaluation Validity and Task Reliability. The reported improvements mainly concern final-outcome correctness rather than actual task completion. Strict task success decreases substantially on RTO and remains very low on CRR, raising concerns about practical effectiveness. Additionally, the deterministic rewrite may introduce inconsistencies between declared outcomes and preserved tool actions (Appendix G.10). The K=4 majority-voting baseline also aggregates externally judged correctness labels rather than model-generated answers, making its comparison with standard self-consistency questionable.
Questions for authors
- How does SETA compare against a simpler baseline that directly retrieves previously verified correct outcomes for the same tasks?
- How do the authors explain the decline in strict task success, and how is consistency between rewritten memories and actual tool execution ensured?
- Could the authors clarify why the K=4 majority-voting baseline aggregates correctness labels rather than model-generated answers?
Thank you for clarifying questions. However, I still have concerns about using four-attempt voting as the reference point for the resource effieciency claim. Response doesn't explain why four-attempt voting can be a fair comparison with SETA. As author said, SETA itself uses more tokens and calls compare with one attempt. Thus, I will take the clarifications and additional results into account in my final assessment.