- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the tool-call failures of agents. It aims to make the boundary of observability predictable, distinguishing recoverble corruptions from others. It proposes LOCO for casual failure attribution, StructAudit for compiling admission predicates, and FreshVERIFY for obtaining extra evidence when local verification fails. It conducts experiments to show the effectiveness of StructAudit.
Critical strengths
- Tackles an important problem of tool-call failures in agents
Critical weaknesses
-
The writing is hard to follow. It seems that this paper first conducts an empirical study, and designs a tool based on it. But it is unclear how LOCO, StructAudit, and FreshVERIFY work together. It is also unclear how FreshVERIFY works. I would recommend the authors to deliver the core idea in a precise, direct, and concise manner. I would also suggest the authors to organize the paper structure in a similar way as other papers in this venue.
-
The evaluation does not show the generalizability of the proposed methods. The LLMs are limited to QWen models. The benchmarks are focused on airline and telecom settings, and have very limited numbers of tasks. The tool-invocation information is also missing. Therefore, it is hard to validate the capability of StructAudit with the given evaluation results. In addition, StructAudit has a lower recovery rate than id-echo alone in some cases.
Questions for authors
-
What is the high-level idea of the proposed solutions?
-
What is the design of FreshVERIFY?