- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks whether agents can perform better simply by improving the tools they use. AT-AT uses a coding agent to read past execution logs and task scores, then edit tool descriptions and code. The target agent stays unchanged, and no new task runs are needed during optimization. The authors test the method on customer-service tasks, coding benchmarks, and three enterprise MCP servers. Results show a significant gain on Telecom, smaller gains on coding tasks, and mixed results on the enterprise tools. Examples show how clearer tool outputs can help agents make better decisions.
Critical strengths
-
The paper addresses a practical problem. Tool providers often cannot change the agents calling their APIs, but they can improve the tools themselves. Using existing logs makes this a useful and relatively low-cost way to improve agent performance.
-
The experiments cover customer service, coding, and enterprise MCP tools. The repeated evaluations help assess whether a particular tool update improves performance. I also appreciate that the authors report mixed results and include concrete examples of how tool changes affect agent behavior.
Critical weaknesses
-
The contribution beyond existing coding-agent optimization is limited. AT-AT mainly asks an off-the-shelf coding agent to inspect logs and edit tools, a workflow already explored by Meta-Harness and Anthropic's tool-design work. The provider-side, offline setting is useful, but the restriction alone is a limited methodological contribution. Adapting DRAFT, ToolObserver, and GEPA to an offline setting is reasonable. However, their editing scope and optimization budgets are not clearly matched to AT-AT. A coding-agent baseline with the same trace access and tool-editing permissions would make the contribution easier to assess.
-
The experiments do not clearly establish what drives the gains or how repeatable they are. The main comparisons use different editing scopes, and the description-only ablation also changes trace presentation. Full AT-AT adds a lookup tool and repairs implementation logic, so its advantage does not isolate the proposed feedback-shaping mechanism. Each condition also produces only one optimized toolset; repeated evaluations cannot establish that the optimization procedure reliably finds good updates. The broader evidence is modest: the multi-model results are not significant, the coding gains are small, and the enterprise tests contain only 5–11 held-out tasks per server. These results demonstrate some useful edits, but do not yet establish a reliable optimizer or a substantial new understanding of tool optimization.
Questions for authors
-
Beyond restricting the editable files and using historical logs, what does AT-AT add to the existing coding-agent optimization workflow? Which result demonstrates a finding not already supported by prior tool-design work?
-
What existing evidence separates feedback shaping from adding capabilities or fixing bugs? In particular, how much of the Telecom gain remains without the new lookup tool?
-
Is there evidence from independent optimizer runs, or a comparison under matched trace access, editing permissions, and optimization budgets? The current repeated trials evaluate one candidate per condition.
-
What evidence supports the shared-tool generalization argument? Table 2 evaluates the same five models whose traces were used for optimization, and none of its reward changes is significant.