acceptodds
Under review as a conference paper at ICLR 2027

AT-AT: Automatic Tool Optimization from Agent Traces

Abstract

Efforts to improve LLM-based agents typically focus on modifying the agent itself, yet success also depends on the environment in which an agent acts. Tools mediate this interaction: a tool's description shapes how an agent calls it, and its implementation determines the results and errors the agent receives afterwards. Many tools, such as APIs and MCP servers, are shared by many agents and maintained by providers who cannot change those agents. We introduce AT-AT (Automatic Tool Optimization from Agent Traces), an agentic tool optimizer that improves a shared toolset offline from historical agent executions while holding the agents fixed. A lightweight coding agent reads agent trajectories and their evaluation results, and selectively edits tool descriptions and, when permitted, implementations. On -bench-verified, AT-AT achieves the highest task completion rates in the Airline and Telecom domains at lower cost than strong prompt- and tool-optimizer baselines, and its Telecom gain is statistically significant under paired permutation tests. Optimizing the tools within a coding agent harness also improves held-out performance on TerminalBench-2 and TBLite. On three enterprise MCP servers in production, applying AT-AT to production logs increased held-out task passes on two servers and reduced tool errors on the third. By turning accumulated execution traces into updates to shared tool environments, AT-AT provides a path toward continual improvement of agent systems through environment-side learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tool optimization, self-improvement, harness optimization, environment learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eH1VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether agents can perform better simply by improving the tools they use. AT-AT uses a coding agent to read past execution logs and task scores, then edit tool descriptions and code. The target agent stays unchanged, and no new task runs are needed during optimization. The authors test the method on customer-service tasks, coding benchmarks, and three enterprise MCP servers. Results show a significant gain on Telecom, smaller gains on coding tasks, and mixed results on the enterprise tools. Examples show how clearer tool outputs can help agents make better decisions.

Critical strengths
  1. The paper addresses a practical problem. Tool providers often cannot change the agents calling their APIs, but they can improve the tools themselves. Using existing logs makes this a useful and relatively low-cost way to improve agent performance.

  2. The experiments cover customer service, coding, and enterprise MCP tools. The repeated evaluations help assess whether a particular tool update improves performance. I also appreciate that the authors report mixed results and include concrete examples of how tool changes affect agent behavior.

Critical weaknesses
  1. The contribution beyond existing coding-agent optimization is limited. AT-AT mainly asks an off-the-shelf coding agent to inspect logs and edit tools, a workflow already explored by Meta-Harness and Anthropic's tool-design work. The provider-side, offline setting is useful, but the restriction alone is a limited methodological contribution. Adapting DRAFT, ToolObserver, and GEPA to an offline setting is reasonable. However, their editing scope and optimization budgets are not clearly matched to AT-AT. A coding-agent baseline with the same trace access and tool-editing permissions would make the contribution easier to assess.

  2. The experiments do not clearly establish what drives the gains or how repeatable they are. The main comparisons use different editing scopes, and the description-only ablation also changes trace presentation. Full AT-AT adds a lookup tool and repairs implementation logic, so its advantage does not isolate the proposed feedback-shaping mechanism. Each condition also produces only one optimized toolset; repeated evaluations cannot establish that the optimization procedure reliably finds good updates. The broader evidence is modest: the multi-model results are not significant, the coding gains are small, and the enterprise tests contain only 5–11 held-out tasks per server. These results demonstrate some useful edits, but do not yet establish a reliable optimizer or a substantial new understanding of tool optimization.

Questions for authors
  1. Beyond restricting the editable files and using historical logs, what does AT-AT add to the existing coding-agent optimization workflow? Which result demonstrates a finding not already supported by prior tool-design work?

  2. What existing evidence separates feedback shaping from adding capabilities or fixing bugs? In particular, how much of the Telecom gain remains without the new lookup tool?

  3. Is there evidence from independent optimizer runs, or a comparison under matched trace access, editing permissions, and optimization budgets? The current repeated trials evaluate one candidate per condition.

  4. What evidence supports the shared-tool generalization argument? Table 2 evaluates the same five models whose traces were used for optimization, and none of its reward changes is significant.

Reviewer_8HkUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a new framework that improves agent performance by optimizing the tool environment rather than the agent itself. Specifically, it uses a coding agent to identify recurring tool-related failures and automatically edit tool descriptions and, when permitted, tool implementations. The method is evaluated on customer-service benchmarks, coding-agent benchmarks, and production MCP servers. Results suggest that tool-side optimization can improve task success without modifying the underlying agent, highlighting environment-side learning as a complementary path to improving agent systems.

Critical strengths
  • Introduces a novel and practically important perspective by treating tool optimization as a first-class mechanism for improving agent performance while keeping agents fixed. Evaluates the approach across multiple settings, including τ²-Bench-Verified, TerminalBench-2, TBLite, and production MCP servers, demonstrating broad applicability.
  • Employs repeated evaluations and paired permutation testing to account for the substantial stochasticity of agent evaluations, resulting in a more rigorous assessment than many agent-optimization papers.
  • Demonstrates practical relevance by showing improvements on real production MCP servers rather than limiting evaluation to academic benchmarks. It also provides qualitative analyses illustrating how improved tool feedback can change downstream agent behavior, offering valuable insights into environment-side learning and feedback shaping.
Critical weaknesses
  • The empirical gains are relatively modest and statistically significant primarily in the Telecom domain; improvements are inconsistent across domains, with Retail showing no improvement and sometimes regression. Can authors demonstrate this approach on τ3 Banking benchmark, which is a better agentic domain compared to others?
  • The optimization process itself is stochastic, yet most experiments evaluate only a single optimized toolset per method. The paper explicitly acknowledges that optimizer variance is not measured, limiting confidence in reproducibility.
  • The production experiments use very small held-out datasets (e.g., 5–11 tasks per server), making it difficult to assess statistical significance or generalizability of reported improvements.
  • AT-AT requires access to editable tool implementations and historical trajectories, assumptions that may not hold for many third-party APIs and commercial tools.
Questions for authors
  • The paper discusses evaluation noise extensively, but optimizer noise remains unmeasured. How consistent are the improvements when AT-AT is run multiple times with different seeds or optimization trajectories?
  • The strongest statistical gains occur primarily in Telecom. What characteristics of a domain make it particularly amenable to tool optimization, and how can practitioners estimate expected gains beforehand?
  • To what extent do the observed improvements arise from implementation-level edits versus improved descriptions and documentation? Can this effect be quantified more rigorously?
  • How robust are the production MCP server results given the relatively small test sets and reliance on LLM-based evaluation metrics?
  • Would the conclusions remain the same if a different evaluator model were used for judging task success and content quality?
  • How does AT-AT behave in environments where tool implementations cannot be modified and only tool schemas or descriptions are available for editing?
  • Have the authors measured whether tool edits discovered for one agent population continue to benefit significantly different agent models and future generations of agents?
Reviewer_hKbXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a framework named AT-AT that automatically optimizes agent tools based on historical execution trajectories. AT-AT uses a Pi coding agent to inspect task trajectories and evaluation results, identify tool-related failures, and modify tool descriptions and implementations while keeping the executor agent fixed. Unlike existing methods that mainly optimize tool descriptions, AT-AT can also modify tool behavior and the feedback returned to agents.

The authors evaluate their method on τ²-bench-verified, TerminalBench-2, TBLite, and three production MCP servers. On τ²-bench-verified, AT-AT achieves the highest task success rates among the compared methods in Airline and Telecom domains. The experiments on coding benchmarks and production MCP servers also show some improvements in several settings.

Critical strengths
  1. The paper is well motivated. The setting is reasonable where tool providers can modify shared tools, such as MCP servers, but cannot directly control the agents using them. Unlike methods that only optimize tool descriptions, AT-AT can modify both tool descriptions and executable tool codes. This design is useful when agent failures arise from missing information or insufficient tool capabilities rather than unclear instructions.

  2. The main cover several different settings and provides useful analysis. The paper evaluates AT-AT on 3 customer service tasks (τ²-bench), agentic coding tasks and production MCP servers from anonymized company.

Critical weaknesses

Current experimental results provide limited evidence that AT-AT can reliably improve shared environment tools across different agents and tasks.

  1. The experimental settings on three types of tasks are different, while the experiments are not consistent with the proposed method. AT-AT's primary setting (as line 135 shows) is updating a shared environmental toolset. And the paper formulates tool optimization as maximizing expected reward over a population of agents (Eq. 1). However, most experiments optimize and evaluate tools for a single executor (Table 1, 3, 4). The cross-LLM experiment in Table 2 is the only direct evaluation across multiple executor models, and it shows no statistically significant improvement in any of the three domains. Moreover, the same five models are used to generate optimization traces and evaluate the updated tools. This makes it difficult to assess whether AT-AT achieves its stated goal of improving a shared tool environment for a heterogeneous population of agents.

  2. A key distinction of AT-AT is that it can modify both tool descriptions and implementations. However, the evaluation of this capability is mainly concentrated in τ²-bench. The TerminalBench experiments optimize tools within an agent harness rather than external tools shared by different agents, while the production MCP experiments only allow description edits.

Questions for authors
  1. Table 2 only reports performance averaged across executor models. Could the authors provide per-model results to show how much each executor model benefits from the optimized tools?

  2. Given the relatively modest improvements on TerminalBench-2 and TBLite, how would AT-AT compare with allocating a similar budget to inference-time scaling, such as parallel rollouts? Since the tool optimization cost can be amortized across requests, a comparison under different numbers of future requests would help clarify when offline tool optimization is more cost-effective than additional inference-time computation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.