acceptodds
Under review as a conference paper at ICLR 2027

-Bench: Pushing Agents toward a Moving Capability Frontier

Abstract

In this paper, we introduce -Bench, an executable agent benchmark that models the structural evolution of large tool ecosystems. In -Bench, independently evolving services continually reshape how tasks can be accomplished: familiar tools may disappear or be replaced, while more efficient alternatives emerge amid overlapping functionality and noisy documentation. Against this moving capability frontier, we evaluate whether agents can proactively discover efficient tool-use paths and promptly revise their accumulated procedural knowledge, bringing together demands that existing benchmarks fail to examine. -Bench comprises 1,588 validated multi-turn tasks across 42 domains and six successive ecosystem states generated through capability-preserving transformations. Our evaluation shows that even the strongest evaluated model or harness achieves only 42.3% task success, while initial gains from existing skill-learning methods can not consistently translate into better adaptation as the ecosystem evolves. These findings motivate Steward, an ecosystem manager that jointly manages tools and procedural knowledge through incremental skill maintenance and dynamic provisioning during task execution. Experiments show that Steward sustains higher task success across ecosystem changes with lower skill-maintenance overhead, while improving both recovery from disrupted workflows and adoption of more efficient alternatives.

Primary area:
datasets and benchmarks
Keywords:
llm tool-use, skill evolution, llm agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fKptOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Δτ-Bench evaluates tool-using agents across six successive versions of an evolving tool ecosystem, with 1,588 tasks spanning 42 domains. Capability-preserving interface transformations create alternative implementations, and reference paths are optimized within the resulting realization space. STEWARD incrementally maintains tool-linked skills and provides tools and procedural guidance during execution. The evaluation measures task success, tool retrieval, path efficiency, and performance on tasks whose reference paths become more or less efficient.

Critical strengths
  1. The benchmark provides a substantial executable testbed. Evaluating the same tasks across validated interface changes enables systematic study of obsolete workflows and newly efficient alternatives (§3). The paper explicitly limits reference-path optimality to the constructed realization space, which is a reasonable scope for the benchmark.

  2. The system evaluation includes relevant controlled comparisons. Incremental maintenance is compared with frozen libraries and full reconstruction, all starting from the same initial library. Dynamic tool and skill provisioning are also examined separately (§6, Table 2, Figure 7). These comparisons provide evidence beyond aggregate performance differences between agent systems.

Critical weaknesses
  1. The success criterion does not explicitly check for prohibited side effects. Equation 4 accepts any set of final database changes that contains the gold changes. Additional changes are not necessarily harmful, but this condition alone does not enforce requirements such as preserving other records or avoiding unauthorized actions. Some restrictions may already be enforced by the environment; the paper should clarify which constraints are enforced there and which are checked by the evaluator. Without this distinction, it remains unclear whether reported successes satisfy the full task requirements. This uncertainty affects SR and the success-weighted efficiency and adaptation scores, rather than only a secondary diagnostic.

  2. IDR does not directly establish the discovery of newly efficient workflows. Equations 7–8 identify tasks through changes in reference-path weight, then score current-version success and efficiency without comparing the agent’s own behavior across versions. For example, an agent that successfully retains the same 10-call workflow still receives a per-task score of 0.5 when the reference path improves from 10 to 5 calls. Partial credit is reasonable, but a higher IDR alone does not demonstrate adoption of a newly efficient realization. The improvement-discovery interpretation in §6 therefore needs a behavioral breakdown or narrower wording. The concern is the interpretation of the metric, not the validity of partial-credit scoring itself.

Questions for authors
  1. Success criteria and existing-trace audit: Which user constraints and preservation requirements are enforced beyond Eq. 4? Could the authors examine a small sample of recorded successes, report the sample size, and distinguish harmless additional changes from task-violating side effects? If violations are found, please re-score the affected saved cases. This does not require exact equality with the entire gold database or rerunning every agent.

  2. Cross-version behavior: On the existing subset of tasks whose reference paths improve, could the authors report how often successful agents adopt a newly efficient realization, retain an older valid workflow, or follow another valid route? Paired comparisons of the agents’ own path weights, together with the subset sizes, would help distinguish stronger current-version performance from actual workflow adaptation. No new global-optimality oracle or model training is needed.

Reviewer_dSsTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents delta-tau bench, which simulate the changes of tool versions, to evaluate whether LLMs can handle these changes. This paper claims that current skill induction methods cannot handle this problem. This paper also proposes Steward, which updates the skill library when tool changes, and provides tools during task execution.

Critical strengths

It is an interesting setting for me to evaluate the LLMs in a context where tools may change.

Critical weaknesses
  • The benchmark details and examples are unclear. Since this benchmark is mainly constructed by simulation, this paper should present concrete examples of tasks, tools, and tool changes between versions, and explain how well they reflect real-world tool ecosystems. This is important for a benchmark paper, because it directly determines the validity of the benchmark and the measurement.
  • More detailed analysis is needed. I would like to see a breakdown that shows how "experience lag", the problem raised in the paper, actually affects performance. For example, what are the root causes of failures, and how are the failure modes distributed across tool versions? (Note that this is just an example; this paper should provide more analysis to strengthen the contribution of the benchmark). Moreover, there is no analysis across different harnesses, which often employ different skill management methods.
  • Steward is not sufficiently described and evaluated. For example, it is unclear whether Steward generalizes to other tool-use benchmarks, since it is only evaluated on he delta-tau bench.
  • I would suggest that the paper clarify its positioning. Is it a benchmark paper or a method paper? Each type requires its own in-depth analysis. Mixing the two prevents the paper from providing more in-depth insights and findings, which I think the paper had the potential to provide.
Questions for authors

See weaknesses.

Reviewer_ZsncOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether agents can discover efficient tool-use paths and update accumulated procedural knowledge as large tool ecosystems evolve. It introduces Δτ-Bench, an executable benchmark that combines stateful tasks, capability-preserving transformations of tool interfaces, and cross-version reference paths to evaluate task success, tool retrieval, execution efficiency, and adaptation to ecosystem changes. It further proposes STEWARD, which tracks skill–tool dependencies, incrementally maintains skills, and dynamically provisions tools together with procedural guidance. Experiments reveal limitations in existing agents and show promising advantages of STEWARD.

Critical strengths

Well-motivated problem formulation. The paper distinguishes between recovering from disrupted workflows and discovering more efficient alternatives while existing workflows remain functional, and jointly evaluates tool discovery, stateful execution, and skill adaptation in a continuously evolving setting. This choice determines what the benchmark can measure: whether an agent merely keeps working workflows functional or also finds more efficient ones as the ecosystem changes.

Systematic benchmark construction and comprehensive experimental analysis. The paper builds tasks using executable MCP services, stateful databases, and multi-turn user simulation; it generates evolving ecosystems through interface modification, splitting, and merging, and validates capability preservation and reference paths through execution. It compares multiple foundation models and four skill-learning methods, uses Oracle Tools/Path diagnostics, skill-maintenance ablations, and dynamic tool/skill retrieval ablations to investigate performance bottlenecks, and separately evaluates improvement discovery and recovery from degraded workflows. Capability preservation and reference paths are validated by execution, and the diagnostics and ablations localize bottlenecks rather than only reporting outcomes.

Critical weaknesses

The search space for reference "optimal paths" is constrained by the original trajectory. Equation (2) in Section 3.2 starts from a single validated, ordered base-tool trajectory and only considers substitutions covering contiguous source blocks. Thus, optimality holds within this constructed space rather than across all valid workflows that achieve the target state; alternative call orders, sharing across blocks, or independently constructed solutions may be excluded. Moreover, Equation (5) evaluates tool retrieval against a single reference path, and since the same reference paths serve both as the efficiency yardstick and as the ground truth for tool retrieval, valid alternatives may be penalized as failures.

The task-success criterion may overlook unintended state changes. Equation (4) in Section 3.3 requires the final database changes to contain the reference changes (Δgold ⊆ Δagent), but does not explicitly exclude unintended insertions, updates, or deletions, or verify all conversational constraints. An agent could therefore satisfy the required changes while modifying unrelated records and still be counted as successful. This directly affects SR, SPE, and the adaptation metrics — the primary evidence for the paper's conclusions.

Questions for authors

The paper mainly evaluates foundation models and automated skill-maintenance methods, without directly testing other adaptation mechanisms for evolving tool ecosystems. ContDa [1] addresses toolset changes through continual documentation adaptation, new-tool discovery, and management of overlapping capabilities, making it closely related to STEWARD; AutoTool [2] studies dynamic selection of unseen tools. Although their training or interface settings differ, can you provide comparisons on compatible tool-selection and cross-version adaptation tasks, or a clear explanation of why fair adaptation is infeasible?

Although Section 6 uses the same task and user models, STEWARD additionally employs dynamic tool retrieval, manager investigations, a skill graph, and tool-schema inspection, whereas the baselines primarily retain their original skill-maintenance mechanisms. Despite the ablations in Figure 7 and Table 2, differences in resource access and reasoning/retrieval budgets may still contribute to the gains. Can you provide comparisons with matched tool-access interfaces and comparable end-to-end computational budgets, along with complete costs for manager calls, retrieval, execution, and maintenance?

Are any additional constraint checks applied beyond the containment test in Equation (4)? Please clarify this, and if such checks are absent, quantify the extra or harmful state changes produced by evaluated agents and examine how stricter state-equivalence or user-constraint checks affect SR, SPE, and the adaptation metrics.

Given that reference paths are derived from a single ordered base trajectory with contiguous-block substitutions, can you run small-scale exhaustive-search or multiple-valid-trajectory comparisons to assess reference-path coverage, the resulting efficiency gaps, and the impact of alternative paths on the retrieval metrics?

Section 3.2 assigns each tool call an LLM-estimated cost score of 1–10 based on its definition and implementation, rather than measured monetary cost or latency, and Cost-SPE/IDR/DAR depend on this proxy. Can you emphasize this limitation in the main text and report the sensitivity of the results to changes in the cost scores?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.