- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents an empirical study of how tool descriptions and execution infrastructure affect agents working with large tool libraries. The evaluation ranges from tool routing with catalogs of up to 1,000 tools to atomic execution and multi-step workflows. In the main experiment, the authors compare two settings: bare+flat exposes a flat tool catalog with short description, and expert uses detailed process templtates and exposes at most 15 tools at each stage, on 30 data science tasks. The results are less uniform elsewhere: richer descriptions improve routing for some models (e.g., Gemini 3.1 Flash Lite and DeepSeek V4 Flash) but substantially hurt others (e.g., Claude 4.5 Haiku), while file-processing tasks remain challenging even with Python execution.
Critical strengths
The results reported in this paper expose meaningful heterogeneity across models. Expert descriptions help Gemini 3.1 Flash Lite and DeepSeek V4 Flash but substantially hurt GPT-4o-mini and Claude 4.5 Haiku on tool routing task in Table 1. Similarly, sandbox execution does not consistently improve GPT-OSS 20B in Tables 3–4. These observations challenge a universal prescription that more detailed descriptions or code execution always improve reliability. A carefully controlled analysis of these differences could constitute a useful empirical contribution.
Critical weaknesses
- The main comparison cannot isolate the mechanisms responsible for improvements as it changes both the descriptions and the tool exposure.
Table 8 compares Expert + partitioned with Bare + flat, so it is unclear whether the higher completion rates come from the expert templates, the smaller tool subsets, or their combination.
The paper also does not explain sufficiently how the stage-specific subsets are constructed. If these subsets are manually selected for each task, they may provide information that the flat setting requires the agent to infer. Similarly, AST validation introduces feedback and retries, but there is no comparison with ordinary execution-error feedback under the same retry budget. These comparisons establish that the combined setup performs better, but do not identify which proposed components are responsible.
- The evaluation is insufficiently specified, and several quantitative claims conflict with the reported tables.
The execution benchmarks contain only 12 clean queries, 18 noisy queries, and 30 multi-step tasks. The manuscript does not provide enough information about task construction, expected outputs or workflow depth to interpret these results.
Reporting inconsistencies further reduce confidence: Section 4.2 claims sandbox accuracy of 91.7%–100.0% across models, whereas Table 3 contains 66.7% (DeepSeek V4 pro, N=60) and 83.3% (GPT-OSS 20B, N=500); the reported 11.1%–33.3% range for file-based execution excludes the 5.6% (GPT-OSS 20b, N=1000) result in Table 5. These are consequential because the paper’s contribution is primarily empirical, and the strength and scope of its conclusions depend on reliable measurement and aggregation.
Questions for authors
-
Do you have results for the full comparison of
Bare/Expertdescriptions ×Flat/Partitionedcatalogs, with execution, validation, and retry budgets held fixed? How are stage-specific tool subsets constructed, what is the flat catalog size, and does constructing the subsets require task-specific knowledge of the correct solution? -
What are the 30 multi-step tasks, and how is successful completion judged?
-
What exactly is the L5 threat model? Specifically, where are malicious instructions inserted, what behavior do they attempt to induce, and how is attack success measured?