acceptodds
Withdrawn from ICLR 2027

From Tool Selection to Autonomous Execution: Process Templates, Sandboxed Code, and Modular Orchestration for Reliable LLM Agents at Scale

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model agents must route and execute tools from large, overlapping libraries. Prior work reported routing degradation under flat catalogs; we study both routing and execution under scale, noise, and multi-step control. Routing: Expert-disambiguating descriptions improve selection for some models and hurt others. The mean nearest-neighbor cosine similarity of target skills increases with catalog size N (Bare peak ≈0.51 at N=500 with MiniLM embeddings); detailed expert text remains more separable than short bare text. Live selection stays near-perfect for strong models on clean queries, while weaker models degrade at large N. Execution: On atomic tasks, bare prompts often fail structured output; expert process templates (SOPs) largely remove schema failures, and sandboxed Python execution removes arithmetic failures, producing large end-to-end (E2E) gains versus prompt-only math. On production file-based tasks, selection can remain high while execution E2E stays modest without sandbox support. Robustness: Under a noise gradient (L0–L5), mild noise is tolerated; adversarial injection collapses E2E for several models, and an AST script-integrity guardrail recovers a substantial fraction of L5 failures (e.g., ∼0%→∼47% on DeepSeek V4 Flash and GPT-4o-mini). A 2×2 query matrix (phrasing novelty × semantic ambiguity) shows high routing accuracy for strong models, with model-dependent weaknesses in harder cells. Orchestration: In a 30-task live multi-step Data Scientist stress test, partitioned Expert stages (N≤15 tools per stage) achieve 93%–100% E2E versus 47%–77% for a flat Bare catalog, with higher loopbacks under Bare. Volume projections for adjacent roles are calibrated simulations, not live API results. Together, these results support a practical recipe—expert templates, sandbox execution, stage-wise tool partitions, and light guardrails—without claiming a single universal cosine threshold or unlimited scaling as a theorem.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, tool selection, agent routing, phase transition, description engineering, schema collapse

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_KQKzOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents an empirical study of how tool descriptions and execution infrastructure affect agents working with large tool libraries. The evaluation ranges from tool routing with catalogs of up to 1,000 tools to atomic execution and multi-step workflows. In the main experiment, the authors compare two settings: bare+flat exposes a flat tool catalog with short description, and expert uses detailed process templtates and exposes at most 15 tools at each stage, on 30 data science tasks. The results are less uniform elsewhere: richer descriptions improve routing for some models (e.g., Gemini 3.1 Flash Lite and DeepSeek V4 Flash) but substantially hurt others (e.g., Claude 4.5 Haiku), while file-processing tasks remain challenging even with Python execution.

Critical strengths

The results reported in this paper expose meaningful heterogeneity across models. Expert descriptions help Gemini 3.1 Flash Lite and DeepSeek V4 Flash but substantially hurt GPT-4o-mini and Claude 4.5 Haiku on tool routing task in Table 1. Similarly, sandbox execution does not consistently improve GPT-OSS 20B in Tables 3–4. These observations challenge a universal prescription that more detailed descriptions or code execution always improve reliability. A carefully controlled analysis of these differences could constitute a useful empirical contribution.

Critical weaknesses
  1. The main comparison cannot isolate the mechanisms responsible for improvements as it changes both the descriptions and the tool exposure.

Table 8 compares Expert + partitioned with Bare + flat, so it is unclear whether the higher completion rates come from the expert templates, the smaller tool subsets, or their combination.

The paper also does not explain sufficiently how the stage-specific subsets are constructed. If these subsets are manually selected for each task, they may provide information that the flat setting requires the agent to infer. Similarly, AST validation introduces feedback and retries, but there is no comparison with ordinary execution-error feedback under the same retry budget. These comparisons establish that the combined setup performs better, but do not identify which proposed components are responsible.

  1. The evaluation is insufficiently specified, and several quantitative claims conflict with the reported tables.

The execution benchmarks contain only 12 clean queries, 18 noisy queries, and 30 multi-step tasks. The manuscript does not provide enough information about task construction, expected outputs or workflow depth to interpret these results.

Reporting inconsistencies further reduce confidence: Section 4.2 claims sandbox accuracy of 91.7%–100.0% across models, whereas Table 3 contains 66.7% (DeepSeek V4 pro, N=60) and 83.3% (GPT-OSS 20B, N=500); the reported 11.1%–33.3% range for file-based execution excludes the 5.6% (GPT-OSS 20b, N=1000) result in Table 5. These are consequential because the paper’s contribution is primarily empirical, and the strength and scope of its conclusions depend on reliable measurement and aggregation.

Questions for authors
  1. Do you have results for the full comparison of Bare/Expert descriptions × Flat/Partitioned catalogs, with execution, validation, and retry budgets held fixed? How are stage-specific tool subsets constructed, what is the flat catalog size, and does constructing the subsets require task-specific knowledge of the correct solution?

  2. What are the 30 multi-step tasks, and how is successful completion judged?

  3. What exactly is the L5 threat model? Specifically, where are malicious instructions inserted, what behavior do they attempt to induce, and how is attack success measured?

Reviewer_GCHWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents an empirical study of LLM-agent reliability across tool routing, atomic execution, robustness, and multi-step orchestration. It combines expert process templates, sandboxed python execution, AST-based validation with feedback, and stage-wise tool exposure. Experiments use synthetic tool catalogs, 59 real-world target skills, small atomic and file-based task sets, and a 30-task data science orchestration benchmark. The reported results suggest model-dependent effects of expert descriptions, substantial execution gains for several models, and higher end-to-end completion for the Expert/partitioned configuration than for Bare/flat exposure.

Critical strengths
  1. This paper addresses a practically important distinction between selecting a tool and completing a task. Evaluating routing, execution, and multi-step completion separately is useful because improvements in one layer do not necessarily imply reliable end-to-end behavior. The low performance on file-based tasks is an informative counterpoint to the stronger results on atomic tasks.

  2. The model-dependent effect of richer descriptions could be a useful empirical finding. Table 1 reports substantial gains for some models and a drop of 42.5 points for Claude 4.5 Haiku under the same intervention. If supported by consistent results and controlled analysis, this would caution against treating expert descriptions as a universally beneficial design choice.

Critical weaknesses
  1. The empirical record is internally inconsistent in ways that affect the main findings. In fig 2, GPT-4o-mini at N=500 appears to achieve about 80%/75% for Bare/Expert, whereas table 1 reports 55%/50%. Fig 3 extends to N=2000 while table 2 ends at N=1000, and its terminal trends for Gemini and DeepSeek Flash favor Bare rather than Expert, so the figure and table do not line up. Fig 7 labels GPT-OSS's L5 result as 6.7%/26.7%, whereas table 6 reports 0%/16.7%. The textual summaries also omit counterexamples: table 3 contains sandbox accuracies of 66.7% despite the stated 91.7%–100% range, and GPT-OSS performs worse with sandbox execution in several settings. These issues prevent me from determining which quantitative findings support the conclusions. Dataset construction, trial counts, and evaluator definitions are also insufficiently specified in the provided PDF. The manuscript mentions an anonymous repository; I have not examined that material and do not assume it is absent.

  2. The experimental comparisons do not isolate the proposed components or establish the contribution beyond existing agent designs. Table 8 changes both description quality and tool partitioning, so it supports a comparison of two combined configurations rather than the stated causal conclusion about stage-wise partitioning. The AST intervention includes feedback and re-prompting without a retry-only comparison at a matched budget. It is also unclear whether the text-based execution baseline has equivalent access to executable tools and files; structured JSON function calling need not require the LLM to perform arithmetic itself. These distinctions matter because PAL, CodeAct, MetaGPT, and AnyTool already cover program execution, code actions with execution feedback, SOP-based workflows, and hierarchical tool retrieval. An empirical contribution remains possible without a novel component, but it needs consistent evidence and comparisons that identify the new insight. The equations do not fill this gap: eq. 1 does not establish statistical error control, and eq. 2 needs a precise definition of the conditional error plus extra assumptions once retries are involved.

Questions for authors
  1. Which figure and table results are authoritative? Figs. 2, 3, and 7 do not match tables 1, 2, and 6, so please reconcile them and state whether the mismatching plots use different datasets or experimental versions; the evaluation denominators and the corresponding records would help. Please also define selection error vs. schema compliance, since the two are exactly complementary in every cell of table 9.

  2. What are the 30 tasks, and how is final success judged independently? The stage and depth distribution, the way per-stage tool sets are assigned, and whether failures in stage selection are counted are all unspecified. The prompts, evaluators, exact model identifiers, and traces are also needed; the anonymous repository is referenced but I could not locate it in the submission materials. Clarifying how these tasks differ from the file-based benchmark, where performance is much lower, would help assess the scope of the result.

  3. Can the authors provide a Bare/Expert × Flat/Partition comparison with execution and retry budgets held fixed, at least for one representative model on the existing task set, or point to existing results that fill this cell? A comparison against the closest executable or retrieval-based baseline would most directly establish the incremental benefit of the proposed configuration.

  4. What do the L5 injected instructions ask the model to do, and what counts as a successful injection? The reported metric only tracks whether the legitimate task completes, so it is unclear whether the recovered runs still executed the injected content. It would also help to know the retry budget allowed in each condition, and whether AST feedback outperforms retry-only feedback at the same budget.

Reviewer_Rgp6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies reliability of LLM agents across tool routing, atomic execution, noise robustness and multi-step orchestration. It compares bare tool descriptions with expert process templates, adds sandboxed Python execution, an AST script guardrail and stage-wise tool partitioning. Experiments use synthetic catalogs of 100 to 1000 tools, a 59-skill library and a 30-task data science workflow. The paper reports better routing with expert descriptions for some models, large gains from sandbox execution on arithmetic tasks and strong gains from stage partitioning in the multi-step test.

Critical strengths

The paper covers routing, execution, robustness and orchestration in one study.

Critical weaknesses

The work is mostly engineering and lacks a new method.

The evaluation is not controlled and several claims exceed the evidence.

The theoretical content is thin.

Stage partitioning reduces the local tool set to at most 15 tools. The comparison with flat catalogs therefore changes both the routing difficulty and the workflow structure. It does not isolate partitioning as a method.

The writing can't reach the standard and there is a big problem for the formatting of pictures.

Questions for authors

Can a length-matched description control separate disambiguation from extra prompt information?

Why does sandbox execution lower accuracy for GPT-OSS in several cells, contrary to the main claim?

How were the 30 multi-step tasks built and scored?

How does the method compare with a standard agent baseline under the same tool catalog?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.