acceptodds
Under review as a conference paper at ICLR 2027

AgentFloor: How Far Up the Tool-Use Ladder Can Small Open-Weight Models Go?

Abstract

Agentic systems make many small tool-using calls, but it is unclear which of them actually require a frontier model. We introduce AgentFloor, a controlled benchmark that organises tool-use tasks into a six-level ladder, from instruction following and single tool calls to branching, recovery, and long-horizon planning, while keeping the tool environment fixed. We evaluate 16 open-weight models alongside five frontier models from three vendors. Small open-weight models perform surprisingly well on the simplest levels: even a sub-billion-parameter model follows instructions reliably, but the gap widens as tasks require chaining, branching, and longer-horizon control. How large that gap is depends strongly on which frontier model serves as the reference: conclusions drawn against GPT-5 differ substantially from those drawn against the strongest Claude models. Models with similar overall accuracy also fail in very different ways, and simple interventions rarely transfer from one model to another. These results suggest that routing decisions should depend on the structure of the tool-use task, not only on model size or aggregate benchmark scores. We release the benchmark, the evaluation harness, and the full run corpus.

Primary area:
datasets and benchmarks
Keywords:
Agentic systems, model routing, agent benchmarks, open-weight models, tool use, long-horizon planning, model efficiency, agent reliability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EybxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a 30-task benchmark to measure instruction following under an increasing number of required tool calls. Results show that models can perform tasks requiring less than two chained tool calls, with significant performance differences for more complex tasks.

Critical strengths

None

Critical weaknesses
  • The tasks or trajectories are not released or described in enough detail to assess the work.
  • The benchmark consists only of 30 tasks.
Questions for authors
  • Could authors release the tasks and trajectories?
Reviewer_fUQrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AgentFloor, a controlled benchmark designed to evaluate the tool-use capabilities of small open-weight models. It organizes 30 tasks into 6 difficulty tiers, from basic instruction following and single tool calls to long-horizon planning. All tasks use a fixed set of eight abstract tools operating on deterministic in-memory databases. The authors evaluate 16 open-weight models and 5 frontier models, finding that smaller models perform competitively on simple tool-use tasks but struggle with more complex, multi-step interactions. The results also highlight differences in failure patterns across models and the importance of task structure for model selection.

Critical strengths
  1. The paper evaluates 21 models across different sizes and families, providing useful comparisons between open-weight and frontier models.
  2. The fixed tool environment and tier-based design help isolate different levels of tool-use complexity and facilitate model comparisons.
  3. The paper reports detailed empirical analysis: per-tier performance, failure categories, statistical comparisons, and cost-efficiency analyses rather than relying solely on aggregate success rates.
Critical weaknesses
  1. There is no task definitions, tool implementations, execution environment, or verification code. This makes it difficult to assess the technical contribution, or reproduce the experiments.
  2. The benchmark is too small, containing only 30 tasks, with 6 per difficulty tier. Prompt paraphrases and repeated runs do not meaningfully increase the diversity. Such a small benchmark is insufficient to reliably characterize general tool-use capabilities, or support robust model routing and training.
  3. Tier E is described as requiring 8–12 tool calls under persistent constraints, but the paper does not provide the concrete task definitions or environment dynamics. Most available tools appear to be read-only or computational, and it is unclear whether tasks involve meaningful state mutations, action-dependent constraints, or nontrivial planning rather than longer sequences of information retrieval.
  4. The main finding that smaller models perform well on simple tool calls but struggle with longer, more complex trajectories is largely expected. The benchmark does not systematically disentangle the effects of trajectory length, branching complexity, information dependencies, and constraint satisfaction, limiting the insights into the underlying causes of model failures.
Questions for authors
  1. Can the author provide one task example for each of the tier?
  2. What is the actual tool definition? Are all of them read only tools?
Reviewer_CmjROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AgentFloor, which evaluates tool-use capability through 30 tasks across six tiers in a fixed environment of eight abstract tools. The tiers progress from instruction following to chaining, branching, recovery, and constrained planning. Experiments cover 16 open-weight and five frontier models, with failure analysis and prompting interventions. Small models perform well on simple tasks, while higher-tier gaps depend strongly on the frontier reference. Models with similar aggregate scores also fail differently. The diagnostic aim is useful, though task coverage and serving differences limit the capability conclusions.

Critical strengths
  • The fixed environment removes API drift and web-grounding confounds. Requiring native tool calls makes the benchmark test whether models actually execute their plans.
  • Multiple frontier references are informative: matching GPT-5 overall does not imply matching the strongest Claude models on chaining or branching (Table 1). Tier-level reporting makes this distinction clear.
  • The failure analysis distinguishes early stopping, planning without acting, wrong tools, and budget exhaustion (Table 3). Testing interventions against these failures adds useful evidence beyond model rankings. The authors also openly disclose the schema problem and unsuccessful interventions.
Critical weaknesses
  1. The schema defect compromises the capability comparison. The Ollama adapter omits parameters for two tools; restoring them improves single-call performance by 16 points in an additional-model probe (Section 6). Models may compensate differently for missing schemas, so this probe does not establish that rankings and reliability thresholds survive correction.

  2. The ladder does not cleanly isolate structural demands. With five tasks per tier, synthesis and recovery are bundled in D, and trajectory length and persistent constraints in E. The ordering checks show shared task difficulty, but the main text lacks matched comparisons identifying which demand causes the decline (Section 4.3).

  3. The intervention gains are too localized to establish reusable model-specific remedies. A submission reminder fixes one model–task combination, and a larger budget helps GPT-5 on one task (Section 4.5). It remains unclear whether these remedies transfer to other tasks with the same failure type.

Questions for authors
  1. After restoring full tool schemas, which model rankings and smallest-model reliability thresholds in Table 2 remain unchanged across the main model families?
  2. Can matched task variants isolate branching or recovery while holding the records, answer, and approximate call count fixed? This would clarify what each tier measures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.