acceptodds
Under review as a conference paper at ICLR 2027

TA-CallBench: Benchmarking Nested Tool Calling for Conditional Trigger–Action Tasks

Abstract

Large language models (LLMs) can select tools and generate arguments from user requests, supporting immediate queries, software operations, and task automation. However, existing benchmarks primarily evaluate whether models generate correct function calls for the current request, and rarely examine their ability to generate nested tool calls for conditional trigger–action tasks, where an operation must be executed only after a specified event or state occurs. Because such a task combines triggering conditions with executable Actions, its representation requires an outer call that organizes these components. To evaluate this ability, we introduce TA-CallBench, which requires a model to select inner tools (the individual Trigger and Action functions) and outer tools (management tools that organize inner calls into complete conditional tasks) from the available tool set and generate the required nested tool calls. TA-CallBench provides thousands of bilingual dialogues across six application scenarios and 12 task types, including single-task creation, multi-task creation, multi-turn modification, and fine-grained component editing, which together require a model to identify task boundaries and organize components within nested calls. We construct the data through Trigger-anchored tool-combination sampling, LLM-based generation, structural validation, and combined model and human review. The evaluation centers on Tool-Call Accuracy (TCA), which checks whether a model correctly generates the Trigger and Action calls and their arguments. All evaluated models achieve TCA below 55% on TA-CallBench. For models evaluated on both benchmarks, their TA-CallBench TCA is substantially lower than their performance on the Berkeley Function Calling Leaderboard (BFCL). This gap indicates that current LLMs still struggle to reliably generate such nested calls, highlighting the need for dedicated evaluation of this capability.

Primary area:
datasets and benchmarks
Keywords:
Large language models, Benchmark, Nested function calling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UdZZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an advanced problem definition extending nested tool calling, called trigger-action. It builds an offline benchmark to evaluate agents in such situations. Experimental results suggest that even though the agents are proficient at ordinary function-calling tasks, they show brittle trigger-action capabilities.

Critical strengths
  • This paper goes one step beyond nested tool calling by defining a problem in which the agent must be aware of a trigger condition and the action that follows it.

  • For precise evaluation, this paper adopts exact-matching-based evaluation.

Critical weaknesses
  • Since end-to-end evaluation frameworks are dominant these days, it is unclear how this dataset helps fill a gap in the current paradigm. Also, trigger-action settings appear frequently in existing online simulation benchmarks, so isn’t analyzing trajectories from those benchmarks sufficient? The authors should justify how this benchmark contributes to current agent evaluation frameworks.

  • It is difficult to fully understand what kind of situation this paper actually deals with. The paper needs many more example situations to describe the problem, and I suggest using running examples.

  • Although the paper generates a large number of dialogues, it provides no full examples at all. Moreover, the dialogue generation pipeline seems too simple, and it is hard to understand how it can produce such hard tasks (GPT-5.4: 23.4%).

  • The authors should include more agent models in the experiments, and should show that the data is genuinely difficult rather than made difficult by internal errors or ambiguous situations.

  • This paper should clearly write the methodology with exact I/O and examples.

Questions for authors
  • What is the motivation for separating inner and outer tools? Using the user–agent interaction as the history context and letting the agent generate the next action while considering the trigger situation seems more intuitive. Is there a specific reason for this more complex setting?

  • Most offline tool-agent benchmarks are in English. Is there a reason for building the Chinese dataset first and then translating it? Generating the Chinese and English datasets separately seems more valid for building a high-quality dataset.

  • Why did this paper not report the DA metric, even though it is clearly defined?

Reviewer_Rk2oOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TA‑CallBench, a benchmark for evaluating organization‑oriented nested tool calling of LLMs on conditional trigger‑action tasks, which differs from prior data‑dependency nested tool‑calling benchmarks. The authors design an outer‑inner tool paradigm and construct a bilingual synthetic dataset covering six scenarios and twelve task types, with Tool‑Call Accuracy (TCA) as the primary metric. Experiments demonstrate that performance on the conventional BFCL function‑calling benchmark cannot transfer to this new task, and explicit structural guidance yields large performance gains. Parameter generation constitutes the dominant failure mode. However, the dataset is entirely LLM‑generated without real‑user samples. There is no human performance baseline. Evaluation relies purely on static JSON matching, and only zero‑shot prompt experiments are conducted.

Critical strengths
  • Identifies under‑studied conditional trigger‑action tasks and distinguishes organization‑oriented nesting from data‑dependency nesting.
  • Complete dataset construction pipeline with multi‑dimensional diagnostic metrics, solid ablation and error analysis.
  • Provides insightful empirical findings: conventional tool‑calling performance does not transfer; structural guidance yields large gains; parameter generation is the primary bottleneck.
  • Bilingual Chinese‑English support, comprehensive appendices and good reproducibility.
Critical weaknesses
  • Fully LLM‑synthetic dataset without real‑user queries, lacking ambiguous or incomplete user inputs and risking distribution shift.
  • No human performance baseline for interpreting absolute model performance.
  • Optimistic bias in multi‑turn evaluation by feeding ground‑truth history; only static JSON matching with no task‑logic simulation.
  • Only zero‑shot prompt evaluation; no fine‑tuning experiments to check whether observed weaknesses are fixable.
  • Several high‑frequency failure modes are only discussed in appendices with insufficient main‑text coverage.
Questions for authors

Same as weakness

Reviewer_gLQoOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces how to test the situation whether a natural-language conditional request is translated into a structured automation definition. Specifically, the proposed bench tests scenarios such as whether model can generate structured function calls for "if A, then B". The benchmark further extends this setting to multiple conditions, independent tasks, and multi-turn modifications. It consists of bilingual dialogues covering 6 application scenarios and 12 task types.

The paper found that many existing models have limited performance in such setting. However, per my understanding, these results do not necessarily indicate a fundamental weakness in the models' ability. One important limitation of this paper is its strict exact-match evaluation, which requires the generated output to completely match the reference structure, which makes it difficult to interpret what the low scores actually reveal. Any relatively small errors can make an otherwise reasonable automation incorrect.

Critical strengths
  1. The paper reveals meaningful limitations in current models, even though it does not fully disentangle failures in conditional reasoning and task composition from difficulties with argument generation, schema compliance, and strict structural matching.
  2. The benchmark provides reasonably broad task coverage and systematic evaluation.
Critical weaknesses
  1. The paper does not show which underlying capability explains those failures.
  2. The benchmark primarily evaluates the translation of natural-language requests into structured conditional workflows. Although the benchmark includes multiple conditions, independent tasks, and modifications, the paper does not sufficiently proves what fundamentally new capability is being evaluated beyond conventional structured workflow generation.
Questions for authors
  1. Are the models failing because they struggle to understand and compose conditional workflows, or because they struggle to reproduce a complex JSON representation?
  2. Could the authors provide more details about the human validation process?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.