- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes an advanced problem definition extending nested tool calling, called trigger-action. It builds an offline benchmark to evaluate agents in such situations. Experimental results suggest that even though the agents are proficient at ordinary function-calling tasks, they show brittle trigger-action capabilities.
Critical strengths
-
This paper goes one step beyond nested tool calling by defining a problem in which the agent must be aware of a trigger condition and the action that follows it.
-
For precise evaluation, this paper adopts exact-matching-based evaluation.
Critical weaknesses
-
Since end-to-end evaluation frameworks are dominant these days, it is unclear how this dataset helps fill a gap in the current paradigm. Also, trigger-action settings appear frequently in existing online simulation benchmarks, so isn’t analyzing trajectories from those benchmarks sufficient? The authors should justify how this benchmark contributes to current agent evaluation frameworks.
-
It is difficult to fully understand what kind of situation this paper actually deals with. The paper needs many more example situations to describe the problem, and I suggest using running examples.
-
Although the paper generates a large number of dialogues, it provides no full examples at all. Moreover, the dialogue generation pipeline seems too simple, and it is hard to understand how it can produce such hard tasks (GPT-5.4: 23.4%).
-
The authors should include more agent models in the experiments, and should show that the data is genuinely difficult rather than made difficult by internal errors or ambiguous situations.
-
This paper should clearly write the methodology with exact I/O and examples.
Questions for authors
-
What is the motivation for separating inner and outer tools? Using the user–agent interaction as the history context and letting the agent generate the next action while considering the trigger situation seems more intuitive. Is there a specific reason for this more complex setting?
-
Most offline tool-agent benchmarks are in English. Is there a reason for building the Chinese dataset first and then translating it? Generating the Chinese and English datasets separately seems more valid for building a high-quality dataset.
-
Why did this paper not report the DA metric, even though it is clearly defined?