acceptodds
Under review as a conference paper at ICLR 2027

ToolTurns: Benchmarking LLM Agents in Multi-Turn Dialogue

Abstract

Tool-using language models are increasingly deployed in conversational settings, yet most large-scale tool-use benchmarks focus on isolated, single-turn requests. We introduce ToolTurns, a benchmark for evaluating language agents end-to-end in multi-turn dialogue with tools. It contains 2,620 problems spanning 1,827 tools and 11,004 API endpoints across 47 categories. Each problem begins with an intentionally underspecified request and unfolds into three user queries, yielding 7,860 independently evaluated subtasks that require agents to seek clarification, maintain conversational context, select tools, and respond to follow-up requests. We additionally introduce an open-source user simulator that reveals withheld context through dialogue and generates context-dependent follow-ups, together with a checker–fixer pipeline for improving simulated user turns. Across nine language models, the best agent resolves 69.6% of individual subtasks but completes only 39.8% of full three-query dialogues, and every evaluated model performs worse on follow-up queries than on the initial request. We further find that dialogue-quality rubric scores and task completion can rank models very differently, highlighting the importance of evaluating whether users' underlying requests are actually resolved. Finally, paired turn-level evaluation shows that our revision pipeline reduces automatically detected dialogue defects by 41.2%, while blind human evaluation prefers revised turns in 63.3% of cases. These results expose substantial room for improvement in reliable, multi-turn tool use and provide a scalable testbed for measuring progress.

Primary area:
datasets and benchmarks
Keywords:
large language models, tool use, language agents, multi-turn dialogue, benchmark, user simulation, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SiR9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a dataset used for benchmarking agents using a generated dataset and automatic evaluation. The proposed framework should measure the performance of agents in multi-turn, multi-tool settings. The benchmark combines rule matching system with a rubric based scores and compares several models across the size and weights availability spectrum. The results show that while frontier models are able to deal somewhat successfully with the initial message (max 67% accuracy), all models fail to achieve substantial performance where multiple steps are involved. Most of the results are based on the fully automatic data generation and evaluation, with minimal human validation, and the paper only briefly address whether the underlying evaluation data is valid.

Critical strengths

-- an interesting problem and first step towards solving it; -- description of the errors discovered

Critical weaknesses

-- no human validation: the generated data set is not validated in terms of solvability (is the information provided enough to solve the problem) -- llm as a judge biases not addressed: how results change when the different backbone is used; how is prompt validated; is there any correlation with human scores; would multiple runs change the results (temperature is set at 0.2) -- is there any data leakage: is the model's performance on the original dataset (from which this one is derived) correlated with the figures reported in the paper; was the original data available before or after the models were trained, and was it perhaps fed to some models; -- how many humans were employed in the study, how were the raters recruited, what were their instructions, were they trained for the task

Questions for authors

How was the generated data validated for answerability (are the task solvable)? How many humans were employed in the study, how were the raters recruited, what were their instructions, were they trained for the task ? Was there any checking of duplicates or contamination done against ToolBench’s public instructions? How was withheld context keywords generated? Do results change if generation or judge models change?

3 comments
AuthorsOpenReview ↗

Response to W1-W4

W1: no human validation: the generated data set is not validated in terms of solvability (is the information provided enough to solve the problem)

We first note that real-world tool-use dialogues are not guaranteed to be successful: tools can be unreliable, and users can be un-cooperative. This can likewise be observed in existing tool-use (Qin et al. 2023) and dialogue benchmarks (Budzianowski et al. 2018. In the paper we conduct human evaluations of the LLM Judge ratings (Section 4.2.1), observing substantial agreement between a human annotator and the LLM Judge (Cohen’s κ\kappa of 0.68) on binary task success and moderate agreement (quadratic weighted Cokens κ\kappa of 0.53) on the quality dimensions.

Human validation of answerability and judgments (Section 4.2.1; Appendix G). One human rater assessed 24 fresh, distinct problems from the fixed judge-check cohort, with four cases per original-label × query-position stratum. Model identities, original IDs, and the target item’s judge label and rationale were hidden during annotation. The packet includes all prior/current semantic messages, API responses and available schemas through the target answer. Written instructions require separate yes/no/unclear ratings for answerability and completion, and notes; no separate calibration round was conducted. Short-record eligibility and seeded selection were frozen before annotation. The record established answerability for 13 cases; 11 were unclear, not established unsolvable. Completion ratings agreed with 19/24 original labels (79.2%, κ=0.58 on this balanced sample): three original fails were judged complete and two original passes incomplete. The disputed passes include treating API descriptions as requested history and phone-number validity as reachability. The notes also flag contradictory or unsupported ancillary details. This is a small, single-rater record-based audit, not independent verification of all tool facts or a population solvability estimate. The paper reports its uncertainty rather than inferring solvability from keywords.

W2: llm as a judge biases not addressed

The core evaluation of our LLM judge is the agreement between its ratings and human ratings conducted in Section 4.2.1, where we find substantial agreement between a human annotator and the LLM Judge (Cohen’s κ\kappa of 0.68) on binary task success and moderate agreement (quadratic weighted Cohens κ\kappa of 0.53) on the quality dimensions. We further evaluate the reliability of the judge by comparing our primary judge, GPT-OSS 120B, to two alternative judges, Gemma 4 and Qwen3. We find substantial agreement on task success (84.0% exact match, Cohen’s of 0.66 for Gemma 4; 83.4% exact match, Cohen’s of 0.66 for Qwen3). We also conduct repeated evaluations with GPT-OSS 120B to evaluate the variance in its estimates, finding re-sampled judgements also show high consistency (91.6–92.3% ‘almost perfect’ agreement, Cohen’s κ\kappa of 0.83–0.84, (Landis & Koch, 1977)). Thus, while some sensitivity to LLM judgment is observed, we see substantial agreement across models and repeated evaluations. Both are reported in Appendix G.

W3: Data Leakage

Duplicates and contamination. Lowercased, quote-stripped opening queries have 0/2,620 exact overlaps with 19,321 distinct ToolBench seed instructions; ToolTurns has 2,590 distinct opening strings. This does not exclude paraphrases, schema exposure or pretraining contamination. Training-data membership is unknown. A matched ToolBench/StableToolBench score correlation has not been computed and would not itself establish contamination.

W4: Other human studies and recruitment. Each human study uses one rater from the author team and conducts label and model-blind ratings; no external raters were recruited.

  • We validate the checker–fixer step in our user simulator with a preference study: using 120 unlabeled pairs of original and revised simulated user messages. Each pair is presented in a randomized order without a source labeled. Annotation instructions prioritize grounding, contextual awareness, task progression, non-templating and naturalness. The revised user message is preferred to the original in 76/120 cases.
  • We refer to the above responses describing our LLM judge and answerability validations, which also use one author annotator each in a model and label blind setting.
AuthorsOpenReview ↗

Response to Q1

Q1

Keyword generation and interpretation. GPT-4o-mini generates the opening request and context keywords together from endpoint descriptions/schemas, without the original ToolBench instruction. Appendix I reproduces the prompt; Sections 3.1 and 3.3 clarify that keywords guide clarification, not a gold answer, required checklist or exhaustive input list. The simulator may add consistent user details without inventing tool facts. The prompt requests 2–4 concrete values; actual lists have 1–7 entries, with 2–4 in 2,591/2,620 problems. Completion is judged against the request, not keyword coverage. Generator sensitivity. Appendix B compares user simulators based on Qwen and GPT-4o-mini. The nine-agent benchmark was not repeated with a different query generator or user simulator; judge swaps do not answer that separate question, and we do not claim generator-invariant scores

Thank you for the response. The added detail is useful, but it confirms that the human validation requires more rigour to support the paper’s main claims.

Every human study uses a single rater from the author team. There is no inter-rater agreement, no external recruitment, and no calibration round, and the raters are not independent of the work. The κ=0.68 and κ=0.53 figures come from an earlier study with no reported sample size or design. The only study that was blind covers 24 problems. In that audit, 11 of 24 cases (46%) could not be established as answerable and were rated incomplete. The 36-judgment check was not blind, since the rater saw agent identities, labels, and rationales. Contrary to the rebuttal, not all studies were label- and model-blind.

The authors acknowledge that none of this estimates population solvability or judge accuracy. So the reported scores cannot be separated from dataset defects. A wrong answer may reflect an unanswerable item, not a model failure.

To meet the standard, the study needs several independent raters, reported agreement, a stated protocol and sample size, and a larger blinded sample large enough to estimate solvability. I keep my concerns about soundness.

Reviewer_t2gcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ToolTurns, a benchmark for evaluating LLM agents in multi-turn tool-use dialogues. It contains 2,620 problems spanning 11,004 API endpoints, with adaptive clarification and follow-up requests. The authors also propose a user simulator with a checker–fixer pipeline and evaluate nine models. The results show lower success rates on follow-up requests, substantial gaps between subtask and full-dialogue completion, and discrepancies between dialogue-quality scores and task success.

Critical strengths
  1. Broad benchmark coverage. Combining a large API catalogue with adaptive multi-turn interaction is a useful contribution. The benchmark extends existing tool-use evaluation settings by assessing agents along their own conversational trajectories.

  2. Comprehensive evaluation. The experiments cover nine models and report both individual task success and full-dialogue completion. The human evaluation of the checker–fixer pipeline also provides useful supporting evidence.

Critical weaknesses
  1. Limited novelty of the empirical findings. Most findings largely confirm known limitations of LLM agents, including multi-turn performance degradation, low end-to-end task completion, and difficulties with tool use. While the discrepancy between dialogue-quality scores and task success is interesting, the paper does not clearly establish what new insights emerge from evaluating a broader API catalogue. The contribution appears to lie primarily in benchmark scale and construction rather than in new empirical understanding. It remains unclear what this benchmark reveals that existing evaluations could not.

  2. Insufficient validation of benchmark reliability and multi-turn difficulty. Tool responses rely on cached or LLM-simulated outputs, while task success is determined by an LLM judge. The paper's own examples show incorrect judgments caused by unreliable tool outputs or incomplete evaluation context. Moreover, follow-up queries differ across agents because they are generated adaptively. Thus, the observed performance degradation cannot be attributed specifically to multi-turn context dependence, as follow-up difficulty may also vary. These issues weaken the interpretation of the main results.

Questions for authors
  1. What new failure modes or insights does ToolTurns reveal beyond existing multi-turn tool-use benchmarks? Can the authors demonstrate how broader API coverage leads to qualitatively different findings?

  2. Can the authors provide a human-verified audit of task solvability and evaluation correctness, along with a controlled comparison using matched follow-up requests to isolate multi-turn context effects?

2 comments
AuthorsOpenReview ↗

Response to Reviewer t2gc: contribution, evaluation reliability and matched follow-ups

Thank you. We clarify the contribution and add human validation, judge-context checks and matched-follow-up evaluation.

W1/Q1: Contribution and insights (Section 2; Table 1; Appendix G).

ToolTurns combines 1,8271{,}827 tools across 4747 categories with evaluation along each agent's own conversational trajectory and a separately assessed user simulator. ToolBench offers broad APIs for a single user request; τ\tau-bench and ToolSandbox evaluate interactive dialogue in stateful applications. We evaluate sustained dialogue across diverse schemas, with a checker-fixer user simulator tested through automatic and human checks.

The catalogue distinguishes reusing an endpoint from using a new operation of a familiar tool. On multi-tool problems with an initial judged pass, follow-ups using only new endpoints of already-used tools score 6.36.3 points lower at the first follow-up (9595\\% interval 2.92.9-9.89.8) and 4.44.4 lower at the second (0.90.9-7.87.8). The analysis accounts for problem and agent differences and adds an exploratory control for distinct endpoints attempted. A tool-level switching measure merges these behaviors. Larger mixed-call gaps shrink after this control, and the adjusted large/small-catalogue interval includes zero. These are associations in chosen calls, not assigned requirements or causal effects.

W2/Q2: Environment and evaluation reliability (Sections 3.2, 3.4.2, 4.2.1; Appendix G).

StableToolBench's cache and simulation reduce dependence on changing live API availability and responses (Guo et al., 2024). They do not guarantee factual correctness or cross-call consistency; the simulator has no shared dialogue state. We retain erroneous-output examples and now state this limitation explicitly.

An LLM judge accommodates heterogeneous outputs, open-ended answers and multiple valid tool paths without a separate verifier for every task. AgentRewardBench finds that rule-based checks can miss valid web-agent successes, while LLM judges can overestimate success (Lù et al., 2025). This motivates validation, not an assumption that either evaluator is correct.

One human rater assessed 2424 target requests from distinct problems, seeing prior/current records and schemas with model identities and target judge labels/rationales hidden. Separate ratings coincided: the same 1313 were answerable and complete; 1111 were unclear for answerability and incomplete. No alternative solution paths were tested in this audit; unclear does not mean unsolvable. Completion agreed with 19/2419/24 original labels (79.279.2\\%): three original fails judged complete and two passes incomplete. This short-record, label-balanced audit assesses recorded evidence, not a perfect agent or population solvability. The earlier unblinded check found 3131 agreements, four disagreements and one uncertain case.

W2: Missing judge context (Appendices F.4 and G).

Appendix F.4 shows a judge calling wallet balances fabricated although they appeared in an earlier tool response outside its input. The rationale is wrong; conflicting outputs leave answer correctness unresolved.

Holding requests, answers and current traces fixed, we rejudge the same 566566 follow-ups with and without full prior evidence. Qwen3-32B-AWQ rises from 61.361.3\\% to 67.867.8\\% (+6.54+6.54 points; dialogue-cluster 9595\\% interval 3.883.88-9.229.22); GPT-OSS-120B rises from 60.660.6\\% to 66.466.4\\% (+5.83+5.83; 2.452.45-9.359.35). Agreement with the blinded ratings rises from 19/2419/24 to 22/2422/24 and 17/2417/24 to 19/2419/24, respectively. Protocols were frozen before unblinding; observations are untruncated and exclusions explicit. Cross-judge agreement is 75.675.6\\%/73.973.9\\% for current/full evidence. On 278278 fully covered dialogues, pooled follow-up scores remain below initial scores under both judges, while agent profiles vary. Main scores retain the original protocol.

W2/Q2: Matched follow-ups (Appendix H).

Adaptive follow-ups can differ in difficulty; we no longer interpret the position decline as a causal history cost. Eight agents instead receive the same saved Opus prefix and first follow-up on 200200 problems with a judged-successful reference initial answer. Success is 52.552.5-76.976.9\\%: failures remain without questions generated from each tested agent's answer. This controls question wording, not solvability or a pure history effect. Exploratory standalone rewrites score 2.52.5-10.510.5 points lower on paired scored problems, but rewrite equivalence is unverified, prior observations are truncated and further clarification is unavailable. Of 3,2003{,}200 planned runs, 3,1463{,}146 were scored; 5454 service/protocol exclusions and denominators are reported. Opus was not re-run.

Keywords guide clarification, not solvability or an exhaustive input list (Sections 3.1/3.3). Appendix F.1 provides a complete interaction and schema; Appendix I gives the generation prompt.

Thank you for the detailed response and additional experiments. I appreciate the matched follow-up evaluation and judge-context analysis, which address several of my concerns.

However, I remain unconvinced that the new endpoint-level findings establish sufficiently distinct insights beyond existing benchmarks. I also have concerns about evaluation reliability, given the limited human audit and the substantial changes in success rates when prior dialogue evidence is included, while the main results retain the original judging protocol.

These concerns prevent me from raising my score, so I will maintain my original rating.

Reviewer_zr5cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ToolTurns, a dataset for benchmarking LLM agents in multi-turn dialog with tools. ToolTurns contains 2,620 problems spanning 1,827 tools and 11,004 API endpoints. Each problem starts with a vague request and unfolds into three user queries, so that the agent has to seek clarification, select tools, and respond to follow-up requests. In addition, a user simulator is provided to reveal withheld context through dialog and generate follow-ups. The paper reports 9 agent models' performance on ToolTurns and argues the hardness of ToolTurns.

Critical strengths
  • The paper introduces a new dataset for benchmarking LLM agents in multi-turn dialog with tools and reports multiple agent models' performance on it.
Critical weaknesses

I do not get a concrete picture of what a real instance of the benchmark looks like after reading the paper (e.g., there's not a single complete sample; there's not even a sample tool schema). The following comments are based on my best understanding of the work.

  • It appears no problem has ground truth (e.g., gold tool path, gold answer, etc.), and it appears problems are not guaranteed to be solvable as the environment is highly simulated. As a result, readers can hardly interpret the real difficulty of the benchmark solely by reading the performance numbers.
  • It appears to me that the main claim of the paper about the hardness of the multi-turn setup is confounded by how the follow-ups are produced. Specifically, a bad first answer could produce a bad second question (and the agent is then scored on a question with no correct answer), which to me is not the same as a genuine multi-turn cost.
Questions for authors
  • Could you please address the weaknesses?
1 comment
AuthorsOpenReview ↗

Response to Reviewer zr5c

Thank you for identifying the missing concrete example and the ambiguity in interpreting follow-up scores. We now show a complete interaction, explain the evaluation choice, and add human validation and a fixed-follow-up replay.

A concrete benchmark instance (Section 3.1; Appendix F.1).

We now explicitly instantiate the problem triple:

  • q251q_{251} is “I need to calculate the result of this equation with some specific values.”
  • G251G_{251} is [x,y,2x+3y,5,10][x, y, 2x + 3y, 5, 10].
  • T251T_{251} contains the symbolic-expression tool's four endpoints and schemas: validexpression, evaluate, differentiate, and simplifybasic (short names).

The saved GPT-4.1-mini dialogue includes clarification, evaluation, and follow-ups about simplification and differentiation. We provide endpoint signatures, a complete callable JSON schema, every API call/response, user message, final answer, and all three judgments. Only transport IDs and duplicate control-function echoes are omitted. Readers can directly check the three mathematical results. Section 3.1 points to this complete example; Appendix F.2 complements it with a non-math Lakers request spanning three tools. Neither example is a representative success estimate.

W1: Ground truth and evaluation reliability.

Keywords guide simulator clarification; they are not a gold answer or exhaustive input list. The agent may ask beyond them, and the simulator may supply consistent user details without inventing tool facts (Sections 3.1/3.3; generation prompt in Appendix I). No gold tool path or independently verified answer is supplied. We use semantic judgments to accommodate heterogeneous API outputs, open-ended answers and multiple valid tool paths without requiring one reference answer or path. Section 3.4.2 now explains this choice. We do not claim formal verification is impossible: the math example admits exact checks. The judge is a scalable evaluation choice whose reliability must be assessed separately; consistency with cached/simulated evidence does not establish real-world correctness.

We added a fresh blinded audit of 24 distinct problems: one human rater saw the complete prior/current record and API schemas, with model identities and the target item’s judge label and rationale hidden during annotation. The record established answerability for 13 requests; 11 remained unclear, not demonstrated unsolvable. Completion ratings agreed with 19/24 original judgments (79.2%): three original fails were judged complete and two original passes incomplete. The sample balances original labels and query positions and is restricted to short records, so it is not a population solvability estimate. The separate earlier unblinded check found 31 agreements, four disagreements and one uncertain case (31/35 determinate agreement). Section 4.2.1 and Appendix G report both protocols. Cached/simulated outputs remain a limitation, and the erroneous-output examples remain in the paper.

W2: Confounds in follow-up generation.

We have removed the interpretation that the position decline itself measures a causal multi-turn cost. Among fully scored dialogues whose initial subtask was judged successful, both follow-ups pass in only 53.3% for Opus 4.8 (n=1,945n = 1{,}945), 50.9% for GLM-5.2 (n=1,938n = 1{,}938), and 49.5% for Gemma 4 (n=1,915n = 1{,}915). Thus an initial failure label is not necessary for later failures. This conditional analysis does not independently certify the first answer or the generated follow-ups.

The fixed-follow-up replay (Appendix H) addresses one part of this confound. Normally, each agent's answer shapes its own follow-up. In the replay, eight agents instead receive the same saved Opus prefix and first follow-up on 200 problems with a judged-successful reference initial answer. Success is 52.5–76.9%, so failures remain without generating a question from each tested agent's own answer. This addresses variation in follow-up wording across agents; it does not certify follow-up solvability. A separate exploratory condition uses generated standalone rewrites, which score 2.5–10.5 points lower, also on paired scored problems. These conditions do not isolate a pure history effect: rewrite equivalence was not verified, prior tool responses are truncated, and no further user clarification is allowed. Of 3,200 planned runs, 3,146 were scored; 54 service/pr

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.