- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents a dataset used for benchmarking agents using a generated dataset and automatic evaluation. The proposed framework should measure the performance of agents in multi-turn, multi-tool settings. The benchmark combines rule matching system with a rubric based scores and compares several models across the size and weights availability spectrum. The results show that while frontier models are able to deal somewhat successfully with the initial message (max 67% accuracy), all models fail to achieve substantial performance where multiple steps are involved. Most of the results are based on the fully automatic data generation and evaluation, with minimal human validation, and the paper only briefly address whether the underlying evaluation data is valid.
Critical strengths
-- an interesting problem and first step towards solving it; -- description of the errors discovered
Critical weaknesses
-- no human validation: the generated data set is not validated in terms of solvability (is the information provided enough to solve the problem) -- llm as a judge biases not addressed: how results change when the different backbone is used; how is prompt validated; is there any correlation with human scores; would multiple runs change the results (temperature is set at 0.2) -- is there any data leakage: is the model's performance on the original dataset (from which this one is derived) correlated with the figures reported in the paper; was the original data available before or after the models were trained, and was it perhaps fed to some models; -- how many humans were employed in the study, how were the raters recruited, what were their instructions, were they trained for the task
Questions for authors
How was the generated data validated for answerability (are the task solvable)? How many humans were employed in the study, how were the raters recruited, what were their instructions, were they trained for the task ? Was there any checking of duplicates or contamination done against ToolBench’s public instructions? How was withheld context keywords generated? Do results change if generation or judge models change?
3 comments
Response to Q1
Q1
Keyword generation and interpretation. GPT-4o-mini generates the opening request and context keywords together from endpoint descriptions/schemas, without the original ToolBench instruction. Appendix I reproduces the prompt; Sections 3.1 and 3.3 clarify that keywords guide clarification, not a gold answer, required checklist or exhaustive input list. The simulator may add consistent user details without inventing tool facts. The prompt requests 2–4 concrete values; actual lists have 1–7 entries, with 2–4 in 2,591/2,620 problems. Completion is judged against the request, not keyword coverage. Generator sensitivity. Appendix B compares user simulators based on Qwen and GPT-4o-mini. The nine-agent benchmark was not repeated with a different query generator or user simulator; judge swaps do not answer that separate question, and we do not claim generator-invariant scores
Thank you for the response. The added detail is useful, but it confirms that the human validation requires more rigour to support the paper’s main claims.
Every human study uses a single rater from the author team. There is no inter-rater agreement, no external recruitment, and no calibration round, and the raters are not independent of the work. The κ=0.68 and κ=0.53 figures come from an earlier study with no reported sample size or design. The only study that was blind covers 24 problems. In that audit, 11 of 24 cases (46%) could not be established as answerable and were rated incomplete. The 36-judgment check was not blind, since the rater saw agent identities, labels, and rationales. Contrary to the rebuttal, not all studies were label- and model-blind.
The authors acknowledge that none of this estimates population solvability or judge accuracy. So the reported scores cannot be separated from dataset defects. A wrong answer may reflect an unanswerable item, not a model failure.
To meet the standard, the study needs several independent raters, reported agreement, a stated protocol and sample size, and a larger blinded sample large enough to estimate solvability. I keep my concerns about soundness.
Response to W1-W4
W1: no human validation: the generated data set is not validated in terms of solvability (is the information provided enough to solve the problem)
We first note that real-world tool-use dialogues are not guaranteed to be successful: tools can be unreliable, and users can be un-cooperative. This can likewise be observed in existing tool-use (Qin et al. 2023) and dialogue benchmarks (Budzianowski et al. 2018. In the paper we conduct human evaluations of the LLM Judge ratings (Section 4.2.1), observing substantial agreement between a human annotator and the LLM Judge (Cohen’s κ of 0.68) on binary task success and moderate agreement (quadratic weighted Cokens κ of 0.53) on the quality dimensions.
Human validation of answerability and judgments (Section 4.2.1; Appendix G). One human rater assessed 24 fresh, distinct problems from the fixed judge-check cohort, with four cases per original-label × query-position stratum. Model identities, original IDs, and the target item’s judge label and rationale were hidden during annotation. The packet includes all prior/current semantic messages, API responses and available schemas through the target answer. Written instructions require separate yes/no/unclear ratings for answerability and completion, and notes; no separate calibration round was conducted. Short-record eligibility and seeded selection were frozen before annotation. The record established answerability for 13 cases; 11 were unclear, not established unsolvable. Completion ratings agreed with 19/24 original labels (79.2%, κ=0.58 on this balanced sample): three original fails were judged complete and two original passes incomplete. The disputed passes include treating API descriptions as requested history and phone-number validity as reachability. The notes also flag contradictory or unsupported ancillary details. This is a small, single-rater record-based audit, not independent verification of all tool facts or a population solvability estimate. The paper reports its uncertainty rather than inferring solvability from keywords.
W2: llm as a judge biases not addressed
The core evaluation of our LLM judge is the agreement between its ratings and human ratings conducted in Section 4.2.1, where we find substantial agreement between a human annotator and the LLM Judge (Cohen’s κ of 0.68) on binary task success and moderate agreement (quadratic weighted Cohens κ of 0.53) on the quality dimensions. We further evaluate the reliability of the judge by comparing our primary judge, GPT-OSS 120B, to two alternative judges, Gemma 4 and Qwen3. We find substantial agreement on task success (84.0% exact match, Cohen’s of 0.66 for Gemma 4; 83.4% exact match, Cohen’s of 0.66 for Qwen3). We also conduct repeated evaluations with GPT-OSS 120B to evaluate the variance in its estimates, finding re-sampled judgements also show high consistency (91.6–92.3% ‘almost perfect’ agreement, Cohen’s κ of 0.83–0.84, (Landis & Koch, 1977)). Thus, while some sensitivity to LLM judgment is observed, we see substantial agreement across models and repeated evaluations. Both are reported in Appendix G.
W3: Data Leakage
Duplicates and contamination. Lowercased, quote-stripped opening queries have 0/2,620 exact overlaps with 19,321 distinct ToolBench seed instructions; ToolTurns has 2,590 distinct opening strings. This does not exclude paraphrases, schema exposure or pretraining contamination. Training-data membership is unknown. A matched ToolBench/StableToolBench score correlation has not been computed and would not itself establish contamination.
W4: Other human studies and recruitment. Each human study uses one rater from the author team and conducts label and model-blind ratings; no external raters were recruited.