- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper measures how LLM tool-selection success compares to a zero-learning lexical-overlap heuristic as the valid action set size ∣V∣ grows, reporting a finite crossover at every open-weight scale tested and examining name-vs-index emission and retrieval filtering as mitigations.
Critical strengths
-
The paper asks a useful diagnostic question and separates the chance floor from model capability with paired controls and a pre-registered design.
-
The empirical scope includes multiple Qwen scales, random controls, a dilution check, Seal-Tools replication, and practical name-emission/filtering analyses.
-
The AI-use and error-correction disclosures are unusually transparent, with raw-data and notebook availability supporting reproducibility.
Critical weaknesses
-
The main ∣V∣ sweep confounds candidate count with prompt length, and the equivalent length control is explicitly left for future work.
-
The central heuristic depends on BFCL’s lexical transparency, while a semantic retriever beats it and name hashing collapses the crossover.
-
Several analysis bugs and post-hoc corrections change or retract reported effects, and between-session noise is not reflected in the CIs.
-
The name-emission mechanism, the superiority of k=∣V∣, and the deployment-cost claim are not cleanly isolated.
-
External validity is limited to BFCL function selection, with weak cross-family checks and an underpowered ALFWorld extension.
Questions for authors
- AI-assisted re-verification is not independent: headline numbers were re-derived by the same AI system that wrote the original analysis code, not by a separate human or system.
others, see weakness
2 comments
Thank you for your reply. I will maintain my score.
Response to Ethics Concerns
Thanks for raising this. We answer it here, separately from the scientific points.
1. AI use is disclosed, as the ICLR 2027 policy requires. Appendix A states that a generative AI system (Claude, Anthropic) helped draft the pre-registered hypotheses, write the experiment and analysis code, interpret results, and draft parts of the manuscript. The policy requires these uses to be disclosed. The author takes full responsibility for every claim in the paper.
2. The 2026 preprints we cite exist. We re-checked all eleven 2026 preprints cited against their arXiv pages (arXiv:2608.04355, 2606.16364, 2605.26731, 2607.17044, 2607.13085, 2602.16805, 2604.07236, 2605.24660, 2605.18857, 2606.29914, 2605.27922). The re-check found two metadata errors, which we will correct: Harness-Bench (arXiv:2605.27922) now appears on arXiv with a revised title and 13 authors (we list 12), and the title of Dhakal et al. (arXiv:2607.13085) is truncated. We will also confirm that the revised Harness-Bench version still supports the point we cite it for.
3. The verification notes in the reference list will be removed. Notes such as "verified 2026-09-15 against arXiv" should not have printed, and they will be removed.
4. Verification of the numbers. The raw per-case CSVs for the submitted experiments are in the supplementary material. The reviewer is right that re-derivation by the same AI system that wrote the analysis code is not independent. [KEEP ONLY IF DONE: On [DATE], the author re-derived the 3B crossover |V|* = 28.7 and the Seal-Tools heuristic rate 0.833 by hand in a spreadsheet from the raw CSVs, with no project code, and both match the paper.]
5. Presentation. We agree the appendix reads like an audit log. In revision, corrections made before submission will be removed from the paper and kept in a supplementary changelog, and the main text will be condensed around a single thesis.