- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes GEAR, a graph-based tool retrieval framework that expands conventional similarity-based retrieval using relationships between tools. It constructs a tool ontology with four types of relations (REQUIRES, INFORMS, semantic-routing, and CONFLICT), allowing the retriever to include tools that may be necessary for task execution but are not explicitly mentioned in user requests. The method is evaluated on CAR-bench and ToolBench, with comparisons against similarity-based retrieval, COLT, and MCP-Zero. Results show improvements in task completion capability, although gains in execution consistency are limited.
Critical strengths
-
The paper addresses a practical problem in tool-using agents. Conventional similarity-based retrieval can miss prerequisite or complementary tools that are not explicitly mentioned in user requests. Exploiting relationships between tools is a reasonable way to address this limitation.
-
The proposed framework is relatively simple and lightweight. It does not require additional model training or per-turn LLM calls, making it potentially easy to integrate into existing agent systems.
-
The paper provides encouraging results on CAR-bench, where GEAR substantially improves Pass@3 over conventional retrieval methods. The experiments also examine repeated-trial consistency, which is an important but often overlooked aspect of agent reliability.
Critical weaknesses
- The comparison with existing methods is insufficient. Several relevant tool retrieval approaches, such as ToolGen, Tool Graph Retriever, Re-Invoke, and ToolRet, are missing from the experiments. In particular, comparisons with existing graph-based retrieval methods are important to establish the novelty and effectiveness of GEAR.
- The benchmark coverage is limited. The end-to-end evaluation mainly relies on only 50 tasks from CAR-bench in a single domain, while ToolBench only evaluates retrieval quality without actual agent execution. More diverse benchmarks and domains are needed to demonstrate generalizability. W3. The paper needs clearer presentation and stronger experimental analysis. Some methodological details, particularly the construction and contribution of different relation types, are insufficiently explained. Moreover, the paper does not adequately isolate the benefits of relational modeling from simply increasing the number of retrieved tools. The overall presentation could also be better organized to clarify the main contributions.
Questions for authors
- Can the authors include comparisons with more recent tool retrieval methods, particularly existing graph-based approaches?
- How much of GEAR's improvement remains when all methods are constrained to the same shortlist size or token budget?
- Can the authors provide a broader end-to-end evaluation across different domains and more diverse tool ecosystems?
Thank you for the review: the ask for graph-based comparisons, sharpened what the paper now claims about where the relations help. We address each point below; the general response summarises the new controls.
W1 / Q1: COLT is graph-based (its second stage reranks over a learned graph of requests, tools and tool sets) and is compared throughout. Re-Invoke is now added: on CAR-bench it equals similarity search at 12 tools and GEAR at 37; on ToolBench it is the strongest method. COLT and MCP-Zero at 37 tools do not differ significantly from GEAR either.
W2 / Q3: The end-to-end evidence covers one domain, 50 tasks and two small models, ToolBench tests retrieval only, and CAR-bench's catalogue fits in context. The limitations state all of this. On ToolBench, the mined graph's small gain comes from the content of its edges. Please see response to Yj1L for more details on the added benchmarks.
W3 / Q2: We reran every retriever at about 37 tools: at equal size GEAR's lead is not significant (p≥0.18), and all 57 tools are level with GEAR's 37.6. More tools help only until the needed ones are in view. After that, the relations help as run-time checks rather than as a way of choosing tools: with all 57 tools in context, removing the ontology lowers Pass@3 from 0.80 to 0.48 (p<0.0001), whereas giving the same knowledge as text lowers it to 0.62. The contribution now restricts GEAR's difficulty advantage to the 12-tool baselines.
Please see the updated paper for more information.