- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates what drives LLM tool selection in MCP systems through controlled metadata interventions across five LLMs and five tasks. It examines four metadata factors: server name, tool name, description, and parameter schema, using conditional logistic regression to estimate position-adjusted effects. The results show that parameter-schema changes produce the largest and most consistent selection shifts, while parameter explanations also increase selection and name effects are weaker. The authors conclude that parameter schemas influence not only how tools are called, but also whether they are selected.
Critical strengths
- Underexplored Role of Parameter Schemas. The paper highlights the underexplored role of parameter schemas in LLM tool selection, demonstrating that schemas influence not only how tools are called but also whether they are selected.
- Controlled Experimental Design. The paper conducts controlled metadata interventions across five LLMs and five tasks, varying one metadata factor at a time while keeping the others fixed. This design helps isolate the effects of individual metadata factors on tool selection.
Critical weaknesses
- Limited Comparability of Metadata Interventions. The paper aims to compare the importance of four metadata factors in LLM tool selection. However, the high/low variants differ substantially in nature and magnitude across factors. Therefore, larger selection effects may reflect the specific variants chosen rather than the general importance of the metadata factors.
- Insufficient Justification for Experimental Variants. The paper does not provide sufficient quantitative evidence to justify the selection of specific high/low variants. Moreover, naming, structuring, and decomposition are evaluated on different tasks, making it difficult to separate schema variation effects from task-specific effects.
- Lack of Comprehensive Validation. Although the paper includes a factorial experiment, it does not directly analyze whether combining favorable metadata characteristics consistently increases the selection probability of a target tool. A dedicated validation experiment would strengthen the findings under combined metadata changes.
- Insufficient Presentation of Large-Scale Metadata Analysis. Although the abstract highlights an analysis of 367,174 MCP tools, detailed schema-variation statistics are largely limited to Send Email. The paper lacks systematic comparisons across the studied task clusters and sufficient frequency statistics to justify the selected high/low variants.
- Incomplete Reporting of Experimental Results. Although the paper evaluates two, three, and five candidate tools, complete results across all tasks, models, and candidate-set sizes are not presented. The full query sets and trial configurations are also not included in the manuscript, with reproducibility relying on the supplementary materials.
Questions for authors
- Comparability of Metadata Interventions. How do the authors ensure that the effects of different metadata factors are comparable when their high/low variants differ in nature and magnitude? Can the observed differences be attributed to the metadata factors themselves rather than the specific variants chosen?
- Selection of Experimental Variants. What quantitative criteria were used to select the high/low variants? Additionally, why are naming, structuring, and decomposition evaluated on different tasks rather than within the same tasks?
- Comprehensive Validation. Can the authors provide a direct evaluation of whether combining the identified favorable metadata characteristics consistently increases the selection probability of a target tool, beyond the reported factorial analysis?
- Connection Between Large-Scale Metadata Analysis and Experimental Design. How does the analysis of 367,174 MCP tools inform the selection of high/low variants and the construction of candidate tool lists? How representative are these experimental tools of real MCP tool collections?
- Completeness of Experimental Results. The paper reports three-tool results across all five tasks, but two- and five-tool results are presented only for Send Email and Place Order. Could the authors provide the missing results for Get Weather, Create Issue, and Create Pull Request across all five models?
2 comments
Response to the Authors' Rebuttal
Thank you for the clarifications and additional experiments in the rebuttal; however, my main concerns remain unresolved, and the responses raise an additional question.
-
Intervention Comparability. The authors attribute schema effects to parameter clarity, but this does not address my original concern about comparing S/T/D/P interventions of different semantic magnitudes; moreover, the to vs. receiver experiment itself demonstrates that the observed effects depend substantially on the specific variants selected.
-
Experimental Design and Generalizability. The authors argue that testing multiple schema dimensions within the same task would produce unrealistic variants, yet the factorial and additional validation experiments also construct synthetic metadata combinations; why are these designs considered acceptable while cross-dimensional schema comparisons are not?
-
New Question: Ecosystem Frequency. Since high/low variants are selected based on their frequency in the MCP ecosystem, why not systematically compare the most frequent forms against multiple less frequent alternatives to examine whether selection preference is associated with ecosystem frequency, rather than relying on a single selected high/low contrast?
Overall, the rebuttal has not sufficiently addressed my concerns about whether the selected interventions support the paper's broader conclusions regarding the relative importance of metadata factors. At this stage, I am inclined to maintain my original rating of Weak Reject.
Thank you for the detailed review and the valuable feedback. We address each comment below.
Q1. Comparability of metadata interventions. Each comparison contrasts clear parameters with less clear ones (
tovs.recipient_id,locationvs.input, separate fields vs.params_dictor one object,{owner, repo}vs.repository), and our tasks are concrete ecosystem instances of this contrast, so the differences reflect parameter clarity rather than a particular variant. The effect recurs across these edits of different kinds under identical queries and position controls (Table 2), and disappears when an edit keeps the parameter clear (to→receiver, Figure 5a).Q2.1 Criteria. Ecosystem frequency: within each task cluster, v_high is the more common form and v_low a less common one (Appendix B.1); descriptions are coded by completeness.
Q2.2 Why different tasks? Each dimension is tested where it arises: tools with few inputs vary in naming (
tovs.recipient_id), tools with several inputs vary in structure (params_dictin Create Issue, one object in Place Order), and tools with a compound input vary in decomposition ({owner, repo}vs.repositoryin Create Pull Request). Applying all dimensions to one task would create variants that do not occur in practice.Q3. Comprehensive validation. Yes. We tested this directly on 100 newly written Send Email requests with GPT-5.4 mini: a target tool competes with an all-v_low competitor, and each request is run with the target in both positions.
Target selection rate against an all-v_low competitor (200 trials per configuration).
Q4. Connection between large-scale metadata analysis and experimental design. We agree. Table below shows the four clusters beyond Send Email: functionally similar tools spread their inputs over many parameter sets, and on the dimension each task tests, v_high is the dominant form and v_low a less common form observed in deployed servers.
Schema variation among ecosystem records with extracted parameters (Send Email: Figure 1a).
location(53.3%)Q5. Completeness of experimental results. Yes. We ran the two- and five-tool comparisons for Get Weather, Create Issue, and Create Pull Request with all five models; Table below summarizes the parameter-schema effect. The results match the three-tool estimates in Table 2: with five tools, all 20 estimates exclude zero, and only Qwen reverses under path queries (−0.72), as with three tools (−1.01); with two tools, the v_high schema is selected in more than 75% of trials in 16 of 20 cells, and 12 of the 14 † estimates come from separation.
Parameter-schema effects with two and five tools, β (H/U).
H: v_high selection rate. U: rate of choices that neither metadata nor the model's most-selected position explains. Separation occurs when H = 100% or U = 0. † 95% CI includes zero.