acceptodds
Under review as a conference paper at ICLR 2027

What Drives LLM Tool Selection in MCP? A Factor-Level Analysis Through Controlled Metadata Interventions

Abstract

In Model Context Protocol (MCP) systems, LLMs select tools from metadata rather than from their implementations. Names and descriptions summarize what a tool does, while the parameter schema specifies the fields a tool accepts and how they are organized. Our analysis of metadata for 367,174 MCP tools reveals substantial schema variation among functionally similar tools. Many of these tools share names and similar descriptions but expose different parameter schemas. This raises a key question: can parameter-schema differences influence tool selection before any call is made? We study this question through controlled metadata interventions across five LLMs and five tasks. Each comparison changes exactly one metadata factor (server name, tool name, description, or parameter schema) while keeping the others fixed. We balance tool positions and use conditional logistic regression to estimate position-adjusted effects. We find that: (1) parameter-schema changes produce the largest and most consistent selection shifts among the tested interventions. Models tend to favor separate top-level parameters over grouped or composite forms. (2) Descriptions that explain parameters also tend to increase selection, while name effects are weaker and less consistent. (3) Targeted comparisons show that schema effects depend on the specific parameter names and on how the request represents the same information. When only one schema satisfies an explicit request requirement, all five models select the compatible tool despite position preferences. These results show that parameter schemas influence not only how tools are called, but also whether they are selected. For MCP developers and evaluators, schema design and tool ordering can therefore affect which implementation is invoked even when functionality is unchanged.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MCP, LLM tool selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YA4gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates what drives LLM tool selection in MCP systems through controlled metadata interventions across five LLMs and five tasks. It examines four metadata factors: server name, tool name, description, and parameter schema, using conditional logistic regression to estimate position-adjusted effects. The results show that parameter-schema changes produce the largest and most consistent selection shifts, while parameter explanations also increase selection and name effects are weaker. The authors conclude that parameter schemas influence not only how tools are called, but also whether they are selected.

Critical strengths
  1. Underexplored Role of Parameter Schemas. The paper highlights the underexplored role of parameter schemas in LLM tool selection, demonstrating that schemas influence not only how tools are called but also whether they are selected.
  2. Controlled Experimental Design. The paper conducts controlled metadata interventions across five LLMs and five tasks, varying one metadata factor at a time while keeping the others fixed. This design helps isolate the effects of individual metadata factors on tool selection.
Critical weaknesses
  1. Limited Comparability of Metadata Interventions. The paper aims to compare the importance of four metadata factors in LLM tool selection. However, the high/low variants differ substantially in nature and magnitude across factors. Therefore, larger selection effects may reflect the specific variants chosen rather than the general importance of the metadata factors.
  2. Insufficient Justification for Experimental Variants. The paper does not provide sufficient quantitative evidence to justify the selection of specific high/low variants. Moreover, naming, structuring, and decomposition are evaluated on different tasks, making it difficult to separate schema variation effects from task-specific effects.
  3. Lack of Comprehensive Validation. Although the paper includes a factorial experiment, it does not directly analyze whether combining favorable metadata characteristics consistently increases the selection probability of a target tool. A dedicated validation experiment would strengthen the findings under combined metadata changes.
  4. Insufficient Presentation of Large-Scale Metadata Analysis. Although the abstract highlights an analysis of 367,174 MCP tools, detailed schema-variation statistics are largely limited to Send Email. The paper lacks systematic comparisons across the studied task clusters and sufficient frequency statistics to justify the selected high/low variants.
  5. Incomplete Reporting of Experimental Results. Although the paper evaluates two, three, and five candidate tools, complete results across all tasks, models, and candidate-set sizes are not presented. The full query sets and trial configurations are also not included in the manuscript, with reproducibility relying on the supplementary materials.
Questions for authors
  1. Comparability of Metadata Interventions. How do the authors ensure that the effects of different metadata factors are comparable when their high/low variants differ in nature and magnitude? Can the observed differences be attributed to the metadata factors themselves rather than the specific variants chosen?
  2. Selection of Experimental Variants. What quantitative criteria were used to select the high/low variants? Additionally, why are naming, structuring, and decomposition evaluated on different tasks rather than within the same tasks?
  3. Comprehensive Validation. Can the authors provide a direct evaluation of whether combining the identified favorable metadata characteristics consistently increases the selection probability of a target tool, beyond the reported factorial analysis?
  4. Connection Between Large-Scale Metadata Analysis and Experimental Design. How does the analysis of 367,174 MCP tools inform the selection of high/low variants and the construction of candidate tool lists? How representative are these experimental tools of real MCP tool collections?
  5. Completeness of Experimental Results. The paper reports three-tool results across all five tasks, but two- and five-tool results are presented only for Send Email and Place Order. Could the authors provide the missing results for Get Weather, Create Issue, and Create Pull Request across all five models?
2 comments

Thank you for the detailed review and the valuable feedback. We address each comment below.

Q1. Comparability of Metadata Interventions.

Q1. Comparability of metadata interventions. Each comparison contrasts clear parameters with less clear ones (to vs. recipient_id, location vs. input, separate fields vs. params_dict or one object, {owner, repo} vs. repository), and our tasks are concrete ecosystem instances of this contrast, so the differences reflect parameter clarity rather than a particular variant. The effect recurs across these edits of different kinds under identical queries and position controls (Table 2), and disappears when an edit keeps the parameter clear (to → receiver, Figure 5a).

Q2. Selection of Experimental Variants.

Q2.1 Criteria. Ecosystem frequency: within each task cluster, v_high is the more common form and v_low a less common one (Appendix B.1); descriptions are coded by completeness.

Q2.2 Why different tasks? Each dimension is tested where it arises: tools with few inputs vary in naming (to vs. recipient_id), tools with several inputs vary in structure (params_dict in Create Issue, one object in Place Order), and tools with a compound input vary in decomposition ({owner, repo} vs. repository in Create Pull Request). Applying all dimensions to one task would create variants that do not occur in practice.

Q3. Comprehensive Validation.

Q3. Comprehensive validation. Yes. We tested this directly on 100 newly written Send Email requests with GPT-5.4 mini: a target tool competes with an all-v_low competitor, and each request is run with the target in both positions.

Target selection rate against an all-v_low competitor (200 trials per configuration).

Target configurationOverallTarget firstTarget second
Baseline (all v_low)50.0%100%0%
Parameter schema only94.5%100%89%
Combined (all v_high)100%100%100%

Q4. Connection Between Large-Scale Metadata Analysis and Experimental Design.

Q4. Connection between large-scale metadata analysis and experimental design. We agree. Table below shows the four clusters beyond Send Email: functionally similar tools spread their inputs over many parameter sets, and on the dimension each task tests, v_high is the dominant form and v_low a less common form observed in deployed servers.

Schema variation among ecosystem records with extracted parameters (Send Email: Figure 1a).

TaskParameter setsMost common setDimensionv_high formv_low form
Get Weather3742.6%Naminglocation (53.3%)Generic name (1.3%)
Create Issue8323.6%StructuringSeparate fields (91.3%)Single container (3.1%)
Place Order584.5%StructuringSeparate fields (79.8%)Single container (14.6%)
Create Pull Request2649.0%DecompositionSeparate fields (78.1%)Combined field (5.2%)

Q5. Completeness of Experimental Results.

Q5. Completeness of experimental results. Yes. We ran the two- and five-tool comparisons for Get Weather, Create Issue, and Create Pull Request with all five models; Table below summarizes the parameter-schema effect. The results match the three-tool estimates in Table 2: with five tools, all 20 estimates exclude zero, and only Qwen reverses under path queries (−0.72), as with three tools (−1.01); with two tools, the v_high schema is selected in more than 75% of trials in 16 of 20 cells, and 12 of the 14 † estimates come from separation.

Parameter-schema effects with two and five tools, β (H/U).

TaskToolsGPTClaudeGeminiDeepSeekQwen
Get Weather22.94 (93%/1%)7.63† (86%/0%)8.05† (79%/0%)1.12 (57%/1%)3.91† (53%/0%)
52.46 (88%/13%)6.09 (93%/1%)2.11 (77%/9%)0.98 (58%/7%)1.73 (60%/3%)
Create Issue211.24† (100%/0%)11.24† (100%/0%)11.24† (100%/0%)3.53 (97%/1%)11.24† (100%/0%)
53.48 (93%/7%)4.35 (93%/5%)8.02 (96%/0.3%)5.27 (93%/2%)12.09 (99.8%/0.1%)
Create PR (split)27.13† (96%/0%)11.25† (100%/0%)11.25† (100%/0%)2.22 (90%/4%)2.22 (90%/4%)
53.20 (91%/8%)5.35 (97%/5%)3.98 (96%/9%)2.64 (85%/8%)5.31 (98%/4%)
Create PR (path)27.44† (90%/0%)7.44† (97%/0%)1.78 (78%/2%)0.57† (64%/16%)0.57† (64%/16%)
51.47 (75%/15%)1.82 (79%/15%)1.58 (76%/15%)0.64 (59%/14%)−0.72 (38%/14%)

H: v_high selection rate. U: rate of choices that neither metadata nor the model's most-selected position explains. Separation occurs when H = 100% or U = 0. † 95% CI includes zero.

Reviewer_YA4gOpenReview ↗

Response to the Authors' Rebuttal

Thank you for the clarifications and additional experiments in the rebuttal; however, my main concerns remain unresolved, and the responses raise an additional question.

  1. Intervention Comparability. The authors attribute schema effects to parameter clarity, but this does not address my original concern about comparing S/T/D/P interventions of different semantic magnitudes; moreover, the to vs. receiver experiment itself demonstrates that the observed effects depend substantially on the specific variants selected.

  2. Experimental Design and Generalizability. The authors argue that testing multiple schema dimensions within the same task would produce unrealistic variants, yet the factorial and additional validation experiments also construct synthetic metadata combinations; why are these designs considered acceptable while cross-dimensional schema comparisons are not?

  3. New Question: Ecosystem Frequency. Since high/low variants are selected based on their frequency in the MCP ecosystem, why not systematically compare the most frequent forms against multiple less frequent alternatives to examine whether selection preference is associated with ecosystem frequency, rather than relying on a single selected high/low contrast?

Overall, the rebuttal has not sufficiently addressed my concerns about whether the selected interventions support the paper's broader conclusions regarding the relative importance of metadata factors. At this stage, I am inclined to maintain my original rating of Weak Reject.

Reviewer_ntMQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines why LLMs choose one MCP tool over another when the tools have similar functions. The authors vary server names, tool names, descriptions, and parameter schemas, while controlling for tool order through balanced positions and conditional logistic regression. Across five LLMs and five tasks, schema changes have the largest and most consistent effects among the tested edits. Follow-up experiments show how these preferences change with parameter names, request wording, and explicit task requirements. The study evaluates which tool is selected, without executing it.

Critical strengths
  1. The controlled comparisons make the results easier to interpret. The authors change one metadata field at a time, use the same requests, and balance tool positions. This helps separate the effect of metadata from the tendency to choose a tool based on its position, giving more convincing evidence than comparisons between tools that differ in several ways.
  2. The follow-up experiments help explain when schema preferences arise. Tests of parameter names, separate versus combined repository inputs, and explicit CC requirements show that preferences depend on how the schema matches the request. These experiments go beyond showing that metadata matters and help explain why the models choose particular tools.
Critical weaknesses
  1. The claim that schemas have the strongest effect is difficult to generalize from the chosen edits. The interventions differ in both size and meaning: renaming "to" as "recipient_id", for example, may suggest a different expected input, whereas some name changes convey little new information. The experiments establish effects for these particular edits, but provide a weaker basis for comparing the importance of metadata fields more broadly.
  2. The same queries are reused across many candidate configurations, but the reported Wald intervals do not appear to account for clustering by query. Complete or quasi-complete separation also makes some coefficient estimates unstable. Query-clustered or hierarchical analyses would help determine how much of the reported significance survives these dependencies.
Questions for authors
  1. How were the edits chosen for each metadata field? Were they matched in semantic change or information content? Since "to" versus "recipient_id" may suggest different input types, how much of the larger schema effect could come from the specific edits rather than the field itself?
  2. Do the reported confidence intervals account for the same queries appearing across multiple candidate configurations? If not, do the main findings hold when uncertainty is estimated by clustering or resampling at the query level?
  3. Clarification: For comparisons with complete or quasi-complete separation, what evidence beyond coefficient magnitude supports the ranking of metadata effects? Are the conclusions unchanged when those comparisons are excluded?
3 comments

We appreciate the reviewer's thoughtful comments and helpful suggestions. Our responses to each point follow.

Q1. How were the edits chosen for each metadata field? Were they matched in semantic change or information content? Since "to" versus "recipient_id" may suggest different input types, how much of the larger schema effect could come from the specific edits rather than the field itself?

Q1.1 How were the edits chosen? By ecosystem frequency. Within each task cluster, v_high is the more common observed form and v_low a less common one, both counted by the same rules (Appendices A.1, B.1). Descriptions are coded by completeness.

Q1.2 Were they matched in semantic change or information content? No. They are matched in origin: both variants of every field are forms observed in the ecosystem, since semantic change has no common scale across names, descriptions, and schemas.

Q1.3 How much of the schema effect comes from the specific edit? For naming, the effect depends on what the new name signals about the input. Selection rate of the renamed tool (Figure 5a; 50% = no preference):

  • to → recipient_id (suggests an identifier): 1.0%–10.0%
  • to → receiver (same role): 35.2%–51.0%
  • body → text (same role): 45.5%–54.6%

The recipient_id edit reflects real naming practice, not a different input type. For example, Gmail-MCP-Server exposes send_email(recipient_id, subject, message) and writes recipient_id directly into the email's To header, so the parameter takes an email address. Beyond naming, the structuring and decomposition edits in Place Order and Create Pull Request change no input type and also shift selection (Table 2), so the schema effect does not rest on this edit alone. Because the parameter schema shows how a tool will be invoked, models favor tools whose parameters are clear: meaningful names such as to over recipient_id, and separate parameters over an undeclared params_dict.

Q2. Do the reported confidence intervals account for the same queries appearing across multiple candidate configurations? If not, do the main findings hold when uncertainty is estimated by clustering or resampling at the query level?

Q2.1 Do the reported intervals account for repeated queries? No. They are model-based Wald intervals that treat trials as independent.

Q2.2 Do the main findings hold under query-level clustering? Yes. We recomputed the intervals with standard errors clustered by query. All 41 intervals that excluded zero continue to exclude zero, so the main findings hold.

Estimates whose 95% intervals exclude zero, before and after clustering by query.

TaskSignificant (original)Remain significant (clustered)Clustered / original SENewly significant (clustered)
Send Email11110.43–1.761
Get Weather14140.76–1.610
Create Issue10101.07–1.871
Place Order660.83–1.640
All41410.43–1.872

Clustered / original SE: range across the originally significant estimates. Newly significant: estimates whose interval includes zero originally but excludes zero after clustering.

Clarification. For comparisons with complete or quasi-complete separation, what evidence beyond coefficient magnitude supports the ranking of metadata effects? Are the conclusions unchanged when those comparisons are excluded?

Q3.1 What evidence beyond coefficient magnitude supports the ranking? Selection rates. Separation occurs when metadata and position together account for every choice, so no finite estimate exists. We therefore report two rates that describe behavior directly: the v_high selection rate H (Section 3.3) and the unexplained selection rate U, the rate of choices that neither cue explains, i.e., skipping a v_high tool in the model's most-selected position or selecting a v_low tool elsewhere. Choices counted in U hold β finite; separation occurs when U = 0 or H = 100%. The table below reports both rates for Create Issue.

Create Issue, three tools: β (H/U).

ModelS (H/U)T (H/U)D (H/U)P (H/U)
GPT-5.4 mini7.12 (97%/1%)0.01† (50%/11%)2.35 (74%/3%)16.23† (99.7%/0%)
Claude Sonnet 4.60.20† (51%/2%)0.16† (51%/5%)1.72 (62%/3%)6.32 (98%/2%)
Gemini 3.5 Flash−2.29 (12%/15%)0.23† (55%/28%)2.36 (89%/15%)sep∗ (100%/3%)
DeepSeek V4 Pro2.81 (59%/0.3%)0.12† (51%/5%)1.71 (70%/6%)4.84 (94%/2%)
Qwen3.7-Plussep (55%/0%)0.20† (51%/3%)4.74 (92%/1%)sep∗ (100%/0%)

Q3.2 Are the conclusions unchanged when those comparisons are excluded? Yes. Our ranking uses only estimates whose intervals exclude zero, so it already omits every separated or near-separated comparison.

Thank you for clarifying.

Q1: Semantic change and information content were not matched across fields, it remains unclear whether the larger schema effects reflect the field itself or the particular edits chosen. The conclusion should therefore be limited to the tested interventions.

Q2: Does the query-clustered analysis also cover Create Pull Request? Neither query format appears in the table, so I cannot tell whether the robustness check covers all the main results.

Q3: My remaining concern is the ranking across fields, rather than whether strong preferences exist. Which comparisons were retained after excluding separated and near-separated cases, and does the same ranking hold for that subset?

Thank you much very for the reply. Considering the quality of the manuscript, I decide to maintain the score.

Reviewer_gWHBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how metadata affects LLM tool selection in MCP environments. The authors examine four factors: server names, tool names, descriptions, and parameter schemas. Across five LLMs and five tasks, using conditional logistic regression to account for positional effects. They find that parameter schemas generally have the strongest influence on tool selection, although the effects vary with parameter naming and request formatting.

I find the question interesting, but I have some concerns about the experimental design and the broader significance of the findings.

Critical strengths
  1. Interesting empirical question. The paper examines the influence of parameter schemas, which has received relatively less attention in previous tool-selection studies. I think this is a worthwhile direction.

  2. Reasonable experimental coverage. The evaluation includes multiple LLMs, tasks, and candidate-set sizes. I particularly appreciate the additional experiments on parameter naming and request formatting.

Critical weaknesses
  1. Insufficient isolation of metadata effects. In the Create Issue experiment, the flattened schema explicitly defines required fields, while the dictionary-based alternative does not. The observed preference may therefore reflect differences in information clarity rather than parameter structure. Similarly, descriptions are adapted to schemas in the factorial experiment, making it difficult to isolate individual effects and compare their relative importance.

  2. Limited generalizability and contribution. Despite the large number of trials, the evaluation relies on only five tasks and a small set of generated queries, with some concerns about repeated observations and near-separated estimates. More importantly, the experiments do not involve actual tool execution. I find the results interesting, but their practical significance remains unclear, and the paper offers limited conceptual insight beyond the reported empirical observations.

Questions for authors

Q1. Would the Create Issue results remain similar if both schemas specified exactly the same fields, types, and constraints?

Q2. Have the authors considered query-clustered uncertainty estimates or alternative methods for handling near-separation? How would schema–description interactions affect the factorial results?

Q3. Do these preferences persist in realistic MCP settings, and do they affect downstream task success?

1 comment

We thank the reviewer for the careful reading and the constructive questions. Below, we respond to each question.

Q1. Would the Create Issue results remain similar if both schemas specified exactly the same fields, types, and constraints?

A1. Yes. In Create Issue, the grouped schema passes all inputs through params_dict, a single parameter whose contents are left undeclared (Appendix B.3.3). An object parameter can declare its inner fields, and Place Order tests exactly this case: both schemas specify identical fields, types, and constraints, either at the top level or grouped under one object (Appendix B.3.4), and most models favor the top-level schema there as well (Table 2). Because the parameter schema shows how a tool will be invoked, models favor the tool whose parameters match the request most directly.

Q2. Have the authors considered query-clustered uncertainty estimates or alternative methods for handling near-separation? How would schema–description interactions affect the factorial results?

Alternative methods. Separation occurs when metadata and position together account for every choice: the likelihood then keeps increasing as their coefficients grow, so no finite estimate exists. As an alternative, we report two selection rates that describe the model's behavior directly: the v_high selection rate H (Section 3.3), and the unexplained selection rate U, the rate of choices that neither cue explains, i.e., skipping a v_high tool in the position the model selects most often, or selecting a v_low tool in another position. When U = 0, every choice follows at least one cue, so β and the position coefficient can grow together without limit and separation occurs; when U > 0, the choices that follow neither cue become less likely as both coefficients grow, so they hold β at a finite value. Separation also occurs when H = 100%, where β alone can grow without limit. The table below reports both rates for Create Issue.

Create Issue, three tools: β (H/U).

ModelS (H/U)T (H/U)D (H/U)P (H/U)
GPT-5.4 mini7.12 (97%/1%)0.01† (50%/11%)2.35 (74%/3%)16.23† (99.7%/0%)
Claude Sonnet 4.60.20† (51%/2%)0.16† (51%/5%)1.72 (62%/3%)6.32 (98%/2%)
Gemini 3.5 Flash−2.29 (12%/15%)0.23† (55%/28%)2.36 (89%/15%)sep∗ (100%/3%)
DeepSeek V4 Pro2.81 (59%/0.3%)0.12† (51%/5%)1.71 (70%/6%)4.84 (94%/2%)
Qwen3.7-Plussep (55%/0%)0.20† (51%/3%)4.74 (92%/1%)sep∗ (100%/0%)

Schema–description interaction. It makes the reported P coefficients conservative. In the factorial experiment (Send Email), the detailed description lists the parameters of the schema it accompanies, so when it is assigned to the {recipient_id, message} tool, it explains those parameters and removes part of that tool's disadvantage. Appendix C.4 isolates this interaction for the same schema contrast (Figure 8, "Both names"): the selection rate of the {to, body} tool drops from 82.8%–99.0% without field descriptions to 50.0%–62.0% with identical ones. P is the largest effect for most models despite this reduction (Section 5.3).

Q3. Do these preferences persist in realistic MCP settings, and do they affect downstream task success?

A3. Yes. We replicated the two-tool Send Email and Place Order comparisons in Cline, a deployed MCP client, using Claude Sonnet 4.6 without extended thinking; only the parameter schema differed, and the selected tools were executed on 20 requests per task.

Tool selection and task success in Cline (20 requests per task).

Taskv_high vs. v_low schemav_highBothv_lowUser askedTask success
Send Email{to, subject, body} vs. {recipient_id, subject, message}17030100%
Place Orderseparate parameters vs. one object16202100%

User asked: Cline asked the user to choose which tool to call.

The preferences persist in realistic MCP settings, and they do not affect downstream task success: every executed call completed the task correctly.

Comments on the paper

1 comment
Area_Chair_FXF3OpenReview ↗

Entering Last 24 hours of the reviewer discussion period

Dear Authors and Reviewers

Just a reminder we are soon to be entering the last 24 hours of the reviewer discussion period.

Authors: If you have not yet responded to the reviews and intend to please do so ASAP. Please note reviewers will most likely not have time to respond to any late rebuttal comments so please make sure any comments are clear and not in need of further clarification.

Reviewers: If the authors have responded to your reviews and you have not yet responded please do so, even if it is a one line response acknowledging you have read the rebuttal.

Thanks for all your time, and I hope you all learnt something interesting during the process.

Kind regards,

Your ICLR Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.