acceptodds
Under review as a conference paper at ICLR 2027

GatewayBench: Unified Capability and Security Evaluation Platform for MCP Gateways

Abstract

AI agents have increasingly interacted with external environments by invoking tools, with Model Context Protocol (MCP) emerging as a standard interface for connecting agents to heterogeneous tools and services. As organizations deploy agents at scale, MCP gateways have become a critical middleware layer for aggregating, governing, and shaping the tool surface exposed to agents. However, gateway design can substantially affect agent behavior: as the number of available tools grows, agents may struggle with tool discovery and selection, leading to degraded task performance, increased computational overhead, and potentially greater exposure to malicious tools. Despite the growing importance of MCP gateways in agentic systems, no prior work has systematically studied how gateway design affects agent capability, efficiency, and security. To address this gap, we introduce GatewayBench, the first unified evaluation framework and benchmark for MCP gateways. We characterize gateway architectures into two major paradigms: , which expose the complete set of available tools to the agent upfront, and , which provide tool-discovery meta-tools that allow agents to selectively retrieve relevant tools on demand. GatewayBench comprises 524 tasks across six domains, together with a pool of 11,118 real-world tools from 251 authentic MCP servers. To emulate large-scale deployments, we systematically inject task-irrelevant tools sampled from this pool into each agent environment, enabling controlled evaluation as the accessible tool surface scales. Each task is evaluated using verifiable judges that inspect the resulting environment state quantitatively. Based on GatewayBench, we conduct extensive experiments across seven widely deployed MCP gateways and diverse agent configurations, covering both benign and adversarial tasks. Our results reveal substantial differences across gateway designs in task completion, tool-use efficiency, and resilience to attacks, and further demonstrate that these effects interact strongly with the underlying model and agent harness. These findings highlight the MCP gateway as a consequential yet underexplored component of the agentic AI stack. By enabling controlled evaluation of gateway architectures under realistic tool ecosystems, GatewayBench provides both practical guidance for MCP gateway design and a rigorous testbed for developing more capable, efficient, and secure agentic systems.

Primary area:
datasets and benchmarks
Keywords:
MCP Gateway, Agent Evaluation, Agent Security

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NzxDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GatewayBench, a framework for evaluating how MCP gateway design affects agent capability, efficiency, and security. It compares seven gateway implementations spanning flat tool exposure and hierarchical, on-demand discovery, using tasks from DTAP and MCP-Universe and distractor tools collected from 251 real MCP servers. The results show that hierarchical gateways substantially reduce context overhead as the tool set grows, while their task performance depends on the model’s ability to discover relevant tools. Lower indirect attack success rates are observed primarily for tool injection. The main contribution is a common evaluation framework and an empirical study of how gateway choices affect agent behavior.

Critical strengths
  1. The paper studies the MCP gateway that deserves evaluation in its own right. Comparing seven gateway implementations under a common set of tasks and metrics helps identify how the tool access layer affects task completion, interaction cost, and security. These results have practical value for developers choosing among gateway designs, and the framework provides a useful basis for subsequent comparisons.

  2. The security analysis explicitly considers the possibility that lower attack success reflects reduced task competence. The authors acknowledge that tool-discovery failures can make weaker models appear more secure. Comparing benign and attacked performance is a useful step toward separating security gains from a general loss of capability.

Critical weaknesses
  1. The security comparison does not sufficiently separate gateway architecture from implementation-specific differences in information exposure. The paper leaves several details of the threat model unclear, including which fields an attacker can modify, the relationship between malicious tools and tasks, and how payloads are adapted for different discovery interfaces. Moreover, Bifrost’s description truncation limits exposure to malicious content, thereby reducing the ASR. However, the current evaluation does not establish how much of the observed reduction comes from selective discovery and how much comes from truncation or other interface choices. These confounds limit the support for architecture-level security recommendations.

  2. The task-performance crossover at scale lacks robustness evidence. The distractors exclude semantically overlapping tools and uniformly return errors, leaving ambiguity among related tools untested. Moreover, the main experiments use fixed tool subsets without repeated sampling, provides limited evidence for a reliable crossover.

Questions for authors
  1. Please clarify the threat model: Which fields can the attacker modify in the tool-injection experiments? Is the payload placed in a task-relevant tool or an additional tool? How is the same attack presented through each gateway’s search, documentation, and truncation mechanisms, and are payloads adapted to individual interfaces?

  2. Could you report clean task success on the same instances used for indirect attacks, together with payload exposure rates and ASR conditional on exposure?

  3. How stable is the crossover in Table 3 across different distractor subsets?

3 comments

Dear Reviewer NzxD,

Thank you for your constructive comments and valuable suggestions. We address your concerns below.

W1 & Q1: Please clarify the threat model: Which fields can the attacker modify in the tool-injection experiments? Is the payload placed in a task-relevant tool or an additional tool? How is the same attack presented through each gateway’s search, documentation, and truncation mechanisms, and are payloads adapted to individual interfaces?

The malicious tasks are divided into direct and indirect attacks. Direct attacks modify the user prompt to include a malicious instruction, and indirect attacks contains both tool injection and environment injection attacks.

For environment injection, the attacker injects a malicious instruction into the database of an existing business tool, such as an email, Slack message, CRM note, or travel record.

For tool injection, the attacker modifies only the description field of an existing tool. The tool name, input schema, and implementation are unchanged, and the attacker does not add a new tool. The attack only targets a tool already required by the task.

Within each task, all gateways receive the same payload configuration, and we do not rewrite payloads for individual gateways. Gateways differ solely in how they present tool metadata. For example, a flattened gateway exposes the complete modified description at the beginning of the conversation, whereas a hierarchical gateway exposes the modified description only when the discovery of the tool is triggered.

W1: Moreover, Bifrost’s description truncation limits exposure to malicious content, thereby reducing the ASR. However, the current evaluation does not establish how much of the observed reduction comes from selective discovery and how much comes from truncation or other interface choices. These confounds limit the support for architecture-level security recommendations. Q2: Could you report clean task success on the same instances used for indirect attacks, together with payload exposure rates and ASR conditional on exposure?

To evaluate clean task success on indirect attack tasks, we restrict the analysis to 24 indirect tasks that have a broadly matched clean task. As shown in Table r.1, we report both clean task success and attack success for each gateway. The clean task success matches our reported benign task results in the paper.

Table r.1: Clean task success rate and Attack Success Rate (ASR) across gateways on matched indirect attack instances.

GatewayGPT-5.1 clean task successGPT-5.1 ASRGPT-5.4 clean task successGPT-5.4 ASR
AgentCore71.9%39.6%87.5%39.6%
AgentGateway76.6%47.9%85.4%32.3%
Bifrost46.9%12.5%84.4%31.2%
Enkrypt26.0%7.3%88.5%26.0%
LiteLLM70.2%39.4%83.3%32.3%
Lunar68.5%44.6%86.5%35.4%
Strata40.6%28.1%72.9%27.1%

We further report payload exposure and ASR conditional on exposure for two hierarchical gateways in Table r.2. A run is marked as exposed if the malicious payload is present in the agent's trace.

Table r.2: Payload exposure rate, overall ASR, and ASR conditional on exposure for hierarchical gateways.

GatewayPayload exposureASRASR given exposure
Bifrost42.6%16.2%37.1%
Strata70.2%27.9%39.8%

These results show that Bifrost has substantially lower payload exposure than Strata, while the ASR conditional on exposure is comparable. For Bifrost, 259 out of 272 runs (95.2%) returned the attacked tool's definition, but 214 out of 259 descriptions (82.6%) were truncated. Although the agent may access the full tool description through another tool, the attack payload is not exposed in 143 out of 259 runs (55.2%) where the attacked tool's definition is returned.

W2 & Q3: How stable is the crossover in Table 3 across different distractor subsets?

We investigate how frequently the model actually invokes a noisy tool. Table r.3 reports the percentage of GPT-5.4 runs containing at least one call to an irrelevant noisy tool.

Table r.3: Percentage of GPT-5.4 runs containing at least one call to an irrelevant noisy tool across different tool-set sizes.

GatewayN=500N=1,000N=1,500
AgentCore0.22%0.22%4.44%
AgentGateway0%0%0%
Bifrost0.44%0%0.22%
Enkrypt0%0%0%
LiteLLM0.44%0.22%5.56%
Lunar0%0.22%7.11%
Strata0%0%0%

The results demonstrate that the model rarely calls a noisy tool. Additionally, the noisy tool set is sufficiently large and diverse, so we assume that the choice of noisy tools does not significantly affect the model's behavior.

We acknowledge that the performance drop of flattened gateways is small even at large tool-set sizes, and the crossover between flattened and hierarchical gateways is not sufficiently pronounced to support a strong architecture-level conclusion regarding TSR alone. However, we note that the excessive token cost of flattened gateways is a more concerning factor than TSR when the tool-set size is large. Therefore, we still conclude that hierarchical gateways are more efficient than flattened gateways for large tool-set sizes.

Thank you for the detailed clarifications and additional results. It remains unclear whether the crossover in TSR holds across different subsets of distractor tools. I suggest presenting it as an observation specific to the setting tested. However, I do not think this limitation is serious enough to negate the contribution of this benchmark study. I will raise my overall rating to 3.

Reviewer_f9YROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GatewayBench, a benchmark that evaluates seven MCP gateways on task completion, security, and interaction cost using 524 tasks and authentic tool interfaces served as mock distractors. The results show substantial differences across implementations and interactions with model capability, agent harness, and tool-set size.

Critical strengths

The benchmark evaluates an important deployment component through execution-based outcomes, revealing trade-offs that model-only evaluations would miss. The large differences in context volume are practically useful.

Critical weaknesses
  1. The main interpretive issue is architectural attribution. Default products differ in discovery, metadata, execution, and failure handling. The main claims should reflect the evaluated implementations. This can be addressed by narrowing the language without requiring a new causal experiment.
  2. Table 4 needs an aggregation rule: none of its nine TSR values is attainable as a rounded integer success count out of 74. Domain-macro averaging could explain this.
  3. The GPT-5.4 capability crossover is small and based on one run: a 2.1-point flattened lead becomes a 2.3-point deficit (Table 3). Paired task-level uncertainty would clarify stability. The exploratory real-execution study in Appendix B.1 is useful but does not test this model and tool-count regime.
  4. Figure 5 reports tokens per turn, not per task. The context-volume and latency findings are useful, but precise total-cost claims need turn counts and cache accounting.
Questions for authors
  1. Can existing traces show how discovery and truncation affect malicious-content exposure?
  2. Can you clarify Table 4's evaluated counts and aggregation, and provide paired task-level support for the small capability crossover?
1 comment

Dear Reviewer f9YR,

Thank you for your constructive comments and valuable suggestions. We address your concerns below.

W1: The main interpretive issue is architectural attribution. Default products differ in discovery, metadata, execution, and failure handling. The main claims should reflect the evaluated implementations. This can be addressed by narrowing the language without requiring a new causal experiment.

Thank you for the constructive comment. We fully agree that implementation-specific details influence performance, and we have provided dedicated qualitative analyses of these individual behaviors in Appendix B.4.

At the same time, we believe our evaluation supports broader architectural insights. By systematically testing a diverse pool of seven widely deployed gateways spanning multiple products per paradigm, the consistent macro-level trends such as token scaling behavior and tool-injection resilience demonstrate fundamental trade-offs between the two architectural designs.

W2 & Q2: Table 4 needs an aggregation rule: none of its nine TSR values is attainable as a rounded integer success count out of 74. Domain-macro averaging could explain this.

Thank you for pointing this out. The TSR values are macro-averaged across the three distinct domains within the MCP-Universe task pool rather than computed as a raw micro-averaged integer count out of the total 74 tasks. We will update the caption and text of Table 4 in the revision to explicitly state this domain-macro-averaging rule.

W3: The GPT-5.4 capability crossover is small and based on one run: a 2.1-point flattened lead becomes a 2.3-point deficit (Table 3). Paired task-level uncertainty would clarify stability. The exploratory real-execution study in Appendix B.1 is useful but does not test this model and tool-count regime.

We investigate how frequently the model actually invokes a noisy tool. Table r.1 reports the percentage of GPT-5.4 runs containing at least one call to an irrelevant noisy tool.

Table r.1: Percentage of GPT-5.4 runs containing at least one call to an irrelevant noisy tool across different tool-set sizes (NN).

GatewayN=500N=1,000N=1,500
AgentCore0.22%0.22%4.44%
AgentGateway0%0%0%
Bifrost0.44%0%0.22%
Enkrypt0%0%0%
LiteLLM0.44%0.22%5.56%
Lunar0%0.22%7.11%
Strata0%0%0%

The results demonstrate that the model rarely calls a noisy tool. Additionally, the noisy tool set is sufficiently large and diverse, so we assume that the choice of noisy tools does not significantly affect the model's behavior.

We acknowledge that the performance drop of flattened gateways is small even at large tool-set sizes, and the crossover between flattened and hierarchical gateways is not sufficiently pronounced to support a strong architecture-level conclusion regarding TSR alone. However, we note that the excessive token cost of flattened gateways is a more concerning factor than TSR when the tool-set size is large. Therefore, we still conclude that hierarchical gateways are more efficient than flattened gateways for large tool-set sizes. We will soften our claims regarding the capability crossover in the revision, presenting it as a qualitative trend rather than a definitive statistical threshold.

W4: Figure 5 reports tokens per turn, not per task. The context-volume and latency findings are useful, but precise total-cost claims need turn counts and cache accounting.

We report tokens per turn in Figure 5 specifically to highlight the high context overhead of flattened gateways when the tool-set size becomes large. Although hierarchical gateways require more interaction turns due to additional discovery steps, this turn-count overhead is minor compared to the linear growth in per-turn token consumption experienced by flattened gateways.

Q1: Can existing traces show how discovery and truncation affect malicious-content exposure?

We report payload exposure and ASR conditional on exposure for two hierarchical gateways in Table r.2. A run is marked as exposed if the malicious payload is present in the agent's trace.

Table r.2: Payload exposure rate, overall ASR, and ASR conditional on exposure for hierarchical gateways.

GatewayPayload exposureASRASR given exposure
Bifrost42.6%16.2%37.1%
Strata70.2%27.9%39.8%

These results show that Bifrost has substantially lower payload exposure than Strata, while the ASR conditional on exposure is comparable. For Bifrost, 259 out of 272 runs (95.2%) returned the attacked tool's definition, but 214 out of 259 descriptions (82.6%) were truncated. Although the agent may access the full tool description through another tool, the attack payload is not exposed in 143 out of 259 runs (55.2%) where the attacked tool's definition is returned.

Reviewer_PVAYOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper suggests a benchmark for testing the performance of an agent given different MCP gateway designs. They found that the MCP should be designed together with the harness and the model. That is, there is no single design that fits all better agents and models .

Critical strengths

No substantial strength can be identified. Testing real-world MCPs, with SOTA agents and models over the created benchmark, is powerful.

Critical weaknesses

Many harnesses today handle the tools inside the harness and implement a specific tool shortlist, for example via cosine similarity, RAG techniques, and even a separate node in the graph that is in charge of this. This enables them to manage the tools, organize them, and even expand their description or output schema when they do not exist (offline and online). I am struggling to see the novelty of the paper.

Questions for authors
  1. Given what I wrote in the weaknesses section, please help me understand the novelty of the paper
  2. Did you test an agent with a shortlister?
  3. What would your results be with small open-source models?
3 comments

Dear Reviewer PVAY,

Thank you for your valuable feedback and thoughtful questions. We would like to summarize the novelty and scope of our work and address your concerns below point by point.

W1 & Q1. Novelty of GatewayBench and existing tool-shortlisting mechanisms

Thank you for raising this important point. We fully agree that various tool-management mechanisms, including tool shortlisting, semantic similarity search, RAG-based retrieval, and hierarchical tool organization, have already been explored in existing agent systems. In fact, the diversity of these existing mechanisms is precisely one of the key motivations for introducing GatewayBench: to systematically evaluate their effectiveness in real-world MCP gateway implementations.

We would like to emphasize that our contribution is not to propose a new gateway or tool-discovery algorithm, but rather to introduce GatewayBench, a unified evaluation platform and benchmark specifically designed to assess the capability, efficiency, and security of MCP gateways.

As a matter of fact, our benchmark already covers several of the mechanisms mentioned by the reviewer. For example, Strata implements semantic search-based tool discovery, AgentCore provides semantic tool search, Enkrypt adopts layered discovery, and Bifrost organizes tools through a virtual filesystem abstraction. These mechanisms are discussed in Section 3 and Appendix B.4 of our paper. We also discuss prior tool-discovery approaches, including RAG-MCP (Gan & Sun, 2025), MCP-Zero (Fei et al., 2025), ScaleMCP (Lumer et al., 2025), and Dynamic ReAct (Gaurav et al., 2025), in Section 2.

While prior works have proposed various tool-management and retrieval mechanisms, there has been a lack of systematic evaluation of how existing MCP gateway implementations affect agent capability, efficiency, and security under a unified experimental setting. To the best of our knowledge, GatewayBench is the first benchmark to systematically evaluate MCP gateways across these dimensions.

Specifically, GatewayBench comprises 524 tasks across six domains and a pool of 11,118 real-world tools collected from 251 MCP servers. We systematically evaluate seven widely deployed MCP gateways across different tool-set sizes, underlying models, and agent harnesses, covering both benign and adversarial tasks. Our findings reveal important trade-offs among existing gateway implementations and demonstrate how gateway effectiveness interacts with the underlying model and agent harness, providing practical insights for gateway selection and agent system design.

Q2: Evaluation of tool-shortlisting mechanisms

Yes, we have evaluated agents using retrieval-based tool-shortlisting mechanisms provided by MCP gateways. For example, Strata implements semantic search-based tool discovery, where the agent retrieves relevant tools through discover_server_actions and subsequently obtains their detailed specifications through get_action_details, rather than receiving the entire tool list upfront. AgentCore also provides a semantic tool-search interface.

We understand that the reviewer may also be referring to tool shortlisters implemented directly within the agent harness, where relevant tools are automatically selected before being presented to the LLM. While this is a related approach, our work specifically focuses on evaluating the MCP gateway layer, which has become an important component of industrial agent deployments. We further examine the interaction between gateways and agent harnesses by evaluating both the OpenAI Agents SDK and OpenClaw. However, we do not separately evaluate an automatic harness-level shortlister as an independent configuration.

Q3: Performance with open-source models

Thank you for this question. We have also conducted preliminary experiments using smaller open-sourced models such as gpt-oss-120b. However, we found in our experimental setup, when more than approximately 100 tool definitions were exposed to the model, the large amount of tool metadata placed substantial pressure on its effective context budget. We observed very low task completion rates for both benign and red-teaming tasks, making these results less informative for meaningful comparisons across gateway implementations.

Therefore, we focused our main evaluation on agent configurations with sufficient capability to complete the underlying tasks, allowing us to more meaningfully distinguish the effects of gateway design on task success, efficiency, and security. We acknowledge that gpt-oss-120b is not a small model and that further experiments would be needed to characterize performance across smaller open-weight models.

Thanks for the reply I am still struggling to see the novelty here. The results of the paper are quite obvious to me. I meant it is an extensive benchmark and experiment, but I did not learn anything new here

Thank you for your follow-up. We understand the reviewer's concern and would like to further clarify that the key novelty of our work lies in identifying and systematically evaluating the MCP gateway as a distinct layer of the agentic system, rather than treating tool management merely as an internal component of the agent harness. As MCP gateways have emerged as independently deployed and widely adopted middleware in real-world agent systems, we believe they deserve systematic evaluation in their own right. To the best of our knowledge, GatewayBench is the first unified benchmark specifically designed to evaluate this layer, enabling systematic comparisons across gateway implementations, underlying models, and agent harnesses while jointly assessing capability, efficiency, and security. This provides a new evaluation perspective and a reusable testbed for an increasingly important component of the agentic AI stack.

Beyond establishing this evaluation perspective, GatewayBench contributes a comprehensive evaluation platform and benchmark comprising more than 500 tasks across 6 domains and 10K real-world tool specifications from more than 250 MCP servers, enabling systematic evaluation of gateway utility, efficiency, and security at scale. Our experiments also reveal several non-trivial empirical findings: (1) gateway effectiveness depends strongly on the underlying model and agent harness, with relative performance across gateway architectures changing under different agent configurations; (2) stronger models substantially close the capability gap between flattened and hierarchical gateways, yet a significant security gap persists, primarily driven by tool-injection rather than environment-injection attacks; and (3) flattened gateways are more efficient with small tool sets, but this advantage reverses as the tool set scales, revealing important trade-offs between discovery overhead and context-processing cost. More importantly, GatewayBench is not merely a one-time empirical comparison: it provides a standardized and reusable evaluation framework that can consistently serve as a signal for future research and gateway development, helping measure and drive improvements in the utility and security of agentic systems.

We sincerely thank the reviewer again for their valuable feedback and thoughtful consideration of our work. We would be happy to provide further clarification or address any remaining concerns or follow-up questions.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank all three reviewers for their insightful, constructive, and encouraging feedback. Based on the reviewers' valuable suggestions, we have conducted additional analyses on our benchmark. Below, we summarize the key improvements made in response to the reviewers:

  • Clarification of scope and methodology (Reviewers PVAY, f9YR, NzxD): we clarified that GatewayBench aims to evaluates diverse real-world MCP gateways for the first time, rather than proposing a new discovery algorithm before we gain an in-depth understanding. We also detailed our threat model, payload injection, and truncation mechanisms.
  • Adding clean task success on matched indirect-attack instances, payload exposure rates, and ASR conditional on exposure (Reviewers f9YR, NzxD): we report clean task success on the indirect-attack instances that have matched benign counterparts, alongside payload exposure and ASR given exposure for hierarchical gateways. We also confirm that Bifrost’s substantially lower payload exposure than Strata is largely attributed to its truncation mechanism.
  • Additional analysis on noisy tool stability and scaling (Reviewers f9YR, NzxD): we measure how often GPT-5.4 actually invokes an injected noisy tool. Our results confirm that noisy tool invocations remain consistently rare across gateways, supporting the stability of our large, domain-filtered distractor pool, which could benefit other related analysis for the community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.