- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces GatewayBench, a framework for evaluating how MCP gateway design affects agent capability, efficiency, and security. It compares seven gateway implementations spanning flat tool exposure and hierarchical, on-demand discovery, using tasks from DTAP and MCP-Universe and distractor tools collected from 251 real MCP servers. The results show that hierarchical gateways substantially reduce context overhead as the tool set grows, while their task performance depends on the model’s ability to discover relevant tools. Lower indirect attack success rates are observed primarily for tool injection. The main contribution is a common evaluation framework and an empirical study of how gateway choices affect agent behavior.
Critical strengths
-
The paper studies the MCP gateway that deserves evaluation in its own right. Comparing seven gateway implementations under a common set of tasks and metrics helps identify how the tool access layer affects task completion, interaction cost, and security. These results have practical value for developers choosing among gateway designs, and the framework provides a useful basis for subsequent comparisons.
-
The security analysis explicitly considers the possibility that lower attack success reflects reduced task competence. The authors acknowledge that tool-discovery failures can make weaker models appear more secure. Comparing benign and attacked performance is a useful step toward separating security gains from a general loss of capability.
Critical weaknesses
-
The security comparison does not sufficiently separate gateway architecture from implementation-specific differences in information exposure. The paper leaves several details of the threat model unclear, including which fields an attacker can modify, the relationship between malicious tools and tasks, and how payloads are adapted for different discovery interfaces. Moreover, Bifrost’s description truncation limits exposure to malicious content, thereby reducing the ASR. However, the current evaluation does not establish how much of the observed reduction comes from selective discovery and how much comes from truncation or other interface choices. These confounds limit the support for architecture-level security recommendations.
-
The task-performance crossover at scale lacks robustness evidence. The distractors exclude semantically overlapping tools and uniformly return errors, leaving ambiguity among related tools untested. Moreover, the main experiments use fixed tool subsets without repeated sampling, provides limited evidence for a reliable crossover.
Questions for authors
-
Please clarify the threat model: Which fields can the attacker modify in the tool-injection experiments? Is the payload placed in a task-relevant tool or an additional tool? How is the same attack presented through each gateway’s search, documentation, and truncation mechanisms, and are payloads adapted to individual interfaces?
-
Could you report clean task success on the same instances used for indirect attacks, together with payload exposure rates and ASR conditional on exposure?
-
How stable is the crossover in Table 3 across different distractor subsets?
3 comments
W2 & Q3: How stable is the crossover in Table 3 across different distractor subsets?
We investigate how frequently the model actually invokes a noisy tool. Table r.3 reports the percentage of GPT-5.4 runs containing at least one call to an irrelevant noisy tool.
Table r.3: Percentage of GPT-5.4 runs containing at least one call to an irrelevant noisy tool across different tool-set sizes.
| Gateway | N=500 | N=1,000 | N=1,500 |
|---|---|---|---|
| AgentCore | 0.22% | 0.22% | 4.44% |
| AgentGateway | 0% | 0% | 0% |
| Bifrost | 0.44% | 0% | 0.22% |
| Enkrypt | 0% | 0% | 0% |
| LiteLLM | 0.44% | 0.22% | 5.56% |
| Lunar | 0% | 0.22% | 7.11% |
| Strata | 0% | 0% | 0% |
The results demonstrate that the model rarely calls a noisy tool. Additionally, the noisy tool set is sufficiently large and diverse, so we assume that the choice of noisy tools does not significantly affect the model's behavior.
We acknowledge that the performance drop of flattened gateways is small even at large tool-set sizes, and the crossover between flattened and hierarchical gateways is not sufficiently pronounced to support a strong architecture-level conclusion regarding TSR alone. However, we note that the excessive token cost of flattened gateways is a more concerning factor than TSR when the tool-set size is large. Therefore, we still conclude that hierarchical gateways are more efficient than flattened gateways for large tool-set sizes.
Thank you for the detailed clarifications and additional results. It remains unclear whether the crossover in TSR holds across different subsets of distractor tools. I suggest presenting it as an observation specific to the setting tested. However, I do not think this limitation is serious enough to negate the contribution of this benchmark study. I will raise my overall rating to 3.
Dear Reviewer NzxD,
Thank you for your constructive comments and valuable suggestions. We address your concerns below.
The malicious tasks are divided into direct and indirect attacks. Direct attacks modify the user prompt to include a malicious instruction, and indirect attacks contains both tool injection and environment injection attacks.
For environment injection, the attacker injects a malicious instruction into the database of an existing business tool, such as an email, Slack message, CRM note, or travel record.
For tool injection, the attacker modifies only the description field of an existing tool. The tool name, input schema, and implementation are unchanged, and the attacker does not add a new tool. The attack only targets a tool already required by the task.
Within each task, all gateways receive the same payload configuration, and we do not rewrite payloads for individual gateways. Gateways differ solely in how they present tool metadata. For example, a flattened gateway exposes the complete modified description at the beginning of the conversation, whereas a hierarchical gateway exposes the modified description only when the discovery of the tool is triggered.
To evaluate clean task success on indirect attack tasks, we restrict the analysis to 24 indirect tasks that have a broadly matched clean task. As shown in Table r.1, we report both clean task success and attack success for each gateway. The clean task success matches our reported benign task results in the paper.
Table r.1: Clean task success rate and Attack Success Rate (ASR) across gateways on matched indirect attack instances.
We further report payload exposure and ASR conditional on exposure for two hierarchical gateways in Table r.2. A run is marked as exposed if the malicious payload is present in the agent's trace.
Table r.2: Payload exposure rate, overall ASR, and ASR conditional on exposure for hierarchical gateways.
These results show that Bifrost has substantially lower payload exposure than Strata, while the ASR conditional on exposure is comparable. For Bifrost, 259 out of 272 runs (95.2%) returned the attacked tool's definition, but 214 out of 259 descriptions (82.6%) were truncated. Although the agent may access the full tool description through another tool, the attack payload is not exposed in 143 out of 259 runs (55.2%) where the attacked tool's definition is returned.