- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CATS, a tool-selection stress-testing framework. An LLM generator inserts distracting tool metadata over multiple rounds, using the selector's previous choices as feedback. Five specifications cover incorrect selection, retrieval saturation, excessive privileges, unnecessary abstention, and intent changes. Each trial succeeds only if every round satisfies the specification. CATS applies a one-sided Clopper-Pearson confidence bound to the resulting success counts. Experiments on BFCL and OpenAPI report low success bounds under the strongest distractor distributions and limited improvement from the evaluated mitigations.
Critical strengths
-
The framework explicitly states the selection requirement and the sampling process being evaluated. This makes the scope of its statistical bound inspectable.
-
The experiments examine several distractor mechanisms, retrievers, and mitigations. Forced inclusion of a satisfactory tool helps separate retrieval failure from selector failure.
Critical weaknesses
-
Practical relevance is insufficiently established. The paper assumes that untrusted tools can compete directly with curated tools, leaving the selector to distinguish them from metadata alone. It does not adequately justify this setup for realistic deployments with registry review, trusted-tool policies, and permission enforcement. The targeted generator is also not a validated model of real tool-pool changes. The main selectors achieve only about 71%-77% clean accuracy, so their failures may partly reflect limited basic tool-selection capability rather than a broadly applicable agent-safety problem.
-
The incremental contribution is limited, and the headline comparison is difficult to interpret. The statistical mechanism is a standard binomial confidence bound; the added value beyond adversarial stress testing with confidence intervals is unclear. Calling it a certificate does not establish that the sampling distribution represents deployment risk. Moreover, clean accuracy measures one selection, while the main bound requires success in every round of a ten-round trial. A clean comparison at the same horizon is needed to separate distractor effects from repeated-call failure accumulation.
Questions for authors
-
Which concrete deployment settings permit these inserted tools to enter the candidate pool? How do the assumptions account for tool curation, trusted publishers, and permission checks?
-
Do the main findings persist with stronger tool-selection models and realistic curated-registry controls? This would help separate general capability limitations from the proposed safety vulnerability.
-
Can the authors report clean-pool success under the same ten-round, all-rounds-success predicate used for the main certificates?
-
What substantive contribution does CATS provide beyond an adaptive attack evaluation with a standard confidence interval? Please clarify why the additional certification terminology is needed.
3 comments
Regarding the contribution
The main contribution of CATS is the mathematical formulation of tool-selection robustness under adversarial changes to the candidate tool pool, together with the adaptive sampling procedure used to evaluate the resulting specifications. Classical adversarial robustness specifications typically consider norm-bounded perturbations around a fixed input and ask whether the model’s outputs would remain correct within the region. These previous specifications cannot directly be used in this scenario because the adversary is able to dynamically modify the set of available tools, affect which tools are retrieved in the slate, and by doing so affect the selector’s subsequent decisions. This means that to capture these properties required for agentic tool selection, the specification needs to be over the entire dynamic system rather than a fixed perturbation region around a given individual input.
CATS specifies the admissible tool-pool modifications, their sampling distribution, and the behavioral requirements over the retrieved slate and selected tool. Constructing such a specification requires modeling the relevant stages of the tool-selection pipeline in such a way that characterizes a meaningful space of possible inputs and distinguishes the failure mechanisms that the evaluation is intended to measure. This is important for the natural language tool metadata where perturbations may contain unrealistic or nonsensical inputs that provide minimal information about the actual threat model. We use the term certification specifically to describe this statistical guarantee. A certificate of 0.20 means that the true trajectory-level satisfaction probability under the declared evaluation distribution is at most 0.20 with 95% confidence.
Many attack evaluations use fixed sets of adversarial examples and report empirical attack success rates. CATS reports each result in respect to the specification and the Delta such that it allows us to evaluate each failure mechanism separately from the process that generates the inserted tools.
Rather than flattening all nuance into a single value, CATS allows for more granular attribution within the pipeline. For example, the frontier results above show that increasing selector model capability can reduce Distractor Selection but not Top-N Saturation, which tells us where each defense must act. This is only possible because CATS allows comparisons across combinations of selectors, retrievers, and defenses as bounds on the same quantity under the same protocol using a common specification.
We hope these clarifications and the additional analysis address the reviewer’s remaining concerns, and we would be grateful if the reviewer would consider raising their score.
We thank the reviewer again for their detailed feedback. We have addressed the concerns about practical relevance with evidence from real-world malicious tool registries, evaluated stronger frontier selectors, and added matched-horizon clean-pool comparisons. These results confirm that the identified vulnerabilities persist beyond weaker models and remain significant under realistic deployment assumptions.
We would be grateful if the reviewer could revisit our response and updated manuscript and reconsider their initial assessment in light of these additional results and clarifications.
We thank the reviewer for their time. We address the remaining concerns below.
CATS applies where third-party tools can enter the candidate pool. Some representative real life examples include:
Controls are modeled at different points of the CATS framework. Forced Inclusion isolates selection failures by ensuring that a satisfactory tool reaches the selector, while publisher verification and admission rules constrain which tools may enter the candidate pool. CATS provides a common framework for evaluating the resulting pipeline under explicitly declared threat-model assumptions.
We have run the evaluation on additional frontier models as the selector, as requested by the reviewer.
The results show that even with stronger selector models, clean accuracy is still limited by retrieval. While using more capable models as selectors increases Distractor Selection (0.45 and 0.52 for Sonnet 5.5 and GLM 5.3 respectively), the bound stays below the matched clean values. Meanwhile, due to the failure in tool selection occurring using retrieval for Top-N Saturation, we can see that even with the more capable selector model the bounds remain low. The main findings therefore hold for frontier selectors.
The clean success at ten rounds is 0.61, with a one-sided lower bound of 0.58. The degradation remains even after accounting for repeated opportunities for failure.