acceptodds
Under review as a conference paper at ICLR 2027

Quantitative Certification of Agentic Tool Selection

Abstract

Large language models (LLMs) are increasingly deployed in agentic systems, where a fundamental task is mapping user intents to relevant external tools. Errors in tool selection can have severe outcomes, such as unauthorized data access, even without modifying the agent's underlying model. Existing evaluations measure performance on curated, benign benchmarks. However, a pipeline's behavior in deployment depends on the tool pool the agent actually encounters, which in open registries is shaped by third parties. We introduce CATS (Certification of Agentic Tool Selection), the first statistical framework that returns high-confidence upper bounds on the probability that a tool selection pipeline satisfies a declared safety specification under a plausible tool distribution. CATS reduces the certification of tool selection to a Bernoulli estimation problem, drawing sequences of inserted tools from a declared distribution paired with the safety specification. To model realistic deployment conditions, we instantiate this distribution as a stochastic process that generates sequences of inserted tools round by round, conditioning each round on the agent's selection in the previous round. CATS aggregates the trial outcomes into a one-sided Clopper-Pearson upper bound on the probability that the specification is satisfied. By returning this bound as a certificate that holds under the distribution of inserted tools, CATS makes safety claims intuitive, actionable, and comparable across models, retrievers, mitigations, and registry policies. Across popular BFCL and OpenAPI tool pools, CATS shows that current LLM agents remain fragile under Distractor Selection and Top- Saturation specifications: their certified upper bounds drop to approximately 20%, far below their lower bounds on the clean pool.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agentic AI safety, tool-augmented language models, tool selection, statistical certification, LLM security, retrieval robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6ZYZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CATS, a tool-selection stress-testing framework. An LLM generator inserts distracting tool metadata over multiple rounds, using the selector's previous choices as feedback. Five specifications cover incorrect selection, retrieval saturation, excessive privileges, unnecessary abstention, and intent changes. Each trial succeeds only if every round satisfies the specification. CATS applies a one-sided Clopper-Pearson confidence bound to the resulting success counts. Experiments on BFCL and OpenAPI report low success bounds under the strongest distractor distributions and limited improvement from the evaluated mitigations.

Critical strengths
  1. The framework explicitly states the selection requirement and the sampling process being evaluated. This makes the scope of its statistical bound inspectable.

  2. The experiments examine several distractor mechanisms, retrievers, and mitigations. Forced inclusion of a satisfactory tool helps separate retrieval failure from selector failure.

Critical weaknesses
  1. Practical relevance is insufficiently established. The paper assumes that untrusted tools can compete directly with curated tools, leaving the selector to distinguish them from metadata alone. It does not adequately justify this setup for realistic deployments with registry review, trusted-tool policies, and permission enforcement. The targeted generator is also not a validated model of real tool-pool changes. The main selectors achieve only about 71%-77% clean accuracy, so their failures may partly reflect limited basic tool-selection capability rather than a broadly applicable agent-safety problem.

  2. The incremental contribution is limited, and the headline comparison is difficult to interpret. The statistical mechanism is a standard binomial confidence bound; the added value beyond adversarial stress testing with confidence intervals is unclear. Calling it a certificate does not establish that the sampling distribution represents deployment risk. Moreover, clean accuracy measures one selection, while the main bound requires success in every round of a ten-round trial. A clean comparison at the same horizon is needed to separate distractor effects from repeated-call failure accumulation.

Questions for authors
  1. Which concrete deployment settings permit these inserted tools to enter the candidate pool? How do the assumptions account for tool curation, trusted publishers, and permission checks?

  2. Do the main findings persist with stronger tool-selection models and realistic curated-registry controls? This would help separate general capability limitations from the proposed safety vulnerability.

  3. Can the authors report clean-pool success under the same ten-round, all-rounds-success predicate used for the main certificates?

  4. What substantive contribution does CATS provide beyond an adaptive attack evaluation with a standard confidence interval? Please clarify why the additional certification terminology is needed.

3 comments

We thank the reviewer for their time. We address the remaining concerns below.

Regarding deployment settings and practical relevance

CATS applies where third-party tools can enter the candidate pool. Some representative real life examples include:

  • OX Security reported that nine of eleven MCP marketplaces accepted its test submission, an example of open marketplaces of tools (OX Security, 2026).
  • Island documented malicious MCP and skill listings in public registries, with assistants themselves discovering and recommending a malicious MCP repository (Zaytsev, 2026).
  • The MCP specification notes that clients that combine many different servers can encounter naming collisions between tools (Model Context Protocol, 2026).
  • The LiteLLM incident shows that trusted publishers can become compromised, so curation at one point in time does not remove this risk (LiteLLM, 2026).

Controls are modeled at different points of the CATS framework. Forced Inclusion isolates selection failures by ensuring that a satisfactory tool reaches the selector, while publisher verification and admission rules constrain which tools may enter the candidate pool. CATS provides a common framework for evaluating the resulting pipeline under explicitly declared threat-model assumptions.

Regarding stronger models

We have run the evaluation on additional frontier models as the selector, as requested by the reviewer.

SelectorClean, R = 1Clean, R = 10Forced Inclusion cleanDistractor SelectionTop-N SaturationPrivilege EscalationAbstention TriggerIntent Shifting
Llama-3.1 8B0.770.610.950.200.200.650.730.75
Sonnet 5.50.810.750.970.450.210.650.790.80
GLM 5.30.800.730.960.520.230.710.770.80

The results show that even with stronger selector models, clean accuracy is still limited by retrieval. While using more capable models as selectors increases Distractor Selection (0.45 and 0.52 for Sonnet 5.5 and GLM 5.3 respectively), the bound stays below the matched clean values. Meanwhile, due to the failure in tool selection occurring using retrieval for Top-N Saturation, we can see that even with the more capable selector model the bounds remain low. The main findings therefore hold for frontier selectors.

Regarding clean success at ten rounds

The clean success at ten rounds is 0.61, with a one-sided lower bound of 0.58. The degradation remains even after accounting for repeated opportunities for failure.

Regarding the contribution

The main contribution of CATS is the mathematical formulation of tool-selection robustness under adversarial changes to the candidate tool pool, together with the adaptive sampling procedure used to evaluate the resulting specifications. Classical adversarial robustness specifications typically consider norm-bounded perturbations around a fixed input and ask whether the model’s outputs would remain correct within the region. These previous specifications cannot directly be used in this scenario because the adversary is able to dynamically modify the set of available tools, affect which tools are retrieved in the slate, and by doing so affect the selector’s subsequent decisions. This means that to capture these properties required for agentic tool selection, the specification needs to be over the entire dynamic system rather than a fixed perturbation region around a given individual input.

CATS specifies the admissible tool-pool modifications, their sampling distribution, and the behavioral requirements over the retrieved slate and selected tool. Constructing such a specification requires modeling the relevant stages of the tool-selection pipeline in such a way that characterizes a meaningful space of possible inputs and distinguishes the failure mechanisms that the evaluation is intended to measure. This is important for the natural language tool metadata where perturbations may contain unrealistic or nonsensical inputs that provide minimal information about the actual threat model. We use the term certification specifically to describe this statistical guarantee. A certificate of 0.20 means that the true trajectory-level satisfaction probability under the declared evaluation distribution is at most 0.20 with 95% confidence.

Many attack evaluations use fixed sets of adversarial examples and report empirical attack success rates. CATS reports each result in respect to the specification and the Delta such that it allows us to evaluate each failure mechanism separately from the process that generates the inserted tools.

Rather than flattening all nuance into a single value, CATS allows for more granular attribution within the pipeline. For example, the frontier results above show that increasing selector model capability can reduce Distractor Selection but not Top-N Saturation, which tells us where each defense must act. This is only possible because CATS allows comparisons across combinations of selectors, retrievers, and defenses as bounds on the same quantity under the same protocol using a common specification.

We hope these clarifications and the additional analysis address the reviewer’s remaining concerns, and we would be grateful if the reviewer would consider raising their score.

We thank the reviewer again for their detailed feedback. We have addressed the concerns about practical relevance with evidence from real-world malicious tool registries, evaluated stronger frontier selectors, and added matched-horizon clean-pool comparisons. These results confirm that the identified vulnerabilities persist beyond weaker models and remain significant under realistic deployment assumptions.

We would be grateful if the reviewer could revisit our response and updated manuscript and reconsider their initial assessment in light of these additional results and clarifications.

Reviewer_LVwKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CATS, a framework for evaluating tool-selection reliability when third parties can add misleading tools to a registry. CATS pairs a behavioral specification with a distribution over tool-pool changes, instantiated through an LLM generator that conditions on the selector’s previous choice. Each multi-round trajectory is reduced to a binary outcome, and a one-sided Clopper–Pearson bound is computed on the probability that the specification holds throughout the trajectory. Experiments on BFCL and OpenAPI tool pools cover four open-weight selectors and five distraction mechanisms. The paper reports particularly low satisfaction bounds under Distractor Selection and Top-N Saturation, with failures persisting under forced inclusion of a satisfactory tool and the evaluated mitigations.

Critical strengths

1.The evaluation setting addresses a relevant weakness of fixed-pool benchmarks. Allowing third parties to modify the tool pool captures a practical source of failure in open registries. Explicitly separating the behavioral specification from the tool-generation distribution makes the scope and assumptions of each reported guarantee clear.

2.Forced Inclusion ensures that a satisfactory tool reaches the selector, yet errors persist. This is a useful diagnostic result. Evaluations across generator–selector combinations, retrievers, and mitigations also provide broader evidence than a single attack configuration.

Critical weaknesses

1.The methodological contribution is insufficiently distinguished from existing multi-turn certification. The C³LLM framework [1] already considers multi-turn outcome probabilities, Markov and response-adaptive sampling, and Clopper–Pearson certification. The trajectory-level statistical mechanism therefore does not establish the claimed distinction. The identifiable additions are the tool-selection setting, metadata generator, and empirical study; the paper needs to explain more clearly what substantive methodological advance or new insight these additions provide.

2.The central comparisons change the quantity being estimated, limiting the conclusions drawn from the reported gaps. Figure 2 compares single-step clean accuracy with the probability of satisfying the specification throughout ten adversarial rounds. Without a clean baseline evaluated over the same horizon, the gap combines the effect of tool insertion with the stricter requirement of repeated success.

[1] Wang C, Chaudhary I, Hu Q, et al. How Catastrophic is Your LLM? Certifying Risks in Conversation. International Conference on Learning Representations. 2026.

Questions for authors

1.What technical contribution distinguishes CATS from applying the multi-turn certification approach of C³LLM to tool-selection trajectories? Please clarify which aspects of the formulation and sampler require new methodology.

2.What is the clean-pool satisfaction probability under the same ten-round predicate, queries, and decoding settings? Do the main conclusions remain after matching the evaluation horizon and reporting empirical satisfaction, trajectory-level ASR, and confidence bounds on identical trials?

2 comments

We thank the reviewer for recognizing that third-party changes to the tool pool are a realistic and important failure mode to study. We address the remaining concerns below.

Regarding comparisons to adjacent work

The main contribution of CATS is the mathematical formulation of tool-selection robustness under adversarial changes to the agent's dynamically evolving action space. We believe this is a distinct certification problem from that of C³LLM, which models safety across multi-turn conversations. In CATS, the adversary can modify the available tool pool across interaction rounds, conditioning subsequent modifications on the agent's previous selections. Thus, CATS certifies the entire retrieval-and-selection pipeline under adversarial changes to its candidate action space, rather than only the LLM's behavior across interactions.

This requires specifications over both the retrieved candidate slate and the selector's decision, as well as distributions over adversarial tool metadata. At each round, the adversary modifies the tool pool, which determines the retrieved slate and subsequently constrains the selector's available choices. The specification must therefore capture both whether satisfactory tools remain available after retrieval and whether the selector chooses a satisfactory tool when one is available. These distinctions allow CATS to isolate and study the distinct failure modes that arise in tool selection. In retrieval-level failures, the satisfactory tools can be displaced from the slate itself, meaning uplift in selector capabilities would not be able to resolve this failure mode. On the other hand, CATS also formalizes selection failures, in which the selector chooses an incorrect tool despite having a satisfactory alternative. This formalization is done over dynamically evolving tool pools and the given tool selection trajectories, meaning the sampling procedure also needs to be defined correspondingly over modifications conditioned on the selector’s previous choices.

These distinctions also lead to a number of empirical findings not available from a problem setting in C3LLM or adjacent works. Under Top-N Saturation, a satisfactory tool is displaced from the retrieved slate in 89% of trials within ten rounds. Under Forced Inclusion, a satisfactory tool is guaranteed to reach the selector, yet the upper bound for Distractor Selection remains at 0.43 (Table 2). The defended retriever improves the Top-N Saturation upper bound from 0.20 to 0.45, but improves Distractor Selection only from 0.20 to 0.22 (Table 3).

Thus, CATS introduces a tool selection specific formulation of robustness that captures how adversarial changes to the available action space can propagate through the retrieval and selection stages in agentic tool selection.

Regarding ten-round clean satisfaction probability

We report the original pool under the same R = 10 all-rounds test as requested by the reviewer.

ConditionRSatisfactionFailure RateOne-sided 95% bound
Clean pool10.770.23lower 0.75
Clean pool100.610.39lower 0.58
Distractor Selection10.480.52upper 0.50
Distractor Selection100.180.82upper 0.20
Top-N Saturation10.530.47upper 0.55
Top-N Saturation100.180.82upper 0.20

Thus, the main empirical conclusion persists after matching the number of rounds. The observed degradation cannot be explained solely by the requirement to succeed repeatedly.

We hope these clarifications and the additional analysis address the reviewer’s remaining concerns, and we would be grateful if the reviewer would consider raising their score.

We thank the reviewer again for their constructive feedback. We have clarified the methodological contributions of CATS relative to existing multi-turn certification frameworks and added the requested matched-horizon clean-pool evaluation, confirming that the main findings persist under the same ten-round specification.

We would be grateful if the reviewer could revisit our response and updated manuscript and consider whether these results address their remaining concerns. We would also appreciate reconsideration of the initial score in light of these additions.

Reviewer_B6xkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CATS, a statistical certification framework designed to provide quantitative, high-confidence upper bounds on the probability that an LLM agent’s tool-selection pipeline satisfies declared safety specifications under realistic, non-benign tool pool distributions. Addressing the vulnerabilities exposed in open tool registries like the Model Context Protocol (MCP), where third-party metadata can manipulate retrieval slates, CATS models tool pool perturbation as a stochastic multi-round generation process. Candidate distractor tools are generated by an LLM generator conditioned on the agent’s previous-round selection, followed by deterministic post-processing transformations including parameter schema collision, homoglyph cloning, and persuasion cue injection . By treating multi-round trajectories as Bernoulli trials, CATS derives exact finite-sample upper bounds using the one-sided Clopper-Pearson method. Across the BFCL and OpenAPI benchmarks with four open-weight models, CATS reveals that while agents achieve clean accuracy lower bounds above 70%, their certified upper bounds plunge under Distractor Selection and Top-N Saturation specifications, and existing mitigations (e.g., StruQ, SecAlign, anomaly filters) fail to close this gap.

Critical strengths

The paper formalizes a certification framework for discrete, non-differentiable tool metadata selection where standard norm-ball or randomized smoothing techniques fail. Reducing multi-round stochastic interactions to Bernoulli trials evaluated via exact Clopper-Pearson bounds establishes a sound mathematical foundation for comparing agent selection robustness.

Critical weaknesses

The certified upper bound is strictly valid only under the declared sampling distribution Delta and its specific post-processing heuristics (such as the predefined Cyrillic homoglyph map and fixed token vocabulary). Unlike distribution-free adversarial robustness guarantees, CATS provides a certificate against a parameterized synthetic attacker; an adversary operating outside this generative process is not bounded by the certificate.

The framework defines a trajectory outcome as a total failure if any single round fails . While appropriate for irreversible actions, this metric is excessively punitive for standard information retrieval tasks where real-world agents recover from incorrect selections via runtime feedback and retry loops. Although preliminary retry experiments are discussed in Appendix, the main paper’s headline certificates reflect a worst-case, zero-tolerance reduction.

Questions for authors

Because the Clopper-Pearson upper bound is conditioned on the declared distribution Delta, what formal guarantees or empirical tests can verify that Delta adequately covers the distribution of real-world malicious tools submitted to public MCP registries?

In Appendix, retry mechanisms were tested at horizon R=1. How does allowing execution feedback-driven retries across the full R=10 interaction trajectory impact the certified upper bounds for Distractor Selection and Top-N Saturation?

3 comments

We thank the reviewer for recognizing that the paper addresses a problem setting in which standard norm-ball and smoothing methods would not apply, and that the proposed Bernoulli reduction with exact Clopper-Pearson bounds is a valid method to resolve this gap in the literature. We address the remaining concerns below.

Regarding the validity of Delta

There exists no feasible way to guarantee that all possible malicious tools are covered by Delta, and we do not claim this in the paper.

In settings where we are working with natural language, unrestricted adversarial perturbation regions can contain unrealistic nonsensical inputs that are not likely in practice. Treating every such input equally can produce specifications that poorly represent the intended threat model. Instead, CATS is a general certification framework that uses a probability distribution over tool pool modifications, allowing the specification to assign greater weight to plausible attack patterns. The framework takes in a specification and a distribution over tool pool modifications which allows different threat models to be evaluated without changing the certification methodology.

To show this in practice, we also certify the same pipeline under a distribution of real malicious listings from public registries.

Island documented the AgentBaiting campaign in July 2026 (Zaytsev, 2026). The campaign published more than 800 GitHub repositories that posed as MCP servers or Skills. These repositories appeared in more than 600 listings on public registries (LobeHub, Glama, MCP.so, and MCP Market), and each one delivered information-stealing malware. We took 200 listings from Island's published list and converted each one into tool metadata: the repository name became the tool name, and the listing text became the description. Verbatim inserts the 5 best-matching listings unchanged. Retargeted keeps each listing's strategy (impersonated name, enterprise framing, version string) but replaces its product name with the function the intent asks for.

DistributionRSatisfactionOne-sided 95% boundInserted tool in slate
Clean pool10.77lower 0.72n/a
Clean pool100.61lower 0.58n/a
AgentBaiting, verbatim10.77upper 0.8220%
AgentBaiting, verbatim100.56upper 0.6220%
AgentBaiting, retargeted10.55upper 0.6192%
AgentBaiting, retargeted100.34upper 0.4092%
CATS Delta (n = 1000)100.18upper 0.20n/a

Even with a real-world distribution, we see the same gap occur without any feedback from the selector (0.40 vs a clean lower bound of 0.58). Delta is even stronger than the real lures (0.20 vs 0.40; upper bounds 0.40 vs 0.33 without deterministic stages). Verbatim settings rarely reach the slate because the campaign targeted products such as Gmail and Databricks, not plausible-sounding tools.

We also evaluate the sensitivity of our results to changes in the generator and tool pool. Removing both deterministic post-processing stages while retaining the LLM generator yields upper bounds of 0.33 for Distractor Selection and 0.37 for Top-N Saturation, compared with 0.20 for both under the full generator. Thus, the observed degradation does not depend entirely on the deterministic transformations. Moreover, doubling the marker vocabulary, changing the homoglyph mapping, or changing marker weights changes the reported bounds by at most 0.03 (Table 11). On a separate OpenAPI-derived tool pool, the corresponding bounds remain 0.19 and 0.22 (Table 18).

It is important to note that Delta is an input to the CATS framework, not a fixed distribution tied to the statistical method. A deployer can define their own distribution that reflects their own tools, acceptance criteria, and other assumptions about the agentic system and obtain a confidence bound for that setting. For example, the distribution can vary the different components of the pipeline such as the metadata generation or the composition of the tool pool, without changing the rest of the certification procedure. Our generator and tool-pool experiments demonstrate this flexibility through generator-stage ablations (Section 4.3), distributional sensitivity analyses (Appendix A.7, Table 11), and evaluation on an alternative OpenAPI tool pool (Appendix B.6, Table 18).

Regarding severity of tool selection

The predicate is made to be strict on purpose. In tool selection, one wrong selection can be enough to cause harm that a later retry cannot undo. For example, executing an incorrectly selected tool can disclose private information or perform an unauthorized external action, shown by the security failures studied in previous works such as AgentDojo (Debenedetti et al., 2024). A later correct call may not be able to undo the damage done by the previous harmful tool call. Thus the primary use case for the CATS framework is not just about whether the agent is able to use a benign tool in the end, but rather determining whether the system is able to avoid a bad selection throughout an interaction. Such a framework needs a single predicate that can be applied across variations of selectors, retrievers, registry policies, and defenses that an agentic system may include.

To address more lenient use cases, such as the one described by the reviewer, we also evaluate retry-based specifications that allow recovery after an incorrect selection.

As requested by the reviewer, we have extended this analysis of retries with execution observation appended per round to R = 10, and find that the upper bound is 0.33 on Distractor Selection and 0.41 on Top-N Saturation. The clean pool under the same predicate gives 0.64 as a lower bound. These results show that retries with execution feedback improve satisfaction, but substantial failures remain. As stated in our earlier response, this is possible because CATS is not restricted to the stricter predicate used in our main experiments. The specification is an input to the framework which gives CATS the flexibility to accommodate both safety critical settings and more lenient settings.

We hope these clarifications and the additional analysis address the reviewer’s remaining concerns, and we would be grateful if the reviewer would consider raising their score.

We thank the reviewer again for their constructive feedback. We have addressed both concerns through additional experiments on real-world malicious MCP listings, sensitivity analyses of the sampling distribution, and ten-round evaluations with execution-feedback retries.

We would be grateful if the reviewer could revisit our response and updated manuscript and consider whether these results address their remaining concerns. We would also appreciate reconsideration of the initial score in light of these additions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair

We thank the reviewers for their time and constructive suggestions.

We appreciate that all three reviewers recognized the relevance of the problem setting addressed by CATS and the value of evaluating tool-selection robustness under adversarial changes to tool pools.

  • Reviewer B6xk highlighted the sound mathematical foundation of CATS and how standard adversarial robustness techniques would not be able to directly address the setting of discrete, non-differentiable tool selection.
  • Reviewer LVwK recognized the practical importance of studying third-party modifications to tool pools and praised the methodological contributions such as the separation of safety specifications from sampling distributions and the value of Forced Inclusion in distinguishing retrieval failures from selection failures.
  • Reviewer 6ZYZ similarly acknowledged the clearly specified evaluation requirements and how it makes the statistical guarantees transparent, along with the breadth of experiments across distractor mechanisms, retrievers, and mitigations.

We have addressed the reviewers’ remaining concerns through additional experiments and clarifications:

  • For Reviewer B6xk, we have evaluated CATS using actual real-world malicious MCP listings from the AgentBaiting campaign, conducted sensitivity analyses across generator configurations, and extended our execution-feedback retry experiments to ten rounds. The real-world evaluation confirms that substantial degradation persists under realistic malicious tool distributions (upper bound 0.40 versus clean lower bound 0.58), while the retry experiments show that failures remain even when recovery is permitted (upper bounds 0.33 and 0.41). These results confirm the the practical relevance of our threat model and demonstrate that CATS is a framework that can be used both in scenarios where strict safety requirements are considered and also in more benign and lenient specifications.
  • For Reviewer LVwK, we have clarified the methodological distinction between CATS and existing multi-turn certification frameworks, emphasizing the formulation of robustness over dynamically changing tool pools and the separation of retrieval and selection failures. We have also evaluated clean-pool satisfaction under the same ten-round predicate as the adversarial experiments. The matched comparison shows that the clean lower bound remains at 0.58, compared with adversarial upper bounds of 0.20 for both Distractor Selection and Top-N Saturation. These results establish that the observed degradation cannot be explained solely by repeated-round failure accumulation and clarify the distinct failure mechanisms captured by CATS that would be difficult to capture with other methodologies.
  • For Reviewer 6ZYZ, we have provided concrete evidence of malicious tool listings in public registries, clarified how registry admission policies and other deployment controls fit within the CATS framework, and extended our evaluation to stronger frontier selector models, including Sonnet 5.5 and GLM 5.3. Despite improved selection capabilities, Top-N Saturation upper bounds remain at 0.21 and 0.23, respectively, demonstrating that stronger models alone do not resolve retrieval-level vulnerabilities, supporting the motivation of our design. Together with the matched-horizon comparisons and methodological clarifications, these results establish that the identified failures are not merely artifacts of weaker base models or the statistical estimation procedure.

We have incorporated the additional experimental results and clarifications into the revised manuscript and provided more detailed explanations in our individual reviewer responses.

We hope these revisions address the remaining concerns raised by the reviewers and would be grateful if the reviewers would reconsider their assessments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.