acceptodds
Under review as a conference paper at ICLR 2027

CoSA: Characterizing the Conflict of Skills Attack in Real-World LLM Agent Skill Ecosystems

Abstract

An agent skill is a modular, reusable package of instructions and supporting resources that extends large language model agents with composable procedures for complex tasks. Prior work on Skill security has primarily exposed malicious instructions, bundled payloads, and unsafe packages, motivating single Skill package inspection and runtime defenses. However, individually benign Skills may violate system-level or user-defined policies when composed, as their intermediate artifacts and decision signals may be consumed downstream beyond the constraints governing their original use. We formalize such composition induced violations as Conflict of Skills (CoS). We first characterize recurring CoS mechanisms in real-world Skill ecosystems and then introduce the Conflict of Skills Attack (CoSA), a novel mechanism-guided framework for discovering composition opportunities and constructing locally admitted Skill additions. CoSA targets two augmentation strategies arising from Skill composition: (1) capability supplementation, which adds missing operations, and (2) relationship reshaping, which changes how existing capabilities exchange information, propagate decisions, or trigger side effects. We evaluate CoSA on four domains and 288 distinct tasks. Across multiple models and agent harnesses, CoSA raises attack success rates for data leakage, unauthorized state modification, and unsafe execution by about 20 percentage points over the baseline. Trajectory analysis shows that risk arises from cross-Skill information flow, decision reuse, and target/action binding. Based on these findings, we design a lightweight harness-level guard that checks information flow and authorization before consequential actions. In a five-model recruiting pilot, the guard reduces attack success rate by 58.61 percent points while largely preserving benign task completion (66.67 percent points to 65.00 percent points).

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Agent Skills, Agent Security

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7niYOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Conflict of Skills (CoS), where individually benign Skills can become unsafe when composed. The proposed CoSA framework analyzes an existing Skill pool and adds a small number of benign-looking Skills through capability supplementation and relationship reshaping to increase policy violations. Experiments on 288 tasks across four domains, five LLMs, and three agent harnesses show a pooled ASR increase of 23.43 pp.

Critical strengths
  1. The paper shifts attention from malicious individual Skills to composition-level risks among benign Skills, which is an important and underexplored attack surface.

  2. The attack is tested across multiple domains, models, and agent frameworks, and the ablations suggest that the effect is not simply due to enlarging the Skill pool.

Critical weaknesses
  1. CoSA assumes that the attacker can inspect the existing Skill pool and then introduce new Skills into it. The paper does not sufficiently justify when such access would realistically be available.

  2. The main evaluation is conducted on the authors’ own CoSA-Bench. It is unclear whether the attack generalizes to independently constructed Skill or agent benchmarks.

Questions for authors
  1. What realistic scenario allows an attacker to inspect the target Skill pool and then add specially constructed Skills? How would CoSA perform with only partial knowledge of the existing Skill pool?
  2. Have you evaluated CoSA on any existing Skill- or agent-security benchmarks, rather than only CoSA-Bench? Such results would help establish generalizability.
  3. What are the individual contributions of capability supplementation and relationship reshaping? A separate ablation would clarify which component drives the attack effectiveness.
7 comments
AuthorsOpenReview ↗

Thank you, reviewer 7niY! (1/4)

Response to Weaknesses

CoSA assumes that the attacker can inspect the existing Skill pool and then introduce new Skills into it. The paper does not sufficiently justify when such access would realistically be available.

We sincerely thank the reviewer for highlighting this important concern regarding the practical applicability of our threat model. We acknowledge that our original threat model assumes substantial knowledge of the victim's Skill pool. However, we would like to clarify that both Skill-pool visibility and Skill admission have realistic deployment scenarios: (i) First, regarding Skill-pool visibility, in repository-based Agent deployments, project-level Skill inventories can be publicly accessible through version-controlled Skill directories. For instance, the open-source SECS project explicitly publishes its inventory of 49 installed project-level Skills along with their implementations [1]. Moreover, Agent frameworks such as OpenClaw incorporate available Skill metadata, including Skill names, descriptions, and file paths, into the agent's system prompt in supported configurations [2]. Prior research (JustAsk, ICML 2026) has demonstrated that hidden system prompts can be recovered through black-box interactions [3], suggesting a potential avenue for attackers to obtain such Skill information without direct access to the victim's environment. Furthermore, prior work (Les Dissonances, NDSS 2026) has investigated cross-tool attacks exploiting interactions and dependencies among existing tools [4], highlighting the relevance of informed, target-aware adversaries. (ii) Second, regarding Skill admission, our threat model also requires that a crafted Skill be admitted into the victim’s pool. This presupposes no privileged access; it only requires passing the same channels open to any third-party contributor. Two are common in practice. Public marketplaces such as ClawHub let third-party developers publish Skills that downstream users install on their own initiative [5], so a crafted Skill advertising a plausibly useful capability can be adopted through ordinary discovery and installation, without the user suspecting its role in a cross-Skill composition. Repository-based deployments accept external contributions through workflows such as pull requests, where an addition that passes review and static screening enters the project-level Skill directory. In both cases, admission succeeds precisely because each added Skill is individually benign under existing screening and raises no flags at entry. Our threat model therefore captures an attacker who inspects an observable Skill configuration and submits crafted additions through legitimate channels, not one who arbitrarily modifies the victim’s environment. Collectively, these observations support the practical relevance of our informed-attacker threat model in certain real-world deployment scenarios, while recognizing that complete visibility and successful Skill admission are not guaranteed for arbitrary downstream users.

Nevertheless, we recognize that attackers may not always have complete visibility into the victim's installed Skill pool. To further address this concern, we conduct additional experiments under two settings: (1) Partially Known Skill Pools, where attackers construct adversarial Skills using only a subset (25%, 50%, or 75%) of the target Skill pool; and (2) Zero-Visibility (Domain-Only) Skill Pools, where attackers receive only a predefined domain description without access to the target pool's Skill packages or contracts. In both settings, generated additions are evaluated against the complete target Skill pool, without exposing hidden Skill information during construction or selection. In the domain-only setting, no target-specific adaptation is performed. The results are summarized in Table 1.

AuthorsOpenReview ↗

Thank you, reviewer 7niY! (2/4)

Table 1: Transferability of CoSA under different levels of attacker knowledge.

HarnessModelOriginal ASR (%)100% ASR (%)75% ASR (Drop)50% ASR (Drop)25% ASR (Drop)0% ASR (Drop)
OpenClawQwen3.634.0359.3861.11 (−1.74)50.00 (9.38)37.50 (21.88)31.60 (27.78)
MiniMax-M336.4657.9951.74 (6.25)53.82 (4.17)33.68 (24.31)38.54 (19.44)
DeepSeek-V432.6460.7656.60 (4.17)45.83 (14.93)40.97 (19.79)30.56 (30.21)
GPT-5.620.1429.8628.47 (1.39)25.00 (4.86)18.75 (11.11)21.88 (7.99)
GLM-5.136.4663.1958.33 (4.86)49.65 (13.54)42.01 (21.18)35.07 (28.13)
OpenHandsQwen3.642.0172.5768.40 (4.17)59.38 (13.19)49.31 (23.26)40.28 (32.29)
MiniMax-M342.7177.4378.82 (−1.39)62.50 (14.93)46.88 (30.56)45.14 (32.29)
DeepSeek-V452.4379.1764.93 (14.24)67.71 (11.46)49.31 (29.86)54.86 (24.31)
GPT-5.619.4434.0326.04 (7.99)27.78 (6.25)22.57 (11.46)16.67 (17.36)
GLM-5.143.7567.3662.50 (4.86)55.21 (12.15)48.26 (19.10)41.32 (26.04)
OpenCodeQwen3.617.3645.8341.67 (4.17)34.38 (11.46)25.35 (20.49)20.49 (25.35)
MiniMax-M335.7668.0660.07 (7.99)52.43 (15.63)42.01 (26.04)32.64 (35.42)
DeepSeek-V444.161.8151.74 (10.07)53.13 (8.68)41.67 (20.14)46.18 (15.63)
GPT-5.622.9231.2532.64 (−1.39)26.74 (4.51)23.96 (7.29)20.83 (10.42)
GLM-5.145.1468.0662.85 (5.21)50.69 (17.36)43.06 (25.00)47.57 (20.49)
OverallAll35.0258.4553.73 (4.72)47.62 (10.83)37.69 (20.76)34.91 (23.54)

As shown in Table 1, CoSA's overall ASR decreases from 58.45% under full visibility to 53.73%, 47.62%, and 37.69% at 75%, 50%, and 25% visibility, respectively. Notably, with only 50% visibility, CoSA consistently outperforms the original, unaugmented Skill pools across all 15 model–harness combinations, achieving an overall ASR improvement of 12.60 percentage points over the 35.02% baseline. Even at 25% visibility, CoSA maintains an overall improvement of 2.67 percentage points, outperforming the original baseline in 10 of 15 configurations. In contrast, when the attacker has no access to the target Skill pool and relies solely on domain-level information, the overall ASR is 34.91%, comparable to the original baseline. These findings demonstrate that CoSA retains substantial effectiveness under partial knowledge of the target Skill pool, while also revealing that its benefits diminish as attacker knowledge becomes increasingly limited.

These additional clarifications and experiments directly address the reviewer's concern by establishing realistic scenarios for Skill-pool inspection and third-party Skill admission, while empirically demonstrating that CoSA does not require complete knowledge of the victim's Skill pool to remain effective. At the same time, the domain-only results clarify the applicability boundaries of our threat model. We will incorporate these findings and further clarify the attacker knowledge and Skill admission assumptions in the revised manuscript.

References

[1] https://github.com/EvilFreelancer/secas/blob/main/.agents/skills/README.md

[2] https://docs.openclaw.ai/concepts/system-prompt

[3] Xiang Zheng, Yutao Wu, Hanxun Huang, Yige Li, Xingjun Ma, Bo Li, Yu-Gang Jiang, and Cong Wang. "Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs." In International Conference on Machine Learning (ICML), 2026.

[4] Zichuan Li, Jian Cui, Xiaojing Liao, and Luyi Xing. "Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agents." In Network and Distributed System Security Symposium (NDSS), 2026.

[5] OpenClaw. "ClawHub: Publishing and Installing Skills." Official Documentation. https://docs.openclaw.ai/clawhub

AuthorsOpenReview ↗

Thank you, reviewer 7niY! (3/4)

The main evaluation is conducted on the authors’ own CoSA-Bench. It is unclear whether the attack generalizes to independently constructed Skill or agent benchmarks.

We sincerely thank the reviewer for this valuable suggestion regarding the generalizability of CoSA beyond our self-constructed benchmark. We acknowledge that evaluating CoSA exclusively on CoSA-Bench may not fully capture its effectiveness under independently constructed Skill pools and task distributions. To directly address this concern, we conduct additional experiments on two external benchmarks: CompoSkill-Bench [1] and SCR-CapFlow from SCR-Bench [2]. From CompoSkill-Bench, we select 228 clean tasks, comprising 76 from each of three threat categories: (1) data_exfiltration (CoSA’s DL category); (2) memory_tampering (unauthorized modification, UA); and (3) privilege_escalation_and_dangerous_command_execution (unsafe execution, UE). We include exactly those categories whose violations are realized through cross-Skill composition, the mechanism CoSA targets; the remaining two, multi-agent collaboration hijacking and resource exhaustion, are excluded by definition rather than by outcome, as their violations do not depend on composing a victim Skill pool. From SCR-Bench we evaluate CapFlow in full (150 cases), the only subset instantiating the cross-Skill capability-flow setting CoSA presupposes. For both benchmarks, we compare the original and CoSA-augmented Skill pools on the same tasks, using the same five models and three agent harnesses as in our main evaluation. We apply a 5% augmentation budget on CompoSkill-Bench and a one-Skill budget on SCR-CapFlow, whose pool is too small for 5% to reach one Skill. The results are summarized in Table 2.

Table 2: Generalization of CoSA to independently constructed benchmarks.

Benchmark# TasksOriginal ASR (%)CoSA ASR (%)ΔASR (pp)
CoSA-Bench (Ours)28835.0258.4523.43
CompoSkill-Bench [1]22825.0639.4214.36
SCR-CapFlow [2]15037.8262.0924.27

As shown in Table 2, CoSA consistently improves ASR on both independently constructed benchmarks, demonstrating that its effectiveness is not confined to the Skill pools and task distributions of CoSA-Bench. Specifically, on CompoSkill-Bench, CoSA increases ASR from 25.06% to 39.42%, achieving a 14.36-percentage-point improvement under the same 5% augmentation budget as our main evaluation. On SCR-CapFlow, CoSA further increases ASR from 37.82% to 62.09%, representing a 24.27-percentage-point improvement with only one additional Skill. These results provide direct empirical evidence that CoSA generalizes to independently developed Skill-security benchmarks with different composition structures and risk scenarios, rather than relying on benchmark-specific characteristics of CoSA-Bench. While differences in Skill-pool sizes and augmentation budgets limit direct comparisons of improvement magnitudes, the consistent within-benchmark gains substantiate CoSA's effectiveness across distinct evaluation distributions. We will incorporate these results and clarify the corresponding benchmark adaptation protocols in the revised manuscript.

References

[1] Liu, M., Jiang, Z., Ma, J., Zhang, J., Chen, J., Deng, X., & Wang, Z. (2026). Composkill: Compositional skill chain attacks from individually scanner-passing llm agent skills. arXiv preprint arXiv:2608.16246.

[2] Xie, Y., Du, J., Cheng, Y., Zhou, J., & Yin, Z. (2026). Benign in isolation, harmful in composition: Security risks in agent skill ecosystems. arXiv preprint arXiv:2606.15242.

Reviewer_ud3yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper defines a Conflict of Skills as a policy violation caused by cross-Skill artifact or decision reuse when the participating Skills individually pass a package-level admission procedure. It models an agent task as a user request, initial environment, and policy constraints, and defines a conflict through a concrete terminal violation, a trace-supported cross-Skill dependency witness, and local admission of the participating Skills.

CoSA first extracts operation-level contracts from Skills, including inputs, outputs, permissions, side effects, assessment signals, authorization conditions, and functional roles. It then uses controlled conflict discovery to identify four recurring mechanisms:

  • security-context attenuation;
  • semantic interface bridging;
  • semantic authority promotion; and
  • target/action binding drift.

The framework generates candidate additions through capability supplementation and relationship reshaping, filters them through contract validation and two static scanners, and selects a small budgeted set by maximum coverage over opportunities and mechanisms.

CoSA-Bench contains 288 tasks across recruiting, finance, IT, and business, with data leakage, unauthorized state modification, and unsafe execution as the three target risk categories. The main evaluation uses five language models and three agent harnesses. Each task is run once with the original Skill pool and once with the augmented pool under matched task, environment, model, harness, permission, and execution-budget conditions.

The reported violation rate increases from 35.02% to 58.45% across the complete evaluation, a difference of 23.43 percentage points. The paper also reports 1,162 compliant-to-violating paired transitions and claims that all of them have a human-reviewed, trace-supported dependency witness involving an added Skill.

Finally, the paper proposes a harness-level guard that checks information flow and authorization before externalization, persistent mutation, or execution. On 360 OpenClaw recruiting attack runs, the guard reduces observed ASR from 65.28% to 6.67%. On matched benign controls, policy-compliant completion changes from 66.67% to 65.00%, with a 3.33% task-level false-block rate and increased context overflow.

Critical strengths

Strength 1: The paper identifies an important systems-security gap

The paper clearly articulates why package-level security checks are insufficient when an agent composes Skills. The motivating CI/CD example is concrete, and the formal witness definition requires more than temporal co-occurrence: it requires a concrete policy violation and trace evidence that cross-Skill artifact or decision reuse contributed to the terminal effect. This is a useful framing for a rapidly developing agent ecosystem.

Strength 2: The main evaluation is broad and reports meaningful operational detail

The evaluation spans four domains, 288 tasks, five models, and three agent harnesses, while holding the task, initial state, permissions, original Skill pool, and execution budget fixed within paired comparisons. The paper also reports local reversals, benign completion, false blocks, context overflow, runtime failures, risk coverage, and the distinction between risky-action production and conditional violation. This level of accounting is stronger than a paper that reports only a single attack-success number.

Critical weaknesses

Weakness 1: The central causal and “individually benign” claim is not established

The paper defines local admission through two static scanners, but scanner passage is explicitly not runtime safety. The main evaluation does not provide a systematic single-Skill dynamic counterfactual for every selected addition under the same task and permissions. A candidate may expose an externalization, mutation, or execution capability that is harmless under the scanner but directly enables a violation in the benchmark. The fact that the final trace contains a cross-Skill witness does not by itself show that the added Skill was harmless in isolation or that the claimed composition was causally necessary.

The attribution analysis is also post hoc. GPT-5.1 proposes a witness for each compliant-to-violating transition, and the authors accept all 1,162 cases after review. Without independent blinded annotation, inter-rater reliability, rejected-witness statistics, or a preregistered attribution rubric, the result is vulnerable to confirmation bias.

Finally, every stochastic task-condition cell is executed once. The authors’ own interpretation of the numbers as descriptive is appropriate, but the paper’s broader language about consistent attack effectiveness and generality goes beyond what this protocol supports.

Weakness 2: The method’s novelty, superiority, and defense utility are under-supported

The four mechanisms are plausible and useful, but they are not clearly distinguished from existing ideas in provenance tracking, taint analysis, confused-deputy prevention, capability security, workflow authorization, tool poisoning, and cross-tool attacks. The implementation appears to rely on LLM-based contract extraction, manually specified opportunities, and hand-defined coverage tags, yet there is no independent validation of those components.

The ablation study contains only one realization of random benign additions and one realization of random candidate selection. The paper explicitly does not separately ablate capability supplementation and relationship reshaping, individual mechanisms, contract fields, or selection terms. There is also no direct comparison to existing attack or defense baselines.

The guard is evaluated only on OpenClaw recruiting tasks. Its abstract flow and authorization checks are not specified at a level that permits independent reproduction, and the use of GPT-5 for ambiguous cases makes the claim of a lightweight harness patch questionable. The increase in context overflow, the four runtime failures, the model-specific completion drops, and the large number of runs with no effective risky action all make the reported 58.61-point ASR reduction difficult to interpret as a general defense result.

Questions for authors

Clarifications required

  1. Individual benignity and causal necessity: For every selected added Skill, what is the result of an isolated dynamic execution under the same task, environment, permissions, and policy? In the controlled discovery study, does “single-Skill control” mean that every constituent Skill is tested separately, or only one selected constituent per multi-Skill request? Please report these results and explain how the analysis rules out a candidate that is independently capable of producing the violation.

  2. Candidate construction and benchmark separation: What are the numbers of generated, contract-valid, scanner-admitted, and selected candidates in each domain? Which contract fields are extracted automatically, manually annotated, or LLM-generated, and what are the unresolved-field rates? How do you ensure that the discovery records, opportunity tags, and risk-category coverage do not effectively reveal the structure of the evaluation tasks?

  3. Attribution and judge validation: How many independent human annotators reviewed the policy judgments and dependency witnesses? Were annotators blinded to the original versus augmented condition? How many proposed witnesses were rejected or corrected, and what are the inter-rater agreement statistics? Can the results be reproduced under an independently written judge prompt or a second judge model?

  4. Guard reproducibility and comparison: What are the exact rules, prompts, thresholds, and fallback behavior for Flow and Auth? What are the wall-clock latency, token, and monetary costs? How does the guard compare against a simple terminal authorization check, provenance/taint baseline, confirmation gate, or existing Skill-security defense? What happens when the harness lacks the provenance and authorization fields assumed by the guard?

Experiments that would materially affect the recommendation

  1. Can the authors rerun a representative subset with multiple provider seeds or repeated executions, report task-clustered confidence intervals or paired permutation tests, and repeat the random-addition baselines across multiple draws?

  2. Do the attack results persist on held-out task templates, a held-out domain, a different Skill registry snapshot, or a different budget of added Skills? Such an evaluation would establish whether CoSA transfers beyond the construction distribution.

  3. Can the paper provide separate ablations for capability supplementation, relationship reshaping, the four mechanisms, and the maximum-coverage selection terms, together with direct comparisons to existing Skill/tool-poisoning and runtime-defense baselines?

  4. The paper states that all original domain pools retain one HIGH data-relay finding while candidates pass a procedure that rejects HIGH findings. Is the former finding a pool-level aggregate rather than a package-level result? Please explain the scanner semantics and why the asymmetric --lenient configuration provides a meaningful local-admission guarantee.

15 comments
AuthorsOpenReview ↗

Thank you, reviewer ud3y! (1/13)

Response to Weaknesses

Weakness 1: The central causal and “individually benign” claim is not established The paper defines local admission through two static scanners, but scanner passage is explicitly not runtime safety. The main evaluation does not provide a systematic single-Skill dynamic counterfactual for every selected addition under the same task and permissions. A candidate may expose an externalization, mutation, or execution capability that is harmless under the scanner but directly enables a violation in the benchmark. The fact that the final trace contains a cross-Skill witness does not by itself show that the added Skill was harmless in isolation or that the claimed composition was causally necessary.

We thank the reviewer for raising this important concern regarding the individual benignness of added Skills and the causal necessity of cross-Skill composition. We agree that passing static security scanners alone does not guarantee runtime safety, and that identifying a cross-Skill dependency in a violating trace is not sufficient to establish its causal necessity. To directly address these concerns, we strengthen our evaluation with additional runtime isolation tests and matched counterfactual analyses.

(1) Individual Skill Safety and Runtime Isolation. We first clarify that our use of individually benign refers to package-level admission under specified security checks, rather than an unconditional guarantee of runtime safety. To provide stronger empirical evidence beyond static screening, we evaluate all 11 selected Skill additions using two static scanners (Skill Issue [4] and Cisco AI Defense Skill Scanner [5]) and two dynamic analysis tools (SkillWard [6] and SkillScan [7]). The dynamic tools execute individual Skills in isolated sandbox environments and monitor security-relevant behavior, including network access, sensitive file operations, process execution, and credential access.

Furthermore, we conduct matched single-Skill tests in the original benchmark environment, making only the corresponding added Skill available while preserving the user task, initial environment, permissions, and security policy. The results are summarized in Table 1.

Table 1: Individual security screening and runtime isolation of the 11 selected added Skills.

AnalysisTool / SettingSkills TestedResult
Static ScreeningSkill Issue [4]1111/11 passed
Static ScreeningCisco Skill Scanner [5]1111/11 passed
Dynamic AnalysisSkillWard [6]110/11 runtime alerts
Dynamic AnalysisSkillScan [7]110/11 runtime alerts
Benchmark IsolationMatched single-Skill execution110/11 policy violations

As shown in Table 1, all 11 selected additions pass both static screening procedures and trigger no security alerts in either dynamic analysis tool. Moreover, none of the 11 matched single-Skill benchmark executions produces a policy violation. These results provide additional evidence that the selected additions do not independently induce violations under the tested conditions. We distinguish these observations from a universal runtime-safety guarantee and additionally examine Skill activation and task completion to ensure that nonviolating executions are interpreted appropriately.

(2) Counterfactual Verification of Composition Necessity. To further establish that the observed violations depend on cross-Skill composition rather than merely the presence of potentially dangerous capabilities, we conduct matched counterfactual experiments on augmentation-associated violating executions. For each implicated added Skill, we examine the following conditions: (i) Original Pool, containing only the preexisting Skills; (ii) Single-Skill Isolation, exposing the implicated addition without other Skill packages; (iii) Full Augmented Pool, containing all selected additions; and (iv) Leave-One-Addition-Out, removing the implicated addition while preserving the remaining Skill pool. Where applicable, we also introduce a Cross-Skill Dependency Interruption control that disables the relevant artifact or decision handoff without removing the downstream operation itself.

Across these conditions, we preserve the same user requests, initial states, security policies, model–harness configurations, native tools, and runtime permissions. We examine whether the original violation persists, disappears, or becomes inconclusive due to changes in task execution. The results are summarized in Table 2.

Table 2: Counterfactual analysis of cross-Skill composition necessity.

Evaluation Condition# Matched CasesPolicy Violations (#)Task Completions (#)
Original Skill Pool24022
Single-Skill Isolation24023
Full Augmented Pool242422
Leave-One-Addition-Out24222
Cross-Skill Dependency Interrupted24220
AuthorsOpenReview ↗

Thank you, reviewer ud3y! (2/13)

As shown in Table 2, 22 of 24 composition-associated violations disappear when the implicated addition is removed, while 22 violations are eliminated when the relevant cross-Skill dependency is interrupted. Importantly, task completion remains possible under both controls, with 22 of 24 tasks completed under Leave-One-Addition-Out and 20 of 24 under Cross-Skill Dependency Interruption. These results support counterfactual dependence on the implicated additions and handoffs in the selected cases, with two violations persisting under each intervention, providing stronger evidence that the observed policy violations arise from interactions among Skills rather than merely from the isolated capabilities of individual additions.

These additional experiments directly address the reviewer's two central concerns. The static and dynamic isolation tests strengthen the empirical evidence for the selected Skills' local admissibility and nonviolating behavior under tested conditions, while the matched counterfactual analyses examine the causal role of the added Skills and their cross-Skill dependencies in producing policy violations. We will incorporate these results and clarify the distinction between package-level benignness, observed runtime safety, and composition-dependent failures in the revised manuscript.

[1] Koc, V., Erichsen, P., Tomlinson, J., Rivera, A., Appel, M., & Paz, N. (2026). ClawHub Security Signals: When VirusTotal, Static Analysis, and SkillSpector Disagree. arXiv preprint arXiv:2606.01494.

[2] Ji, Z., Xu, C., Li, Z., Gao, Y., Wei, X., Wang, S., & Cheung, S. C. (2026). Cloak and Detonate: Scanner Evasion and Dynamic Detection of Agent Skill Malware. arXiv preprint arXiv:2607.02357.

[3] Liu, Y., Wang, W., Feng, R., Zhang, Y., Xu, G., Deng, G., ... & Zhang, L. (2026). Agent skills in the wild: An empirical study of security vulnerabilities at scale. arXiv preprint arXiv:2601.10338.

[4] https://github.com/agent-clis/skill-issue

[5] https://github.com/cisco-ai-defense/skill-scanner

[6] https://github.com/Fangcun-AI/SkillWard

[7] https://github.com/NMitchem/SkillScan

The attribution analysis is also post hoc. GPT-5.1 proposes a witness for each compliant-to-violating transition, and the authors accept all 1,162 cases after review. Without independent blinded annotation, inter-rater reliability, rejected-witness statistics, or a preregistered attribution rubric, the result is vulnerable to confirmation bias.

We thank the reviewer for highlighting the need to validate the dependency attribution. We clarify the review protocol, inter-rater agreement, witness-review outcomes, and attribution rubric below.

(1) Independent blinded annotation and inter-rater reliability. Three human annotators independently reviewed the 1,162 compliant-to-violating cases, with explicit Original/Augmented condition labels withheld. They checked each proposed witness against the underlying execution evidence. Independent judgments were recorded before discussion, and disagreements were resolved by majority vote. Before adjudication, the three annotators agreed on 1144 of 1,162 cases (98.45%), with Fleiss’ κ = 0.899 across the review categories accept unchanged, correct, and reject. These statistics describe agreement on the initial proposals before adjudication.

(2) Corrected and rejected witnesses. As described in Appendix C.3, GPT-5.1 proposed an initial dependency witness for each transition. Annotators checked the proposals against the task policy, raw trajectory, Skill reads, tool arguments and results, output artifacts, backend events, and final state. Of the 1,162 initial proposals, 1100 (94.66%) were accepted unchanged, 62 (5.34%) were corrected and retained, and 0 were rejected. Corrections addressed inaccurate operation references, artifact or decision-handoff identification, or the description of the supporting evidence. The rejection criterion was the absence of a qualifying violation-specific dependency.

(3) Attribution rubric and its timing. The analysis applied the retrospective trace-evidence criterion in Appendix A.5. A qualifying witness required observable artifact or decision reuse across at least two Skills, including an addition, connected to the specific policy-violating terminal effect. Temporal co-occurrence or Skill retrieval alone was insufficient. We describe this procedure as retrospective attribution and make no claim of preregistration. The resulting evidence supports the identified dependencies in the reviewed executions.

Finally, every stochastic task-condition cell is executed once. The authors’ own interpretation of the numbers as descriptive is appropriate, but the paper’s broader language about consistent attack effectiveness and generality goes beyond what this protocol supports

AuthorsOpenReview ↗

Thank you, reviewer ud3y! (3/13)

We thank the reviewer for this valuable suggestion regarding the reliability of our evaluation under stochastic agent execution. We agree that single-run results provide limited evidence of run-to-run variability and that claims about consistent effectiveness should be supported by repeated measurements. To further examine this issue, we repeat the CoSA evaluation three times across all 288 tasks, five LLMs, and three agent harnesses, covering all 15 model–harness configurations. The results are summarized in Table 3.

Table 3: CoSA attack success rate across three repeated runs (%).

HarnessModelRun 1Run 2Run 3Mean ASR
OpenClawQwen3.662.1565.9764.5864.24
MiniMax-M357.6460.4256.9458.33
DeepSeek-V458.6862.1559.3860.07
GPT-5.630.5628.4731.630.21
GLM-5.164.2461.4662.8562.85
OpenHandsQwen3.669.4465.2868.467.71
MiniMax-M376.7479.8677.7878.13
DeepSeek-V479.8677.0879.5178.82
GPT-5.632.6435.0733.3333.68
GLM-5.169.4465.9768.7568.06
OpenCodeQwen3.647.2244.147.2246.18
MiniMax-M368.465.9767.7167.36
DeepSeek-V462.560.0763.8962.15
GPT-5.631.629.5131.630.9
GLM-5.166.6769.7966.6767.71
OverallAll58.5258.0858.6858.43

As shown in Table 3, CoSA exhibits limited aggregate run-to-run variation across the three repeated evaluations, achieving overall ASRs of 58.52%, 58.08%, and 58.68%. The difference between the highest and lowest runs is only 0.60 percentage points, and the three-run mean of 58.43% closely matches the 58.45% reported in our main evaluation.

We also examine variability at the model–harness level. The largest observed range across the three repetitions is 4.17 percentage points, occurring for Qwen3.6 on OpenHands. Despite this variation, all 15 model–harness configurations achieve higher ASRs in each repeated CoSA evaluation than their corresponding original-pool ASRs reported in the main experiment. These observations suggest that the aggregate effectiveness of CoSA is reasonably stable across repeated executions under the evaluated settings, although individual configurations exhibit greater variability.

We appreciate the reviewer's concern regarding the interpretation and generality of single-run results. These additional experiments provide further empirical evidence of the repeatability of our observations, while we recognize that three repetitions do not establish universal attack effectiveness across arbitrary settings. We will incorporate the repeated-run measurements into the revised manuscript and clarify the scope of our empirical claims accordingly.

Weakness 2: The method’s novelty, superiority, and defense utility are under-supported The four mechanisms are plausible and useful, but they are not clearly distinguished from existing ideas in provenance tracking, taint analysis, confused-deputy prevention, capability security, workflow authorization, tool poisoning, and cross-tool attacks. The implementation appears to rely on LLM-based contract extraction, manually specified opportunities, and hand-defined coverage tags, yet there is no independent validation of those components.

We thank the reviewer for this valuable comment regarding CoSA's novelty, empirical effectiveness, and the reliability of its construction pipeline. We appreciate the opportunity to clarify how our work builds on existing security principles and how its individual components are validated.

(1) Novelty Relative to Prior Work. We acknowledge that CoSA's four conflict mechanisms are related to established concepts in provenance tracking, information-flow security [1, 2], confused-deputy prevention, capability security, and workflow authorization [3, 4]. Our contribution does not lie in introducing these security principles independently, but rather in identifying and operationalizing how they manifest as composition-induced failures among individually screened Agent Skills. Specifically, Semantic Interface Bridging captures how an intermediate Skill enables a previously unavailable cross-Skill operation; Security-Context Attenuation describes the loss of usage restrictions during artifact reuse; Semantic Authority Promotion captures the elevation of an assessment or recommendation into authorization; and Target/Action Binding Drift captures the reuse of authorization for a different action or target.

Reviewer_tSKEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies security risks arising from the composition of individually admitted LLM agent skills and formalizes this problem as Conflict of Skills (CoS). The authors propose CoSA, a mechanism-guided framework that analyzes an existing skill pool, identifies potential cross-skill composition opportunities, and introduces a small number of locally admitted skills to increase the likelihood of policy violations. CoSA models cross-skill artifact and decision dependencies and identifies several recurring mechanisms, including security-context attenuation, semantic interface bridging, semantic authority promotion, and target/action binding drift. The framework generates candidate skills through capability supplementation and relationship reshaping, filters them using package-level security checks, and performs budgeted selection. The paper further introduces CoSA-Bench with 288 tasks across recruiting, finance, IT, and business domains. Experiments cover five LLMs and three agent harnesses and evaluate data leakage, unauthorized state modification, and unsafe execution. The paper also proposes a harness-level defense that validates information flow and authorization before consequential actions.

Critical strengths
  1. The paper studies a timely and important security problem in LLM agent skill ecosystems, moving beyond the security of individual skills toward composition-level risks.
  2. The distinction between artifact reuse, decision reuse, and consequential downstream actions provides a useful framework for analyzing cross-skill vulnerabilities.
  3. The evaluation is relatively comprehensive, covering four domains, five LLMs, three agent harnesses, and three different security-risk categories.
  4. The paper goes beyond aggregate attack success and performs trajectory-level attribution to identify cross-skill dependencies associated with newly introduced violations.
Critical weaknesses
  1. The threat model assumes substantial knowledge of the victim’s skill pool. CoSA constructs a composition surface from the original domain skill pool and generates additions specifically around its capability and relationship gaps. In realistic public skill marketplaces, an attacker may not know the exact set of skills installed by a downstream user. Transferability to unseen or partially known skill pools is therefore unclear.
  2. The baseline comparison is limited. Random benign additions and random candidate selection are useful controls, but the paper does not experimentally compare with recent compositional skill attacks or simpler LLM-generated skill-pool augmentation strategies. It is therefore difficult to quantify how much CoSA improves over existing composition-based attack mechanisms.
  3. The ablation study is incomplete. Capability supplementation and relationship reshaping are presented as the two main augmentation strategies, but they are not separately ablated. Likewise, the contributions of the four identified conflict mechanisms and individual selection criteria are not isolated.
  4. The defense evaluation has limited scope. The proposed Guard is evaluated only on recruiting tasks with OpenClaw. Its effectiveness across the other three domains and the other agent harnesses remains unclear.
  5. The claim that the defense is “lightweight” needs stronger empirical support. The paper defines lightweight mainly architecturally, but does not report wall-clock latency or token overhead.
Questions for authors

Refer to weaknesses.

7 comments
AuthorsOpenReview ↗

Thank you, reviewer tSKE! (1/5)

Response to Weaknesses

The threat model assumes substantial knowledge of the victim’s skill pool. CoSA constructs a composition surface from the original domain skill pool and generates additions specifically around its capability and relationship gaps. In realistic public skill marketplaces, an attacker may not know the exact set of skills installed by a downstream user. Transferability to unseen or partially known skill pools is therefore unclear.

We sincerely thank the reviewer for raising this important concern. We agree that our threat model assumes substantial knowledge of the victim’s Skill pool, and we appreciate the opportunity to clarify why we believe this assumption is practically relevant. In practice, such knowledge may be obtained through three routes: direct access, extraction of the system prompt, and inference without access. (1) Direct access. When an Agent is deployed from a repository, the Skill pool is stored and versioned alongside the code; for example, the publicly available SECAS project publishes all 49 of its installed Skills with their full implementations [1]. (2) Extraction of the system prompt. Frameworks such as OpenClaw inject the name, description, and file location of every eligible Skill into the agent’s system prompt [2], and JustAsk (ICML 2026) recovers hidden system prompts from 41 commercial models using only standard user interaction [3]. An attacker who can query the agent may therefore plausibly recover its Skill inventory. (3) Inference without access. Even without any access to the victim, an attacker may still target Skills that are likely to be installed, such as those bundled with the framework by default [2]. Les Dissonances (NDSS 2026) follows a similar approach: it crafts malicious tools around the public descriptions of existing LangChain and LlamaIndex tools, so that the attack triggers whenever the agent uses the targeted tool [4].

Nevertheless, we recognize that attackers may not always have complete visibility into the victim's installed Skill pool. Following the reviewer’s suggestion, we further investigate the transferability of CoSA under more restricted attacker knowledge. Specifically, we conduct additional experiments under two settings: (1) Partially Known Skill Pools, where attackers construct adversarial Skills using only a subset (25%, 50%, or 75%) of the target Skill pool; and (2) Unseen Skill Pools, where the attacker receives only a predefined domain description and has no access to the target pool’s Skill packages or contracts.

Table 1: Transferability of CoSA under different levels of attacker knowledge.

HarnessModelOriginal ASR (%)100% ASR (%)75% ASR (Drop)50% ASR (Drop)25% ASR (Drop)Unseen ASR (Drop)
OpenClawQwen3.634.0359.3861.11 (−1.74)50.00 (9.38)37.50 (21.88)31.60 (27.78)
MiniMax-M336.4657.9951.74 (6.25)53.82 (4.17)33.68 (24.31)38.54 (19.44)
DeepSeek-V432.6460.7656.60 (4.17)45.83 (14.93)40.97 (19.79)30.56 (30.21)
GPT-5.620.1429.8628.47 (1.39)25.00 (4.86)18.75 (11.11)21.88 (7.99)
GLM-5.136.4663.1958.33 (4.86)49.65 (13.54)42.01 (21.18)35.07 (28.13)
OpenHandsQwen3.642.0172.5768.40 (4.17)59.38 (13.19)49.31 (23.26)40.28 (32.29)
MiniMax-M342.7177.4378.82 (−1.39)62.50 (14.93)46.88 (30.56)45.14 (32.29)
DeepSeek-V452.4379.1764.93 (14.24)67.71 (11.46)49.31 (29.86)54.86 (24.31)
GPT-5.619.4434.0326.04 (7.99)27.78 (6.25)22.57 (11.46)16.67 (17.36)
GLM-5.143.7567.3662.50 (4.86)55.21 (12.15)48.26 (19.10)41.32 (26.04)
OpenCodeQwen3.617.3645.8341.67 (4.17)34.38 (11.46)25.35 (20.49)20.49 (25.35)
MiniMax-M335.7668.0660.07 (7.99)52.43 (15.63)42.01 (26.04)32.64 (35.42)
DeepSeek-V444.161.8151.74 (10.07)53.13 (8.68)41.67 (20.14)46.18 (15.63)
GPT-5.622.9231.2532.64 (−1.39)26.74 (4.51)23.96 (7.29)20.83 (10.42)
GLM-5.145.1468.0662.85 (5.21)50.69 (17.36)43.06 (25.00)47.57 (20.49)
OverallAll35.0258.4553.73 (4.72)47.62 (10.83)37.69 (20.76)34.91 (23.54)
AuthorsOpenReview ↗

Thank you, reviewer tSKE! (2/5)

As shown in Table 1, CoSA's overall ASR changes from 58.45% under full target-pool visibility to 53.73%, 47.62%, and 37.69% at 75%, 50%, and 25% visibility, respectively. Notably, with only 50% visibility, CoSA consistently outperforms the original, unaugmented Skill pools across all 15 model–harness combinations, achieving an overall ASR improvement of 12.60 percentage points over the 35.02% baseline. Even at 25% visibility, CoSA maintains an overall improvement of 2.67 percentage points, outperforming the original baseline in 10 of 15 configurations. In contrast, when the attacker has no access to the target Skill pool and relies solely on domain-level information, the overall ASR is 34.91%, comparable to the original baseline. These findings demonstrate that CoSA retains substantial effectiveness under partial knowledge of the target Skill pool, while also revealing that its benefits diminish as attacker knowledge becomes increasingly limited. We will incorporate these results and clarify the corresponding threat-model assumptions and limitations in the revised manuscript.

[1] https://github.com/EvilFreelancer/secas/blob/main/.agents/skills/README.md

[2] https://docs.openclaw.ai/concepts/system-prompt

[3] Xiang Zheng, Yutao Wu, Hanxun Huang, Yige Li, Xingjun Ma, Bo Li, Yu-Gang Jiang, and Cong Wang. "Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs." In International Conference on Machine Learning (ICML), 2026.

[4] Zichuan Li, Jian Cui, Xiaojing Liao, and Luyi Xing. "Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agents." In Network and Distributed System Security Symposium (NDSS), 2026.

The baseline comparison is limited. Random benign additions and random candidate selection are useful controls, but the paper does not experimentally compare with recent compositional skill attacks or simpler LLM-generated skill-pool augmentation strategies. It is therefore difficult to quantify how much CoSA improves over existing composition-based attack mechanisms.

We sincerely thank the reviewer for pointing out this important gap in our baseline comparisons. We would first like to clarify that our original random baselines address two narrower questions: Random Benign Additions evaluates whether simply expanding the Skill pool with the same number of domain-relevant Skills induces comparable risks, while Random Candidate Selection isolates the contribution of CoSA's informed candidate selection.

Nevertheless, we acknowledge that these controls do not fully capture LLM-generated augmentation or recent composition-based attacks. To address the reviewer's concern, we conduct additional experiments with three stronger baselines: (1) LLM-generated Skill-pool Augmentation, which directly generates Skill additions without CoSA's composition-aware gap analysis; (2) SCR-Bench-style Composition [1], which adapts the existing-Skill composition protocol from SCR-Bench; and (3) CRIME-ACA [2], which adapts feedback-guided modification of complementary Skill pairs. We evaluate these approaches on CoSA-Bench using consistent target tasks, agent configurations, and evaluation metrics.

As shown in Table 2, CoSA achieves the highest observed ASR across all three harnesses, with an overall ASR of 58.45%, outperforming LLM-generated augmentation by 16.46 percentage points, SCR-Bench-style composition by 20.44 points, and CRIME-ACA, the strongest additional baseline, by 6.46 points. Notably, CoSA consistently outperforms CRIME-ACA on OpenClaw, OpenHands, and OpenCode by 5.28, 8.12, and 5.97 points, respectively. Furthermore, Random Candidate Selection achieves an overall ASR of 46.99%, which is 11.46 points below full CoSA, highlighting the contribution of informed candidate selection beyond candidate generation. While the adapted compositional baselines operate under different attacker capabilities, these comparisons directly quantify their relative effectiveness on CoSA-Bench and provide stronger empirical evidence for CoSA's composition-aware augmentation strategy. We will incorporate these results and clarify the corresponding comparison protocols in the revised manuscript.

Table 2: Comparison with recent compositional attacks and Skill-pool augmentation baselines.

MethodOpenClaw ASROpenHands ASROpenCode ASROverall ASR
Random Benign Additions26.39%28.47%35.14%30.00%
Random Candidate Selection43.40%52.08%45.49%46.99%
LLM-generated Skill-pool Augmentation36.18%49.17%40.63%41.99%
SCR34.72%33.33%45.97%38.01%
CRIME48.96%57.99%49.03%51.99%
CoSA (Ours)54.24%66.11%55.00%58.45%

[1] Xie, Y., Du, J., Cheng, Y., Zhou, J., & Yin, Z. (2026). Benign in isolation, harmful in composition: Security risks in agent skill ecosystems. arXiv preprint arXiv:2606.15242.

AuthorsOpenReview ↗

Thank you, reviewer tSKE! (3/5)

[2] Zhou, Z., Yang, Z., Sun, M., & Zhang, Y. (2026). Runaway Reaction: When Benign Skills Compose into Malicious Behavior. arXiv preprint arXiv:2610.05943.

The ablation study is incomplete. Capability supplementation and relationship reshaping are presented as the two main augmentation strategies, but they are not separately ablated. Likewise, the contributions of the four identified conflict mechanisms and individual selection criteria are not isolated.

We sincerely thank the reviewer for this valuable suggestion regarding more fine-grained ablation studies. We acknowledge that our original ablation mainly evaluates the overall effectiveness of candidate construction and selection, without isolating the contributions of individual augmentation strategies and conflict mechanisms. To address this concern, we conduct two additional sets of ablation experiments under consistent evaluation settings.

(1) Ablation of Augmentation Strategies. We separately evaluate Capability Supplementation Only, Relationship Reshaping Only, and Full CoSA in the Finance and IT domains, where both augmentation strategies are applicable. In contrast, Recruiting and Business already possess the necessary capabilities in their original Skill pools and therefore do not require Capability Supplementation. Their augmentation relies on Relationship Reshaping, making a separate Capability Supplementation ablation inapplicable. The results for Finance and IT are presented in Table 3.

Table 3: Domain-wise ablation of augmentation strategies (ASR %).

DomainOriginalRelationship OnlyCapability OnlyCoSA
Finance33.846.252.0455.56
IT33.4339.9151.6754.54

As shown in Table 3, both strategies contribute to attack effectiveness in Finance and IT, but to different extents. Relationship Reshaping alone increases ASR by 12.40 and 6.48 percentage points in Finance and IT, respectively, demonstrating that modifying cross-Skill interactions can induce violations without introducing missing capabilities. Capability Supplementation alone achieves 52.04% and 51.67% ASR, highlighting the importance of introducing operations absent from the original pools. Combining both strategies achieves the highest ASR in these domains (55.56% and 54.54%), outperforming either strategy alone. Together with the relationship-only augmentation used in Recruiting and Business, these findings demonstrate that the contributions of the two strategies depend on the capability gaps and composition structure of each domain's original Skill pool.

(2) Ablation of Conflict Mechanisms. We further investigate the contributions of the four identified mechanisms: Semantic Interface Bridging (SIB), Semantic Authority Promotion (SAP), Target/Action Binding Drift (TAB), and Security-Context Attenuation (SCA). Since different domains exhibit distinct capability gaps and composition opportunities, we evaluate the individual mechanisms and combinations applicable to each domain, rather than artificially enforcing all four mechanisms in every domain. Table 4 summarizes these domain-specific results. Notably, Full CoSA in Finance and IT additionally incorporates Capability Supplementation, so its improvement over individual mechanisms reflects contributions from both augmentation strategies.

Table 4: Domain-wise ablation of applicable conflict mechanisms (ASR %).

DomainIndividual MechanismsApplicable CombinationsCoSA
RecruitingSIB: 51.85; SCA: 55.56SIB + SCA: 65.1965.19
FinanceSIB: 43.15; TAB: 40.56SIB + TAB: 46.2055.56
ITSIB: 35.83; SAP: 38.33SIB + SAP: 39.9154.54
BusinessSAP: 43.52; TAB: 41.20; SCA: 47.69SAP + TAB: 48.15; SAP + SCA: 52.87; TAB + SCA: 50.2858.52

To further isolate the individual and combined contributions of all four mechanisms under a unified experimental setting, we conduct a full-factorial ablation covering all 16 combinations in an expanded Business Skill pool. Unlike the original Business configuration in Table 4, this supplementary experiment expands the candidate pool to enable all four mechanisms, including SIB. Specifically, among the top 150 retrieved Business Skills, 89 pass individual static screening, from which we select one relationship-reshaping Skill per mechanism. The four additions account for 4/89 (4.49%) of the expanded pool, remaining within the original 5% augmentation budget. We evaluate all 16 configurations on the same 72 Business tasks across five models and three harnesses, yielding 1,080 executions per configuration. The expanded pool without additions serves as the common baseline. The results are summarized in Table 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.