- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Conflict of Skills (CoS), where individually benign Skills can become unsafe when composed. The proposed CoSA framework analyzes an existing Skill pool and adds a small number of benign-looking Skills through capability supplementation and relationship reshaping to increase policy violations. Experiments on 288 tasks across four domains, five LLMs, and three agent harnesses show a pooled ASR increase of 23.43 pp.
Critical strengths
-
The paper shifts attention from malicious individual Skills to composition-level risks among benign Skills, which is an important and underexplored attack surface.
-
The attack is tested across multiple domains, models, and agent frameworks, and the ablations suggest that the effect is not simply due to enlarging the Skill pool.
Critical weaknesses
-
CoSA assumes that the attacker can inspect the existing Skill pool and then introduce new Skills into it. The paper does not sufficiently justify when such access would realistically be available.
-
The main evaluation is conducted on the authors’ own CoSA-Bench. It is unclear whether the attack generalizes to independently constructed Skill or agent benchmarks.
Questions for authors
- What realistic scenario allows an attacker to inspect the target Skill pool and then add specially constructed Skills? How would CoSA perform with only partial knowledge of the existing Skill pool?
- Have you evaluated CoSA on any existing Skill- or agent-security benchmarks, rather than only CoSA-Bench? Such results would help establish generalizability.
- What are the individual contributions of capability supplementation and relationship reshaping? A separate ablation would clarify which component drives the attack effectiveness.
7 comments
Thank you, reviewer 7niY! (2/4)
Table 1: Transferability of CoSA under different levels of attacker knowledge.
| Harness | Model | Original ASR (%) | 100% ASR (%) | 75% ASR (Drop) | 50% ASR (Drop) | 25% ASR (Drop) | 0% ASR (Drop) |
|---|---|---|---|---|---|---|---|
| OpenClaw | Qwen3.6 | 34.03 | 59.38 | 61.11 (−1.74) | 50.00 (9.38) | 37.50 (21.88) | 31.60 (27.78) |
| MiniMax-M3 | 36.46 | 57.99 | 51.74 (6.25) | 53.82 (4.17) | 33.68 (24.31) | 38.54 (19.44) | |
| DeepSeek-V4 | 32.64 | 60.76 | 56.60 (4.17) | 45.83 (14.93) | 40.97 (19.79) | 30.56 (30.21) | |
| GPT-5.6 | 20.14 | 29.86 | 28.47 (1.39) | 25.00 (4.86) | 18.75 (11.11) | 21.88 (7.99) | |
| GLM-5.1 | 36.46 | 63.19 | 58.33 (4.86) | 49.65 (13.54) | 42.01 (21.18) | 35.07 (28.13) | |
| OpenHands | Qwen3.6 | 42.01 | 72.57 | 68.40 (4.17) | 59.38 (13.19) | 49.31 (23.26) | 40.28 (32.29) |
| MiniMax-M3 | 42.71 | 77.43 | 78.82 (−1.39) | 62.50 (14.93) | 46.88 (30.56) | 45.14 (32.29) | |
| DeepSeek-V4 | 52.43 | 79.17 | 64.93 (14.24) | 67.71 (11.46) | 49.31 (29.86) | 54.86 (24.31) | |
| GPT-5.6 | 19.44 | 34.03 | 26.04 (7.99) | 27.78 (6.25) | 22.57 (11.46) | 16.67 (17.36) | |
| GLM-5.1 | 43.75 | 67.36 | 62.50 (4.86) | 55.21 (12.15) | 48.26 (19.10) | 41.32 (26.04) | |
| OpenCode | Qwen3.6 | 17.36 | 45.83 | 41.67 (4.17) | 34.38 (11.46) | 25.35 (20.49) | 20.49 (25.35) |
| MiniMax-M3 | 35.76 | 68.06 | 60.07 (7.99) | 52.43 (15.63) | 42.01 (26.04) | 32.64 (35.42) | |
| DeepSeek-V4 | 44.1 | 61.81 | 51.74 (10.07) | 53.13 (8.68) | 41.67 (20.14) | 46.18 (15.63) | |
| GPT-5.6 | 22.92 | 31.25 | 32.64 (−1.39) | 26.74 (4.51) | 23.96 (7.29) | 20.83 (10.42) | |
| GLM-5.1 | 45.14 | 68.06 | 62.85 (5.21) | 50.69 (17.36) | 43.06 (25.00) | 47.57 (20.49) | |
| Overall | All | 35.02 | 58.45 | 53.73 (4.72) | 47.62 (10.83) | 37.69 (20.76) | 34.91 (23.54) |
As shown in Table 1, CoSA's overall ASR decreases from 58.45% under full visibility to 53.73%, 47.62%, and 37.69% at 75%, 50%, and 25% visibility, respectively. Notably, with only 50% visibility, CoSA consistently outperforms the original, unaugmented Skill pools across all 15 model–harness combinations, achieving an overall ASR improvement of 12.60 percentage points over the 35.02% baseline. Even at 25% visibility, CoSA maintains an overall improvement of 2.67 percentage points, outperforming the original baseline in 10 of 15 configurations. In contrast, when the attacker has no access to the target Skill pool and relies solely on domain-level information, the overall ASR is 34.91%, comparable to the original baseline. These findings demonstrate that CoSA retains substantial effectiveness under partial knowledge of the target Skill pool, while also revealing that its benefits diminish as attacker knowledge becomes increasingly limited.
These additional clarifications and experiments directly address the reviewer's concern by establishing realistic scenarios for Skill-pool inspection and third-party Skill admission, while empirically demonstrating that CoSA does not require complete knowledge of the victim's Skill pool to remain effective. At the same time, the domain-only results clarify the applicability boundaries of our threat model. We will incorporate these findings and further clarify the attacker knowledge and Skill admission assumptions in the revised manuscript.
References
[1] https://github.com/EvilFreelancer/secas/blob/main/.agents/skills/README.md
[2] https://docs.openclaw.ai/concepts/system-prompt
[3] Xiang Zheng, Yutao Wu, Hanxun Huang, Yige Li, Xingjun Ma, Bo Li, Yu-Gang Jiang, and Cong Wang. "Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs." In International Conference on Machine Learning (ICML), 2026.
[4] Zichuan Li, Jian Cui, Xiaojing Liao, and Luyi Xing. "Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agents." In Network and Distributed System Security Symposium (NDSS), 2026.
[5] OpenClaw. "ClawHub: Publishing and Installing Skills." Official Documentation. https://docs.openclaw.ai/clawhub
Thank you, reviewer 7niY! (3/4)
The main evaluation is conducted on the authors’ own CoSA-Bench. It is unclear whether the attack generalizes to independently constructed Skill or agent benchmarks.
We sincerely thank the reviewer for this valuable suggestion regarding the generalizability of CoSA beyond our self-constructed benchmark. We acknowledge that evaluating CoSA exclusively on CoSA-Bench may not fully capture its effectiveness under independently constructed Skill pools and task distributions. To directly address this concern, we conduct additional experiments on two external benchmarks: CompoSkill-Bench [1] and SCR-CapFlow from SCR-Bench [2]. From CompoSkill-Bench, we select 228 clean tasks, comprising 76 from each of three threat categories: (1) data_exfiltration (CoSA’s DL category); (2) memory_tampering (unauthorized modification, UA); and (3) privilege_escalation_and_dangerous_command_execution (unsafe execution, UE). We include exactly those categories whose violations are realized through cross-Skill composition, the mechanism CoSA targets; the remaining two, multi-agent collaboration hijacking and resource exhaustion, are excluded by definition rather than by outcome, as their violations do not depend on composing a victim Skill pool. From SCR-Bench we evaluate CapFlow in full (150 cases), the only subset instantiating the cross-Skill capability-flow setting CoSA presupposes. For both benchmarks, we compare the original and CoSA-augmented Skill pools on the same tasks, using the same five models and three agent harnesses as in our main evaluation. We apply a 5% augmentation budget on CompoSkill-Bench and a one-Skill budget on SCR-CapFlow, whose pool is too small for 5% to reach one Skill. The results are summarized in Table 2.
Table 2: Generalization of CoSA to independently constructed benchmarks.
| Benchmark | # Tasks | Original ASR (%) | CoSA ASR (%) | ΔASR (pp) |
|---|---|---|---|---|
| CoSA-Bench (Ours) | 288 | 35.02 | 58.45 | 23.43 |
| CompoSkill-Bench [1] | 228 | 25.06 | 39.42 | 14.36 |
| SCR-CapFlow [2] | 150 | 37.82 | 62.09 | 24.27 |
As shown in Table 2, CoSA consistently improves ASR on both independently constructed benchmarks, demonstrating that its effectiveness is not confined to the Skill pools and task distributions of CoSA-Bench. Specifically, on CompoSkill-Bench, CoSA increases ASR from 25.06% to 39.42%, achieving a 14.36-percentage-point improvement under the same 5% augmentation budget as our main evaluation. On SCR-CapFlow, CoSA further increases ASR from 37.82% to 62.09%, representing a 24.27-percentage-point improvement with only one additional Skill. These results provide direct empirical evidence that CoSA generalizes to independently developed Skill-security benchmarks with different composition structures and risk scenarios, rather than relying on benchmark-specific characteristics of CoSA-Bench. While differences in Skill-pool sizes and augmentation budgets limit direct comparisons of improvement magnitudes, the consistent within-benchmark gains substantiate CoSA's effectiveness across distinct evaluation distributions. We will incorporate these results and clarify the corresponding benchmark adaptation protocols in the revised manuscript.
References
[1] Liu, M., Jiang, Z., Ma, J., Zhang, J., Chen, J., Deng, X., & Wang, Z. (2026). Composkill: Compositional skill chain attacks from individually scanner-passing llm agent skills. arXiv preprint arXiv:2608.16246.
[2] Xie, Y., Du, J., Cheng, Y., Zhou, J., & Yin, Z. (2026). Benign in isolation, harmful in composition: Security risks in agent skill ecosystems. arXiv preprint arXiv:2606.15242.
Thank you, reviewer 7niY! (1/4)
Response to Weaknesses
We sincerely thank the reviewer for highlighting this important concern regarding the practical applicability of our threat model. We acknowledge that our original threat model assumes substantial knowledge of the victim's Skill pool. However, we would like to clarify that both Skill-pool visibility and Skill admission have realistic deployment scenarios: (i) First, regarding Skill-pool visibility, in repository-based Agent deployments, project-level Skill inventories can be publicly accessible through version-controlled Skill directories. For instance, the open-source SECS project explicitly publishes its inventory of 49 installed project-level Skills along with their implementations [1]. Moreover, Agent frameworks such as OpenClaw incorporate available Skill metadata, including Skill names, descriptions, and file paths, into the agent's system prompt in supported configurations [2]. Prior research (JustAsk, ICML 2026) has demonstrated that hidden system prompts can be recovered through black-box interactions [3], suggesting a potential avenue for attackers to obtain such Skill information without direct access to the victim's environment. Furthermore, prior work (Les Dissonances, NDSS 2026) has investigated cross-tool attacks exploiting interactions and dependencies among existing tools [4], highlighting the relevance of informed, target-aware adversaries. (ii) Second, regarding Skill admission, our threat model also requires that a crafted Skill be admitted into the victim’s pool. This presupposes no privileged access; it only requires passing the same channels open to any third-party contributor. Two are common in practice. Public marketplaces such as ClawHub let third-party developers publish Skills that downstream users install on their own initiative [5], so a crafted Skill advertising a plausibly useful capability can be adopted through ordinary discovery and installation, without the user suspecting its role in a cross-Skill composition. Repository-based deployments accept external contributions through workflows such as pull requests, where an addition that passes review and static screening enters the project-level Skill directory. In both cases, admission succeeds precisely because each added Skill is individually benign under existing screening and raises no flags at entry. Our threat model therefore captures an attacker who inspects an observable Skill configuration and submits crafted additions through legitimate channels, not one who arbitrarily modifies the victim’s environment. Collectively, these observations support the practical relevance of our informed-attacker threat model in certain real-world deployment scenarios, while recognizing that complete visibility and successful Skill admission are not guaranteed for arbitrary downstream users.
Nevertheless, we recognize that attackers may not always have complete visibility into the victim's installed Skill pool. To further address this concern, we conduct additional experiments under two settings: (1) Partially Known Skill Pools, where attackers construct adversarial Skills using only a subset (25%, 50%, or 75%) of the target Skill pool; and (2) Zero-Visibility (Domain-Only) Skill Pools, where attackers receive only a predefined domain description without access to the target pool's Skill packages or contracts. In both settings, generated additions are evaluated against the complete target Skill pool, without exposing hidden Skill information during construction or selection. In the domain-only setting, no target-specific adaptation is performed. The results are summarized in Table 1.