acceptodds
Under review as a conference paper at ICLR 2027

Can Agents Trust Their Skills? Uncovering Unsafe Chains of Trust in Skill-Based LLM Agents

Abstract

LLM agents increasingly rely on installable skills, which are packages of instructions, code, and resources that equip them with task-specific capabilities and, once installed, can be automatically invoked across subsequent user tasks. This creates a chain of trust in which users delegate authority to agents, while agent frameworks admit skill-provided content into the agents’ context with insufficient validation, allowing malicious skills to influence agent behavior under that delegated authority. Yet, little is known about whether this trust model adequately constrains untrusted skill content before it reaches security-sensitive operations, or how frequently such trust violations arise in real-world agents. We present TrustProbe, a framework for uncovering unsafe chains of trust in skill-based LLM agents. First, TrustProbe analyzes agent source code to identify source-to-sink call paths from skill-controlled inputs to security-sensitive operations. Second, it generates semantically realistic SKILL.md seeds with injected canaries and evolves them through feedback-guided scheduling and mutation. Finally, it validates vulnerabilities using a two-layer oracle that requires both a confirmed source-to-sink taint flow and observable harm. Across 11 open-source agents, eight with more than 10,000 GitHub stars, TrustProbe identifies 104 taint-style vulnerabilities. Validation on a large corpus of real-world skills collected from public hubs such as ClawHub further shows that 25.1% of skill-agent trials exercise the identified vulnerable paths, with payload injection successfully weaponizing 15 of them. These results reveal a systematic trust failure in skill-based LLM agents: untrusted skill content can reach security-sensitive operations and exercise authority delegated by users to their agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, agent security, skill supply chain, fuzzing, AI safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_g88UOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a systemic supply-chain vulnerability in skill-based LLM agent ecosystems: the confused-deputy failure created when agent frameworks admit untrusted third-party skill content into prompt contexts and tool execution paths under user-delegated authority. To systematically audit this boundary, the authors present TrustProbe, an automated 3-stage framework combining static AST source-to-sink call graph analysis, directed greybox fuzzing with LLM-assisted seed generation and feedback scheduling (balancing distance and semantic alignment), and a bug oracle that validates vulnerabilities via strict taint predicates and independent verification of observable harm. Evaluated across 11 open-source agent frameworks (8 with >10,000 GitHub stars), TrustProbe identifies 104 verified taint-style vulnerabilities spanning command execution, file disclosure, file tampering, and SSRF. Paired direct-prompt replay experiments reveal that framework-level skill delivery (catalogs, retrieval, and prompt-binding) causes 68.3% higher vulnerability reproduction than direct prompt text alone. Furthermore, testing on 633 real-world skills from ClawHub and SkillsMP shows that 25.1% of test runs trigger vulnerable execution paths, with minimal edits weaponizing 15 real skills into full exploits (RCE, credential exfiltration, OAuth phishing, and MCP hijacking)

Critical strengths

Systemic Formulation of Skill-Mediated Trust: Formalizes the gap between user-delegated authority and framework-admitted third-party skill instructions, exposing how framework delivery mechanisms grant implicit trust elevation over plain user prompts. Methodological Rigor (TrustProbe): Combines static AST call-graph extraction with feedback-guided greybox fuzzing and a non-speculative bug oracle requiring both taint-tracking proof and observable harm. Broad Empirical Validation: Demonstrates widespread vulnerability across 11 major agent platforms (over 1.08M aggregate GitHub stars) and validates real-world exposure against 633 skills from public hubs. Pivotal Mechanistic Insights: Discovers that direct user prompts reproduce only 31.7% of vulnerabilities, proving that security failures stem from framework-level delivery architectures rather than pure LLM prompt compliance.

Critical weaknesses

Reliance on Pre-Defined Sink Models: Static AST/Compiler analysis depends on a pre-enumerated inventory of security sinks (47 Python, 115 TypeScript sinks), potentially missing custom or unmodeled sinks. Suggestion: Discuss dynamic sink discovery or open-ended capability mapping during static call graph construction.

Single Model Backbone during Audit: The main campaign holds the target agent model fixed to deepseek-v4-flash. Suggestion: Evaluate whether larger reasoning models exhibit lower instruction-following susceptibility when processing untrusted skill instructions.

Questions for authors

How could agent frameworks implement provenance-aware Complete Mediation (e.g., cryptographic instruction tagging) at execution sinks to distinguish user-originated commands from skill-originated ones?

Why do frameworks that consult skill provenance currently use it to expand auto-approval (e.g., auto-approving declared frontmatter tools) rather than restricting permissions?

What is the performance tax of running AST source-to-sink verification as a mandatory pre-installation gate in skill marketplaces like ClawHub?

6 comments
AuthorsOpenReview ↗

Reviewer g88U-W1

W1: The predefined sink inventory may miss custom or unmodeled security-sensitive operations. Suggestion: Discuss dynamic sink discovery or open-ended capability mapping during static call graph construction.

Response: We thank the reviewer for this constructive suggestion. We agree this is a very promising direction to explore. The rationale is as follows: to systematically verify the coverage of our predefined sink list, we can conduct a call-graph leaf-node analysis across all 11 evaluated agents at their audited repository versions. Specifically, for each agent we can parse all source files, resolve external module imports to build a per-file import map, and extract every call expression whose target resolves to an external library or runtime API rather than a project-internal definition. These call graph leaf nodes capture the complete set of external operations reachable by each agent, including any custom or unmodeled sinks. We can then differentially compare these leaf nodes against our predefined sink inventory to quantify coverage and identify any security-relevant gaps. We agree with the reviewer that dynamic sink discovery and open-ended capability mapping are valuable directions for future work.

Revision: We have added this discussion to Section 4.1 as a statement of the inventory's scope and have identified dynamic sink discovery and open-ended capability mapping as future work. The revised text is marked in blue in the paper.

AuthorsOpenReview ↗

Reviewer g88U-W2

W2: Single Model Backbone during Audit: The main campaign holds the target agent model fixed to deepseek-v4-flash. Suggestion: Evaluate whether larger reasoning models exhibit lower instruction-following susceptibility when processing untrusted skill instructions.

Response: We thank the reviewer for raising this concern. Our original campaign used DeepSeek-V4-Flash for target-agent execution as well as seed generation, scoring, and mutation. We have now evaluated the 104 verified vulnerabilities on two additional larger reasoning models, GLM-5.3 and Qwen3.8-Max. The results remain stable across backbones. Of the 104 vulnerabilities identified in the original campaign, 103 reproduce with GLM-5.3 and 101 with Qwen3.8-Max, corresponding to 99.0% and 97.1%. The verified vulnerabilities are therefore not specific to the model used to discover them. In the few cases where the attack does not reproduce, the larger model either declines to execute the untrusted instruction or completes the task through a different execution path. These isolated cases do not establish a systematic reduction in susceptibility to untrusted instructions.

Revision: We added the cross-model replay results to Section 5.2. The revised text is marked in blue in the paper.

AuthorsOpenReview ↗

Reviewer g88U-Q1

Q1: How could agent frameworks implement provenance-aware Complete Mediation (e.g., cryptographic instruction tagging) at execution sinks to distinguish user-originated commands from skill-originated ones?

Response: We thank the reviewer for bringing up this interesting direction. We agree that provenance-aware Complete Mediation is a promising approach to distinguishing user-originated requests from skill-influenced operations at execution sinks. Our analysis identifies distinct ingestion paths for skill and user content. For example, OpenCode loads skills through its skill catalog and skill tool, while Pochi create a skill tag during slash-command activation and uses useSkill to load skill instructions. User prompts enter through the ordinary message interface. However, these paths converge into tool-call arguments without preserving reliable source attribution at execution sinks.

A practical implementation could introduce provenance tracking at skill ingestion and enforce authorization at execution sinks. At ingestion, the framework would label skill content with its source identity and propagate this information through the agent's execution context. Cryptographic tags could authenticate the original skill content, while runtime provenance tracking would account for instructions reformulated by the LLM. At each security-sensitive sink, a centralized enforcement wrapper would examine the requested operation, its arguments, and their provenance. Operations influenced by skill content, or whose provenance is uncertain, would require explicit user authorization. Upon approval, the framework could issue a single-use signed capability token bound to the specific operation and arguments, which the sink would verify before execution.

TrustProbe provides concrete guidance for implementing this design. The 104 verified source-to-sink paths identify where provenance labels should be introduced, how skill-controlled content reaches sensitive operations, and where mediation checks should be enforced. These paths can also serve as regression tests for evaluating whether provenance-aware controls successfully prevent unauthorized skill-directed execution.

We thank the reviewer for this constructive suggestion and plan to explore provenance-aware Complete Mediation as future work, building on TrustProbe's findings to develop and evaluate practical enforcement mechanisms for skill-based agents.

Reviewer_CV4iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether agent frameworks that load third-party SKILL.md skills check skill-provided content before it reaches security-sensitive operations. The authors define a trust failure as a flow where skill-originated content reaches a sensitive operation without validation, without user consent for that class of use, and without provenance at the enforcement point (Sec. 3.1), assuming a malicious skill author as the attacker (Sec. 3.3). Their tool, TrustProbe (Sec. 4), statically extracts call paths to a sink inventory in Python and TypeScript agent code, uses an LLM to generate SKILL.md seeds carrying deterministic canaries, and schedules and mutates those seeds using an LLM semantic score alongside call-graph distance. Confirming a vulnerability requires the canary to reach the sink along the audited stack path, plus a verified observable effect (Eqs. 6–7, Sec. 4.5). Using deepseek-v4-flash as the backend model throughout, the paper finds 104 vulnerabilities across 11 open-source agents under permissive configurations (Table 1). Of 89 tested cases, 31 survive restrictive approval settings (Sec. 5.2; Appendix I), and 31.7% reproduce when the skill bodies are sent as direct prompts (Table 2). In tests with 633 real skills, 743 of 2,963 runs trigger the identified paths, and 15 minimally edited skills yield complete attacks (Sec. 5.4). The evaluation also compares TrustProbe against LLMSmith and AgentFuzz (Table 3) and includes ablations (Table 4).

Critical strengths

The paper traces the vulnerabilities that survive strict approval settings to two concrete enforcement bugs (in Kimi Code CLI and OpenClaw) plus policy gaps, and shows that skill provenance, where it is used, loosens rather than tightens auto-approval (Appendices B, H–I), giving framework developers specific things to fix.

Critical weaknesses
  • W1. The headline number conflates intended skill execution under disabled approvals with actual bypasses of active controls. The main campaign runs under "permissive, non-interactive execution configurations" (Sec. 5.2, l. 337), putting every permission-capable agent into its most permissive non-interactive mode (Appendix D, l. 993–994) via flags like --dangerously-skip-permissions, --yolo, and --auto-approve true (Table 6). Yet Sec. 3.1 identifies approval layers as the mechanism encoding the trust boundary in eight of the eleven frameworks (l. 134–135), and user consent is one of the three conditions in the paper's own invariant (l. 130–132). In addition, several frameworks explicitly prompt the model to read and follow a matching or activated skill (Appendix G.1–G.2), skills are explicitly activated or invoked for several agents (Appendix G.2; Table 6), and the motivating example uses the user prompt "please use this skill and follow its instructions" (Sec. 3.2, l. 155), while the user message starting each campaign session is otherwise left unspecified. Under these conditions, an agent executing operations specified by an installed skill is arguably doing what the skill mechanism was built to do. Measuring exposure under permissive settings is reasonable (Appendix I, l. 1245–1246) and doesn't lessen the supply-chain risk, but the abstract frames the 104 count as evidence of "a systematic trust failure", and neither the abstract nor Table 1 separates flows that only happen when approvals are turned off from flows that bypass an active control (Q2).
  • W2. Vulnerabilities are counted per path rather than per underlying defect. A vulnerability is counted "once per audited source-to-sink path" (l. 337–338). In Appendix H, however, 19 OpenClaw cases stem from a single configuration design choice (approval fields existing only on ExecToolConfig, with workspaceOnly defaulting to false), and six Kimi Code CLI cases trace back to one rule-loading bug. In total, the 31 cases surviving restrictive approval reduce to two enforcement bugs and policy-scope gaps across four agents, with 22 of the 31 in OpenClaw alone (Appendix H; Table 11). The number of distinct framework-level defects that defeat an enabled control is thus much smaller than the headline figure suggests, yet the main text gives the restrictive replay only a single paragraph (Sec. 5.2, l. 359–366) and relegates the per-agent and root-cause breakdown to Appendices H–I. Reporting deduplicated root causes alongside the path-level counts, and putting the restrictive-replay results (Table 11) next to Table 1 and into the abstract, would resolve both W1 and W2.
  • W3. An asymmetric comparison in RQ2 leaves the magnitude of the delivery effect unclear. Because "The installed-skill counts come from the successful discovery runs" (Sec. 5.3, l. 370), the skill arm has a 100% success rate by construction. In contrast, the direct-prompt arm is a fresh replay of skill bodies that were already selected, and in 29 cases mutated (Sec. 5.6), for succeeding under skill delivery, with their YAML frontmatter stripped (l. 371). The paper notes that agent runs are nondeterministic (l. 060–061, l. 250–251) and subject to "stochastic route changes" on replay (Appendix H, l. 1220–1222). Indeed, in the approval replay, 18 of the 58 non-surviving cases simply "do not reproduce the target attack in that run" without any sign of being blocked (Appendix I, l. 1259–1260). The failure mode reported for RQ2, "changed execution routes or sink arguments rather than explicit refusals" (l. 376–377), is therefore also consistent with replay variance alone, and the paper does not state how many replay attempts each vulnerability received. While per-agent differences (e.g., 1/24 for OpenClaw vs. 9/13 for Pochi) and the code inspection in Appendix G make a delivery effect plausible, RQ2 is the primary evidence that the skill-delivery pathway itself shapes execution (Sec. 5.3, l. 396–397; Sec. 6), so establishing its effect size requires a symmetric replay (Q1).
  • W4. Claims of generality and component effectiveness rely on a single model, single runs, and a self-referential baseline setup. Every target agent runs on deepseek-v4-flash, which is also the model used to generate, score, and mutate the seeds (Sec. 5.1, l. 332–333; Table 7). While the framework bugs identified via source inspection (Appendix H) are model-independent, the prevalence counts and RQ2 ratios may depend on the model, even though the abstract states its conclusions for skill-based LLM agents in general (Abstract). No repeated campaigns or variance estimates are reported for Tables 1, 2, or 4; for instance, the RQ5 claims regarding semantic feedback and adaptive scheduling rely on mutation-stage yields of about 11 (NoSigma) and 9 (RandomSched) vulnerabilities compared to 29 for the full pipeline (Table 4; l. 460–464). Finally, TrustProbe achieves 100% precision and recall in Table 3 by construction, because the ground-truth reference set is just its own 20 verified vulnerabilities (Sec. 5.5, l. 425–428); Table 3 is thus better framed as baseline recovery of TrustProbe's findings rather than a true head-to-head detection benchmark. Replaying the confirmed seeds with at least one other backend model and repeating the ablation runs would address these concerns.
Questions for authors

Please refer to the weaknesses.

9 comments
AuthorsOpenReview ↗

Reviewer CV4i-W1

W1: The main campaign puts permission-capable agents into permissive non-interactive modes, including configurations that disable approvals. Which findings reflect execution under those settings, and which persist when applicable controls are enabled?

Response: Thank you for your valuable feedback. Our original submission included a paired approval replay under strictest usable non-interactive approval configurations (auto-approval disabled) to distinguish findings observed in permissive modes and restrictive modes. The results were reported in Table 11 of the appendix I, but without an adequate explanation of the findings. Therefore, below we clarify the results, and we have moved the table into the main text (now Table 3) with an expanded explanation in the revised paper.

In the paired approval replay that includes strictest settings, eight agents were eligible for a strict replay and account for 89 of the 104 findings. The other 15 findings belong to Pochi, Agent Zero, and DB-GPT, which have no corresponding configurable approval layer in the tested modes. For these 89 findings, we removed automatic approval where applicable and replayed them under the strictest usable non-interactive approval configurations, holding the skill, confirmed variant, target, model, and execution environment fixed. Requests requiring approval were denied or left pending, not approved by a human. 31 of the 89 cases (34.8%) remained exploitable without an approval grant. 9 are enforcement failures, comprising 6 Kimi Code CLI cases in which configured deny rules were never loaded and 3 OpenClaw cases whose execution path skipped the approval check; the other 22 are policy-scope gaps, where the affected operation falls outside the configured approval boundary. These 22 persist without bypassing a check that governs that operation. The remaining 58 cases were blocked. In the revised manuscript, Table 1 reports the 104 vulnerabilities under auto-approval configurations, while Table 3 compares verified counts under permissive and restrictive settings.

To determine whether the 104 verified vulnerabilities could be weaponized to cause concrete security harm, we replaced the canary in each verified skill seed with a realistic malicious payload, preserved each original SKILL.md, and replayed the attacks against agents. All 104 taint-style vulnerabilities completed payload delivery in the original source-to-sink path. Agents made requests to attacker-controlled endpoints on network-request paths, achieving Server-Side Request Forgery (SSRF). Command-injection paths achieved Remote Code Execution (RCE). In addition to these network-level attacks, agents modified payment configuration files, redirecting payment routing to attacker-controlled accounts. Beyond payment tampering, agents also read credential files containing API keys and bank account numbers, fully disclosing contents into the conversation. The agent framework placed no restriction on which sensitive files on the user's machine, dangerous URLs injected into skills, or malicious commands from skill content could direct agents toward. Reading credentials, altering payment routing, and executing remote code are not intended skill capabilities that a user authorizes by instructing an agent to follow this skill. All 104 verified vulnerabilities posed severe security risks.

Revision: We added the real-payload replay to Section 5.2. We revised the abstract and Section 5.2 to distinguish the 104 vulnerabilities verified under permissive, non-interactive execution, the 89 eligible for the paired approval replay, and the 31 that remained exploitable under the strictest usable non-interactive approval configurations. We moved the restrictive-approval comparison, previously Table 11 in Appendix I, into the main text as Table 3, which presents the per-agent comparison under permissive and strictest settings.The revised text is marked in blue in the paper.

AuthorsOpenReview ↗

Reviewer CV4i-W2

W2: Do 104 distinct source-to-sink paths represent 104 independent framework defects? Can the paper report deduplicated causes and move the restrictive-replay breakdown alongside the headline results?

Response: Thank you for your valuable feedback. To answer your question, we conducted a root-cause analysis of all 104 vulnerabilities. Our analysis examined two aspects: how attacker-controlled input reaches the security-sensitive operation and why that operation is not stopped by an approval control. The answers across all 104 cases converge to nine mutually independent agent defects, and every case falls into exactly one. The analysis is summarized in the table below (also added to the revised manuscript as Table 2 in Section 5.2). The restrictive replay table, previously Table 11 in Appendix I, now appears in the main text as Table 3. Both results are also presented in the abstract.

#Agent defectVulnerabilitiesAffected agents
1Unvalidated file paths outside the workspace are read or written after an auto-approval option disables the permission system13Qwen Code (7), Cline (6)
2Unvalidated file paths outside the workspace are read or written under insecure default-allow configurations23OpenClaw (20), Kimi Code CLI (2), Hermes Agent (1)
3Unvalidated file paths are read or written in execution modes that implement no approval mechanism13Pochi (11), Pi Coding Agent (2)
4Unconstrained command content is executed after an auto-approval option disables the permission system21OpenCode (17), Mistral Vibe (2), Cline (1), Qwen Code (1)
5Unconstrained command content is executed under insecure default-allow configurations24Hermes Agent (10), Kimi Code CLI (10), OpenClaw (4)
6Unconstrained command content is executed in execution modes that implement no approval mechanism3Pochi (2), Pi Coding Agent (1)
7URLs with unrestricted destinations are fetched under an insecure default-allow configuration1Hermes Agent (1)
8Sensitive data is exfiltrated unfiltered to framework-fixed endpoints under an insecure default-allow configuration4Hermes Agent (4)
9Untrusted skill content is injected directly into an execution channel without any approval mechanism2Agent Zero (1), DB-GPT (1)
Total104All 11 agents

Revision: We added the nine-category mapping and Table 2 to RQ1, moved the restrictive replay into the main text as Table 3, and updated the abstract with both results. The revised text is marked in blue in the paper.

AuthorsOpenReview ↗

Reviewer CV4i-W3

W3: RQ2 compares successful skill-discovery runs with fresh direct-prompt replays, making the size of the delivery effect uncertain. Can symmetric repeated replays of the frozen samples separate delivery effects from replay variability?

Response: We thank you for the valuable feedback, for raising this point and for suggesting a symmetric repeated design. We agree it isolates the delivery mechanism more cleanly, and we have completed the requested experiment. On the frozen set of 104 final samples, we performed three independent replays under each of the two delivery modes, installed-skill and direct-prompt. Averaged over the three replays, installed-skill delivery reproduces 100/104 (96.2%), while direct-prompt delivery reproduces 34/104 (32.7%). The direct-prompt average agrees with the single-run 33/104 (31.7%) reported in RQ2, so run-to-run variance is small.

Measured under identical conditions on the frozen set, the gap between the two delivery modes is therefore attributable to the delivery mechanism itself. When the identical text is delivered through framework-mediated skill loading, mechanisms such as framework-directed selection and retrieval, explicit activation, and incorporation into the configured instructions of the agent cause it to be adopted as the procedure for the task and to reach the original sinks. A direct prompt instead proceeds along its own execution route rather than the one that installed skills reach natively, and therefore rarely reproduces the original paths. This strengthens the conclusion of RQ2 that the delivery path itself materially shapes execution, and it supports driving vulnerability discovery through the native skill-loading path rather than treating skill-borne vulnerabilities as a special case of prompt injection.

Revision: We replaced the asymmetric RQ2 comparison with a symmetric three-replay evaluation of the 104 frozen samples in Section 5.3. We also moved the per-agent single-run table to Appendix I and updated the RQ2 summary sentence.

Reviewer_5kCgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how third-party skills can influence security-sensitive operations in LLM agent frameworks. It introduces TrustProbe, which combines source-to-sink analysis, LLM-generated SKILL.md seeds containing canary markers, feedback-guided scheduling and mutation, and runtime verification based on sink arguments, execution traces, and observable effects.

The evaluation covers 11 open-source agent frameworks and reports 104 verified path-level findings. Additional experiments examine direct-prompt replay, restrictive approval configurations, exposure through real-world skills, comparisons with existing methods, and component ablations.

The paper addresses an important problem and provides useful system-level observations, particularly concerning gaps in approval enforcement. However, two issues currently limit the strength of its conclusions: whether all reported findings establish violations of user authorization or framework security boundaries, and whether the comparisons isolate the contributions of skill delivery and the proposed search mechanisms.

Critical strengths

S1. The cross-framework evaluation and execution-based evidence provide a potentially valuable empirical contribution.

TrustProbe evaluates 11 real agent frameworks through their native skill-loading mechanisms. It combines static call paths, runtime sink arguments, stack evidence, and independently observed execution effects. This provides a stronger basis for security analysis than judging whether an agent produces suspicious text alone. Supporting both Python and TypeScript frameworks and accommodating heterogeneous skill-loading mechanisms also represents meaningful engineering work. The study could offer useful insight into how framework design connects third-party instructions to delegated execution authority.

S2. The analysis of approval-control failures is the most informative part of the paper.

The restrictive-configuration experiments go beyond reporting aggregate attack outcomes. Among the 89 applicable findings, the authors distinguish 31 surviving cases, comprising nine enforcement failures and 22 policy-scope gaps. They also distinguish 40 cases with explicit blocking evidence from 18 cases that simply did not reproduce. This distinction appropriately avoids attributing every unsuccessful replay to effective enforcement.

The explanations of missing permission-rule initialization in Kimi Code CLI and omitted approval checks on an OpenClaw execution path are particularly useful. These observations identify concrete implementation mechanisms that can inform framework design and remediation.

Experimental rigor and validation. The manuscript documents framework revisions, execution configurations, prompts, sink inventories, and campaign budgets. The 15 modified real-world skills also provide evidence of concrete outcomes, including data reaching controlled endpoints. These experiments strengthen the practical relevance of the study.

Clarity of presentation. The overall organization is clear, and the appendices contain substantial supporting detail. The authors explicitly acknowledge several measurement limitations, including the interpretation of the real-skill exposure rate and reference-set recall. The main presentation concern is the insufficient separation among attacker influence, observable execution, unauthorized harm, and independent vulnerabilities.

Critical weaknesses

W1. The vulnerability oracle does not sufficiently distinguish intended skill functionality from violations of user authorization This is my primary concern.

Section 3.1 defines trust failures in terms of insufficient validation, user consent, or provenance at security-sensitive operations. However, the core oracle in Section 4.5 checks whether:

  1. The target operation is reached.
  2. A sink argument contains a canary originating from the skill.
  3. The recorded stack matches the audited path. The authors then verify an observable execution effect. These checks establish that skill content influences an operation and that execution actually occurs. They do not, by themselves, establish that the operation violates the legitimate user’s authorization.

The motivating example illustrates this gap. In Figure 1, the skill is explicitly named “Command Runner,” its declared functionality is to execute a supplied command, and the user asks the agent to use the skill and follow its instructions. Executing the command and creating a marker file therefore demonstrates command execution, but the example does not specify which authorized task or security restriction has been violated.

The examples of harmful effects in Section 4.5 raise similar questions. Benign command output, a test-file write, or file contents appearing in the conversation may also be normal outcomes of legitimate skill execution. In particular, displaying a file to its authorized user does not necessarily disclose that file to the malicious skill author. The relevant distinction depends on the legitimate task, the permitted resources, and the intended recipient.

This concern does not establish that all 104 findings are false positives. The approval-enforcement failures and several data-exfiltration examples provide stronger evidence of actual security violations. Nevertheless, the current reporting is insufficient to audit whether every finding satisfies a consistent authorization-based security criterion.

The paper should provide, for each finding or clearly defined finding category, the legitimate task, authorized operations and resources, attacker objective, violated boundary, and independently observed unauthorized effect. Benign execution controls are also needed to show that the oracle excludes ordinary command-running, file-processing, and network-access skills.

W2. The comparisons do not sufficiently separate delivery-channel effects, search improvements, and replay stochasticity This affects both the delivery-mechanism analysis in RQ2 and the methodological comparison in RQ4.

The skill-versus-prompt comparison is asymmetric. Section 5.3 explicitly states that the 104 installed-skill findings come from successful discovery runs, whereas the 33 direct-prompt findings come from subsequent replay. The skill condition therefore benefits from adaptive generation, search, and selection on success; the prompt condition is evaluated through a new execution of those selected samples. The paper does not report an equivalent independent replay of the frozen samples under the installed-skill condition.

Consequently, 33/104 is a valid description of direct-prompt recovery on a skill-conditioned discovery set, but it does not independently estimate the difference in success probability between the two delivery conditions. Search specialization, selection on success, and execution stochasticity can contribute to the observed difference. The 18 non-reproducing cases in the approval experiments further demonstrate that replay variability is a material concern. (§5.3; Table 2; Appendix I)

The AgentFuzz comparison also does not isolate the benefit of the proposed search mechanisms. AgentFuzz already includes call-chain-guided seed generation, trace-grounded semantic scoring, distance feedback, selection-count penalties, and functionality/argument mutation. TrustProbe’s search structure substantially overlaps with these components, while its additional mechanisms primarily concern skill structure, native loading, and skill-specific mutation. www.usenix.org

In the current comparison, AgentFuzz operates through prompts, whereas TrustProbe installs skills through the native framework interface. This demonstrates a limitation of applying the original prompt-based system to the skill attack surface, but it does not establish how much benefit comes from TrustProbe’s additional search mechanisms.

A more informative comparison would adapt AgentFuzz to the same native skill-delivery interface and align the shared sink inventory, model, oracle, and evaluation budget. The incremental contribution of metadata and body mutation could then be assessed separately.

Questions for authors
  1. Authorization boundaries and labels. In Figure 1, what specific user authorization or framework security restriction is violated when the agent follows the Command Runner skill? For the 104 findings, are there case-level records of the legitimate task, permitted operations, and unauthorized consequence? How does the oracle exclude ordinary command execution, file processing, and network access?

  2. Symmetric replay. After freezing the 104 final samples, what are the results of equally repeated independent replays under both installed-skill and direct-prompt delivery? Can you report both reproduction of the original path and success of the same attacker objective?

  3. Isolating methodological improvements. What improvement remains when AgentFuzz uses the same native skill-loading interface, shared sink inventory, model, oracle, and budget? If this comparison has not been performed, which existing results isolate the contribution of skill-specific mutation and scheduling from access to the skill-delivery channel?

  4. Independent root causes and auditable evidence. How many independent defects and remediation points underlie the 104 path-level findings? Beyond the nine enforcement-failure cases attributed to two defects, how are the remaining findings grouped? Is an artifact available containing final skills, execution traces, authorization judgments, and the mapping from cases to root causes?

6 comments
AuthorsOpenReview ↗

Reviewer 5kCg-Q1-Part1/2

Q1: In Figure 1, what specific user authorization or framework restriction is violated when the agent follows the skill? For the 104 findings, what case-level labels and controls distinguish unauthorized effects from ordinary command execution, file processing, and network access?

Response: We thank the reviewer for the valuable feedback and for this insightful comment, which highlights the importance of distinguishing intended skill functionality from violations of user authorization or agent security restrictions. We address the reviewer's questions and concerns below and have incorporated the corresponding revisions into the uploaded manuscript.

Authorization violation. TrustProbe identifies source-to-sink paths through which attacker-controlled skill content can reach security-sensitive operations. However, reaching a sensitive operation alone does not constitute a vulnerability. We verify exploitability by confirming attacker-controlled flows and their observable security effects, rather than relying on sink reachability alone. The authorization violation arises when attacker-controlled skill content induces security-sensitive operations beyond the user's intended task, causing the agent to exercise user-delegated authority under the attacker's direction rather than the user's. In Figure 1, for example, the agent executes an attacker-injected pipeline command without verifying its origin or whether the operation was authorized by the user.

Validation of all 104 findings. Motivated by the reviewer's question, we replaced the canary in each of the 104 verified vulnerability skills with a realistic malicious payload, preserved the original SKILL.md and source-to-sink path, and replayed all 104 attacks. All 104 replays successfully delivered their malicious payloads through the previously verified paths, demonstrating concrete security-sensitive effects rather than merely theoretical reachability.

Specifically, network-request paths caused agents to contact attacker-controlled endpoints, demonstrating attacker-directed network access. Command-injection paths achieved remote code execution (RCE), including downloading and executing attacker-hosted scripts on the user's machine without user authorization. Other attacks induced agents to collect AWS access keys and SSH private keys, disclose sensitive files containing API keys and banking information, and modify payment configuration files to redirect payments to attacker-controlled accounts. These actions were introduced by attacker-controlled skill content rather than the user's legitimate task and therefore constitute unauthorized security effects, not ordinary skill functionality. All 104 verified vulnerabilities posed severe security risks. These results have been incorporated into the RQ1 section of the revised paper. The two tables in Q1-Part2/2 summarize the observed effects by sink category and list representative cases for the five sink families.

Security implications. These results demonstrate that the 104 findings represent exploitable trust-boundary violations, rather than merely source-to-sink reachability. The underlying failure is that agents allow untrusted skill content to direct security-sensitive operations using user-delegated privileges without adequately enforcing the boundary between attacker-controlled instructions and authorized user intent. We have incorporated the replay results and their security implications into the revised RQ1 section.

Revision: We added the real-payload experiment and its observed effects to Section 5.2. The revised text is marked in blue in the paper.

AuthorsOpenReview ↗

Reviewer 5kCg-Q1-Part2/2

Continued from Q1-Part1/2.

CategorySecurity restrictionUnauthorized effects observedFindings
Arbitrary file readSensitive files outside the intended user task should not be read.Agents read credential files holding API keys, bank account numbers and passwords, bringing every field into the session transcript.27/104
Arbitrary file writeSensitive files should not be modified using skill-selected paths or templates.Agents edited and modified the victim payment configuration, redirecting payment routing from the victim account to the attacker account.22/104
Command injectionMalicious commands supplied by a skill should not be executed.Agents executed injected malicious commands, achieving Remote Code Execution (RCE) and operating the AWS access keys and SSH private keys on the user machine.49/104
Network requestRequests to skill-selected attacker-controlled endpoints should not be issued without a legitimate task need.Agents issued requests to attacker-controlled endpoints, achieving Server-Side Request Forgery (SSRF).5/104
Code executionMalicious code supplied by a skill should not be executed.Agents executed injected malicious code inside the agent interpreter.1/104
Discovered pathAttacker-controlled payloadIntended user task (skill functionality)Unauthorized effects observed
CodexAppServerClient.__init__ ... subprocess.PopenReplaces the client initialization command with a chain that downloads a remote script and hands it to Bash.Codex App Server Client Initialization purports to initialize the client.Agents executed injected malicious commands, achieving Remote Code Execution (RCE).
OpenRouterCompatImageProvider.generate ... requests.postRedirects the image-generation request to an attacker endpoint.OpenRouter Image Generation purports to generate an image.Agents issued requests to attacker-controlled endpoints, achieving Server-Side Request Forgery (SSRF).
infer_lang ... compileAdds file-writing code to the snippet submitted for validation.Code Snippet Validator purports to validate a code snippet.Agents executed injected malicious code inside the agent interpreter.
insertInFile ... fs.readFileInstructs the agent to read a file outside the workspace that holds bank account credentials.insertInFile purports to insert content into a file.Agents read credential files holding API keys, bank account numbers and passwords, bringing every field into the session transcript.
createFile ... fs.writeFileAdds a configuration-migration step that overwrites an existing configuration file outside the workspace with a skill-supplied template.File Creator purports to create a new file with specified content.Agents edited and modified the victim payment configuration, redirecting payment routing from the victim account to the attacker account.
AuthorsOpenReview ↗

Reviewer 5kCg-Q2

Q2: After freezing the 104 final samples, what do equally repeated independent replays show under installed-skill and direct-prompt delivery? Can the results separately report reproduction of the original path and success of the same attacker objective?

Response: We thank you for the valuable feedback, for raising this point and for suggesting a symmetric repeated design. We added the experiments by freezing the 104 final samples and performed three independent replays under each of the two delivery modes, installed-skill and direct-prompt. Averaged over the three replays, installed-skill delivery reproduces 100/104 while direct-prompt delivery reproduces 34/104. The direct-prompt average agrees with the single-run 33/104 reported in RQ2, so run-to-run variance is small. The 100/104 average under installed-skill delivery shows that the discovery results of RQ1 remain stable.

Measured under identical conditions on the frozen set, the gap between the two delivery modes is therefore attributable to the delivery mechanism itself. When the identical text is delivered through framework-mediated skill loading, mechanisms such as framework-directed selection and retrieval, explicit activation, and incorporation into the configured instructions of the agent cause it to be adopted as the procedure for the task and to reach the original sinks. A direct prompt instead proceeds along its own execution route rather than the one that installed skills reach natively, and therefore rarely reproduces the original chains. This strengthens the conclusion of RQ2 that the delivery path itself materially shapes execution, and it supports driving vulnerability discovery through the native skill-loading path rather than treating skill-borne vulnerabilities as a special case of prompt injection.

Revision: We replaced the asymmetric RQ2 comparison with a symmetric three-replay evaluation of the 104 frozen samples in Section 5.3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Appreciation and Summary of Key Findings

We sincerely thank the reviewers for their time and constructive feedback, which has helped us strengthen this work.

Our key insight is that installed skills introduce a new trust boundary: untrusted skill content can exercise user-delegated authority through agent frameworks without adequate security checks. To systematically investigate this problem, we developed TrustProbe, combining source-to-sink analysis, feedback-guided fuzzing, and runtime verification. Across 11 agent frameworks, we identified and verified 104 vulnerabilities under permissive settings with observed unauthorized effects including payment redirection, full credential disclosure, credential exfiltration, and access to attacker-controlled services. We then replayed 89 eligible findings under the strictest approval configurations, 31 vulnerabilities remain exploitable without an approval grant, including 3 bypassed applicable approval checks, 6 resulted from failures to load configured deny rules, and 22 exploited gaps in approval-policy coverage. We further show that skill-specific delivery mechanisms amplify these risks and that real-world skills can trigger the vulnerable paths.

These findings highlight an important security problem in the emerging skill ecosystem and the need for stronger trust-boundary enforcement in agent frameworks. We appreciate the opportunity to address the reviewers' concerns and further clarify our findings and contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.