acceptodds
Withdrawn from ICLR 2027

SkillJect: Effectively Automating Skill-Based Prompt Injection for Skill-Enabled Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agent skills are increasingly used to extend LLM agent systems with task-specific instructions, executable scripts, and auxiliary resources. While this modular design improves reusability, it also creates a new supply-chain attack surface: a malicious or compromised skill can be repeatedly loaded as trusted guidance and steer an agent's tool use during downstream execution. Existing skill-based prompt-injection attacks are largely manual and brittle, since explicit malicious instructions are often rejected or ignored when they are not well aligned with the original skill workflow. We propose SkillJect, the first automated framework for generating effective poisoned skills against skill-enabled agent systems. SkillJect separates the attack into two coordinated channels. In the artifact channel, it hides the malicious payload inside an auxiliary helper script. In the instruction channel, it rewrites SKILL.md with a front-loaded inducement strategy, placing the injected content at the beginning of the document and framing the helper script as a mandatory prerequisite or required first step. The injected instruction explicitly references the helper-script path and provides an executable example command, making the helper appear to be a legitimate initialization step before normal skill operations. SkillJect further uses a closed-loop multi-agent-system process to improve attack performance. An Attack Agent generates injected skills, a Victim Agent executes downstream tasks with the poisoned skill, and an Evaluate Agent inspects execution traces to determine whether the hidden payload is executed. The Attack Agent then uses this feedback to identify why the payload was not executed and rewrites SKILL.md, producing an updated poisoned skill while keeping the payload fixed. Experiments across skill-enabled platforms, backend LLMs, and attack categories show that SkillJect substantially outperforms naive direct injection and prior manual skill-injection attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Agent Skills, Prompt Injection, Automated Red Teaming, Supply Chain Security

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9vCtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SkillJect places malicious operations in an auxiliary skill script, uses front-loaded instructions in SKILL.md to induce the agent to execute it, and iteratively revises the document using execution-trace feedback. The authors evaluate 100 ClawHub skills across two agent platforms, four main backend models, and four target behaviors. They report mean cumulative attack success rates of 82.1% on Claude Code and 80.5% on OpenClaw, compare with Skill-Inject, and study transfer and defenses. Success is defined by observed execution of the target script.

Critical strengths
  1. Automated attack pipeline. SkillJect keeps the payload fixed in an auxiliary script, uses front-loaded SKILL.md instructions to induce its execution, and revises the documentation based on execution traces. This provides a concrete workflow for generating and refining poisoned skills.
  2. Broad evaluation and a shared-payload comparison. The main experiments cover 100 skills, four target behaviors, two platforms, and four backend models. With the same benign skills and malicious scripts, SkillJect achieves 69.1% mean ASR, compared with 32.3% for Skill-Inject. This supports its effectiveness in the evaluated setting.
Critical weaknesses
  1. The incremental value relative to related methods is unclear. Table 4 compares SkillJect with Skill-Inject under shared skills and payloads, but the paper does not discuss Poise [1], Fujinuma et al. [2], SCH [3], AutoSkillHarm [4], or SkillTrojan [5]. Skill-document prompt injection, feedback-guided optimization, and malicious payloads embedded in executable skill artifacts each have precedents in this literature. Although Table 7 ablates components within SkillJect, the current analysis and experiments leave the added benefit of their combination over these related methods unclear.
  2. Task variation in the transfer experiment is unclear. Figure 4 tests 20 additional LLM-generated instructions per skill, but their generation, filtering, and semantic distance from the original task are not reported. This makes the scope of the cross-task transfer result difficult to assess.
Questions for authors
  1. How does SkillJect differ from Poise [1], Fujinuma et al. [2], SCH [3], AutoSkillHarm [4], and SkillTrojan [5], and what additional benefit comes from combining document inducement, auxiliary scripts, and feedback-based refinement? For methods with comparable settings, could you provide a comparison using the same tasks, attempt budgets, and success criteria?
  2. How were the 20 additional instructions per skill in Figure 4 generated and filtered? Please report the generation model, prompt, filtering criteria, and the distribution of semantic similarity to the original tasks, including how similarity was measured.
  3. Where did the “previously reported” percentages used in Table 3 come from? What were the original sample size, attempt budget, and success criterion? Why are the raw success counts unavailable, and are execution traces still available?

References

[1] Hao, H., et al. (2026). Poise: Position-Aware One-Instruction Skill Injection for Silent Execution on LLM Agents. arXiv:2606.07943.

[2] Fujinuma, Y., et al. (2026). Defenses & Enablers For Skill Injection Attacks on Terminal Based Agents. arXiv:2606.01567.

[3] Liu, X., et al. (2026). Exploiting LLM Agent Supply Chains via Payload-less Skills. arXiv:2605.14460.

[4] Ning, Y., et al. (2026). SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction. arXiv:2606.02540.

[5] Feng, Y., et al. (2026). SkillTrojan: Backdoor Attacks on Skill-Based Agent Systems. ICML 2026. arXiv:2604.06811.

Reviewer_83U7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SKILLJECT, an automated framework for prompt injection attacks against skill-enabled llm agents. SKILLJECT hides malicious payloads in helper scripts and modifies SKILL.md to instruct agents to execute these scripts. It also uses execution feedback to refine the injected instructions. The authors evaluate SKILLJECT on 100 real-world skills, four attack types, two agent platforms, and four main LLMs. SKILLJECT achieves an average attack success rate of 82.1% on claude code and 80.5% on openclaw.

Critical strengths
  1. Automated skill-based attacks. SKILLJECT combines malicious helper scripts with instructions in SKILL.md. It also uses execution feedback to improve the attack. This approach reduces manual effort and supports attacks targeting multiple skills.

  2. Evaluation in various environments. The authors conducted experiments on two agent platforms, four major llms, and four types of attacks. SKILLJECT achieves a higher ASR than Skill-Inject when both methods use the same skills and payloads. These results support its effectiveness in the tested settings.

Critical weaknesses
  1. The paper does not discuss or compare with related automated attacks. SKILLJECT uses an LLM as an optimizer in a closed loop with feedback. Prior work already uses this method. AutoHijacker [1] uses an LLM as an optimizer to make indirect prompt injection payloads. Claudini [2] uses a Claude Code agent in a loop to find attack algorithms for prompt injection. Recent skill attacks also have the same components as SKILLJECT. SkillHarm [3] makes poisoned skills automatically with a fixed payload. Proteus [4] changes skills again and again with runtime feedback. Poise [5] puts one instruction near the prerequisite steps in SKILL.md. The related work section does not mention these methods. The only baselines are Naive (0.0% in all settings) and Skill-Inject (a manual setting). Because of this, the new contribution of SKILLJECT is not clear.
  2. Table 3 does not contain measured results. The authors state that the original success counts are not available. They calculate the values from an assumed denominator of 120. Unverified values should not be in an experimental evaluation.
  3. The comparisons do not control the attempt budget. SkillJect uses up to five attempts with feedback. Naive and the “w/o refinement feedback” ablation (table 7) use one attempt. The authors agree that the Skill-Inject comparison does not control the budget (sec. 4.2.3). Thus, the gain can come from more attempts and not from the feedback.

The paper has problems in its evaluation. It also does not compare SKILLJECT with related work. Because of this, I think that future researchers will learn little from this paper. My score is 1 (clear rejection).


References

[1]: AutoHijacker: Automatic Indirect Prompt Injection Against Black-box LLM Agents (Liu et al., 2025)

[2]: Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs, arxiv.org/pdf/2603.24511

[3]: SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction, arxiv.org/pdf/2606.02540

[4]: Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems, arxiv.org/pdf/2605.11891

[5]: Poise: Position-Aware One-Instruction Skill Injection for Silent Execution on LLM Agents, arxiv.org/pdf/2606.07943

Questions for authors
  1. Can authors compare SKILLJECT with a no-feedback baseline that uses the same five-attempt budget?

  2. Can authors report how often the malicious scripts cause the intended harmful outcomes, rather than only script execution?

  3. Can authors provide verified experimental results and original success counts for the six additional LLMs in Table 3?

Reviewer_7dRPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates prompt injection through agent skills (the SKILL.md + scripts/resources packages used by tools like Claude Code, Codex CLI, and Gemini CLI) as a supply-chain attack surface. A motivating study in Fig. 2 shows that malicious skills gathered from public sources trigger their intended behavior only 2.5%–7.5% of the time once loaded, pointing to a clear gap between malicious-content presence and activation. To close this gap, the authors introduce SkillJect, which (i) hides the payload inside an auxiliary helper script (.sh/.py), and (ii) rewrites SKILL.md with a "front-loaded" inducement that frames running the helper as a required first step, complete with its file path and an example command. A closed loop of three agents (an Attack generator based on GPT-3.5-Turbo, a Victim, and an Evaluate trace verifier that returns a binary success signal alongside an Ignored/Rejected diagnostic) refines the documentation over up to five attempts while holding the payload fixed. Evaluated on a 100-skill ClawHub benchmark across four target behaviors and four backend LLMs, SkillJect hits a mean ASR of 82.1% on Claude Code and 80.5% on OpenClaw, compared to 0.0% for naive injection (Tables 1–2); these ASRs are cumulative across up to five attempts on the same task used for optimization. When the payload is shared with the baseline, SkillJect reaches 69.1% versus 32.3% for Skill-Inject (Table 4). The paper also evaluates cross-model and cross-task transfer, scanner detection rates, a prompt-level defense, ablations, and a naturalness score. Table 3 includes estimates for six more backends, though the authors explicitly note these are not verified measurements.

Critical strengths

This paper proposes an interesting point in solving a "transferability problem" in existing skill-based prompt injection red teaming methods. Separating the mere presence of malicious content from its actual activation after loading (§1, Fig. 2) complements prior ecosystem studies that identify whether shared skills contain malicious content.

Critical weaknesses

Table 3 presents numbers that are not measurements as part of the evaluation. §4.1 says the evaluation is extended to six more backends (l. 287–289), but §4.2.2 and the Table 3 caption state that the success counts are unavailable, that the values are previously reported percentages "rounded downward to a multiple of 100/120", and that they "are not verified empirical measurements"; a mean (68.1%) and a gap (66.7 pp) are still computed from them. Results that cannot be traced to recorded outcomes should not appear in the paper, and this raises the question of whether per-case outcomes exist for the other tables.

Questions for authors

Q1 Were tool/command executions auto-approved for the victim agents? If so, how would ASR change with approvals enabled?
Q2 How is the verifier M(τ;B) implemented (rules or an LLM judge), and how were its labels validated?
Q3 How many attempts did each method get in the Skill-Inject comparison (Table 4), and what is SkillJect's first-attempt ASR there?
Q4 Where were the Table 3 percentages previously reported, and how were they obtained? Do per-case outcomes exist for all other tables (e.g., what explains the MiniMax denominators of 92/92/93/91 in Table 1)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.