- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates supply-chain "skill poisoning" of LLM agents, in which malicious SKILL.md instruction files steer an agent's decision loop during benign user requests. The main conceptual idea is to decompose a payload's realization into two Risk-Realization Factors (RRFs): an actuation factor (the concrete harmful operation, or what is executed) and a pretext factor (the situational rationale, or why it looks warranted) (Sec. 3.1, Eq. 1). Rather than fragmenting the actuation across skills as in prior distributed attacks, the proposed approach keeps intact inside a downstream "Steering Skill" and offloads the pretext to an upstream "Grounding Skill" that writes a persistent workspace artifact , which is later read as justification (Sec. 3.2, Eqs. 2–3). The attack operates strictly over authentic, pre-existing benign coordination between two skills (Sec. 3.3). The authors implement this in CoordPoison, an automated pipeline featuring dependency screening, plan-then-render decoupling, and failure-guided refinement up to , using GPT-5.5 for generation (Sec. 4). They evaluate the method on 62 qualified skill pairs gathered from public GitHub skill repositories, paired with 7 fixed payloads adapted from the SkillJect and Skill-Inject benchmarks (434 instances total), across three primary victim models. The experiments also cover cross-model transfer (MiniMax-M3, GPT-5.5), ablations, a cross-lifecycle analysis, and a prompt-level defense (Sec. 5, Tables 1–4, 8).
Critical strengths
The attack transfers, and the cross-session effect is explained. Attacks built on one model reach roughly 79–95% ASR on a held-out model (MiniMax-M3) vs. 0% for the naive baseline (Table 2), and splitting the attack across sessions raises success further (e.g., DeepSeek 68.25% → 95.31%; Table 3).
Critical weaknesses
- W1. Headline within-model efficacy numbers are hard to interpret and potentially optimistic, with inconsistent metric reporting and no variance. Table 1 is titled "Generation-stage attack performance," and Figure 3 shows that its values are the cumulative fraction of instances resolved by the optimizer over up to 12 refinement iterations (the endpoints 76.2% / 51.2% / 44.4% match Table 1's overall ASRs). This makes it unclear whether Table 1 reports in-sample optimization success against each victim or a held-out re-evaluation of the finalized poisoned skills; both the title and Figure 3 point to the former. Three other issues compound this: (a) the column is labeled "ASR" (caption: successes "over eligible Steering-only failures"), but the text calls the exact same values "conditional ASR"/"cASR" (Sec. 5.2, lines 317–321), and ASR and cASR can differ substantially elsewhere (e.g., GLM cross-lifecycle is 60.58% ASR vs 91.30% cASR in Table 3), so knowing which metric is reported matters; (b) identical configurations produce different numbers across tables, as DeepSeek overall is 76.19 in Table 1 while Same-Lifecycle CoordPoison is 68.25 ASR / 69.35 cASR in Table 3, with no explanation of the denominators; and (c) no repeated runs, seeds, or confidence intervals are reported for ASR/cASR (the only is for benign-baseline task-completion gating, Appendix A.1, ~line 621), despite stochastic LLM sampling. Because these are the core efficacy claims of the paper, their interpretation depends on whether they reflect optimizer success or deployment success, which metric is actually shown, and how noisy the measurements are. To resolve this, the authors should state clearly what each table measures, reconcile the denominators across Tables 1–3, and add a held-out re-evaluation of the finalized poisoned skills across multiple runs with variance.
- W2. The specific novelty of decoupling pretext over other multi-skill attacks and the "evades isolated audits" claim are argued but not empirically demonstrated. All three baselines (Naive, Skill-Inject, SkillJect) are localized single-skill attacks (Sec. 5.3; Appendix C.3), and the "w/o Pretext" ablation (Table 3) is also a single-skill variant. The held-out transfer table (Table 2) does show CoordPoison beating these single-skill baselines (e.g., 95.19% vs 74.04% for SkillJect on MiniMax for GLM-source), which supports the view that multi-skill coordination helps. However, it does not test the paper's more specific claim that decoupling pretext beats the conceptually nearest attacks that fragment actuation (SkillTrojan, ColluSkill) or exploit persistent cross-lifecycle state (SkillHarm, SkillJack); those are only discussed in related work (Sec. 2.2) and never compared empirically. Along similar lines, the central claim of exposing a "blind spot in isolated Skill audits" (Abstract, line 32; Conclusion, lines 484–485) is only tested against a prompt-level runtime auditing defense (Table 4), without running any concrete single-skill scanner or auditor on the poisoned pairs to show they actually pass. For an attack paper, isolating the advantage of the proposed mechanism and showing the claimed evasion in practice are central to the contribution. The authors should run a concrete single-skill audit tool on the poisoned pairs (which requires only their existing artifacts and should be feasible) and, where possible, compare against at least one representative multi-skill or persistent-state attack. (p.s., I note that these competing attacks are all 2026 preprints and likely contemporaneous with this submission, so a head-to-head comparison is a strengthening suggestion rather than a requirement; the scanner evaluation is the more actionable request.)
Questions for authors
Please refer to the weaknesses.
We thank all reviewers for their constructive feedback. We summarize the recognized strengths and our responses to the main concerns.
1. Key Strengths Recognized by the Reviewers
2. Substantive Responses to the Major Concerns
Attack effectiveness and reliability (AQoz, UQv7, mcRM). Our original evaluation already provides complementary evidence through construction success (Table 1), independent execution of finalized attacks (Table 3), and cross-model transfer (Table 2). As clarified in our rebuttal, Steering-only screening excludes instances where strong localized poisoning succeeds, preventing these successes from being incorrectly attributed to coordination and overstating its effectiveness. We further clarified ASR/cASR and evaluation protocols, and provided full-set pipeline analysis, repeated-run statistics, confidence intervals, and per-pair distributions. These results strengthen the interpretation and reliability of our original findings.
Mechanistic evidence and causal attribution (mcRM, AQoz). Our original experiments already support the contribution of execution justification. Table 3 shows substantial ASR reductions when justification components are removed while preserving the harmful action: 68.25% to 10.32% on DeepSeek-V4-Flash and 52.38% to 30.95% on Claude-Sonnet-4.6. Sec. 4.2 verifies Grounding–Steering dependencies in successful execution traces, while Table 8 provides complementary lifecycle controls. We further conducted the requested matched artifact-content counterfactual, holding the downstream Skill, task, artifact path, and environment fixed. Valid pretext-bearing artifacts increase ASR by 37.50, 16.54, and 7.14 percentage points over neutral artifacts across three models. Together, the original evidence and supplementary counterfactual substantiate the contribution of execution justification to harmful-action invocation, beyond the mere presence of an artifact.
Automated construction and baseline comparison (mcRM). Our agent-driven framework automatically discovers authentic workflow dependencies and uses runtime feedback to diagnose and repair failed coordinated attack constructions. Refinement preserves the fixed harmful action while establishing and repairing execution justification across Skills. The original Table 2 already demonstrates effective cross-model transfer compared with the baselines. To address optimization-budget concerns, we additionally evaluated SkillJect with its original five-iteration refinement setting. CoordPoison maintains higher overall transfer ASR across all source-model settings: +15.38 points on GLM-5.3-Flash, +8.33 on Claude-Sonnet-4.6, and +1.36 on DeepSeek-V4-Flash. These results reinforce the original comparison. Moreover, CoordPoison's iterative refinement addresses the distinct challenge of discovering authentic workflow dependencies and constructing coordinated attacks through runtime feedback, enabling the systematic discovery and realization of previously underexplored attack opportunities in complex multi-Skill workflows. This agent-driven construction process is itself a central methodological contribution of our work.
Defense scope (AQoz, UQv7). Our original defense evaluation (Table 4, Appendix E) focuses on runtime security assessment. Following the reviewers' suggestions, we additionally tested a static Skill scanner and the composition-aware
skil composeframework. These supplementary evaluations clarify the scope of our findings beyond the original study's focus on runtime execution decisions.3. Positive Feedback Following the Rebuttal
Reviewer AQoz expressed satisfaction with the additional results and remained positive. Reviewer UQv7 explicitly confirmed that the main interpretability concern was resolved and maintained a positive assessment. Reviewer mcRM acknowledged that our responses meaningfully addressed several concerns and raised the Soundness assessment.
The reviewers' recognition of CoordPoison's strengths, together with the original evidence and supplementary results, provides strong support for its novelty, significance, and methodological contributions. We respectfully hope the Area Chair will consider these points in the final decision.