- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies skill injection, where an attacker hides text in a third-party skill bundle (SKILL.md, reference files, helper scripts) so that the coding agent loading the skill carries out the attack, and keeps doing so every time the skill is used. The authors label two things separately: whether the bundle is poisoned (skill stage, Eq. 1), and whether a recorded execution actually ran injected content that was permitted and completed (execution stage, Eq. 2), so that . The corpus applies the SkillJect procedure to 27 public carrier skills, giving 3,492 poisoned bundles across nine attack families, which are run under four agent frameworks and six backends for 4,165 executions; 46 clean carriers serve as extra skill-stage negatives. SkillVigil is a single LoRA adapter, and a system prompt picks the stage and the output channel. Three nested arms are trained under a fixed per-epoch budget: verdict only (Arm A), plus citation (Arm B, a character span for skills and an event ID for traces), and plus attribution (Arm C, which adds a quote and an attack class), each on five backbones from 1.2B to 4B. The main results are that in-distribution verdict AUC is saturated across all 15 models, while on a length-matched MalSkillBench subset the 4B model reaches 0.891 AUC against 0.714 for the best of six general-purpose guards; that at the execution stage the adapter picks the right event on 0.998 of windows, compared with 0.040 and 0.050 for zero-shot GPT-4o and Qwen3.8-27B; that citation supervision makes abstention consistent across backbones; and that at the skill stage the model quotes the injected line correctly on 0.780 of documents but gets the character span right on only 0.116. Linear probes find payload tokens at about 0.99 AUC even in the base weights, which the authors read as the model knowing where the payload is but being unable to say it. The appendices contain a long series of shortcut audits, including length matching, a counting baseline, a filename lookup that leads the authors to drop the attack-class metric, stratification by recording style, a label audit, extra seeds, and an untrained baseline.
Critical strengths
S1. The measurement is careful, and the authors are honest about their own shortcuts. For most headline numbers they also report the simplest statistic that could have produced it, and they act on what they find. Length matching drops a trace-counting baseline from 0.781 to 0.500 while the detectors stay at 0.944-0.958. A ten-entry filename lookup reaches 0.956 on the attack class, so they withdraw that metric. Splitting by recording style shows that a pooled 0.793 vs. 0.781 result hides a detector that loses to counting on argument-free traces and wins easily on the rest. They also disclose and fix a checkpoint mix-up in Table 8. I trust the reported numbers more because of this, and several of my concerns below come directly from the authors' own diagnostics.
S2. The two-stage framing is useful and is backed by real executions. Separating the bundle label from the execution label, and then tracing where each failed attack stopped, makes a concrete case for an external guardrail: of the 2,360 poisoned but uncompromised executions, 80.7% have the payload in context with no action and no comment, and only 4.8% contain an explicit refusal. An agent that quietly does not comply is not something an operator can audit. The instrumentation finding is also practically important. One framework logs tool names without arguments, which drops the detector to 0.603 on those traces, and recovering the arguments lifts it to 0.938 with the model unchanged. If the corpus is released with the label audit as planned, I expect this to be the most reused part of the paper.
=== Post-rebuttal update ===
The rebuttal strengthens S1 and adds the controls I asked for. On all 812 execution windows the 4B locator is correct on 797, against 757 for the best invocation rule. It recovers all 55 positives the rule misses, including 9 Markdown payloads that no script rule can reach, and the result holds on all five backbones. When the attack call is rewritten outside the rule’s pattern (python3 helpers/x.py, or copied and run from /tmp), the rules fall to 0/300 while the locator keeps 275-300/300. Holding out an attack family leaves the locator at 61/62. On a second generator (Skill-Inject) the skill-stage guard reaches AUC 0.859 against 0.632 for Qwen3Guard-Gen-8B, and 0.810 on skills absent from the training corpus. The new same-protocol baselines are informative: PromptLocate reaches character F1 0.145 against 0.282 at about 200 times the latency, and Sentry-static reaches recall 0.286 against 0.784 at a similar FPR. I also appreciate that the authors ran the offset-regression probe, reported that it does not beat the position-only control, and withdrew the probe claim from the abstract. The abstract and contributions are now scoped to one construction procedure, which resolves my scope concern.
Critical weaknesses
W1. The headline evidence numbers are not shown to go beyond what the corpus construction gives away, and evidence quality is never checked outside the corpus. This is the main reason for my score.
(a) Execution stage. 395 of the 406 attributed windows have a shell-script payload, and the rule "first Bash call on resources/*.sh" already gets 0.865 without any model (Table 15). Script bodies are withheld, so the invocation string is all the model sees, and filename randomization keeps the bash resources/<name>.sh pattern intact. Benign helper calls of the same form are never tested. The fair comparison is therefore 0.998 against 0.865 on one generator, not against 0.040.
(b) Frontier baselines. GPT-4o and Qwen3.8-27B score below the 0.227 chance level, and Table 2 shows they abstain on 0.990 and 0.940 of windows. With a prompt that hides the script and asks for a judgment from the call alone, declining to flag bash resources/workspace_validator.sh seems like the reasonable answer. To me this says the target is not identifiable from the window without the corpus prior, which is different from "a gap of training exposure rather than capability". No few-shot or prior-stating prompt is tried, so the abstract's reading is untested.
(c) Skill stage. This is where the load-or-not decision is made and the only stage tested externally, yet its evidence channel is weak. Arm C span citation is 0.045-0.235 against a 0.100 null, and no cell clears 0.025 under the containment criterion (Tables 13 and 19). Evidence accuracy is never measured on MalSkillBench or SkillSafetyBench; the checkmarks in Figure 2 and Table 8 only mark output formats. For detection, the relevant comparison is the skill-specific scanners in Table 9, and the 4B model (F1 0.812) does not beat Sentry (0.886) or AI-Infra-Guard (0.856), although the splits differ.
(d) One generator, no adaptive attacker. All injections come from a single procedure over 27 carriers. There is no held-out template, carrier split, or second generator, and nothing tests an attacker who moves the payload out of resources/, runs it through python, or splits the call across windows. The model's score on the 11 non-script rows, where the rule falls to 0.182, is not reported.
Sec. 6 scopes the execution-stage results to corpus-specific auditing, and I agree with that framing. The abstract, introduction, and contribution list do not; they present the evidence results as general properties of evidence-supervised guardrails.
W2. Some explanatory claims go further than the controls allow, and closely related work on localization is missing.
(a) Abstention. Arm A never saw a citation prompt, so its "abstention" is simply how a verdict-tuned model reacts to an unfamiliar output format. That this varies by backbone and becomes uniform once "none" targets are supervised is what SFT would be expected to do. It is a reasonable sanity check, but I don't see it as a contribution. The Discussion also quotes the Arm B range, while the shipped Arm C model spreads out again to 0.861-0.989 (Table 13).
(b) Probe. The probe labels each token as payload or not, and spans are rebuilt through the tokenizer's offset mapping (App. D.3). It never reads out a character index. It therefore shows the model knows which tokens belong to the payload, which the 0.780 quote accuracy already tells us, but not that "the position it fails to report" is encoded. The 0.993 AUC in the untouched base weights also suggests it may just be detecting the template line. A probe on the span's start offset or line index would actually test the claim.
(c) Related work. Localizing injected content already has dedicated methods that should at least be discussed: PromptLocate (Jia et al., IEEE S&P 2026), SkillGuard-Robust (arXiv:2604.25109), HINTBench (arXiv:2604.13954), and the step-level attribution in AgentDoG. Locate-and-Judge (arXiv:2606.23416) and VIGIL (arXiv:2606.26524) appeared close to the deadline but are directly relevant, and VIGIL's near-identical name should be addressed. I did not find prior work covering the specific combination here (one stage-conditioned adapter, nested evidence arms, the quote/offset gap). Concurrent preprints such as SkillSecurer, SkillSentry, SkillAtlas, and SkillLite should not count against the paper, though a sentence on each would help. At minimum, I would like to see a PromptLocate-style baseline at the skill stage and the invocation rule in a main-text table.
W3. Minor issues.
(a) The paper often reads like a revision log ("we withdraw that column", cells that "did not previously satisfy" the rule). A short table of claims, supporting experiments, and current status would make it much easier to follow.
(b) The abstain column in Table 2 (e.g., 0.968 for Qwen3-1.7B) does not match Arm C abstention in Table 13 (0.861) and seems to come from the attribution channel. Please label which channel each column uses.
(c) The midpoint criterion is lenient when some models emit spans up to 611 characters wide against a 360-character reference. IoU should be in the main text.
=== Post-rebuttal update ===
The rebuttal addresses most of my concerns. The new rule comparisons, call rewrites, held-out family, and Skill-Inject experiments resolve W1(a), W1(b), and W1(d). The abstract and contributions are now scoped to one construction procedure, and the probe claim has been revised as I asked in W2(b). The related work and presentation issues in W2(c) and W3 are fixed. One limitation remains and should be stated in the final version: evidence accuracy drops outside the training generator (F1 0.355 at the execution stage on Skill-Inject, and file recall 0.017 for the 4B model on SkillSafetyBench). Because the authors met the conditions I stated, I raise my rating to weak accept.
However, I cannot raise it to strong accept, as the major technical contributions are finetuning and evaluations, which are relatively achievable with the help of AI tools.
Questions for authors
Q1 (analysis). How does the trained locator do on the 11 non-script rows, where the rule gets 0.182, and on any window where the payload call does not look like bash resources/*.sh? A clear margin here would go a long way toward addressing W1(a).
Q2 (analysis). How many no-target windows contain a call to a bundled helper (App. B.3 mentions env_checker.sh in 10 negative windows and arxiv_search.py in 163), and what is the false-citation rate on those windows? This is the closest available substitute for the benign-helper test described in App. G.
Q3 (experiment). For the frontier baselines, could you add (i) a few-shot prompt, (ii) a prompt that states the prior that the attacker's file is a bundled helper, and (iii) a forced-choice version without "none"? If (ii) closes most of the gap, the abstract should say the target is not identifiable without that prior.
Q4 (experiment). Can you test at least one held-out axis, such as leave-one-family-out, carrier-disjoint splits, or a second generator like Skill-Inject or SkillAttack? An adaptive variant that renames and relocates the helper, or invokes it some other way, would test W1(d) directly.
Q5 (experiment). SkillSafetyBench records the injected files. Can you report quote and locator accuracy there, even at file level, so that at least one evidence number is external?
Q6 (analysis). Can you probe for the quantity that actually fails, for example by regressing the start offset or line index of the span from the hidden state at the first payload token, with the position-only control? If that works, the expression-versus-content claim holds; if not, the abstract should be revised.
Q7 (clarification). In App. B.5, where in the record do the recovered arguments come from, and would a deployed auditor have access to them? Execution windows also include the agent's closing narrative. How do the verdict and locator change if it is removed?
Q8 (clarification). How was the length-matched MalSkillBench subset sampled, and what are AUC and F1 on the full benchmark and on its published split?
If Q1 and Q2 show the locator clearly beating the rule on non-template payloads and benign helper calls, and Q4 or Q5 shows the evidence channel holding up on a new generator or external data, I would raise my rating to weak accept. Even without new experiments, bringing the abstract and contribution list in line with Sec. 6 and with what the probe actually shows would resolve the scope concern in W1 and most of W2(b).
=== Post-rebuttal update ===
The authors answered all eight questions with new experiments or clarifications. Q1: the locator finds all 11 non-script payloads and all 55 positives the rule misses. Q2: it makes 4 false citations on the 38 benign-helper windows (0 for the 1.7B model). Q3: with the best few-shot prompt, the frontier baseline reaches 0.640, against 0.995 for SkillVigil. Q4: the held-out family scores 61/62 and the rewritten calls 275-300/300, and Skill-Inject gives a second generator. Q5: SkillSafetyBench now provides an external evidence number. Q6: the offset regression does not beat the position-only control, and the probe claim was revised to match. Q7: the recovered arguments come from the run record, and removing the closing narrative changes little. Q8: the subset sampling is documented, with AUC 0.863 on the full benchmark. I have no further questions.
5 comments
Response to Reviewer ak23 (Part 2/5)
W1(b). "GPT-4o and Qwen3.8-27B score below the 0.227 chance level, and Table 2 shows they abstain on 0.990 and 0.940 of windows. […] To me this says the target is not identifiable from the window without the corpus prior, which is different from "a gap of training exposure rather than capability". No few-shot or prior-stating prompt is tried, so the abstract's reading is untested."
Q3. "For the frontier baselines, could you add (i) a few-shot prompt, (ii) a prompt that states the prior that the attacker's file is a bundled helper, and (iii) a forced-choice version without "none"? If (ii) closes most of the gap, the abstract should say the target is not identifiable without that prior."
Response. We thank the reviewer for these well-posed controls. We ran all three and rescored SkillVigil on the same sample IDs (Table 21, App. E.5). App. C reproduces these prompts verbatim:
| Prompt | GPT-4o event / quote | Qwen3.8-27B event / quote | SkillVigil Qwen3-4B event / quote |
|---|---|---|---|
| Few-shot | 0.500 / 0.500 | 0.640 / 0.635 | 0.995 / 0.995 |
| Prior stated | 0.410 / 0.330 | 0.615 / 0.590 | 0.995 / 0.995 |
Forced choice, no none | 0.475 / 0.120 | 0.465 / 0.455 | 0.995 / 0.995 |
| Same IDs plus 200 negatives | GPT-4o | Qwen3.8-27B | SkillVigil Qwen3-4B |
|---|---|---|---|
Correct none, few-shot / prior | 0.940 / 0.920 | 0.915 / 0.550 | 0.975 |
Joint event-or-none, few-shot / prior | 0.720 / 0.665 | 0.778 / 0.583 | 0.985 |
SkillVigil is not prompted, so its row repeats its standard output. The four few-shot examples are stratified training rows with all test IDs excluded.
On (ii). Stating the corpus prior raises Qwen's event accuracy from 0.050 to 0.615, closing about 60% of the gap to SkillVigil (0.995), but correct none falls to 0.550 (joint: 0.583 vs. 0.985).
On (i) and the abstract. Few-shot prompting raises Qwen to 0.640 (joint: 0.778), still below SkillVigil's 0.995 (joint: 0.985) on the same IDs. We removed "training exposure rather than capability" from the paper. The abstract now reports the matched few-shot comparison. §5.1 attributes the remaining gap to in-domain supervision, and App. E.5 reports the prior-stating control.
W1(c). "Arm C span citation is 0.045-0.235 against a 0.100 null, and no cell clears 0.025 under the containment criterion (Tables 13 and 19). Evidence accuracy is never measured on MalSkillBench or SkillSafetyBench; the checkmarks in Figure 2 and Table 8 only mark output formats. For detection, the relevant comparison is the skill-specific scanners in Table 9, and the 4B model (F1 0.812) does not beat Sentry (0.886) or AI-Infra-Guard (0.856), although the splits differ."
Q5. "SkillSafetyBench records the injected files. Can you report quote and locator accuracy there, even at file level, so that at least one evidence number is external?"
Response. On Q5, external evidence. Thank you for asking for an external evidence number. On all 255 SkillSafetyBench bags (Table 11), with 1,525 windows, 881 files, and 231 injected or modified files:
| SkillSafetyBench | Correct target files (231) | False citations (650 other files) | File precision | File recall | Quote verbatim in cited window |
|---|---|---|---|---|---|
| Qwen3-1.7B | 27 | 2 | 0.931 | 0.117 | 19/29 |
| Qwen3-4B | 4 | 6 | 0.400 | 0.017 | 10/10 |
The 1.7B model gives selective, high-precision evidence. The 4B model cites no file in any of the 72 clean bags. SkillSafetyBench has no gold quotes, so quote grounding checks occurrence in the cited window. MalSkillBench skills are malicious as a whole, with no clean counterpart from which an injected span could be marked, so its evidence cannot be scored. On the span channel. §5.3 now reports the stricter criteria, and in the same responses the model's quote recovers the location (App. E.13):
| Skill stage, same responses, five backbones | Generated character span | Model's quote resolved in the window |
|---|---|---|
| Localization | 0.019–0.116 | 0.675–0.856 |
The quote is therefore the evidence a skill-stage audit should request. On Figure 2 and Table 8. The marks record which outputs a system emits, and their accuracy is reported in Tables 1, 11, and 13. On the scanners. We ran the official static component of Sentry (Sentry-static) on identical MalSkillBench rows, with the SkillVigil threshold frozen on disjoint calibration rows for 80% recall (Table 10, Figure 6):
| 3,034 identical reporting rows | Recall | FPR | F1 |
|---|---|---|---|
| Sentry-static | 0.286 | 0.107 | 0.411 |
| SkillVigil Qwen3-4B, seed 42 | 0.782 | 0.155 | 0.807 |
| SkillVigil Qwen3-4B, seed 1 | 0.786 | 0.112 | 0.828 |
| SkillVigil Qwen3-4B, seed 2 | 0.784 | 0.075 | 0.844 |
The published Sentry-full (0.886) and AI-Infra-Guard (0.856) figures come from a random split whose membership the official scorer does not expose, so Table 10 keeps them in a separate block.
Response to Reviewer ak23 (Part 3/5)
W1(d). "All injections come from a single procedure over 27 carriers. There is no held-out template, carrier split, or second generator, and nothing tests an attacker who moves the payload out of resources/, runs it through python, or splits the call across windows. The model's score on the 11 non-script rows, where the rule falls to 0.182, is not reported."
Q4. "Can you test at least one held-out axis, such as leave-one-family-out, carrier-disjoint splits, or a second generator like Skill-Inject or SkillAttack? An adaptive variant that renames and relocates the helper, or invokes it some other way, would test W1(d) directly."
Response. Carrier split and held-out family. The submitted split is already carrier-disjoint (App. B.3). We also removed information_disclosure from every training and calibration channel and retrained, with test rows and decoding unchanged:
| Held-out family | POST locator, full→held out | POST grounded quote | POST correct none |
|---|---|---|---|
| Qwen3-4B | 62/62→61/62 | 62/62→61/62 | 27/27→27/27 |
| Qwen3-1.7B | 62/62→62/62 | 60/62→62/62 | 26/27→25/27 |
Moved, run through python, split across windows. We rewrote in the trace the 300 attack calls that R1 matches, keeping event IDs, labels, and the 406 no-target windows (Table 23). In the last row the copy happens in an earlier window, so the evaluated window never names the bundle:
| Attack call located (300) | R1 | R2 | Qwen3-4B | Qwen3-1.7B |
|---|---|---|---|---|
python3 resources/x.py | 0 | 245 | 296 | 271 |
python3 helpers/x.py | 0 | 0 | 275 | 239 |
copied in an earlier window, run from /tmp | 0 | 0 | 300 | 292 |
The rewrite leaves the 406 no-target windows unchanged, and correct none stays at 379/406 (4B seed 42) and 386/406 (1.7B). R1 falls to 0 on all three rewrites and R2 on the last two, while the learned locator keeps 275–300 of 300. We also renamed resources/ to helpers/ before retraining a 1.7B model (Table 23):
| Path intervention, 406 positives | Original paths | resources/→helpers/ |
|---|---|---|
| SkillVigil Qwen3-1.7B, retrained | 388/406 | 387/406 |
| R1 | 351/406 | 0/406 |
Second generator. SkillAttack leaves skills unmodified, so its skills are clean under Eq. 1. We therefore use Skill-Inject (App. G.1, Table 29), whose templates, scripts, and tasks share nothing with our construction. We scored its 245 bundles and recorded 88 Claude Code runs on its script attacks and clean counterparts. R3, the first call to a script shipped with the skill, is R1 adapted to this layout:
| Skill stage, bag AUC | All bundles (212 / 33) | Skills absent from our corpus (73 / 28) |
|---|---|---|
| Qwen3Guard-Gen-8B | 0.632 | 0.591 |
| SkillVigil Qwen3-1.7B | 0.785 | 0.692 |
| SkillVigil Qwen3-4B | 0.859 | 0.810 |
| Execution stage, 643 windows | R1 | R2 | R3 | SkillVigil Qwen3-4B |
|---|---|---|---|---|
| Attack windows located (56) | 0 | 0 | 39 | 33 |
Benign bundled-script windows, none (178) | 178 | 178 | 0 | 152 |
All no-target windows, none (587) | 587 | 587 | 409 | 490 |
| F1 | 0 | 0 | 0.286 | 0.355 |
F1 counts a located attack window as a true positive and any citation on a no-target window as a false positive. Three 4B seeds give AUC 0.810–0.859 and F1 0.355–0.412. A paired bootstrap over bundles puts the 4B model's skill-stage margin over Qwen3Guard-Gen-8B at [0.124, 0.338] overall and [0.110, 0.338] on the skills absent from our corpus. Skill-Inject templates and agent runs are all unseen in training. Of the 178 benign script calls, 147 run ooxml/scripts/ code that occurs nowhere in our training data. An attacker who optimizes against the guard itself is the case §6 states as open.
W1, scope. "Sec. 6 scopes the execution-stage results to corpus-specific auditing, and I agree with that framing. The abstract, introduction, and contribution list do not; they present the evidence results as general properties of evidence-supervised guardrails."
Response. Following your final paragraph, the abstract now reads: "At the execution stage, on attacks from one construction procedure, it names the implicated call on 0.998 of windows and quotes it on 0.995, against 0.865 for a rule matching the attack's invocation, and names it on 0.995 of a 200-window subset where the strongest few-shot frontier prompt reaches 0.640." It attributes the margins over general guards to in-domain training rather than evidence supervision alone. The introduction and the contribution list now state that the findings hold within one attack-construction procedure.
Response to Reviewer ak23 (Part 1/5)
We sincerely thank the reviewer for the careful review, for the strengths noted in S1 and S2, and for stating clear conditions. As the Ethics Statement says, the corpus will be released with the label audit of App. B.3 attached. Changes in the revised PDF are marked in red, and table numbers refer to the revision. We quote each point and answer it in the order of the review.
Response. We apply the paper's two no-model rules (Table 20, submitted Table 15) to the windows the model sees. R1 is the first
Bashcall whose command names aresources/*.shfile (0.865). R2 is the first event namingresources/(0.798). Model columns are saved Arm C outputs, and §5.1 now reports this table as Table 3:arxiv_search.pycall (38)On Q1, the 11 non-script rows. The 4B locator names the implicated event in all 11, against 2 for R1. Nine of the eleven payloads are Markdown files, which no
resources/*.shrule can reach, and the 4B model locates all nine. The other two have no file extension, and both the rule and the model locate them.On Q1, windows whose payload call does not look like
bash resources/*.sh. We tested three kinds, each in a different way:Readof the injected fileGlobeventBashcall of another formpython3 helpers/x.py(Part 3)/tmp(Part 3)The first three rows are the 55 positives R1 misses in our own test set. The rewritten rows keep every event, label, and no-target window of the original test set. Skill-Inject's attacks run bundled Python scripts that share no template with ours.
The result holds on every backbone we trained, not only on the 4B model:
App. E.4 now reports this breakdown.
On Q2. 38 of the 406 no-target windows contain an actual
arxiv_search.pycall. None contains an actualenv_checker.shevent, and three mention it outside any event, so App. B.3's 10/163 is a pool count rather than the evaluated set. False citations on the 38 are 4 (4B) and 0 (1.7B). R1 cites none because it never fires outside its pattern, and widening it to recover the missed attacks (R2: 46/55) makes it cite 27 of the 38.On the fair comparison. We agree, and the abstract now reports 0.998 against 0.865 for the rule, on attacks from one construction procedure. On all 812 windows the 4B locator is the most accurate system (797 against 757). Benign helper calls of the attack's own form come from a second generator, Skill-Inject (Schmotz et al., 2026, arXiv:2602.20156), detailed in Part 3. A rule adapted to that layout (R3) cites all of them, and the 4B model leaves most uncited:
none(178)none(587)