acceptodds
Under review as a conference paper at ICLR 2027

SkillVigil: Evidence-Supervised Guardrails Against Skill Injection in LLM Agents

Abstract

Coding agents now install third-party skills: bundles of instructions and executable scripts that enter the agent's context and run with its privileges. A poisoned bundle does not act on its own but recruits the agent that loads it, and it is reloaded every time the skill is used. The guardrails deployed for this surface check a bundle before it is loaded and stop there. Runtime monitoring is a separate line that watches the execution, rarely attributing it to the bundle that introduced it. Either way the operator learns at most that something is wrong, not which step is implicated or what text implicates it. Scored the same way on the same inputs, three of six general-purpose guardrails fall within of chance on skill bundles: a bundle reads as ordinary documentation until one line invokes a bundled script. We train a single stage-conditioned adapter that screens the bundle before execution and audits the resulting trace afterwards, supervised to return a citation and an explicit abstention alongside the verdict. It reaches AUC on MalSkillBench, where the best of those six reaches . It names the implicated call on of execution windows and quotes it on , whereas GPT-4o and Qwen3.8-B prompted zero-shot reach and on the locator, a gap of training exposure rather than capability. Adding the channels one at a time under a fixed budget moves verdict AUC by at most in paired estimates on the same executions and bundles, and makes abstention, otherwise ranging from to across backbones, uniform. Localization succeeds where the answer can be copied from the input and degrades where it must be counted to, a limit we trace to the output format: probes read from the same hidden states the position it still fails to report.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agent security, prompt injection, LLM agents, guardrail models, agent skills

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ak23OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies skill injection, where an attacker hides text in a third-party skill bundle (SKILL.md, reference files, helper scripts) so that the coding agent loading the skill carries out the attack, and keeps doing so every time the skill is used. The authors label two things separately: whether the bundle is poisoned (skill stage, Eq. 1), and whether a recorded execution actually ran injected content that was permitted and completed (execution stage, Eq. 2), so that Ypost≤YpreY^{\text{post}} \le Y^{\text{pre}}. The corpus applies the SkillJect procedure to 27 public carrier skills, giving 3,492 poisoned bundles across nine attack families, which are run under four agent frameworks and six backends for 4,165 executions; 46 clean carriers serve as extra skill-stage negatives. SkillVigil is a single LoRA adapter, and a system prompt picks the stage and the output channel. Three nested arms are trained under a fixed per-epoch budget: verdict only (Arm A), plus citation (Arm B, a character span for skills and an event ID for traces), and plus attribution (Arm C, which adds a quote and an attack class), each on five backbones from 1.2B to 4B. The main results are that in-distribution verdict AUC is saturated across all 15 models, while on a length-matched MalSkillBench subset the 4B model reaches 0.891 AUC against 0.714 for the best of six general-purpose guards; that at the execution stage the adapter picks the right event on 0.998 of windows, compared with 0.040 and 0.050 for zero-shot GPT-4o and Qwen3.8-27B; that citation supervision makes abstention consistent across backbones; and that at the skill stage the model quotes the injected line correctly on 0.780 of documents but gets the character span right on only 0.116. Linear probes find payload tokens at about 0.99 AUC even in the base weights, which the authors read as the model knowing where the payload is but being unable to say it. The appendices contain a long series of shortcut audits, including length matching, a counting baseline, a filename lookup that leads the authors to drop the attack-class metric, stratification by recording style, a label audit, extra seeds, and an untrained baseline.

Critical strengths

S1. The measurement is careful, and the authors are honest about their own shortcuts. For most headline numbers they also report the simplest statistic that could have produced it, and they act on what they find. Length matching drops a trace-counting baseline from 0.781 to 0.500 while the detectors stay at 0.944-0.958. A ten-entry filename lookup reaches 0.956 on the attack class, so they withdraw that metric. Splitting by recording style shows that a pooled 0.793 vs. 0.781 result hides a detector that loses to counting on argument-free traces and wins easily on the rest. They also disclose and fix a checkpoint mix-up in Table 8. I trust the reported numbers more because of this, and several of my concerns below come directly from the authors' own diagnostics.

S2. The two-stage framing is useful and is backed by real executions. Separating the bundle label from the execution label, and then tracing where each failed attack stopped, makes a concrete case for an external guardrail: of the 2,360 poisoned but uncompromised executions, 80.7% have the payload in context with no action and no comment, and only 4.8% contain an explicit refusal. An agent that quietly does not comply is not something an operator can audit. The instrumentation finding is also practically important. One framework logs tool names without arguments, which drops the detector to 0.603 on those traces, and recovering the arguments lifts it to 0.938 with the model unchanged. If the corpus is released with the label audit as planned, I expect this to be the most reused part of the paper.

=== Post-rebuttal update ===

The rebuttal strengthens S1 and adds the controls I asked for. On all 812 execution windows the 4B locator is correct on 797, against 757 for the best invocation rule. It recovers all 55 positives the rule misses, including 9 Markdown payloads that no script rule can reach, and the result holds on all five backbones. When the attack call is rewritten outside the rule’s pattern (python3 helpers/x.py, or copied and run from /tmp), the rules fall to 0/300 while the locator keeps 275-300/300. Holding out an attack family leaves the locator at 61/62. On a second generator (Skill-Inject) the skill-stage guard reaches AUC 0.859 against 0.632 for Qwen3Guard-Gen-8B, and 0.810 on skills absent from the training corpus. The new same-protocol baselines are informative: PromptLocate reaches character F1 0.145 against 0.282 at about 200 times the latency, and Sentry-static reaches recall 0.286 against 0.784 at a similar FPR. I also appreciate that the authors ran the offset-regression probe, reported that it does not beat the position-only control, and withdrew the probe claim from the abstract. The abstract and contributions are now scoped to one construction procedure, which resolves my scope concern.

Critical weaknesses

W1. The headline evidence numbers are not shown to go beyond what the corpus construction gives away, and evidence quality is never checked outside the corpus. This is the main reason for my score.

(a) Execution stage. 395 of the 406 attributed windows have a shell-script payload, and the rule "first Bash call on resources/*.sh" already gets 0.865 without any model (Table 15). Script bodies are withheld, so the invocation string is all the model sees, and filename randomization keeps the bash resources/<name>.sh pattern intact. Benign helper calls of the same form are never tested. The fair comparison is therefore 0.998 against 0.865 on one generator, not against 0.040.

(b) Frontier baselines. GPT-4o and Qwen3.8-27B score below the 0.227 chance level, and Table 2 shows they abstain on 0.990 and 0.940 of windows. With a prompt that hides the script and asks for a judgment from the call alone, declining to flag bash resources/workspace_validator.sh seems like the reasonable answer. To me this says the target is not identifiable from the window without the corpus prior, which is different from "a gap of training exposure rather than capability". No few-shot or prior-stating prompt is tried, so the abstract's reading is untested.

(c) Skill stage. This is where the load-or-not decision is made and the only stage tested externally, yet its evidence channel is weak. Arm C span citation is 0.045-0.235 against a 0.100 null, and no cell clears 0.025 under the containment criterion (Tables 13 and 19). Evidence accuracy is never measured on MalSkillBench or SkillSafetyBench; the checkmarks in Figure 2 and Table 8 only mark output formats. For detection, the relevant comparison is the skill-specific scanners in Table 9, and the 4B model (F1 0.812) does not beat Sentry (0.886) or AI-Infra-Guard (0.856), although the splits differ.

(d) One generator, no adaptive attacker. All injections come from a single procedure over 27 carriers. There is no held-out template, carrier split, or second generator, and nothing tests an attacker who moves the payload out of resources/, runs it through python, or splits the call across windows. The model's score on the 11 non-script rows, where the rule falls to 0.182, is not reported.

Sec. 6 scopes the execution-stage results to corpus-specific auditing, and I agree with that framing. The abstract, introduction, and contribution list do not; they present the evidence results as general properties of evidence-supervised guardrails.

W2. Some explanatory claims go further than the controls allow, and closely related work on localization is missing.

(a) Abstention. Arm A never saw a citation prompt, so its "abstention" is simply how a verdict-tuned model reacts to an unfamiliar output format. That this varies by backbone and becomes uniform once "none" targets are supervised is what SFT would be expected to do. It is a reasonable sanity check, but I don't see it as a contribution. The Discussion also quotes the Arm B range, while the shipped Arm C model spreads out again to 0.861-0.989 (Table 13).

(b) Probe. The probe labels each token as payload or not, and spans are rebuilt through the tokenizer's offset mapping (App. D.3). It never reads out a character index. It therefore shows the model knows which tokens belong to the payload, which the 0.780 quote accuracy already tells us, but not that "the position it fails to report" is encoded. The 0.993 AUC in the untouched base weights also suggests it may just be detecting the template line. A probe on the span's start offset or line index would actually test the claim.

(c) Related work. Localizing injected content already has dedicated methods that should at least be discussed: PromptLocate (Jia et al., IEEE S&P 2026), SkillGuard-Robust (arXiv:2604.25109), HINTBench (arXiv:2604.13954), and the step-level attribution in AgentDoG. Locate-and-Judge (arXiv:2606.23416) and VIGIL (arXiv:2606.26524) appeared close to the deadline but are directly relevant, and VIGIL's near-identical name should be addressed. I did not find prior work covering the specific combination here (one stage-conditioned adapter, nested evidence arms, the quote/offset gap). Concurrent preprints such as SkillSecurer, SkillSentry, SkillAtlas, and SkillLite should not count against the paper, though a sentence on each would help. At minimum, I would like to see a PromptLocate-style baseline at the skill stage and the invocation rule in a main-text table.

W3. Minor issues.

(a) The paper often reads like a revision log ("we withdraw that column", cells that "did not previously satisfy" the rule). A short table of claims, supporting experiments, and current status would make it much easier to follow.

(b) The abstain column in Table 2 (e.g., 0.968 for Qwen3-1.7B) does not match Arm C abstention in Table 13 (0.861) and seems to come from the attribution channel. Please label which channel each column uses.

(c) The midpoint criterion is lenient when some models emit spans up to 611 characters wide against a 360-character reference. IoU should be in the main text.

=== Post-rebuttal update ===

The rebuttal addresses most of my concerns. The new rule comparisons, call rewrites, held-out family, and Skill-Inject experiments resolve W1(a), W1(b), and W1(d). The abstract and contributions are now scoped to one construction procedure, and the probe claim has been revised as I asked in W2(b). The related work and presentation issues in W2(c) and W3 are fixed. One limitation remains and should be stated in the final version: evidence accuracy drops outside the training generator (F1 0.355 at the execution stage on Skill-Inject, and file recall 0.017 for the 4B model on SkillSafetyBench). Because the authors met the conditions I stated, I raise my rating to weak accept.

However, I cannot raise it to strong accept, as the major technical contributions are finetuning and evaluations, which are relatively achievable with the help of AI tools.

Questions for authors

Q1 (analysis). How does the trained locator do on the 11 non-script rows, where the rule gets 0.182, and on any window where the payload call does not look like bash resources/*.sh? A clear margin here would go a long way toward addressing W1(a).

Q2 (analysis). How many no-target windows contain a call to a bundled helper (App. B.3 mentions env_checker.sh in 10 negative windows and arxiv_search.py in 163), and what is the false-citation rate on those windows? This is the closest available substitute for the benign-helper test described in App. G.

Q3 (experiment). For the frontier baselines, could you add (i) a few-shot prompt, (ii) a prompt that states the prior that the attacker's file is a bundled helper, and (iii) a forced-choice version without "none"? If (ii) closes most of the gap, the abstract should say the target is not identifiable without that prior.

Q4 (experiment). Can you test at least one held-out axis, such as leave-one-family-out, carrier-disjoint splits, or a second generator like Skill-Inject or SkillAttack? An adaptive variant that renames and relocates the helper, or invokes it some other way, would test W1(d) directly.

Q5 (experiment). SkillSafetyBench records the injected files. Can you report quote and locator accuracy there, even at file level, so that at least one evidence number is external?

Q6 (analysis). Can you probe for the quantity that actually fails, for example by regressing the start offset or line index of the span from the hidden state at the first payload token, with the position-only control? If that works, the expression-versus-content claim holds; if not, the abstract should be revised.

Q7 (clarification). In App. B.5, where in the record do the recovered arguments come from, and would a deployed auditor have access to them? Execution windows also include the agent's closing narrative. How do the verdict and locator change if it is removed?

Q8 (clarification). How was the length-matched MalSkillBench subset sampled, and what are AUC and F1 on the full benchmark and on its published split?

If Q1 and Q2 show the locator clearly beating the rule on non-template payloads and benign helper calls, and Q4 or Q5 shows the evidence channel holding up on a new generator or external data, I would raise my rating to weak accept. Even without new experiments, bringing the abstract and contribution list in line with Sec. 6 and with what the probe actually shows would resolve the scope concern in W1 and most of W2(b).

=== Post-rebuttal update ===

The authors answered all eight questions with new experiments or clarifications. Q1: the locator finds all 11 non-script payloads and all 55 positives the rule misses. Q2: it makes 4 false citations on the 38 benign-helper windows (0 for the 1.7B model). Q3: with the best few-shot prompt, the frontier baseline reaches 0.640, against 0.995 for SkillVigil. Q4: the held-out family scores 61/62 and the rewritten calls 275-300/300, and Skill-Inject gives a second generator. Q5: SkillSafetyBench now provides an external evidence number. Q6: the offset regression does not beat the position-only control, and the probe claim was revised to match. Q7: the recovered arguments come from the run record, and removing the closing narrative changes little. Q8: the subset sampling is documented, with AUC 0.863 on the full benchmark. I have no further questions.

5 comments

Response to Reviewer ak23 (Part 1/5)

We sincerely thank the reviewer for the careful review, for the strengths noted in S1 and S2, and for stating clear conditions. As the Ethics Statement says, the corpus will be released with the label audit of App. B.3 attached. Changes in the revised PDF are marked in red, and table numbers refer to the revision. We quote each point and answer it in the order of the review.

W1(a). "395 of the 406 attributed windows have a shell-script payload, and the rule "first Bash call on resources/*.sh" already gets 0.865 without any model (Table 15). […] Benign helper calls of the same form are never tested. The fair comparison is therefore 0.998 against 0.865 on one generator, not against 0.040."

Q1. "How does the trained locator do on the 11 non-script rows, where the rule gets 0.182, and on any window where the payload call does not look like bash resources/*.sh?"

Q2. "How many no-target windows contain a call to a bundled helper (App. B.3 mentions env_checker.sh in 10 negative windows and arxiv_search.py in 163), and what is the false-citation rate on those windows?"

Response. We apply the paper's two no-model rules (Table 20, submitted Table 15) to the windows the model sees. R1 is the first Bash call whose command names a resources/*.sh file (0.865). R2 is the first event naming resources/ (0.798). Model columns are saved Arm C outputs, and §5.1 now reports this table as Table 3:

Correct outputsQwen3-4BQwen3-1.7BR1R2
Positives that R1 misses (55)5551046
Non-script positives (11)119210
No-target windows with a real arxiv_search.py call (38)34383811
All 812 windows (406 positive, 406 no-target)797791757683

On Q1, the 11 non-script rows. The 4B locator names the implicated event in all 11, against 2 for R1. Nine of the eleven payloads are Markdown files, which no resources/*.sh rule can reach, and the 4B model locates all nine. The other two have no file extension, and both the rule and the model locate them.

On Q1, windows whose payload call does not look like bash resources/*.sh. We tested three kinds, each in a different way:

Payload call outside the R1 patternWindowsR1Qwen3-4B
Natural traces: Read of the injected file48048
Natural traces: Glob event404
Natural traces: Bash call of another form303
Calls rewritten to python3 helpers/x.py (Part 3)3000275
Calls copied in an earlier window, run from /tmp (Part 3)3000300
Skill-Inject attacks, a second generator (Part 3)56033

The first three rows are the 55 positives R1 misses in our own test set. The rewritten rows keep every event, label, and no-target window of the original test set. Skill-Inject's attacks run bundled Python scripts that share no template with ours.

The result holds on every backbone we trained, not only on the 4B model:

Arm C backboneAll positives (406)R1-missed positives (55)Non-script positives (11)
Qwen3-4B4055511
Qwen3-1.7B398519
Llama-3.2-3B4055410
Llama-3.2-1B4035511
Phi-4-mini400539
R135102

App. E.4 now reports this breakdown.

On Q2. 38 of the 406 no-target windows contain an actual arxiv_search.py call. None contains an actual env_checker.sh event, and three mention it outside any event, so App. B.3's 10/163 is a pool count rather than the evaluated set. False citations on the 38 are 4 (4B) and 0 (1.7B). R1 cites none because it never fires outside its pattern, and widening it to recover the missed attacks (R2: 46/55) makes it cite 27 of the 38.

On the fair comparison. We agree, and the abstract now reports 0.998 against 0.865 for the rule, on attacks from one construction procedure. On all 812 windows the 4B locator is the most accurate system (797 against 757). Benign helper calls of the attack's own form come from a second generator, Skill-Inject (Schmotz et al., 2026, arXiv:2602.20156), detailed in Part 3. A rule adapted to that layout (R3) cites all of them, and the 4B model leaves most uncited:

Skill-Inject, 643 windows from 88 runsR3Qwen3-4B
Benign bundled-script windows, none (178)0152
All no-target windows, none (587)409490
F10.2860.355

Response to Reviewer ak23 (Part 2/5)

W1(b). "GPT-4o and Qwen3.8-27B score below the 0.227 chance level, and Table 2 shows they abstain on 0.990 and 0.940 of windows. […] To me this says the target is not identifiable from the window without the corpus prior, which is different from "a gap of training exposure rather than capability". No few-shot or prior-stating prompt is tried, so the abstract's reading is untested."

Q3. "For the frontier baselines, could you add (i) a few-shot prompt, (ii) a prompt that states the prior that the attacker's file is a bundled helper, and (iii) a forced-choice version without "none"? If (ii) closes most of the gap, the abstract should say the target is not identifiable without that prior."

Response. We thank the reviewer for these well-posed controls. We ran all three and rescored SkillVigil on the same sample IDs (Table 21, App. E.5). App. C reproduces these prompts verbatim:

PromptGPT-4o event / quoteQwen3.8-27B event / quoteSkillVigil Qwen3-4B event / quote
Few-shot0.500 / 0.5000.640 / 0.6350.995 / 0.995
Prior stated0.410 / 0.3300.615 / 0.5900.995 / 0.995
Forced choice, no none0.475 / 0.1200.465 / 0.4550.995 / 0.995
Same IDs plus 200 negativesGPT-4oQwen3.8-27BSkillVigil Qwen3-4B
Correct none, few-shot / prior0.940 / 0.9200.915 / 0.5500.975
Joint event-or-none, few-shot / prior0.720 / 0.6650.778 / 0.5830.985

SkillVigil is not prompted, so its row repeats its standard output. The four few-shot examples are stratified training rows with all test IDs excluded.

On (ii). Stating the corpus prior raises Qwen's event accuracy from 0.050 to 0.615, closing about 60% of the gap to SkillVigil (0.995), but correct none falls to 0.550 (joint: 0.583 vs. 0.985).

On (i) and the abstract. Few-shot prompting raises Qwen to 0.640 (joint: 0.778), still below SkillVigil's 0.995 (joint: 0.985) on the same IDs. We removed "training exposure rather than capability" from the paper. The abstract now reports the matched few-shot comparison. §5.1 attributes the remaining gap to in-domain supervision, and App. E.5 reports the prior-stating control.

W1(c). "Arm C span citation is 0.045-0.235 against a 0.100 null, and no cell clears 0.025 under the containment criterion (Tables 13 and 19). Evidence accuracy is never measured on MalSkillBench or SkillSafetyBench; the checkmarks in Figure 2 and Table 8 only mark output formats. For detection, the relevant comparison is the skill-specific scanners in Table 9, and the 4B model (F1 0.812) does not beat Sentry (0.886) or AI-Infra-Guard (0.856), although the splits differ."

Q5. "SkillSafetyBench records the injected files. Can you report quote and locator accuracy there, even at file level, so that at least one evidence number is external?"

Response. On Q5, external evidence. Thank you for asking for an external evidence number. On all 255 SkillSafetyBench bags (Table 11), with 1,525 windows, 881 files, and 231 injected or modified files:

SkillSafetyBenchCorrect target files (231)False citations (650 other files)File precisionFile recallQuote verbatim in cited window
Qwen3-1.7B2720.9310.11719/29
Qwen3-4B460.4000.01710/10

The 1.7B model gives selective, high-precision evidence. The 4B model cites no file in any of the 72 clean bags. SkillSafetyBench has no gold quotes, so quote grounding checks occurrence in the cited window. MalSkillBench skills are malicious as a whole, with no clean counterpart from which an injected span could be marked, so its evidence cannot be scored. On the span channel. §5.3 now reports the stricter criteria, and in the same responses the model's quote recovers the location (App. E.13):

Skill stage, same responses, five backbonesGenerated character spanModel's quote resolved in the window
Localization0.019–0.1160.675–0.856

The quote is therefore the evidence a skill-stage audit should request. On Figure 2 and Table 8. The marks record which outputs a system emits, and their accuracy is reported in Tables 1, 11, and 13. On the scanners. We ran the official static component of Sentry (Sentry-static) on identical MalSkillBench rows, with the SkillVigil threshold frozen on disjoint calibration rows for 80% recall (Table 10, Figure 6):

3,034 identical reporting rowsRecallFPRF1
Sentry-static0.2860.1070.411
SkillVigil Qwen3-4B, seed 420.7820.1550.807
SkillVigil Qwen3-4B, seed 10.7860.1120.828
SkillVigil Qwen3-4B, seed 20.7840.0750.844

The published Sentry-full (0.886) and AI-Infra-Guard (0.856) figures come from a random split whose membership the official scorer does not expose, so Table 10 keeps them in a separate block.

Response to Reviewer ak23 (Part 3/5)

W1(d). "All injections come from a single procedure over 27 carriers. There is no held-out template, carrier split, or second generator, and nothing tests an attacker who moves the payload out of resources/, runs it through python, or splits the call across windows. The model's score on the 11 non-script rows, where the rule falls to 0.182, is not reported."

Q4. "Can you test at least one held-out axis, such as leave-one-family-out, carrier-disjoint splits, or a second generator like Skill-Inject or SkillAttack? An adaptive variant that renames and relocates the helper, or invokes it some other way, would test W1(d) directly."

Response. Carrier split and held-out family. The submitted split is already carrier-disjoint (App. B.3). We also removed information_disclosure from every training and calibration channel and retrained, with test rows and decoding unchanged:

Held-out familyPOST locator, full→held outPOST grounded quotePOST correct none
Qwen3-4B62/62→61/6262/62→61/6227/27→27/27
Qwen3-1.7B62/62→62/6260/62→62/6226/27→25/27

Moved, run through python, split across windows. We rewrote in the trace the 300 attack calls that R1 matches, keeping event IDs, labels, and the 406 no-target windows (Table 23). In the last row the copy happens in an earlier window, so the evaluated window never names the bundle:

Attack call located (300)R1R2Qwen3-4BQwen3-1.7B
python3 resources/x.py0245296271
python3 helpers/x.py00275239
copied in an earlier window, run from /tmp00300292

The rewrite leaves the 406 no-target windows unchanged, and correct none stays at 379/406 (4B seed 42) and 386/406 (1.7B). R1 falls to 0 on all three rewrites and R2 on the last two, while the learned locator keeps 275–300 of 300. We also renamed resources/ to helpers/ before retraining a 1.7B model (Table 23):

Path intervention, 406 positivesOriginal pathsresources/→helpers/
SkillVigil Qwen3-1.7B, retrained388/406387/406
R1351/4060/406

Second generator. SkillAttack leaves skills unmodified, so its skills are clean under Eq. 1. We therefore use Skill-Inject (App. G.1, Table 29), whose templates, scripts, and tasks share nothing with our construction. We scored its 245 bundles and recorded 88 Claude Code runs on its script attacks and clean counterparts. R3, the first call to a script shipped with the skill, is R1 adapted to this layout:

Skill stage, bag AUCAll bundles (212 / 33)Skills absent from our corpus (73 / 28)
Qwen3Guard-Gen-8B0.6320.591
SkillVigil Qwen3-1.7B0.7850.692
SkillVigil Qwen3-4B0.8590.810
Execution stage, 643 windowsR1R2R3SkillVigil Qwen3-4B
Attack windows located (56)003933
Benign bundled-script windows, none (178)1781780152
All no-target windows, none (587)587587409490
F1000.2860.355

F1 counts a located attack window as a true positive and any citation on a no-target window as a false positive. Three 4B seeds give AUC 0.810–0.859 and F1 0.355–0.412. A paired bootstrap over bundles puts the 4B model's skill-stage margin over Qwen3Guard-Gen-8B at [0.124, 0.338] overall and [0.110, 0.338] on the skills absent from our corpus. Skill-Inject templates and agent runs are all unseen in training. Of the 178 benign script calls, 147 run ooxml/scripts/ code that occurs nowhere in our training data. An attacker who optimizes against the guard itself is the case §6 states as open.

W1, scope. "Sec. 6 scopes the execution-stage results to corpus-specific auditing, and I agree with that framing. The abstract, introduction, and contribution list do not; they present the evidence results as general properties of evidence-supervised guardrails."

Response. Following your final paragraph, the abstract now reads: "At the execution stage, on attacks from one construction procedure, it names the implicated call on 0.998 of windows and quotes it on 0.995, against 0.865 for a rule matching the attack's invocation, and names it on 0.995 of a 200-window subset where the strongest few-shot frontier prompt reaches 0.640." It attributes the margins over general guards to in-domain training rather than evidence supervision alone. The introduction and the contribution list now state that the findings hold within one attack-construction procedure.

Reviewer_2BKZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces SkillVigil, a unified security protection framework based on evidence monitoring. It divides defense into two stages: one is to scan files before the loading of Skill to identify the malicious content implanted therein; the other is to analyze the tool invocation trajectory after the Agent performs its task, to locate the specific operation that triggers the attack, and provide corresponding evidence references and the original text of the malicious content, rather than merely outputting a judgment of security or insecurity. Experiments show that SkillVigil achieved a detection AUC of 0.891 on MalSkillBench, which is superior to the 0.714 of the best general security protection model; in the real Agent online intervention experiment, by combining pre-execution scanning and runtime tool invocation interception, the number of successful attacks was reduced from 7/8 to 1/8, and no normal tasks were mistakenly intercepted. The core contribution of this paper is to advance the traditional Agent security detection from simple risk classification to verifiable evidence location and attribution, and to further transform the detection capability into proactive defense mechanisms before execution and during runtime.

Critical strengths
  1. The paper addresses the practically relevant risk of persistent prompt injection through third-party skills.
  2. The experimental setup is systematic and comprehensive.
  3. The paper reveals a gap between identifying relevant text and accurately reporting its location, providing practical guidance for designing evidence-reporting interfaces.
Critical weaknesses
  1. The method primarily relies on multitask supervised fine-tuning with a shared LoRA adapter, offering relatively limited technical novelty.
  2. Simple invocation rules already achieve high localization scores, making it difficult to determine whether strong performance reflects reliable discrimination of malicious behavior.
  3. Execution-stage evaluation is limited to the internal corpus, without validation on unseen attack templates.
  4. Benign task completion rates and real-time inference latency are not evaluated.
Questions for authors
  1. Could you provide additional generalization experiments evaluating the model on unseen attack templates or execution traces from independent sources?
  2. If benign scripts with similar invocation patterns were included, and all methods were given access to script contents or side-effect records, how much of an advantage would SkillVigil retain over simple invocation rules?
  3. How would online deployment affect attack success rates, benign task completion rates, and per-call latency?
7 comments

Response to Reviewer 2BKZ (Part 1/3)

We sincerely thank the reviewer for the constructive review and for recognizing the practical relevance, the systematic evaluation, and the quote/offset finding. Changes in the revised PDF are marked in red, and table numbers refer to the revision. We quote each point and answer it in order.

W1. "The method primarily relies on multitask supervised fine-tuning with a shared LoRA adapter, offering relatively limited technical novelty."

Response. We appreciate this concern. The contribution is a formulation and a controlled study of evidence supervision, delivered through one adapter. (i) Two supervised targets that skill injection creates, the bundle before loading (Eq. 1) and the completed call (Eq. 2), each served by one stage-conditioned interface. Eq. 4 defines the input when the same execution-stage model scores a proposed call before it runs. (ii) Nested supervision under a fixed budget across five backbones. Verdict AUC moves by at most 0.011, while evidence accuracy and abstention vary by an order of magnitude and are invisible to verdict-only evaluation (Table 17):

Five backbones, execution stageArm A (verdict only)Arm B (+ citation)Arm C (+ attribution)
Correct none, citation channel0.000–0.8930.907–0.9890.861–0.989

(iii) A deployed two-stage guard whose online effect is measured. Part 3 reports that measurement and places these points among existing defenses.

W2. "Simple invocation rules already achieve high localization scores, making it difficult to determine whether strong performance reflects reliable discrimination of malicious behavior."

Q2. "If benign scripts with similar invocation patterns were included, and all methods were given access to script contents or side-effect records, how much of an advantage would SkillVigil retain over simple invocation rules?"

Response. Benign scripts with similar invocation, contents shown. Our own traces contain no such call, but agent traces from Skill-Inject (Schmotz et al., 2026, arXiv:2602.20156), a second generator, do (Table 29). In 178 no-target windows the agent runs a benign script shipped with the skill in the attacks' form (python3 /skills/<skill>/.../x.py). R3, the first call to a script shipped with the skill, is the invocation rule adapted to this layout:

Skill-Inject, 643 windows from 88 runsR3Qwen3-4B, bodies withheldQwen3-4B, bodies shown
Benign bundled-script windows, none (178)0152133
Attack windows located (56)393338
F10.2860.3550.362

The rule cites every benign script of the attack's form. The model leaves most of them uncited and keeps the higher F1 with or without the bodies. A benign call constructed to render exactly as the attack can only be told apart by its content or effects, the identifiability limit §6 states.

Side-effect records. Side-effect records can inform retrospective auditing, but they are unavailable when deciding whether a proposed call should execute. We therefore evaluate online intervention using only the information available at the pre-execution hook (Eq. 4, Part 3).

On our own corpus, model columns are the saved Arm C outputs (Table 3):

Correct outputsQwen3-4BQwen3-1.7BR1R2
Positives that R1 misses (55)5551046
No-target windows with a real benign helper call (38)34383811
All 812 windows (406 positive, 406 no-target)797791757683

R1 cites no helper window but misses all 55 implicated events outside its pattern. Widened to recover them (R2), the rule cites 27 of the 38 helper windows. The locator recovers them while citing 4/38 (4B) and 0/38 (1.7B), and it is the most accurate system on all 812 windows. The advantage on R1-missed positives also holds across all five backbones (App. E.4):

Arm C backboneAll positives (406)R1-missed positives (55)Non-script positives (11)
Qwen3-4B4055511
Qwen3-1.7B398519
Llama-3.2-3B4055410
Llama-3.2-1B4035511
Phi-4-mini400539
R135102

Of the 55 positives R1 misses, 48 are reads of the injected file, 4 are Glob events, and 3 are Bash calls of another form. Nine of the 11 non-script payloads are Markdown files, which no invocation rule over scripts can reach.

Response to Reviewer 2BKZ (Part 2/3)

W3. "Execution-stage evaluation is limited to the internal corpus, without validation on unseen attack templates."

Q1. "Could you provide additional generalization experiments evaluating the model on unseen attack templates or execution traces from independent sources?"

Response. Execution traces from an independent source. Skill-Inject (App. G.1, Table 29) shares no template, script, or task with our construction. We scored its 245 bundles (212 poisoned, 33 clean) and recorded 88 Claude Code agent runs on its script attacks and clean counterparts, with a DeepSeek-V4-Pro backend in our sandbox (51 poisoned runs, 37 clean):

Skill stage, bag AUCAll bundles (212 / 33)Skills absent from our corpus (73 / 28)
Qwen3Guard-Gen-8B0.6320.591
SkillVigil Qwen3-1.7B0.7850.692
SkillVigil Qwen3-4B0.8590.810
Execution stage, 643 windows from 88 runsR1R2R3SkillVigil Qwen3-4B
Attack windows located (56)003933
Benign bundled-script windows, none (178)1781780152
All no-target windows, none (587)587587409490
F1000.2860.355

Over the 587 no-target windows the 4B model makes 97 false citations against 178 for R3, and it has the highest F1. F1 counts any citation on a no-target window as a false positive. Three 4B seeds give AUC 0.810–0.859 and F1 0.355–0.412. A paired bootstrap over bundles puts the 4B model's skill-stage margin over Qwen3Guard-Gen-8B at [0.124, 0.338] overall and [0.110, 0.338] on the skills absent from our corpus. Skill-Inject templates and agent runs are all unseen in training, and 147 of the 178 benign script calls run ooxml/scripts/ code that occurs nowhere in our training data.

Held-out attack family. The submitted split is already carrier-disjoint (App. B.3). We removed information_disclosure from every training and calibration channel, with test rows and decoding unchanged:

Held-out familyPOST locator, full→held outPOST grounded quotePOST correct none
Qwen3-4B62/62→61/6262/62→61/6227/27→27/27
Qwen3-1.7B62/62→62/6260/62→62/6226/27→25/27

Unseen invocation forms. We rewrote in the trace, without retraining or re-executing, the 300 attack calls that R1 matches, with event IDs, labels, and the 406 no-target windows unchanged (Table 23). In the last row the copy happens in an earlier window, so the evaluated window never names the bundle:

Attack call located (300)R1R2Qwen3-4BQwen3-1.7B
python3 resources/x.py0245296271
python3 helpers/x.py00275239
copied in an earlier window, run from /tmp00300292

Correct none stays at 379/406 (4B seed 42) and 386/406 (1.7B). We also renamed resources/ to helpers/ before retraining a 1.7B model (Table 23):

Path intervention, 406 positivesOriginal pathsresources/→helpers/
SkillVigil Qwen3-1.7B, retrained388/406387/406
R1351/4060/406

Independent skill-stage data. On SkillSafetyBench (Table 11), file-level evidence is selective:

SkillSafetyBench, 231 target filesCorrect target filesFalse file citationsPrecisionRecall
SkillVigil Qwen3-1.7B2720.9310.117
SkillVigil Qwen3-4B460.4000.017

The 4B model cites no file in any of the 72 clean bags. Across these shifts, the held-out family, the rewritten invocations, and a second generator whose templates and runs were unseen in training, the learned locator keeps its advantage over the invocation rules. We thank the reviewer for this question, which led to the independent evaluation now in App. G.1.

Response to Reviewer 2BKZ (Part 3/3)

W4. "Benign task completion rates and real-time inference latency are not evaluated."

Q3. "How would online deployment affect attack success rates, benign task completion rates, and per-call latency?"

Response. Thank you for asking for measured prevention. PRE rejects a bundle before the agent runs. POST denies a proposed tool call at Claude Code's pre-execution hook. Its input (Eq. 4) is the task, the prior events, and the current tool name and arguments, with no tool result, later event, or closing narrative. We evaluated four existing tasks, all with the same file-write attack, two runs per condition, and the same DeepSeek-V4-Pro backend: 64 runs in total (Table 28). Task completion uses deterministic artifact checkers. Attack success is read from the runtime consequence record after any retries, independently of the guard's verdict.

Attack success and benign completion.

OutcomeNo guardPRE onlyPOST onlyPRE+POST
Successful attacks on poisoned tasks7/82/83/81/8
Benign tasks completed5/85/86/85/8
Benign runs blocked by the guard—0/80/80/8
Attacks succeeding, per task (out of 2)No guardPREPOSTPRE+POST
algorithmic-art2000
react-native-skills1020
web-development2000
webapp-testing2211

PRE+POST reduces observed attack success from 7/8 to 1/8, and benign completion is the same as without a guard. No benign run is blocked in any of the 24 guarded benign runs. Under PRE alone, only two of eight poisoned runs result in a successful attack. In the combined configuration, POST stops one webapp-testing attack that PRE misses, and PRE prevents both react-native-skills attacks that POST alone misses.

Per-call latency (App. F). The complete hook includes communication and guard processing and excludes agent and tool execution:

Guard operationBenign inputPoisoned inputTiming denominator
PRE, per bundle1.35 s2.56 s8 + 8 PRE-only runs
POST guard computation, per call (verdict, plus evidence when flagged)0.059 s0.508 s84 + 93 POST-only calls
POST complete hook, per call0.209 s1.855 ssame calls

The poisoned-call average includes evidence generation after a suspicious verdict. On the prefix-only input, the same 4B model localizes 406/406 positives with 360/381 correct none on event-bearing negatives.

W1 (continued). "offering relatively limited technical novelty."

Response. To help assess the contribution against what existing defenses already provide, we summarize the related work that §2 of the revision now covers:

Line of workExamples in §2What it returnsWhat our study adds
Pre-load skill scanners and auditorsMalSkillBench scanners, SkillGuard-Robust (Lv et al., 2026), Locate-and-Judge (Etteib et al., 2026), SkillSieve, SkillSecurer (Mecca et al., 2026), SkillLite (Ou et al., 2026)A label before loading, sometimes with findingsThe execution stage, and a same-row comparison: Sentry-static F1 0.411 against 0.807–0.844 for SkillVigil (Table 10)
Defenses at the tool callCaMeL (Debenedetti et al., 2025), Progent (Shi et al., 2025), Task Shield (Jia et al., 2024), MELON (Zhu et al., ICML 2025), VIGIL (Li et al., 2026)Allow or deny for a proposed call, under a policy or a task-alignment checkThey do not point to the injected content, and they complement our hook at the same point (Eq. 4)
Diagnostic guardrailsAgentDoG (Liu et al., 2026), TRUSS, Runtime Skill AuditStep-level diagnosis, or links from violations back to skill contentA controlled test of whether supervision makes such evidence checkable and abstention consistent
Localization of injected textPromptLocate (Jia et al., IEEE S&P 2026)Spans in input known to be contaminatedClean inputs and abstention. Character F1 0.282 against 0.145 on the same 1,302 documents, at 0.43 s against 89.9 s per document (Table 13)

Under a fixed budget and across five backbones, adding evidence channels moves verdict AUC by at most 0.011, while correct abstention ranges over 0.000–0.893 under verdict-only training and narrows to 0.861–0.989 once a none target is supervised. Within the same responses, the model quotes the injected line on 0.780 of documents but places it on only 0.116. These effects are invisible to the verdict-level evaluation that most of these systems report, and they determine whether a guard's output can be acted on.

We are grateful that this question pushed us to state the positioning explicitly, and we would be glad to discuss any of these comparisons further.

Reviewer_umPZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates skill injection, where attacker-controlled content reaches a coding agent through an installed third-party skill bundle. It defines separate labels for two stages: whether a bundle contains attacker-controlled text (the skill stage, Eq. 1), and whether a tool call in the resulting trace invoked that injected content, was permitted, and completed (the execution stage, Eq. 2, with Eq. 3 relating the two labels). The proposed method, SkillVigil, is a single LoRA adapter that uses one of six system prompts to select the evaluation stage and output channel. The channels are a verdict, a citation (a character interval in a document or an event identifier in a trace), and an attribution (a quote and attack class). The authors train three nested supervision arms (verdict; +citation; +attribution) under a fixed per-epoch budget across five 1.2B–4.0B backbones. The training corpus consists of 3,492 poisoned bundles built from 27 public carrier skills using the SkillJect procedure, paired with 4,165 real executions across four agent frameworks and six backend models. On MalSkillBench, where external evaluation covers skill-stage verdicts only, SkillVigil (Qwen3-4B) reaches 0.891 AUC compared to 0.714 for the best general-purpose guard. At the execution stage, its locator scores 0.980–0.998 versus 0.040/0.050 for prompted GPT-4o/Qwen3.8-27B. Across supervision arms, verdict AUC shifts by at most 0.011, while citation supervision yields more uniform abstention. Finally, on Qwen3-4B at the skill stage, the authors observe a large gap between quoting the injected line (0.780) and locating it by character interval (0.116), which they analyze using format interventions and probes.

Critical strengths

The authors actively rule out shortcut explanations for their results (e.g., a trace-length baseline falls to chance after length matching while their models stay at 0.944–0.958; Table 1) and drop a metric that turned out to reflect a shortcut, so I find the in-corpus results trustworthy.

Critical weaknesses
  • W1. The evidence results rely on a single attack-construction procedure and aren't tested against unseen or adaptive attacks, while the paper's own diagnostics suggest the model partly encodes that procedure. Although the train and test splits use different carriers (App. B.3, "Split hygiene"), they share injection templates and attack families, and a simple lookup on (normalized template, quote length) fit on the training rows predicts the family of held-out rows at 0.985 (App. B.3). In this corpus, clean bundles rarely reference executable helpers (App. A.1–A.2), script bodies are withheld from every window (App. C.5), and simple no-model rules already reach 0.865 (Bash on resources/*.sh) and 0.798 (any event naming resources/) on the execution-stage locator (Table 15). Indeed, the paper notes that "identically rendered benign and malicious calls cannot be distinguished from the window alone" (§6). Getting 0.998 on the 406 rows of Table 15 means the trained channel makes at most one error, so it locates at least 10 of the 11 non-script payloads (compared to 2 for the Bash rule). Still, these are all positive windows, and we don't see how the model behaves on matched benign helper calls, on held-out templates, or against an attacker who adapts to the guardrail ("evasion by one who does remains untested", §6). For a defense paper, this is the central question. While the authors narrow the execution-stage claim to "corpus-specific event auditing" in the body and appendix (§6; App. G), the abstract reports the 0.998 locator against 0.040/0.050 for prompted models without that caveat (Q1, Q2).
  • W2. The baseline comparisons do not yet establish that SkillVigil is a stronger skill guardrail. The headline result (0.891 vs. 0.714; abstract, Fig. 2) compares against six general-purpose guardrails, none of which were trained on skill bundles (§2; App. C.2). Much of that gap is plausibly due to in-domain training rather than the evidence-supervision design: a verdict-only adapter on Llama-3.2-3B already gets 0.808 (App. E.6), and Arm C training boosts Qwen3Guard-Gen-4B from 0.665 to 0.871 (Table 18). Although §5.1 states that the paper does not rest its claim on this ordering, the abstract still highlights it. Meanwhile, dedicated skill scanners are not run under the paper's evaluation protocol. Table 9 puts published MalSkillBench random-split numbers next to SkillVigil's numbers on a threshold-calibrated, length-matched subset (the caption notes the splits differ), and on those numbers two scanners actually report higher F1 (0.886 and 0.856 vs. 0.812). The remark in §5.1 that scanners "fall to 0.28–0.35 macro-F1 held out by source" comes from a different benchmark (Wang et al., 2026b), largely different systems, and, by the paper's own wording, a different metric—even though Table 9 labels both columns "F1". Because Table 9 is used to position SkillVigil, the paper needs either a same-protocol comparison with at least one public scanner or a revised framing of Table 9 (Q3).
  • W3. The operating point and training variance are not sufficiently characterized. No FPR is reported for SkillVigil (Table 9 shows "—"). Because the in-distribution skill stage has only 16 negatives (§5.1), the skill-stage claims rely on the external corpora. However, those external AUCs come from a single training run per backbone, meaning the intervals in Table 3 do not capture training variance. In App. D.1, a change to the training set moved Qwen3-1.7B on MalSkillBench from 0.685 to 0.806—a swing comparable to the 0.154 cross-backbone spread that the paper presents as a finding (Contributions; Table 3). Reporting FPR at a fixed high recall along with multi-seed runs for the external AUCs is needed to make the skill-stage results interpretable (Q3, Q4).
  • W4. Practical deployment value is limited at both stages. The strongest evidence results occur at the execution stage, which by design audits completed traces (including the agent's closing narrative); per-call, prefix-only execution is not evaluated (App. F; §6), making this stage useful for post-hoc remediation rather than prevention. At the pre-load skill stage, citation precision is at most 0.235 against a 0.100 null, and no cell exceeds 0.025 under the containment criterion (Tables 13, 19). While quote search recovers 0.675–0.856, App. E.10 does not claim robustness to the edge cases a deployed resolver would have to handle. An alternative would be to frame the paper primarily as an empirical study of evidence supervision rather than as a deployable guardrail.
Questions for authors

Please refer to the weaknesses.

11 comments

Response to Reviewer umPZ (Part 1/3)

We sincerely thank the reviewer for the thorough review and for finding the in-corpus results trustworthy after the shortcut controls. Changes in the revised PDF are marked in red, and table numbers refer to the revision. W1 contains several distinct points, so we quote and answer each in turn.

W1, rules and benign helpers. "simple no-model rules already reach 0.865 (Bash on resources/*.sh) and 0.798 (any event naming resources/) on the execution-stage locator (Table 15). […] Getting 0.998 on the 406 rows of Table 15 means the trained channel makes at most one error, so it locates at least 10 of the 11 non-script payloads (compared to 2 for the Bash rule). Still, these are all positive windows, and we don't see how the model behaves on matched benign helper calls"

Response. We apply both rules (Table 20, submitted Table 15) to the windows the model sees, now Table 3 in §5.1. R1 is the first Bash call whose command names a resources/*.sh file, and R2 the first event naming resources/. Model columns are the saved Arm C outputs:

Correct outputsQwen3-4BQwen3-1.7BR1R2
Positives that R1 misses (55)5551046
Non-script positives (11)119210
No-target windows with a real benign helper call (38)34383811
All 812 windows (406 positive, 406 no-target)797791757683

The 4B locator finds all 11 non-script payloads, as you inferred, and nine of them are Markdown files that no resources/*.sh rule can reach. The table adds the 406 no-target windows, so it scores citations on benign windows as well as localization. Because the 55-row slice is selected by R1's failure, we also give the unselected 812-window total. R1 cites no helper window but misses every implicated event outside its pattern (48 of the 55 are reads of the injected file). R2 recovers 46 of them but cites an event on 27 of the 38 benign-helper windows. The learned locator recovers 55 (4B) and 51 (1.7B) while citing 4/38 and 0/38. The locator is therefore not reducible to resources/*.sh matching. Benign helper calls of the attack's own form, from a second generator, follow below.

W1, abstract. "the abstract reports the 0.998 locator against 0.040/0.050 for prompted models without that caveat (Q1, Q2)."

Response. We have revised the abstract to clarify the scope of this comparison: "At the execution stage, on attacks from one construction procedure, it names the implicated call on 0.998 of windows and quotes it on 0.995, against 0.865 for a rule matching the attack's invocation, and names it on 0.995 of a 200-window subset where the strongest few-shot frontier prompt reaches 0.640."

W1, templates and families. "they share injection templates and attack families, and a simple lookup on (normalized template, quote length) fit on the training rows predicts the family of held-out rows at 0.985 (App. B.3)."

Response. The submitted split is carrier-disjoint (App. B.3). We removed information_disclosure from every training and calibration channel and trained both sizes again, with test rows and decoding unchanged:

Held-out familyPOST locator, full→held outPOST grounded quotePOST correct none
Qwen3-4B62/62→61/6262/62→61/6227/27→27/27
Qwen3-1.7B62/62→62/6260/62→62/6226/27→25/27

The 0.985 lookup classifies families at the skill stage. Templates that share nothing with ours are tested below with Skill-Inject.

W1, matched benign helpers and held-out templates. "we don't see how the model behaves on matched benign helper calls, on held-out templates"

Response. Skill-Inject (Schmotz et al., 2026, arXiv:2602.20156, App. G.1, Table 29) shares no template, script, or task with our construction. We scored its 245 bundles and recorded 88 Claude Code runs on its script attacks and clean counterparts. R3, the first call to a script shipped with the skill, is R1 adapted to this layout:

Skill stage, bag AUCAll bundles (212 / 33)Skills absent from our corpus (73 / 28)
Qwen3Guard-Gen-8B0.6320.591
SkillVigil Qwen3-1.7B0.7850.692
SkillVigil Qwen3-4B0.8590.810
Execution stage, 643 windows from 88 runsR1R2R3SkillVigil Qwen3-4B
Attack windows located (56)003933
Benign bundled-script windows, none (178)1781780152
All no-target windows, none (587)587587409490
F1000.2860.355

In 178 no-target windows the agent runs a benign script shipped with the skill, in the attacks' form. These are the matched benign helper calls, in natural traces. R3 cites all of them, and the 4B model leaves 152 uncited. Over all 587 no-target windows the 4B model makes 97 false citations against 178 for R3, which gives it the highest F1. Both orderings hold for all three 4B seeds (AUC 0.810–0.859, F1 0.355–0.412).

Response to Reviewer umPZ (Part 2/3)

W1, construction cues. "In this corpus, clean bundles rarely reference executable helpers (App. A.1–A.2), script bodies are withheld from every window (App. C.5) […] Indeed, the paper notes that "identically rendered benign and malicious calls cannot be distinguished from the window alone" (§6)."

Response. Clean bundles with helpers. Skill-Inject includes clean skills that invoke bundled scripts, yielding 178 benign-script windows. This allows us to test a setting in which bundled-script invocation is not exclusive to attacks. On this dataset, the 4B model achieves a skill-stage AUC of 0.859.

Script bodies. Skill-Inject's benign scripts run in the attacks' own form, so we also scored the 4B model with every invoked body shown (Table 29):

Skill-Inject, 643 windows from 88 runsR3Qwen3-4B, bodies withheldQwen3-4B, bodies shown
Benign bundled-script windows, none (178)0152133
Attack windows located (56)393338
F10.2860.3550.362

With bodies shown the model locates 38 of the 56 attacks, against 39 for R3, while leaving 133 of the 178 benign script calls uncited, against none for R3. A call rendered identically to the attack can only be told apart by its content or effects, the limit §6 states.

W1, adaptive attacks. "against an attacker who adapts to the guardrail ("evasion by one who does remains untested", §6)."

Response. We rewrote in the trace the 300 attack calls that R1 matches, without retraining or re-executing, keeping event IDs, labels, and the 406 no-target windows (Table 23):

Attack call located (300)R1R2Qwen3-4BQwen3-1.7B
python3 resources/x.py0245296271
python3 helpers/x.py00275239
copied in an earlier window, run from /tmp00300292

Correct none stays at 379/406 for the seed-42 4B checkpoint. Renaming resources/ to helpers/ before retraining moves a 1.7B model from 388/406 to 387/406, and R1 from 351/406 to 0/406. An attacker who optimizes against the guard itself is stated in §6 as the case left open.

Prompted baselines. For the 0.040/0.050 comparison, we added few-shot, prior-stating, and forced-choice prompts and rescored SkillVigil on the same 200 positive IDs (Table 21, App. E.5):

PromptGPT-4o event / quoteQwen3.8-27B event / quoteSkillVigil Qwen3-4B event / quote
Few-shot0.500 / 0.5000.640 / 0.6350.995 / 0.995
Prior stated0.410 / 0.3300.615 / 0.5900.995 / 0.995
Forced choice, no none0.475 / 0.1200.465 / 0.4550.995 / 0.995

SkillVigil is not prompted, so its row repeats its standard output. With 200 negatives added, joint event-or-none accuracy under few-shot prompting is 0.720 / 0.778 / 0.985 for GPT-4o / Qwen / SkillVigil.

W2, in-domain training. "Much of that gap is plausibly due to in-domain training rather than the evidence-supervision design: a verdict-only adapter on Llama-3.2-3B already gets 0.808 (App. E.6), and Arm C training boosts Qwen3Guard-Gen-4B from 0.665 to 0.871 (Table 18). Although §5.1 states that the paper does not rest its claim on this ordering, the abstract still highlights it."

Response. We agree, and the abstract now states that its margins "reflect in-domain training rather than evidence supervision alone". Evidence supervision is studied through the nested arms, and guardrail performance through identical-row scanner runs. The margin over general guards also holds on a generator absent from our training: on Skill-Inject, 0.859 against 0.632 for Qwen3Guard-Gen-8B, with a paired bootstrap margin of 0.124 to 0.338 (Table 29).

W2, scanners and Table 9. "dedicated skill scanners are not run under the paper's evaluation protocol. […] The remark in §5.1 that scanners "fall to 0.28–0.35 macro-F1 held out by source" comes from a different benchmark […] the paper needs either a same-protocol comparison with at least one public scanner or a revised framing of Table 9 (Q3)."

Response. We did both. We ran the official static component of Sentry (Sentry-static) on identical rows, with the SkillVigil threshold frozen on calibration bags for 80% recall (Table 10, Figure 6):

Same 3,034 reporting packagesRecall↑FPR↓F1↑
Sentry-static0.2860.1070.411
SkillVigil Qwen3-4B, seed 420.7820.1550.807
SkillVigil Qwen3-4B, seed 10.7860.1120.828
SkillVigil Qwen3-4B, seed 20.7840.0750.844

At about the scanner's false-positive rate, SkillVigil recovers 2.7 times as many attacks. On all 4,332 matched packages, Sentry-static reaches F1 0.395. Table 10 now lists the published Sentry-full and AI-Infra-Guard figures (0.886 and 0.856) in a separate block with no SkillVigil row, because they come from a random split whose membership the official scorer does not expose. The macro-F1 remark is now cited in §2 on its own, apart from our results.

Response to Reviewer umPZ (Part 3/3)

W3. "No FPR is reported for SkillVigil (Table 9 shows "—"). […] those external AUCs come from a single training run per backbone, meaning the intervals in Table 3 do not capture training variance. […] Reporting FPR at a fixed high recall along with multi-seed runs for the external AUCs is needed to make the skill-stage results interpretable (Q3, Q4)."

Response. We trained the 4B Arm C model at three seeds. AUC is 0.892 ± 0.016 (sample SD) on matched MalSkillBench and 0.735 ± 0.036 on SkillSafetyBench. The Table 4 caption (submitted Table 3) now states that its brackets are bootstrap intervals over bags from one training run. We freeze each threshold on the 30% calibration bags and measure the disjoint reporting bags (Table 19, mean ± SD over seeds):

Frozen recall targetMalSkillBench recall / FPRSkillSafetyBench recall / FPR
0.800.784 ± 0.002 / 0.114 ± 0.0400.711 ± 0.035 / 0.379 ± 0.023
0.900.894 ± 0.007 / 0.362 ± 0.0780.796 ± 0.022 / 0.431 ± 0.034
0.950.953 ± 0.005 / 0.616 ± 0.0780.922 ± 0.048 / 0.608 ± 0.068

The 0.685→0.806 change in App. D.1 reflects a change of training data, not seed noise, and we report the seed SD and the 0.154 cross-backbone range as separate quantities.

W4, prevention. "The strongest evidence results occur at the execution stage, which by design audits completed traces (including the agent's closing narrative); per-call, prefix-only execution is not evaluated (App. F; §6), making this stage useful for post-hoc remediation rather than prevention."

Response. We now measure prevention directly. PRE gives a load/no-load verdict before the agent runs, and POST can deny a proposed tool call before it executes. For a proposed call E~t\widetilde{E}_t, the hook supplies the window (Eq. 4)

Wtprecall=(Et−k+1,…,Et−1,E~t),W_t^{\mathrm{precall}} = (E_{t-k+1}, \ldots, E_{t-1}, \widetilde{E}_t),

where E~t\widetilde{E}_t holds the tool name and arguments, with no tool result, completion status, later event, or closing narrative. On this prefix-only input the 4B model localizes 406/406 positives with 360/381 correct none. On 1,042 executions, removing only the closing narrative keeps AUC within 0.005 (4B 0.970→0.965, 1.7B 0.965→0.969). Four tasks, one file-write attack, two runs per condition, DeepSeek-V4-Pro backend (Table 28):

OutcomeNo guardPRE onlyPOST onlyPRE+POST
Successful attacks7/82/83/81/8
Benign tasks completed5/85/86/85/8
Benign runs blocked—0/80/80/8
Attacks succeeding, per task (out of 2)No guardPREPOSTPRE+POST
algorithmic-art2000
react-native-skills1020
web-development2000
webapp-testing2211

PRE stops both react-native-skills attacks before the agent starts. Attack success is read from the executed-consequence record after any retries, and task completion from deterministic artifact checkers. None of the 24 guarded benign runs is blocked. Mean overhead (App. F):

Guard operationBenign inputPoisoned inputTiming denominator
PRE, per bundle1.35 s2.56 s8 + 8 PRE-only runs
POST guard computation, per call (verdict, plus evidence when flagged)0.059 s0.508 s84 + 93 POST-only calls
POST complete hook, per call0.209 s1.855 ssame calls

W4, skill-stage evidence. "At the pre-load skill stage, citation precision is at most 0.235 against a 0.100 null, and no cell exceeds 0.025 under the containment criterion (Tables 13, 19). While quote search recovers 0.675–0.856, App. E.10 does not claim robustness to the edge cases a deployed resolver would have to handle. An alternative would be to frame the paper primarily as an empirical study of evidence supervision rather than as a deployable guardrail."

Response. At the skill stage the verdict is the blocking signal and the citation is an audit output. App. E.13 (submitted App. E.10) now specifies the cases a resolver must handle: multiple occurrences, missing matches, and incomplete quotes. §5.3 reports the stricter criteria on the 1,302 documents of the PromptLocate comparison (Table 13). The main-checkpoint cells you cite remain in Table 27:

Skill-stage character citation, precisionMidpointIoU ≥ 0.5Containment
PromptLocate, 1,302 documents0.1010.0290.047
SkillVigil Qwen3-4B seed 42, same documents0.1690.0820.054
Skill stage, same responses, five backbonesGenerated character spanModel's quote resolved in the window
Localization0.019–0.1160.675–0.856

We agree with your suggested reading: the paper's central measurement is what evidence supervision changes, and the guard is the instrument of that study. We hope these results address your concerns, and we would be glad to provide further details.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of the Revision for the Program Chairs, Area Chairs, and Reviewers

Summary of the Revision for the Program Chairs, Area Chairs, and Reviewers

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We are sincerely grateful for the time and care the reviewers devoted to our submission. Their questions identified exactly the controls needed to show what SkillVigil adds beyond the conventions of its training corpus, and the paper is substantially stronger for them. All changes in the revised PDF are marked in red, and the main text stays within 10 pages.

What the paper does. SkillVigil is a single stage-conditioned guard for agent skills. It screens a skill bundle before loading and audits the resulting execution, returning a verdict, a citation to the responsible content, and an explicit abstention when there is none. The paper measures what this evidence supervision changes beyond the verdict: across five backbones, verdict AUC moves by at most 0.011, while evidence accuracy and abstention change by an order of magnitude.

What the reviewers asked for, and what we added.

Request (reviewers)WhereResult
Compare with invocation rules, including benign helper calls (ak23, umPZ, 2BKZ)§5.1, Table 3On all 812 windows, 797 correct for the 4B model against 757 for the best rule. All 55 attacks the rule misses are located, and 4 of 38 benign helper calls are cited, against 27 for the widened rule
Test unseen attacks and an independent generator (ak23, umPZ, 2BKZ)Tables 23 and 29, App. G.1Held-out family 61/62. For helpers/ and /tmp rewrites, both rules score 0/300, and the 4B model scores 275/300 and 300/300. Skill-Inject (Schmotz et al., 2026): AUC 0.859 against 0.632, execution-stage F1 0.355 against 0.286, and 152 of 178 benign scripts left uncited
A same-protocol scanner baseline, false-positive rates, and seed variance (umPZ)Tables 10 and 19, Figure 6On identical rows, with the threshold frozen for 80% recall, recall 0.784 at FPR 0.114 ± 0.040, against 0.286 at 0.107 for Sentry-static. AUC 0.892 ± 0.016 over three seeds
Stronger prompted and localization baselines (ak23)Tables 3, 13, and 21Best few-shot frontier prompt 0.640 against 0.995 on the same IDs. PromptLocate: character F1 0.145 against 0.282, at 89.9 s against 0.43 s per document
External evidence accuracy (ak23)Table 11On SkillSafetyBench, the 1.7B model cites 27 of 231 target files with 2 false citations (precision 0.931, recall 0.117)
Online prevention, benign completion, and latency (umPZ, 2BKZ)§3 (Eq. 4), §6, Table 28, App. FSuccessful attacks fall from 7/8 to 1/8 with both stages. No benign run is blocked, benign completion stays at 5/8, and the hook adds 0.209 s per benign tool call
Probe interpretation, span metrics, and presentation (ak23)§5.3, App. D.4, Tables 22 and 27The probe claim is limited to payload membership, stricter span criteria are reported, and a claims table is added

Changes to the text. The abstract and contributions now state the scope of the execution-stage result, report the rule and few-shot baselines, and attribute the margin over general guards to in-domain training. §2 adds the related work the reviewers listed (PromptLocate, Locate-and-Judge, SkillGuard-Robust, VIGIL, HINTBench, AgentDoG, defenses at the tool call, and concurrent skill-auditing systems) and corrects our description of SkillAttack. §3 defines the proposed-call decision used online (Eq. 4). §6 states the scope of the results, including an attacker who optimizes against the guard.

Each comment quotes relevant excerpts and addresses the weaknesses and questions in review order:

ReviewerCommentsPoints, in review order
ak235W1(a)+Q1+Q2, W1(b)+Q3, W1(c)+Q5, W1(d)+Q4, W1 scope, W2(a), W2(b)+Q6, W2(c), W3(a)–(c), Q7, Q8
umPZ3W1 (rules, abstract, templates, benign helpers, construction cues, adaptive attacks, prompted baselines), W2, W3, W4
2BKZ3W1, W2+Q2, W3+Q1, W4+Q3, W1 continued

We thank the reviewers once more for their constructive and generous engagement. Our detailed replies follow in each thread, and we would be glad to answer any further question during the discussion.

Respectfully, The Authors

AuthorsOpenReview ↗

Update to the Summary of the Revision: Results Added During the Discussion

Update to the Summary of the Revision: Results Added During the Discussion

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We are sincerely grateful to the reviewers for their time and careful questions throughout the discussion, and to the Area Chairs for coordinating it. As the discussion closes, we respectfully update our earlier Summary of the Revision with the results added after it. All are in the revised PDF, marked in red. Tables 1–29 keep their numbers, the new tables are numbered 30–32 at the end of the appendix, and the main text stays within 10 pages.

Discussion so far. Reviewer ak23 re-evaluated the paper and raised the rating. Reviewer umPZ wrote that the revision resolves W2 and W3 and asked two follow-up questions on W1, which the last two rows below answer. Reviewer 2BKZ's review noted that online prevention, benign task completion, and per-call latency were not evaluated. The revision now reports all three (Table 28, App. F): successful attacks fall from 7/8 to 1/8, no benign run is blocked, and the guard adds 0.209 s per benign tool call. Our comments under each review end with a brief summary of our responses in review order.

Results added during the discussion.

Request (reviewer)WhereResult
Training variance across backbones (umPZ W3)Table 30, App. E.3Five backbones at three seeds each. On MalSkillBench the range of backbone means is 0.150, against a largest seed SD of 0.026. At the threshold frozen for 80% recall, the 4B model's FPR is 0.114 ± 0.040
More scanners on identical rows (umPZ W2)Table 10Five public scanners, run with their official runners on the same 3,034 packages. SkillVigil has the highest recall and F1 at all three seeds (0.782–0.786 and 0.807–0.844). The lower-FPR scanners recover at most 0.339 of the attacks
Filename and invocation form removed at once (umPZ W1 follow-up)Table 31, App. G.1Each injected script gets a random name and runs with python3 from helpers/ or /tmp. All three invocation rules locate 0 of 300 attacks, and the 4B model locates 283–299 over three seeds
Skill-Inject split by script family and skill overlap (umPZ W1 follow-up)Table 32, App. G.1Benign ooxml/scripts/ calls left uncited: 136–141 of 147. With the preset verdict gate of the online guard, 0.013–0.026 false citations per tool call in clean runs, at overall precision 0.492–0.556

A correction to Table 10. MalSkillBench reports its own scanner figures on the full release, not on a split. Table 10 now labels that block accordingly and adds the published false-positive rates. On the full release, three LLM-assisted scanners report a higher F1 than SkillVigil. Each queries an API model for every skill, whereas SkillVigil is a local 4B adapter that also audits the execution stage.

Limits stated in the revision. In our corpus, the injected script is the first script run in the window. A rule that takes the first script run also locates 299 of 300 attacks, so the corpus cannot separate the locator from that cue (App. G.1). On the unseen Skill-Inject skills, which run almost no benign scripts, the bundled-script rule has the higher F1 (Table 32). On SkillSafetyBench, the cross-backbone range does not exceed seed variance, so we claim no ordering there (Table 30).

We thank the reviewers once more for their constructive and generous engagement, which has made the paper clearer and more careful. We are grateful to the Area Chairs for their time and consideration, and we would be glad to answer any further question.

Respectfully, The Authors

AuthorsOpenReview ↗

Summary for the AC Discussion: Findings, Revision, and Status of Every Concern

Dear Program Chairs, Senior Area Chairs, and Area Chairs,

We sincerely thank you for coordinating the discussion, and we are deeply grateful to all three reviewers for their careful and constructive reviews, which made the paper substantially stronger. Two of the three reviewers raised their ratings. All numbers are in the revised PDF or in our replies.

Bottom line. Agent skills are third-party bundles that run with an agent's privileges, and existing defenses return only a label before a bundle loads. SkillVigil is one local guard that, before a bundle loads and before a proposed tool call executes, also returns which tool call carried the attack and which bundle text caused it. Across five backbones and three supervision conditions, evidence supervision changes these outputs while moving verdict AUC by at most 0.011. Online, both stages together reduce successful attacks from 7/8 to 1/8, with no benign run blocked.

Core findings.

FindingEvidence
Verdict accuracy does not separate guards, but evidence doesFifteen models (five backbones, three supervision conditions) span 0.958 to 0.978 verdict AUC (Table 2). Correct abstention ranges from 0.000 to 0.893 under verdict-only training and narrows to 0.861 to 0.989 with an explicit none target (Table 17)
The output format decides whether evidence is usableIn the same responses, the model quotes the injected line on 0.780 of skill documents but places its character span on 0.116 (Table 1)
The agent is not an auditable defenseOf 2,360 poisoned but uncompromised executions, 4.8% end in an explicit refusal and 80.7% in silent non-compliance (§3, Table 6)

We were encouraged that Reviewer 2BKZ found the second finding to provide "practical guidance for designing evidence-reporting interfaces."

What the revision adds.

Request (reviewers)Result
Invocation rules and benign helpers (all three)On 812 windows, 797 correct against 757 for the best rule (Table 3). Moving attack calls to other paths drops the three rules to 0/300, while the 4B model keeps 275 to 300 (Tables 23, 31). With same-form benign scripts and every body shown to every method, a model trained with such calls reaches F1 0.822, against 0.595 for the invocation rule and 0.621 for a content rule (reply to 2BKZ)
Unseen attacks and a second generator (all three)Held-out family 61/62. On Skill-Inject, skill-stage AUC 0.859 against 0.632 for Qwen3Guard-Gen-8B (0.810 against 0.591 on unseen skills), execution-stage F1 0.355 to 0.448 against 0.286 (Table 29)
Same-protocol baselines (ak23, umPZ)Five public scanners on the same 3,034 packages: SkillVigil has the highest recall (0.782 to 0.786) and F1 (0.807 to 0.844) (Table 10). Best few-shot frontier prompt 0.640 against 0.995 (Table 21). PromptLocate span F1 0.145 against 0.282 (Table 13)
FPR and seed variance (umPZ)Recall 0.784 at FPR 0.114 ± 0.040, against 0.286 at 0.107 for Sentry-static (Table 19, Figure 6). Across five backbones at three seeds, the range of backbone means is 0.150 against a largest seed SD of 0.026 (Table 30)
Online prevention and cost (umPZ, 2BKZ)Successful attacks 7/8 to 1/8, no benign run blocked, benign completion 5/8 with and without the guard, 0.209 s per benign tool call (Table 28, App. F)
External evidence, probe, related work (ak23)SkillSafetyBench file citations: 27 of 231 with 2 false for the 1.7B model (Table 11). Probe claim limited to payload membership (§5.3). §2 adds the listed work

Status of each review.

ReviewerStatus
ak23All points answered with new controls. Rating raised
umPZW2 and W3 resolved in the reviewer's words. Two W1 follow-ups answered (Tables 31, 32). Rating raised
2BKZEvery point answered with measured results, including Q2 with script contents and side-effect records. No reply yet

The concern raised most often. All three reviewers asked whether the locator relies on how our attacks are invoked. In our corpus the injected script is also the first script run in its window, so the corpus alone cannot separate the locator from that cue (App. G.1). The same-form benign result goes beyond it: when the first script run is benign, the trained model leaves 279 of 406 such calls uncited. The first two findings compare output channels and training conditions on the same inputs, so the attack construction cannot account for them. Table 10 now also labels MalSkillBench's published scanner figures as full-release results.

Contribution. Training is standard LoRA fine-tuning, so the comparisons isolate supervision. The contribution is the problem definition (separate labels for the skill and the execution, and a pre-call decision), nested controls measuring what evidence supervision changes beyond the verdict, and the finding that copyable references are reliable where counted offsets are not.

We thank you once more for your time and consideration, and we would be glad to answer any further question.

Respectfully, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.