- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper detects malicious skills by comparing a skill’s declared capabilities (from SKILL.md) against its actual behavior. It builds a progressive-disclosure graph rooted at SKILL.md, reconstructs security-sensitive information flows across files and modalities, and scores each flow by a declaration-compatibility gap plus a covert-intent signal, aggregated via soft-OR. On SkillTrustBench (5,520 skills) it reaches F1 0.946 and AP 0.984.
Critical strengths
- Sharp, well-motivated reframing. Moving the decision from “which capabilities a skill has” to “which dangerous flows are undeclared” is a genuinely useful shift. It directly explains why benign power-tools (which legitimately read env vars and make network calls) are over-flagged by capability scanners, and the taint-vs-co-occurrence distinction (Fig. 5) is clearly argued.
- Strong, auditable design. Tracing each warning to a concrete source, sink, and declaration makes the output inspectable for human reviewers — a real advantage for pre-deployment vetting. The conservative, rule-based compatibility coefficients (rather than learned similarity) and verbatim-excerpt requirement for declaration extraction are sensible defenses against gaming.
- Thorough, honest evaluation. Five paradigm baselines, a six-way ablation, a τ/w_c sensitivity sweep, six evasion families, and a leave-one-attack-type-out protocol. The LOATO result (0.90 macro-recall held-out vs 0.934 overall) is the paper’s best evidence that detection rests on a structural signal rather than threshold overfitting. The w/o-LLM ablation (still F1 0.823) cleanly shows the gain isn’t just the LLM.
Critical weaknesses
- Single-benchmark, single-vendor evaluation. All headline numbers come from one benchmark (SkillTrustBench), and the LLM backend is a single model (Qwen-plus). There’s no cross-benchmark check against MalSkillBench or MASB, which the other submissions use, so generalization beyond this one dataset is unverified.
- The evasion test uses a synthetic corpus, not the benchmark. Robustness (RQ4) is measured on a self-constructed adversarial set with recall anchored at 92/94% — a different and seemingly easier operating point than the 0.934 main result. Synonym rewriting still costs 16%/8%, and adaptive attackers who co-design the SKILL.md to look consistent aren’t tested at all.
- Several hand-tuned design choices. The severity weights, class-specific caps (Table 6), b_x=0.9 exfiltration bump, and w_c=0.85 are fixed by hand. The sensitivity sweep covers only τ and w_c, not the severity/compatibility tables, so robustness to the many other magic numbers is unestablished.
- Analyzer-injection risk is acknowledged but not evaluated. Since SKILL.md is fully attacker-controlled and feeds an LLM, prompt injection against the analyzer is a live threat. The paper offers verbatim-evidence as mitigation but provides no experiment showing it actually resists a crafted injection.
Questions for authors
see weekness
4 comments
We have submitted our revised manuscript and sincerely look forward to your comments. We greatly appreciate your time and constructive feedback. Should you have any questions regarding our revisions, please feel free to contact us,we would be more than happy to discuss them with you and make further revisions as needed.
Dear Reviewer,
We hope this message finds you well. As the discussion period draws to a close, we would like to check whether our responses have adequately addressed your concerns. We added MalSkillBench as a second benchmark, evaluated without retuning, and DeepSeek-V4-Flash as a second LLM backend, on MalSkillBench our method keeps the highest precision among all systems. We replaced the synthetic evasion study with adaptive attacks on SkillTrustBench that include benign skills and white-box attackers, and we directly evaluated injection against the analyzer, reporting the hidden-instruction attack as a limitation. Finally, perturbing all hand-set constants keeps F1 between 0.940 and 0.948.
If any questions remain, or if further clarification would be helpful, we would be glad to hear your thoughts before the discussion ends.
Thank you for your careful reading and constructive feedback.
Thanks for the rebuttal. I update the score to 3. However, I still think the evaluation of this paper is weak.
Response to Reviewer 3
Thank you for the thorough review.
Q1: One benchmark, one LLM. We added MalSkillBench and evaluated our method on it without retuning after removing two sources of label leakage. We also added DeepSeek-V4-Flash as a second LLM backend. The results are reported in Table 1 of the revised paper. Compared with SkillTrustBench, the performance on MalSkillBench decreases primarily in recall, while precision remains the highest among all evaluated systems.
Complete detectors vs. our method on SkillTrustBench and MalSkillBench . MethodSkillTrustBenchSkillSieveCisco ScannerOurs (Qwen-Plus)Ours (DeepSeek)MalSkillBenchSkillSieveCisco ScannerOurs (Qwen-Plus)Ours (DeepSeek)P0.8640.8220.9570.9780.6500.7630.8540.880R0.5770.9520.9340.9800.3560.9140.6920.719F10.6920.8820.9460.9790.4600.8320.7650.792AP0.8550.9140.9840.9830.5840.8660.7530.773 Revision: Sec. 4.1, Table 1, and Sec. 4.2 (results), App. J (MalSkillBench and label-leakage removal), App. I (LLM backends), Sec. 5 (limitation).
Q2: Synthetic evasion set and adaptive attackers. We agree that the original evasion set was limited because it was synthetic and contained no benign skills. The new attack experiment addresses both limitations by using 100 malicious and 100 benign skills from SkillTrustBench and evaluating attackers that rewrite SKILL.md, including white-box attackers with knowledge of our static analysis and the output schema of our extractor. Declaring malicious behavior does not evade detection. Instead, recall increases from 0.84 to 0.96 when all malicious capabilities are declared and to 0.88 under the white-box setting. Among the three injections targeting the analyzer, only the hidden instruction reduces recall, from 0.84 to 0.80. Combining this attack with white-box declarations causes no additional loss.
Adaptive attacks on 100 malicious SkillTrustBench skills AttackUnmodifiedDeclare all capabilitiesDeclare detected flows (white-box)Authority claimForged analysis result (white-box)Hidden instruction to the analyzerDeclared flows + hidden instructionRecall0.840.960.880.850.900.800.84ΔR–+0.12+0.04+0.01+0.06−0.040.00Flipped–016084 Revision: Sec. 4.5, paragraph “Adaptive attacks” and Table 3; App. M (Table 12). The synthetic evasion results (Table 13) are moved to App. N and marked as synthetic in App. C and Sec. 5.
Q3: Hand-set constants. We perturbed all constants in Table 6, namely the severity weights wκf and the declaration-compatibility constants dx(f), as well as the contradiction term bx in Eq. 2. Specifically, we varied each severity weight by ±30% or set all severity weights equal, varied each compatibility constant by ±50%, and varied bx over [0,1]. F1 remains between 0.940 and 0.948, while AP remains between 0.973 and 0.986. This stability arises from two properties of the detector: the constants only scale the contributions of recovered flows, which occur in 18% of the evaluation set, and the bimodal score distribution keeps most skills far from τ. Consequently, no perturbation changes F1 by more than 0.006.
Revision: Sec. 4.4 (last part); App. K (Table 11 and the accompanying note).
Q4: Analyzer injection. We now evaluate three injection attacks against the analyzer: a claim that the skill passed a security review, a forged analysis result that follows the output schema of our extractor (white-box), and an HTML comment instructing AI reviewers to classify the skill as benign. As shown in the adaptive-attack table of our previous comment (Q2), neither the claim nor the forged result reduces recall. The hidden instruction is the only effective attack, it lowers the LLM covert score and reduces recall from 0.84 to 0.80, and combining it with white-box declarations causes no further loss. Requiring a verbatim excerpt for each declared capability removes forged declarations but does not counter this attack. We report it as a limitation and leave hardening the covert-intent judgment to future work.
Revision: Sec. 4.5 and Table 3, App. M (Table 12), App. C .