- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MaliciousSkillBench, a malicious-detection benchmark for Agent Skills. Starting from 13 public sources, the authors apply normalization, deduplication, structural-similarity grouping, and removal of label-conflicting items, obtaining 9,740 Skills (7,505 malicious, 2,235 benign), and map part of the malicious samples to 11 attack categories. Using only the SKILL.md text as input, the authors evaluate TF-IDF models, a fine-tuned ModernBERT, a zero-shot LLM judge, and three static scanners. The core findings are: ModernBERT's Macro-F1 is 0.964 under the random split and drops to 0.734 under the source-disjoint split, a gap of 0.230 that persists after reweighting and balanced retraining; supervised models have a false-positive rate of about 48.8% on unseen benign sources, while the zero-shot LLM has a false-positive rate of only 0.7% but a malicious recall of only 41.6%. Based on these results, the authors recommend reporting ranking quality, fixed-threshold performance, and the benign false-positive rate separately.
Critical strengths
-
Careful and transparent data consolidation. The authors point out that naively concatenating multiple datasets causes double counting and train/test leakage, and provide an actionable pipeline: 8,414 raw malicious records are deduplicated into 7,539 normalized identities, 34 cross-label conflicting items are removed, metadata such as source and label strength is retained throughout, and uncertain samples are placed in an Auxiliary set rather than forcibly labeled benign. The appendix also discloses the data recovery process, licensing, and release restrictions. For a benchmark paper, this traceability is a substantive contribution.
-
A fairly careful evaluation protocol with candid discussion of limitations. The authors report three splits (random, structural-family-disjoint, and source-disjoint) and run several diagnostics such as reweighting, balanced retraining, and leave-one-source-out, and in many places explicitly state the scope of their conclusions. Their recommendation to report AUROC, fixed-threshold metrics, and the benign false-positive rate separately is a useful reference for evaluating security detectors.
Critical weaknesses
-
The source-disjoint test set is imbalanced across sources. Appendix Table 26 shows that the source-disjoint test set contains 1,384 samples from only 3 sources: SRC009 has 728 samples, all malicious; SRC011 has 476 samples but only 21 are malicious; and only two sources contain benign samples. As a result, the Macro-F1 and benign FPR on this split are essentially determined by two benign sources, and malicious recall is essentially determined by a single source. This suggests that the test split is considerably imbalanced.
-
The detector input does not match where the label evidence lies, making it hard to distinguish detector failure from an undecidable task. The paper acknowledges that some malicious labels depend on package files or runtime evidence, yet all detectors read only the primary instruction document. Appendix M shows that on the source-disjoint test set, the agreement between the LLM review and the malicious labels is only 18.19%, and only 12/123 (9.76%) for SRC009; Section 6.3 also notes that among SRC009's 566 analyzable samples, 423 place the payload in additional resources. This means that many malicious samples contain no visible malicious content in SKILL.md. Therefore, whether the zero-shot LLM's 41.6% recall and the cross-source drop of supervised models reflect detector deficiencies or insufficient input information is not distinguished in the paper.
-
Label-quality and deduplication validation is weak, baselines are few, and novelty is limited. Labels are entirely inherited from the sources, with no independent manual spot-check to estimate label noise, and the Cohen's κ between the two initial reviewers in Appendix M's LLM review is only 0.479. Structural families are applied only to the malicious side, with no structural grouping on the benign side, and the 72 sample pairs used to validate the structural families only verify similarity within a family, without assessing whether near-duplicates still exist across different families. On the baseline side, there is only a single-prompt, single-model zero-shot LLM; there is no stronger fine-tuned LLM baseline, and no package-level baseline that looks at both SKILL.md and scripts. Finally, that distribution shift degrades performance and that multiple metrics should be reported separately are already common understanding in security machine learning; this paper mainly provides a concrete instance in this domain, with limited methodological or theoretical novelty.
Questions for authors
See the Weaknesses section above.
Individual Response to Reviewer uQuu
We thank Reviewer uQuu for the positive and constructive review. We answer W1–W3 below; Tables R1–R4 and R6 are in the general response (Comments 1A–1H).
W1: Imbalanced Source-Disjoint test set
We agree that, on the frozen test set, recall is driven mainly by SRC009 and benign FPR by SRC011/SRC012. Table R3 therefore holds out each of six sources in turn and reports recall, FPR and AUROC, which do not depend on the class ratio, together with Macro-F1 standardized to a 1:1 ratio. Supervised models degrade on every held-out source (1:1 Macro-F1 with the base training set: .307–.747 for TF-IDF and .318–.818 for ModernBERT, against .882–.964 on Random).
W2: Detector input vs. where the label evidence lies
We now report every held-out result separately for the 297 malicious items whose evidence is inside SKILL.md and the 542 whose payload is in a companion file (Tables R2, R4), which separates detector failure from missing input. The LLM judge's low recall comes almost entirely from the second group, where it answers INSUFFICIENT_CONTEXT; counting those answers as escalations raises its Source-Disjoint recall from 41.6% to 85.5% (Macro-F1 .636 to .898). The supervised models' high recall on the same group is mostly driven by source and carrier cues rather than by the payload (Comment 1B); a retrained ModernBERT flags 77.1% of unattacked carriers. The same missing input explains the low agreement in Appendix M that you cite: that review also read SKILL.md alone, and on the 123 sampled SRC009 items it answered benign 61 times and insufficient context 49 times. In our package-level audit, which reads every companion file, 26 of the 27 confirmed external-file items place the harm in a companion file that SKILL.md delegates to.
W3: Benign deduplication, near-duplicates, label quality, baselines and novelty