acceptodds
Under review as a conference paper at ICLR 2027

MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection

Abstract

Agent Skills extend LLM agents with reusable instruction packages that can carry scripts, resources, and service configuration, turning the Skill distribution channel into a direct vector for malicious behavior. Progress on pre-installation detection is limited by its data: existing malicious Skill resources are fragmented across sources, artifact formats, evidence regimes, and benign coverage, and they overlap enough that concatenating published rows both inflates apparent scale and places related content on both sides of any evaluation split. We present MaliciousSkillBench, a consolidated benchmark for malicious Agent Skill detection. From 13 frozen public sources, 11 of which yield eligible malicious Skill artifacts, we recover artifact-level Skills and apply canonicalization, deduplication, structural analysis, and conflict filtering, preserving provenance throughout. The malicious Core reduces from 8,414 raw to 7,539 normalized unique identities organized into 4,588 structural families, yielding 9,740 Skills: 7,505 malicious and 2,235 benign. Using only labels native to each source, we harmonize 11 attack categories for 4,983 malicious identities and find that threat composition differs sharply across sources. On static primary-instruction text, the benchmark reshapes detector conclusions: the strongest sparse lexical baseline falls from 0.932 Random Macro-F1 to 0.665 once whole sources are held out, and a fine-tuned long-context ModernBERT encoder reaches 0.964/0.940/0.734 on Random/structural-disjoint/Source-Disjoint but still carries a 0.230 gap. Class prevalence accounts for little of this gap: evaluation-only standardization removes 5.9% of it, and fully balanced retraining still yields 0.966 vs. 0.662 Macro-F1 and 0.991 vs. 0.718 AUROC. Held-out-source failures are heterogeneous and model-specific: some sources lose ranking quality outright, others rank well but transfer their decision boundary poorly; the encoder over-flags unseen benign sources at 47.3% FPR, whereas a zero-shot LLM judge keeps 0.7% benign FPR but only 41.6% malicious recall. Malicious Skill detection should therefore be reported using three separate quantities: ranking quality, fixed-operating-point performance, and benign false-positive rate, each capturing a distinct failure mode. Code and data: [https://anonymous.4open.science/r/msb-F725/](https://anonymous.4open.science/r/msb-F725/)

Primary area:
datasets and benchmarks
Keywords:
AI agent security, agent skills, malicious skill detection, benchmark, dataset, distribution shift

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uQuuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MaliciousSkillBench, a malicious-detection benchmark for Agent Skills. Starting from 13 public sources, the authors apply normalization, deduplication, structural-similarity grouping, and removal of label-conflicting items, obtaining 9,740 Skills (7,505 malicious, 2,235 benign), and map part of the malicious samples to 11 attack categories. Using only the SKILL.md text as input, the authors evaluate TF-IDF models, a fine-tuned ModernBERT, a zero-shot LLM judge, and three static scanners. The core findings are: ModernBERT's Macro-F1 is 0.964 under the random split and drops to 0.734 under the source-disjoint split, a gap of 0.230 that persists after reweighting and balanced retraining; supervised models have a false-positive rate of about 48.8% on unseen benign sources, while the zero-shot LLM has a false-positive rate of only 0.7% but a malicious recall of only 41.6%. Based on these results, the authors recommend reporting ranking quality, fixed-threshold performance, and the benign false-positive rate separately.

Critical strengths
  • Careful and transparent data consolidation. The authors point out that naively concatenating multiple datasets causes double counting and train/test leakage, and provide an actionable pipeline: 8,414 raw malicious records are deduplicated into 7,539 normalized identities, 34 cross-label conflicting items are removed, metadata such as source and label strength is retained throughout, and uncertain samples are placed in an Auxiliary set rather than forcibly labeled benign. The appendix also discloses the data recovery process, licensing, and release restrictions. For a benchmark paper, this traceability is a substantive contribution.

  • A fairly careful evaluation protocol with candid discussion of limitations. The authors report three splits (random, structural-family-disjoint, and source-disjoint) and run several diagnostics such as reweighting, balanced retraining, and leave-one-source-out, and in many places explicitly state the scope of their conclusions. Their recommendation to report AUROC, fixed-threshold metrics, and the benign false-positive rate separately is a useful reference for evaluating security detectors.

Critical weaknesses
  • The source-disjoint test set is imbalanced across sources. Appendix Table 26 shows that the source-disjoint test set contains 1,384 samples from only 3 sources: SRC009 has 728 samples, all malicious; SRC011 has 476 samples but only 21 are malicious; and only two sources contain benign samples. As a result, the Macro-F1 and benign FPR on this split are essentially determined by two benign sources, and malicious recall is essentially determined by a single source. This suggests that the test split is considerably imbalanced.

  • The detector input does not match where the label evidence lies, making it hard to distinguish detector failure from an undecidable task. The paper acknowledges that some malicious labels depend on package files or runtime evidence, yet all detectors read only the primary instruction document. Appendix M shows that on the source-disjoint test set, the agreement between the LLM review and the malicious labels is only 18.19%, and only 12/123 (9.76%) for SRC009; Section 6.3 also notes that among SRC009's 566 analyzable samples, 423 place the payload in additional resources. This means that many malicious samples contain no visible malicious content in SKILL.md. Therefore, whether the zero-shot LLM's 41.6% recall and the cross-source drop of supervised models reflect detector deficiencies or insufficient input information is not distinguished in the paper.

  • Label-quality and deduplication validation is weak, baselines are few, and novelty is limited. Labels are entirely inherited from the sources, with no independent manual spot-check to estimate label noise, and the Cohen's κ between the two initial reviewers in Appendix M's LLM review is only 0.479. Structural families are applied only to the malicious side, with no structural grouping on the benign side, and the 72 sample pairs used to validate the structural families only verify similarity within a family, without assessing whether near-duplicates still exist across different families. On the baseline side, there is only a single-prompt, single-model zero-shot LLM; there is no stronger fine-tuned LLM baseline, and no package-level baseline that looks at both SKILL.md and scripts. Finally, that distribution shift degrades performance and that multiple metrics should be reported separately are already common understanding in security machine learning; this paper mainly provides a concrete instance in this domain, with limited methodological or theoretical novelty.

Questions for authors

See the Weaknesses section above.

1 comment
AuthorsOpenReview ↗

Individual Response to Reviewer uQuu

We thank Reviewer uQuu for the positive and constructive review. We answer W1–W3 below; Tables R1–R4 and R6 are in the general response (Comments 1A–1H).

W1: Imbalanced Source-Disjoint test set

We agree that, on the frozen test set, recall is driven mainly by SRC009 and benign FPR by SRC011/SRC012. Table R3 therefore holds out each of six sources in turn and reports recall, FPR and AUROC, which do not depend on the class ratio, together with Macro-F1 standardized to a 1:1 ratio. Supervised models degrade on every held-out source (1:1 Macro-F1 with the base training set: .307–.747 for TF-IDF and .318–.818 for ModernBERT, against .882–.964 on Random).

W2: Detector input vs. where the label evidence lies

We now report every held-out result separately for the 297 malicious items whose evidence is inside SKILL.md and the 542 whose payload is in a companion file (Tables R2, R4), which separates detector failure from missing input. The LLM judge's low recall comes almost entirely from the second group, where it answers INSUFFICIENT_CONTEXT; counting those answers as escalations raises its Source-Disjoint recall from 41.6% to 85.5% (Macro-F1 .636 to .898). The supervised models' high recall on the same group is mostly driven by source and carrier cues rather than by the payload (Comment 1B); a retrained ModernBERT flags 77.1% of unattacked carriers. The same missing input explains the low agreement in Appendix M that you cite: that review also read SKILL.md alone, and on the 123 sampled SRC009 items it answered benign 61 times and insufficient context 49 times. In our package-level audit, which reads every companion file, 26 of the 27 confirmed external-file items place the harm in a companion file that SKILL.md delegates to.

W3: Benign deduplication, near-duplicates, label quality, baselines and novelty

  • Benign structural families. Clustering the 2,235 benign items at τ = 0.68 gives 1,897 families (1,716 singletons; 181 multi-member families with 519 items). In the frozen Malicious-Structural-Disjoint split, 69 benign families span train/validation and test (95/447 benign test items, 21.3%). With a fixed per-source allocation over 5 seeds, enforcing zero family overlap on both classes keeps malicious recall stable but raises benign FPR from 9.8% to 16.6% (Word LR) and from 7.8% to 16.2% (Word SVM), lowering Macro-F1 from .861 to .828 and from .910 to .881 (Table R6). Shared benign templates do lower the FPR, as you suspected, and the revised PDF adds this split (Appendix O.5); its split assignments are in the repository.
  • Near-duplicates across families. Among the 50 closest cross-family malicious pairs just below τ = 0.68, 35 share a skill name; such pairs involve 89 of the 1,500 malicious test items (5.9%) of the structural split (Comment 1G).
  • Label quality and κ = 0.479. The reference labels come from upstream construction (SRC001 human- and runtime-verified, SRC006 runtime-verified; SRC009/SRC010/SRC012/SRC013 constructed), not from any LLM judge. In our 100-package audit (two independent annotators who used an LLM assistant only to help read files; author adjudication of all 19 disagreements without AI), no benign item was judged malicious and 85/100 items match the reference (95% CI 76.5–91.4%; 5 undeterminable from missing files, 10 static-unconfirmable malicious labels). In Appendix M, the moderate κ mostly reflects abstention: when both passes give a definite answer they agree on 86.7% (κ = 0.672), and 99 of the 165 disagreements involve INSUFFICIENT_CONTEXT or UNCERTAIN.
  • More judges and baselines. Running the identical Appendix N prompt and inputs through three further judges (gpt-6-luna, gpt-6.1-terra, gpt-6.1-sol; temperature 0; model name, response id and timestamp logged per item; no parse failures) reproduces the effect of abstention scoring: .596–.610 Macro-F1 when abstentions count as benign (vs. .636 for deepseek-v4.1-flash) and .887–.906 when they are escalated (vs. .898). The cost differs by judge: these judges escalate 14.7–22.6% of held-out benign skills against 2.9%, mostly SRC011 skills that defer to unseen scripts (κ with DeepSeek .68–.75; among themselves .86–.91). The package-input baseline (Table R2, 2B) and MiniLM (Appendix Table 46) complement TF-IDF and ModernBERT; we have not added a fine-tuned LLM baseline.
  • Novelty. Beyond consolidation, the benchmark separates three shortcuts behind the source-shift failures (Table R2), shows how much evidence location and abstention scoring change the conclusions (Tables R1, R2), and adds benign-side deduplication and a tiered, license-compliant release (Table R6, Comment 1H).
Reviewer_5WgDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Agent Skills are installable instruction packages, and malicious ones are a new supply-chain risk. Detection research is held back by fragmented, overlapping datasets, where naive merging inflates scale and leaks near-duplicates across splits. This paper consolidates thirteen public sources into a deduplicated, family-partitioned benchmark with explicit provenance, defines random, source-balanced, structure-disjoint and source-disjoint splits, and evaluates text classifiers, a fine-tuned encoder, a zero-shot LLM judge and scanners on the main instruction file. The central message is that supervised detectors that look strong on random splits break under source shift, mostly by flagging benign Skills, while the judge stays precise but misses much.

Where it sits (October 2026): MalSkillBench (arXiv 2606.07131, cited) offers a runtime-verified benchmark, and industry scans (Snyk's ToxicSkills) and bypass tests of deployed scanners show the same trade-off between recall and false positives. This paper's distinct value is the consolidation and accounting of existing sources rather than new verified ground truth, so its claims about detector behaviour have to stand on its own labels.

Critical strengths
  1. The accounting is careful and reusable. Exact and normalised deduplication, structural families, conflict removal and per-item provenance are exactly what a merged benchmark needs, and everything I checked regenerated from the released files. Others can build on the deduplicated registry directly.

  2. Separating ranking from operating point is the right lens for deployment, and the payload-placement analysis makes a clear case for package-aware detection: a text-only judge cannot see payloads routed to other files.

Critical weaknesses
  1. The headline detection gap depends on how the judge's "insufficient context" answers are scored. The paper counts them as benign, but a pre-installation screen would normally escalate what it cannot verify. Scored that way, the zero-shot judge reaches high recall at a low false-positive rate under source shift and shows no gap at all. The gap is therefore a property of supervised training on this pool, not of the benchmark's difficulty, and the abstract and Fig. 1 should say so.

Recommendation: report both abstention policies and frame the result as "supervised training on this pool does not transfer".

  1. Label is entangled with where the data came from. Most benign Skills come from one source, the malicious items of that source are injected variants of its benign carriers with a recognisable wrapper style, and most other malicious items have no benign counterpart at all. A supervised model can therefore learn "looks like source X" rather than "is malicious", which fits both the high random-split score and the encoder flagging every benign item of one held-out source. Many held-out malicious items also carry their payload outside the file the detectors see. The control that would settle this, retraining with a source-diverse benign pool, is marked as not run in the supplement.
Questions for authors

Questions that could change my assessment:

  1. What do the main tables look like with abstentions scored as escalations?
  2. Does retraining with a source-diverse benign pool (the auxiliary benign set, or clean host carriers for the injected sources) shrink the source gap and the benign false-positive rate?
  3. Can you report held-out results separately for items whose malicious evidence is inside SKILL.md and outside it?
  4. On what basis is full text released for sources recorded as "academic-research-only" or "pending author clarification", and have the upstream maintainers consented?

Recommendations (not affecting my score):

  1. Position the benchmark explicitly against MalSkillBench's runtime-verified labels, for example by evaluating on its verified subset or reporting agreement with it.
  2. Add package-aware inputs, since that is where the field is moving and where your own payload analysis points.
  3. Add negative pairs to the structural validation, and state in Appendix M that the judge identities are unverified, as your own integration notes record.
3 comments
AuthorsOpenReview ↗

Individual Response to Reviewer 5WgD

We thank Reviewer 5WgD for checking our released files in such detail. Tables R1–R4 and R6 are in the general response (Comments 1A–1H).

Q1 / W1: Abstentions scored as escalations

We agree. Scoring INSUFFICIENT_CONTEXT as benign penalizes the judge whenever SKILL.md delegates to a companion file, and our argument in Appendix N.3 that this is the deployment-faithful reading does not hold: a pre-installation screen would escalate such cases for package inspection. We withdraw that claim; the revised PDF reports both policies in the Abstract, Table 2, Figure 1 and Table 67 (Table R1). As you suggested, the Figure 1 banner now reads "Supervised training on this pool does not transfer to unseen benign sources."

  • On Source-Disjoint, escalation raises recall from 41.6% to 85.5% (external-file items: 15.3% to 82.8%) and Macro-F1 from .636 to .898, while escalating 2.9% of held-out benign skills (SRC011 2.9%, SRC012 3.3%).
  • Per source, recall rises from 83.4% to 90.8% on SRC001, 92.2% to 94.4% on SRC012, 97.2% to 98.5% on SRC010, and 33.7% to 83.9% on SRC009. Escalation has a cost: it sends 11.0% of SRC010 benign skills, most of which ship helper scripts, to package inspection, and on SRC006 (AgentTrap) the judge still reaches only 51.2% recall at 40.0% benign escalation (AUROC .568).

Q2 / W2: Source-diverse benign retraining

We agree that label and source are entangled. 97.2% (1,478/1,521) of the training benign items come from SRC010, and the supervised models flag 62.0–62.4% (LR/SVM) and 48.8% (ModernBERT) of unseen benign items; ModernBERT flags every SRC012 benign item when SRC012 is held out (Table R3). The class-conditional diagnostic in Appendix Table 43 points the same way: holding out only the malicious side of the test sources gives 5.6% benign FPR, while holding out only the benign side gives 47.3%. Two further cues inflate external-file recall (Table R2): (1) SRC013 and SRC009 wrap attacks around the same SkillsBench carriers—removing SRC013 lowers the false-positive rate on 70 unattacked carriers from 60.0% to 45.7% (LR); and (2) the routing sentence alone is flagged 92.0–93.9% of the time.

For the requested control, we added 8,102 source-diverse benign skills (none overlapping the 70 carriers; 3 seeds). Held-out benign FPR falls from 62.0% to 12.2% (LR) and 8.0% (SVM), the carrier false-positive rate falls from 60.0% to 18.1%, and external-file recall at 5% FPR rises from 11.6% to 47.8% (AUROC .707 to .852).

Q3 and Rec 2: Evidence inside vs. outside SKILL.md

We now report every held-out result separately for the 297 malicious items whose evidence is inside SKILL.md and the 542 whose payload is in a companion file (Tables R2, R4). After diverse-benign retraining, SKILL.md-only recall is 70.8–73.2% on the first group and 52.3–59.5% on the second. Adding the companion files raises external-file recall to 72.6–76.7% (AUROC .914–.938). However, these package-level models also flag 40.9–54.5% of the 22 multi-file Clean70 carriers and 77.8–88.9% of the 27 held-out multi-file SRC013 packages, so we present package input as a baseline with a clear limitation.

Q4 / Ethics: Redistribution

The reviewer is right that releasing 9,735 inline texts in primary.parquet conflicted with the per-source policies in Table 5 (Appendix A). We did not obtain separate waivers for the 8,711 Tier 2 rows (SRC001 3,426; SRC006 131; SRC009 728; SRC010 4,426) or the 186 Tier 3 rows (SRC002 153; SRC004 33). The anonymous repository now keeps inline text only for the 843 rows released under MIT or Apache-2.0 (SRC005, SRC008, SRC011, SRC012, SRC013). For all other rows it keeps identifiers, labels, hashes and split assignments but no text. We also added SECURITY.md, which documents the tier policy and safe-handling rules, and removed the earlier file from the repository's git history. SRC001, which Table 5 lists as "adapter only", is now metadata-only. We have not added access gating, since all remaining inline text is MIT/Apache-2.0 (details in Comment 1H).

Recommendations

  • Rec 1. The revised Section 2 contrasts MalSkillBench (SRC001: 3,426 runtime-verified skills from a single source) with our multi-source, leave-one-source-out design, and Appendix O.4 evaluates it as a held-out source. When SRC001 is held out, supervised models reach 35.1–52.0% recall on it, compared with 90.8% for the escalating judge (Table R3).
  • Rec 3. Among cross-family malicious pairs just below the 0.68 threshold, 35 of the 50 closest share a skill name, so the threshold does leave some variants of the same skill in different families. Such pairs touch 89 of the 1,500 malicious test items (5.9%) in the malicious-structural split. Random cross-family pairs below 0.55 share neither a base skill nor a name (0/50). The revised Appendix M states that the secondary judge model names are unverified log metadata.
Reviewer_5WgDOpenReview ↗

Follow-up: two remaining points

Thank you for the detailed response and the new experiments. They answer my questions, mostly in the direction I expected, and the tiered release resolves my licensing concern. I have removed my ethics flag.

Two things keep me at my current score. First, the default Source-Disjoint split is unchanged, so models trained on the benchmark still learn mostly from one benign source. The diverse benign data that fixes this is not part of the release, and it comes from sources the benchmark itself treats as weak negatives. I would like the paper to state which training pool future work should use, and to report that setting in the main results.

Second, most of the benchmark text now has to be fetched from upstream, and the repository does not yet include a script for this. A commitment to add it for the camera-ready would be enough for me.

If the first point is addressed, and the second committed to, I will raise my score to 3.

AuthorsOpenReview ↗

Recommended training pool and upstream fetch script

Thank you for the follow-up and for removing the ethics flag.

  1. Recommended training pool for Source-Disjoint

We agree that the benchmark should name a default training pool. For Source-Disjoint we now recommend training on the base training set plus 7,645 benign SKILL.md files from outside the Main pool (3,488 SRC001, 76 SRC005, 4,000 SRC007, 33 SRC003, 48 unattacked SRC013 carriers), and keep the original training set as a shortcut diagnostic. The test set is unchanged.

While freezing this pool we removed every candidate that shares a skill name, at least 50% of its lines, or at least 50% of its word 5-grams with any Source-Disjoint test item, Clean70 skill or held-out SRC013 carrier (5-grams are needed because SRC007 stores each file on one line), plus 38 exact duplicates. In Comment 1C we had applied the name and line rule only to the SRC013 carriers, so the 2A/2B pool still contained near-copies of 64 of the 70 Clean70 skills, and our statement that no training file overlaps Clean70 (Comments 1C, 2 and 3) was wrong. The results below supersede the 2A/2B numbers in Comments 1A, 1C, 1D, 2 and 3; the 2A rows of Table R3 were rerun with this pool for the revised PDF.

DetectorTraining poolMacro-F1RecallBenign FPRAUROCExternal-file recallClean70 FPR
Word TF-IDF + LRoriginal.66194.4%62.0%.73892.8%60.0%
recommended.80779.9%16.7%.91277.3%41.4%
Word TF-IDF + SVMoriginal.66595.6%62.4%.80993.9%58.6%
recommended.78571.2%9.9%.91065.9%30.0%
Char TF-IDF + SVMoriginal.65395.7%64.4%.82194.8%—
recommended.80181.9%20.9%.89777.3%35.7%
ModernBERT (3 seeds)original.734 ± .00793.8 ± 1.9%48.8 ± 2.8%.812 ± .03992.4 ± 2.9%77.1% (1 seed)
recommended.837 ± .03387.8 ± 5.8%20.7 ± 0.7%.874 ± .01488.3 ± 7.1%21.4 ± 10.3%
LLM judge, abstain as escalatenone (zero-shot).89885.5%2.9%.97182.8%—

Sparse results are deterministic on the fixed pool; three different SRC007 draws give Macro-F1 within .02. Removing the overlap makes diverse benign data more useful on the test set than Table R2 showed (LR Macro-F1 .807 instead of .735), likely because the earlier pool taught the models that SRC009 carriers are benign. The shortcut controls improve less than we reported: Clean70 FPR falls to 41.4% (LR) and 30.0% (SVM), not to 18.1–20.0%, and CarrierOnly flagging falls from 49.4% to 34.0% (LR), not to 14.9%. ModernBERT (3 seeds, paper configuration, one A100 per seed) gains the most, from .734 to .837 Macro-F1 and from 48.8% to 20.7% benign FPR; its Clean70 and CarrierOnly rates fall from 77.1% and 80.4% to 21.4% and 20.0% on average but vary widely across seeds (10.0–30.0% and 8.0–32.6%). These controls therefore remain part of the recommended evaluation. With package input (2B), external-file AUROC is .932–.959 and the same models flag 45.5–68.2% of the 22 multi-file Clean70 skills and 77.8–88.9% of the 27 held-out multi-file SRC013 skills, so that conclusion is unchanged.

Weak negatives. You are right that these sources do not pass our Main-benign gate (Appendix Table 18). We therefore use them only for training: the test negatives remain the Main benign items, so label noise in the added files can change what a model learns but not how it is scored. The gain does not hinge on the silver scanner labels of SRC007: without them (3,645 added files), Macro-F1 is .820 (LR) and .811 (SVM), while benign FPR rises to 24.8% and 15.8%, so SRC007 mainly lowers the false-positive rate.

The revised PDF now names this pool as the recommended Source-Disjoint training setting (Section 6.1), reports it in Table 2 next to the original pool, and gives its construction and the corrected 2A, 2B and LOSO rows in Appendix O.3–O.4. The pool is now in the anonymous repository: rebuttal/recommended_pool/ holds its manifest (source, pinned upstream commit, licence and SHA-256 per file; no text) and the training split, and rebuttal/06_recommended_pool.py and 07_modernbert_recommended_pool.py reproduce the numbers above (the ModernBERT predictions and control scores are in rebuttal/modernbert_controls/recommended_pool/).

  1. Upstream fetch script

We commit to adding it for the camera-ready release. The script will download every Tier 2 text and every file of the recommended pool from the pinned upstream revisions, check each file against its released SHA-256 hash, and list any item that can no longer be retrieved. As of today, all four Tier 2 upstreams (SRC001, SRC006, SRC009, SRC010) are public and their pinned revisions are still available.

Reviewer_2W55OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces MaliciousSkillBench, a benchmark for detecting malicious agent skills. The authors combine 13 existing datasets to create a benchmark of 9,740 skills, including 7,505 malicious and 2,235 benign examples. They test different detection methods and show that performance drops a lot when models are tested on data from new sources. For example, ModernBERT's Macro-F1 drops from 0.964 on random splits to 0.734 on source-disjoint splits.

Critical strengths
  • The paper studies an important and timely security problem.
  • The authors put considerable effort into combining existing datasets, removing duplicates, and checking data quality.
  • The experiments cover different evaluation settings and provide useful results about how well models work on unseen data.
  • The analysis of false positives and class imbalance is helpful.
Critical weaknesses
  1. The models may be learning dataset patterns instead of detecting malicious behavior.

Some skills are labeled malicious because of code or behavior in other files, but the models only look at the main SKILL.md file. ModernBERT still detects many of these examples correctly, even when the actual malicious content is not in the file it sees. This makes me wonder whether the model is learning patterns specific to the datasets rather than identifying real threats. Testing models with access to the full skill package would help answer this.

  1. The evaluation on unseen sources is not fully convincing.

The three datasets used for testing on unseen sources have very different numbers of malicious and benign examples. One contains only malicious examples, while another is mostly benign. Although the authors try to address this issue, it is still hard to tell whether the performance drop is because the data is new or because the datasets are very different. Testing on more balanced datasets would make the results more convincing.

  1. The novelty is limited.

The main contribution is combining existing datasets and evaluating detection methods on them. The dataset is useful, but most of the methods used to build and evaluate it are already known. The finding that models perform worse on unseen sources is also not very surprising. I would like to see more new insights beyond showing this performance drop.

Questions for authors

Clarifications:

  1. Some malicious skills have their harmful content in external files rather than SKILL.md. How should a model detect these attacks when it can only see the main file? Do you think these examples provide a fair test of text-only detection?
  2. Why were SRC009, SRC011, and SRC012 chosen for the main source-disjoint evaluation? Given their very different class distributions, how confident are you that the results reflect generalization problems rather than differences between datasets?

Additional evidence:

  1. ModernBERT performs well even when the malicious payload is stored in another file (Table 71). Could you test whether it is relying on dataset-specific wording or templates, similar to the checks performed for the simpler models?
  2. Could you provide a small human review of the benchmark labels, especially for cases where malicious behavior is not visible in SKILL.md? This would help establish how reliable the labels are for the text-only detection task.
5 comments
AuthorsOpenReview ↗

Individual Response to Reviewer 2W55

We thank Reviewer 2W55 for the careful review. Tables R1–R4 and R6 are in the general response (Comments 1A–1H).

W1, Q1: External-file attacks and what SKILL.md-only models detect

On Q1, we agree that external-file items are not a fair test of payload detection from SKILL.md alone: at best, a text-only screen can notice the visible delegation and escalate the Skill for package inspection. We therefore report the 297 held-out malicious items with evidence inside SKILL.md separately from the 542 whose payload is in a companion file (Tables R2, R4), for all detectors (Appendix O.2).

On W1, your concern is confirmed: the supervised models' 92.4–93.9% recall on external-file items is mostly not payload detection (Table R2).

  1. 97.2% (1,478/1,521) of the training benign items come from SRC010, so the base models flag 62.0–62.4% (LR/SVM) and 48.8% (ModernBERT) of unseen benign items. On external-file items their recall at 5% FPR is only 11.6% (LR) and 14.9% (ModernBERT), with AUROC .707–.795.
  2. SRC013 (training) and SRC009 (test) share 26 SkillsBench carrier families (42.4% of SRC009 rows; Appendix C). Removing SRC013 lowers the false-positive rate on the 70 unattacked carriers (Clean70) from 60.0% to 45.7% (LR) and from 58.6% to 38.6% (SVM).
  3. The models react to the delegation sentence. They flag 49.4% (LR) and 47.5% (SVM) of the 423 routed items once the routing line is removed, but 92.0% and 93.9% when only the routing line is kept.

Adding 8,102 diverse benign skills (2A; 3 seeds; no overlap with Clean70) cuts the Clean70 false-positive rate from 60.0% to 18.1% (LR) and from 58.6% to 20.0% (SVM), CarrierOnly flagging to 14.9% (LR), and held-out benign FPR to 12.2% (LR) and 8.0% (SVM). External-file recall at the 0.5 threshold falls to 59.5% (LR) and 52.3% (SVM), while recall at 5% FPR rises from 11.6% to 47.8% (AUROC .707 to .852). Appending the companion files (2B) raises external-file recall to 72.6–76.7% (66.6–72.7% at 5% FPR; AUROC .914–.938), but these models also flag 40.9–54.5% of the 22 multi-file Clean70 skills and 77.8–88.9% of the 27 held-out multi-file SRC013 skills.

Q3: Does ModernBERT also rely on templates?

Yes. Its high routed-payload recall (Table 71) is mostly a carrier effect. We retrained ModernBERT (seed 42, paper configuration; test Macro-F1 .719 vs. .729 archived) and scored the sparse models' controls (Table R2 †). It flags 77.1% of the unattacked Clean70 carriers, 80.4% of CarrierOnly and 98.1% of RouteLineOnly (TF-IDF LR: 60.0%, 49.4%, 92.0%). When the carrier body is replaced by a 36-word stub (FPP_ROUTE_PDF, 121 items), ModernBERT recall drops by 20 points to 76.3 ± 12.2%, while the escalating LLM judge flags 96.7% (117/121; Table R4). In the class-conditional diagnostic (Appendix Table 43), holding out only the benign sources gives 47.3 ± 3.9% benign FPR (Macro-F1 .740), against 5.6 ± 1.6% when only the malicious sources are held out (88.7% external-file recall at 5% FPR; AUROC .970).

W2, Q2: Class ratio and the choice of held-out sources

Malicious recall, benign FPR and AUROC do not depend on the class ratio of the test set, so the 62.0% benign FPR on held-out sources (Table R1) reflects a failure to generalize to new sources. Table R3 holds out each of six sources in turn (SRC012, SRC011, SRC006, SRC010, SRC009 with Clean70, SRC001) and reports Macro-F1 standardized to a 1:1 ratio and averaged over 100 random 1:1 subsamples. With the base training set, 1:1 Macro-F1 is .307–.747 for TF-IDF and .318–.818 for ModernBERT (.333 on SRC012, where its benign FPR is 100%; Appendix Tables 40 and 48). The frozen split held out SRC009 together with SRC011 and SRC012, both dual-class, keeping 85.7% of the data for train/val; Table R3 shows the conclusion does not depend on this choice.

Q4, W3: Label audit and what is new

  • Label audit (Q4). Two annotators independently labeled 100 stratified packages, blinded to label, source and stratum. They read SKILL.md first and then all companion files, using an LLM assistant only to help read files, and the authors adjudicated all 19 disagreements without AI assistance. No benign item was judged malicious (0/30), 85/100 match the reference label (95% CI 76.5–91.4%; 90.5% design-weighted on Source-Disjoint), and 5 are undeterminable (missing files). All 10 contradictions are malicious labels that static reading cannot confirm: 4/4 runtime-verified SRC002/SRC004 items, 4/39 SRC009 items, and 1 each from SRC006 and SRC010. On external-file items the SKILL.md-only verdict was ESCALATE in 29/30.
  • What is new (W3). The revision collects the new findings in Appendix O, apart from the original contributions (harmonizing 11 sources, τ = 0.68 deduplication, evidence-location labels): the three shortcut factors (Table R2), the effect of abstention scoring and package input (Tables R1–R2), and benign-side deduplication (Table R6).
AuthorsOpenReview ↗

Update to W1 and Q3: corrected 2A/2B numbers and 3-seed ModernBERT controls

Thank you again for your constructive comments. Following Reviewer 5WgD's request for a default training pool, the revised PDF now names a recommended Source-Disjoint training pool (Section 6.1, Table 2, Appendix O.3–O.4). This changes several 2A/2B and Q3 numbers in our response above, which we correct below.

  1. Recommended training pool and corrected 2A/2B numbers (W1)

The 8,102-file 2A pool in our response above was filtered against Clean70 only for the SRC013 carriers, and its line-based rule could not catch SRC007, which stores each file on one line. It still contained near-copies of 64 of the 70 Clean70 skills, so our statement that it had no overlap with Clean70 was wrong. We now remove every candidate that shares a skill name, at least 50% of its lines, or at least 50% of its word 5-grams with any Source-Disjoint test item, Clean70 skill or held-out SRC013 carrier, plus 38 exact duplicates. This leaves 7,645 added benign SKILL.md files, the recommended pool in Table 2.

On this pool, Word TF-IDF + LR reaches .807 Macro-F1 and .912 AUROC (.661 and .738 with the original training set). The shortcut controls improve less than we reported above: Clean70 FPR is 41.4% (LR) and 30.0% (SVM), not 18.1% and 20.0%; CarrierOnly flagging is 34.0% (LR), not 14.9%; and held-out benign FPR is 16.7% (LR) and 9.9% (SVM), not 12.2% and 8.0%. External-file recall is higher than we reported: 77.3% (LR) and 65.9% (SVM) at the 0.5 threshold, not 59.5% and 52.3%, and 59.2% (LR) at 5% FPR (AUROC .905), not 47.8% (.852). With package input (2B), external-file recall is 83.4–88.9% (71.0–79.0% at 5% FPR; AUROC .932–.959), but the same models flag 45.5–68.2% of the 22 multi-file Clean70 skills (not 40.9–54.5%) and 77.8–88.9% of the 27 held-out multi-file SRC013 skills, so the conclusion on package input is unchanged.

  1. ModernBERT on the recommended pool (W1, Q3)

We retrained ModernBERT-base on this pool with three seeds in the paper configuration. Compared with the three original-pool runs, Macro-F1 rises from .734 ± .007 to .837 ± .033 (AUROC .812 ± .039 to .874 ± .014) and held-out benign FPR falls from 48.8 ± 2.8% to 20.7 ± 0.7%, while recall falls from 93.8 ± 1.9% to 87.8 ± 5.8% (external-file recall 88.3 ± 7.1%).

On the Q3 controls, the single-seed rates we reported above (77.1% of Clean70, 80.4% of CarrierOnly, 98.1% of RouteLineOnly) become 21.4 ± 10.3%, 20.0 ± 12.3% and 82.0 ± 7.8%; the first two vary widely across seeds (10.0–30.0% and 8.0–32.6%). Diverse benign data thus reduces the carrier shortcut, but RouteLineOnly flagging stays high: the model still reacts mainly to the delegation sentence, which for a SKILL.md-only screen is at best a reason to escalate the package for inspection (Q1), not payload detection. These controls therefore remain part of the recommended evaluation.

The pool manifest (source, pinned upstream commit and licence per file; no text) and the training split are in rebuttal/recommended_pool/ of the anonymous repository. rebuttal/06_recommended_pool.py and 07_modernbert_recommended_pool.py reproduce the numbers above, and the ModernBERT predictions and control scores are in rebuttal/modernbert_controls/recommended_pool/.

Please let us know if our response above and these updates address your concerns.

AuthorsOpenReview ↗

A gentle request for your feedback (Reviewer 2W55)

Dear Reviewer 2W55,

With the discussion window closing in about two days, we know this is a busy week and just wanted to check whether our responses and new experiments have addressed your concerns.

We are very grateful for your questions, which directly prompted our new experiments on full skill packages, template checks for ModernBERT, and the label audit of 100 packages in the revised paper (Appendix O). If anything still feels unconvincing, or if we can clarify any detail before the deadline, we would be very glad to hear from you.

With sincere thanks,

The Authors

Comments on the paper

9 comments
AuthorsOpenReview ↗

General Response (8/8): Redistribution and licensing

4. Redistribution and licensing (5WgD Ethics, Q4)

Reviewer 5WgD found a real inconsistency in our release. Our per-source redistribution policies (Table 5 in Appendix A, and benchmark_master.csv) imply three tiers: redistributable (843 items), metadata and links only (8,711) and pending author clarification (186). Yet data/primary.parquet and RESPONSIBLE_USE.md (Appendix L) released all 9,735 non-secret SKILL.md texts inline, on the mistaken view that extracting SKILL.md text from a package was exempt from package-level limits. That distinction does not hold, and we have corrected the release.

Licensing by source (5WgD Q4):

  • Tier 1, inline text (843 items). Explicit MIT or Apache-2.0 licenses, checked against the upstream license files: SRC005 (revision 5cff435d, MIT, 30), SRC008 (2864c752, Apache-2.0, 3), SRC011 (7219b10d, MIT, 476), SRC012 (eb9d5a9c, MIT, 180) and SRC013 (e0589d7b, Apache-2.0, 154).
  • Tier 2, metadata and hashes only (8,711 items). Sources with restrictive, non-commercial or unstated terms: SRC001 (06e08312, academic research only, 3,426), SRC006 (ab13f59d, terms not provided, 131) and SRC010 (f90517b7, CC-BY-NC-SA-4.0, 4,426). We also keep SRC009 (cc21f909, CC-BY-4.0, 728) in this tier as a conservative dual-use choice consistent with Table 5, although CC-BY-4.0 allows redistribution with attribution. We did not obtain separate written permission to rehost any Tier 2 source. SRC001, which Table 5 lists as "adapter only", is therefore now released as metadata and hashes only, as Table 5 states.
  • Tier 3, withheld (186 items). Author-provided research snapshots of SRC002 (422bf340, MIT, 153) and SRC004 (8264436a, MIT, 33), for which written permission to redistribute had not been finalized before the freeze.

Changes in the repository. The anonymous repository now ships data/primary.parquet with inline text only for the 843 Tier 1 items. For the 8,711 Tier 2 and 186 Tier 3 items it keeps identifiers, labels, split assignments, pinned upstream URLs and revisions, and SHA-256 hashes, but no text. We added SECURITY.md and scripts/build_tiered_release.py, updated RESPONSIBLE_USE.md, and rebuilt the git history so that the earlier full-text file is no longer in it. Automated download of the Tier 2 texts from upstream is planned for the camera-ready release. We have not added access gating: the inline text is now limited to the 843 MIT/Apache-2.0 records, the reviewer did not consider the dual-use risk severe, and SECURITY.md documents the tier policy and safe-handling rules (static analysis only; credential tokens redacted).

Reproducibility trade-off. This correction costs some reproducibility: the full-text file let anyone refit Table 2 with one command, whereas retraining on the full corpus now requires fetching the Tier 2 texts from the pinned upstream revisions in Table 5. All split manifests, hashes and the offline recomputation scripts under reproducibility/ still run unchanged.

AuthorsOpenReview ↗

General Response (7/8): Boundary pairs and the label audit

3.2 Cross-family pairs near the threshold (5WgD Rec 3)

The paper's validation checked 72 positive pairs (Appendix C, Table 17). To check negatives, we scored the 3,699,575 cross-family pairs among the 7,505 malicious items that share at least one indexed template shingle or line:

  • Only 1,392 of them fall in the band [0.55, 0.68).
  • Among 50 random cross-family pairs below 0.55 (similarity 0.028–0.118), none share a base skill or a skill name (0/50).
  • Among the 50 closest pairs just below the threshold ([0.662, 0.680)), 35 share a skill name (e.g. ASB04_000009 and ASB04_000607, both prime-numbers, 0.6789), and in 9 of the 17 pairs with a recorded base skill both items come from the same base skill, so τ = 0.68 leaves some variants of the same skill in different families. The other 15 pairs have different skill names (e.g. ASB04_003426 load-balancer-probe and ASB04_005014 audit-logger, 0.6779).
  • Of the 1,392 boundary pairs, 488 (392 with the same skill name) cross the train/test boundary of the Malicious-Structural-Disjoint split and involve 89 of its 1,500 malicious test items (5.9%). In Source-Disjoint, 312 pairs cross train/test, all with the same skill name; the training-side item comes from SRC001 in 254 pairs, SRC006 in 40 and SRC013 in 18. They involve 150 of the 839 malicious test items (17.9%), all from SRC009; 142 of them are among the 320 SRC009 items whose family already appears in training or validation (Table R4).

3.3 Package-level label audit and the κ = 0.479 agreement (2W55 Q4; uQuu W3)

  • Label audit. We drew 100 stratified items (Source-Disjoint test: 30 external-file malicious, 15 in-document malicious, 15 benign; development pool: 25 malicious across eight sources, 15 benign), blinded to label, source and stratum. Two annotators labeled them independently, reading SKILL.md first and then every companion file, and used an LLM assistant to help read and explain files; every label was decided by the annotators, and the authors adjudicated all 19 disagreements without AI assistance. Before adjudication, κ was .65 for the package verdict, .76 for evidence location and .89 for the SKILL.md-only verdict. After adjudication, no benign item was judged malicious (28/30 confirmed), 85/100 items match the reference label (95% CI 76.5–91.4%; design-weighted 90.5% on the Source-Disjoint test and 87.3% on the development pool), 5/100 are undeterminable because referenced files are missing (2 benign, 3 malicious), and all 10 contradictions are malicious labels that static reading cannot confirm: 4/4 SRC002/SRC004 items, whose labels rest on runtime evidence while only SKILL.md is archived; 4/39 Source-Disjoint SRC009 items whose realized payload shows no concrete harm; and one item each from SRC006 and SRC010. On external-file items, the SKILL.md-only verdict was ESCALATE in 29/30, and 26/27 confirmed items place the harm in a companion file that SKILL.md delegates to. Both annotators' verdicts, the adjudicated labels and the 10 contradicting items are in rebuttal/human_audit/ of the anonymous repository, and the 10 items are listed in Table 79 of the revised PDF.
  • Agreement in Appendix M. When both initial review passes give a definite answer (BENIGN or MALICIOUS; 497 of 610 items, 81.5%), they agree on 86.7% (431/497; κ = 0.672): 325 both benign, 106 both malicious, and 66 direct contradictions. 99 of the 165 disagreements (60.0%) concern abstention: 52 where pass B answers UNCERTAIN, 40 where A answers INSUFFICIENT_CONTEXT and B answers BENIGN, and 7 the other way round. Treating MALICIOUS against all other answers, the passes agree on 85.9% of the 610 items (524/610; κ = 0.622). The model names recorded in that log (Gemini-3.8-flash, GPT-5.6-luna) are unverified metadata, and we do not use them for any claim about model families.
AuthorsOpenReview ↗

General Response (6/8): LOSO findings and benign-side deduplication (Table R6)

Findings from Table R3:

  1. The drop appears on every held-out source (2W55 W2, Q2). Even at a 1:1 class ratio, the TF-IDF models trained on the base pool reach only .307–.747 Macro-F1 on the six held-out sources, and ModernBERT .318–.818 on the four where it was run, against .882–.964 on Random. ModernBERT's benign FPR is 100.0% on SRC012, 88.1% on SRC006 and 66.7% on SRC010 (Appendix Table 48).
  2. Choice of the frozen Source-Disjoint split (2W55 Q2). We held out SRC009, SRC011 and SRC012 to combine one malicious benchmark covering several realizations (SRC009) with two sources that contain both classes (SRC011, SRC012), while keeping 85.7% of the items for training and validation. Table R3 shows that the conclusion does not depend on this choice.
  3. Relation to MalSkillBench (5WgD Rec 1). MalSkillBench (SRC001) provides 3,426 execution-verified malicious skills from a single source. Our benchmark harmonizes 11 sources, deduplicates structural templates (τ = 0.68), records whether the evidence lies in SKILL.md or in a companion file, and evaluates with whole sources held out. When SRC001 is held out, supervised models trained on the remaining sources recall only 35.1–52.0% of its malicious items, compared with 90.8% for the escalating LLM judge, so the two resources complement each other.

3. Structural families, boundary pairs and label audit (uQuu W3; 5WgD Rec 3; 2W55 Q4)

3.1 Benign structural families and a fully structure-disjoint split (uQuu W3)

Running the structural clustering of Section 4.3 (τ = 0.68) on all 2,235 benign items gives 1,897 benign families: 1,716 singletons and 181 multi-member families with 519 items (largest: 35). In the frozen Malicious-Structural-Disjoint split, 69 multi-member benign families span train/validation and test, covering 95/447 (21.3%) of the benign test items. To measure the effect of this leakage without also changing the source mix, we built 5-seed splits with a fixed 60/20/20 per-source benign allocation (e.g. SRC012's 90 benign items are split 54/18/18, against 70/7/13 in the frozen split), once with zero family overlap on both classes and once with random benign assignment as a control (Table R6). Enforcing benign family disjointness raises benign FPR from 9.8 ± 1.5% to 16.6 ± 1.3% (Word LR) and from 7.8 ± 1.6% to 16.2 ± 0.8% (Word SVM), and lowers Macro-F1 from .861 ± .006 to .828 ± .005 (LR) and from .910 ± .007 to .881 ± .004 (SVM). Shared benign templates therefore do lower the FPR in the frozen split, as Reviewer uQuu suspected.

Table R6: Frozen Malicious-Structural-Disjoint split vs. a split that is structure-disjoint on both classes (τ = 0.68)

SplitDetectorMacro-F1RecallBenign FPRAUROC
Frozen Malicious-Structural-DisjointWord LR / Word SVM.861 / .91589.4% / 94.6%10.7% / 8.9%.957 / .977
Control: random benign assignment, same per-source counts (5 seeds)Word LR / Word SVM.861±.006 / .910±.00789.0±0.3% / 93.7±0.3%9.8±1.5% / 7.8±1.6%.958±.005 / .977±.003
Structure-disjoint on both classes, same per-source counts (5 seeds)Word LR / Word SVM.828±.005 / .881±.00487.9±0.2% / 93.8±0.2%16.6±1.3% / 16.2±0.8%.933±.004 / .962±.001

All rows are refits with the Table 2 configuration; for the frozen split, the archived Table 2 predictions give Macro-F1 .860 / .916.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.