- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Skill Behavioral Rules (SBRs) for malicious skill detection. An LLM extracts a skill graph from SKILL.md and referenced script filenames; a multiple-instance learning (MIL) model scores textual descriptions to instantiate ML predicates; Horn rules are mined from the augmented graphs; and sparse logistic regression learns rule weights. Predictions can be traced to matched rules and evidence in the extracted graph. After excluding 1,300 prompt-injection malicious skills from MalSkillBench, SBR achieves 87.17% F1 and a 5.88% false positive rate (FPR) on 1,329 test skills. It achieves 71.36% F1 on the cross-source Agent Skill Malware (ASM) dataset. The paper also reports ablations and a case study.
Critical strengths
- Captures concrete behavior combinations. SBR incorporates entities, operation relations, and maliciousness scores of local descriptions into rules. This allows decisions to consider operations, their objects, and their textual context together, such as a credential read paired with a send to an external endpoint.
- Broad experimental evaluation. The paper evaluates a skill-family-disjoint MalSkillBench split and the cross-source ASM dataset, compares nine baseline configurations, and includes ablations. SBR achieves 87.17% F1 with a 5.88% FPR on MalSkillBench and 71.36% F1 on ASM.
Critical weaknesses
- Script contents are outside the input. SBR reads
SKILL.mdand the filenames of referenced scripts, whereas code-injection and mixed attacks may place decisive malicious behavior in script bodies. Without results stratified by attack vector and evidence visibility, its performance on this subset remains unclear. - No equally supervised classifier comparison. SBR trains on MalSkillBench labels, while LLM Guard, the trained-classifier baseline in Table 1, uses its official configuration. A malicious-skill classifier trained with the same data and label budget is needed to assess the gain from learned behavioral rules over direct classification.
Questions for authors
- For code-injection and mixed-attack samples whose key malicious evidence lies in script bodies, what visible signals does SBR rely on? Could you report sample counts and recall by attack vector and by whether that evidence appears in the model input, to clarify the method's scope?
- Could you train a malicious-skill text classifier under the same train/test split, label budget, and input content (
SKILL.mdand referenced script filenames), and compare it with SBR? This would help quantify the gain from the graph-and-rule design. - Appendix A.7 reports an overall structural-relation recall of 0.269 on 20 ASM documents, but does not report extraction performance for relations associated with malicious behavior. Could you report the extraction recall of key malicious relations visible in the input and analyze whether their omission contributes to false negatives on ASM?
6 comments
Q3. Extraction of key malicious relations and its effect on false negatives on ASM
We appreciate your suggestion to focus on key malicious relations. Following your suggestion, we have evaluated the extraction of key malicious relations and examined whether missing relations contribute to false negatives on ASM. We found that SBR can still detect malicious skills using the retained behavioral evidence when key relations are missing.
For the 10 malicious samples in the original 20-document ASM set, we marked 24 key malicious relations in their reference graphs. MiniMax-M2.5 correctly extracted 4 of these relations, giving an extraction recall of 16.7%.
To further examine how extraction results relate to false negatives, we kept the SBR detector fixed and evaluated it using graphs extracted by MiniMax-M2.5 and the reference graphs. Both graph sources led to the detection of 5 of the 10 malicious skills, with 3 detected skills shared between them. On this ASM subset, SBR achieved the same overall detection rate with the extracted and reference graphs. We have added the key relation extraction results and the comparison of SBR's detection results in Appendix A.7.
To further explain how a malicious skill can still be detected when key relations are missing, we have added the ASMFULL_150 video downloader case in Figure 10 of Appendix A.12.
The document presents downloading and running the openclaw-agent utility as a prerequisite for the task.
The graph extracted by MiniMax-M2.5 omits the utility's execution relation while retaining the GitHub download and log-reading relations.
The download description's risk score of 0.999 satisfies the corresponding ML predicate.
This semantic evidence and the retained behavioral relations jointly support matches of rules 263 and 367.
The execution relation in the reference graph participates in a match of rule 133.
Despite differences in the matched rules, SBR predicts the skill as malicious with either graph after aggregating all matched rule contributions.
This case illustrates how SBR can preserve a correct detection by combining the retained behavioral relations with ML predicates, even when a key relation is missing.
Response to Authors
Thank you for the additional analyses. The fine-tuned DeBERTa result addresses my baseline concern, and the breakdown by malicious-content location is helpful.
My remaining concern is the reported 85.51% recall on 207 “scripts only” samples. SBR does not read script bodies, and the case study's SKILL.md still mentions downloading remote executables. How many of these samples lack visible risk cues in SKILL.md and script filenames, and what is recall on that subset? A benign-looking document can invoke a malicious helper script without exposing its behavior to SBR.
This leaves the detector's coverage of concealed script attacks unclear, so I maintain my original rating of 2.
We report detection results by malicious content location and attack vector, a fine-tuned DeBERTa comparison, and an analysis of key relation extraction and downstream detection on ASM.
Thank you for your careful evaluation and constructive comments. Our responses are as follows.
W1 & Q1. Document evidence and detection results by malicious content location and attack vector
Thank you for suggesting this breakdown of detection performance. As you suggested, we have added analyses by malicious content location and attack vector. We found that, even when malicious behavior is implemented in scripts, SBR can extract features indicative of malicious behavior from the skill document for detection.
Specifically, we grouped the 529 malicious skills in the test set by the location of malicious content.
Recall is at least 80% in all three groups and exceeds 85% in both groups with malicious content in scripts. These results show that SBR can detect most samples in these groups using document input. We have added the results and analysis by malicious content location in Table 14 of Appendix A.10.
In addition to malicious content location, we grouped the generated malicious samples in the test set by the attack vectors defined in MalSkillBench. The code injection and mixed attack groups contain 272 and 123 samples, respectively, with SBR achieving recalls of 81.62% and 97.56%. These results further support SBR's detection capability across these attack vectors.
To further illustrate how SBR detects a sample whose malicious content resides only in scripts, we have added the
perplexity-search__CI_B7case in Figure 7 of Appendix A.12. The ransom-note operation is implemented in the script, while the document describes runningsonar_guard.pyduring initialization to download remote executables. The operation description has a risk score of 0.992, satisfying the machine learning (ML) predicate in rule 30. This semantic condition and the script execution relation jointly support the rule match. SBR predicts the sample as malicious after aggregating the contributions of all matched rules.To further illustrate how SBR distinguishes such behavior from normal installation, we have added the benign
lightpanda-browsercase in Figure 8 of Appendix A.12. It also contains a script execution relation. The installation description has a local risk score of 0.055. The final maliciousness probability is 0.364, yielding a benign prediction. The two cases share the same type of script execution relation. Their different semantic scores illustrate how SBR combines ML predicates and behavioral relations to distinguish these invocations.W2 & Q2. Comparison with a text classifier under the same training supervision
Thank you for suggesting a comparison under matched supervision. Following your suggestion, we have added a comparison with DeBERTa fine-tuned on the same training data and skill labels as SBR. We found that SBR still achieves better overall detection performance than fine-tuned DeBERTa under the configurations evaluated here.
We initialized DeBERTa from the checkpoint which is used by LLM Guard. We then fine-tuned it on the same MalSkillBench training split and skill labels as SBR. We evaluated both methods on the same 1,329 test skills. Both methods use information from
SKILL.mdand the script filenames referenced in it. We retained LLM Guard's input settings after fine-tuning.All values below are percentages.
SBR achieves an F1 score of 87.17% and an accuracy of 90.14%. These are 25.35 and 12.26 percentage points higher than those of fine-tuned DeBERTa, respectively. The results show that SBR retains an overall performance advantage under these detector configurations with matched training data and label supervision. We have added this experiment and analysis in Table 13 of Appendix A.9.