- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies malicious-skill detection for agent skills (SKILL.md, bundled code, and declared permissions) and proposes CARVE, a static framework that does not execute code. It runs three checks in parallel: (1) knowledge-guided code screening; (2) evidence-guided iterative investigation, which includes adversarial review and source verification and separates findings into "attack findings" and "governance concerns"; and (3) an injection check and a consequence check on SKILL.md. The three outputs are represented as strong, weak, or abstaining signals, and a skill is flagged as malicious if there is any strong signal or at least two weak signals. On MalSkillBench and MaliciousSkillBench, using Qwen2.5-7B/72B and GLM-5.3, CARVE achieves the highest F1 in most settings; on MalSkillBench with Qwen2.5-72B, its F1 is 89.11, 21.44 points above the strongest baseline.
Critical strengths
-
The problem and deployment constraints are realistic, and the gains are clear on small and mid-sized models. The paper targets a practical need: heterogeneous artifacts combining instructions and code, no code execution, and local deployability. With Qwen2.5-72B, CARVE's F1 on the two benchmarks (89.11 and 81.32) is clearly higher than that of the strongest baselines (67.67 and 58.75), which suggests that the multi-view design can mitigate the recall collapse of single-pass LLM judgments on mid-sized models.
-
The experiments are fairly broad and include module-level analysis. They cover two benchmarks, three open-weight models, and two closed-source models, with 4 LLM-based and 6 rule-based baselines. The ablation shows that removing the semantic checks reduces recall on MaliciousSkillBench by about 38 percentage points, which is valuable evidence for the role of instruction-level semantic checking.
Critical weaknesses
-
Numbers in different sections contradict each other, and the paper does not explain this. For Qwen2.5-72B on MaliciousSkillBench, Section 4.4 reports values (CARVE recall 69.5, Cisco 42.5) that differ from Table 1 (CARVE 74.00, Cisco-LLM 31.00). For GLM-5.3, Section 4.4 states that CARVE has the highest F1 (92.0), whereas in Table 1 Sentry-LLM has higher recall and F1 (86.0, 89.82) than CARVE (84.5, 89.42).
-
The comparison and evaluation setup do not suffice to support the claim that the coverage-first architecture is better. CARVE makes more LLM calls than the baselines; the authors explicitly do not adjust for budget, and CARVE's token usage is about 1.8× that of AIG (as read from Fig. 4). Baselines are also not given an equivalent budget (multi-prompt ensembling, voting, chunked scanning, threshold sweeps), so it is impossible to tell whether the gains come from the architecture or from more calls and a recall-oriented operating point. The advantage largely disappears with the stronger GLM-5.3: on MalSkillBench CARVE is only 1.16 F1 above AIG, and on MaliciousSkillBench it is on par with or slightly below AIG and Sentry-LLM. Table 2 (closed-source models) contains no baselines at all.
-
Key design choices and selling points lack supporting evidence. The fusion rule is presented as a methodological contribution, yet it is not compared against simple alternatives such as plain OR, majority voting, or strong-signal-only. Moreover, Section 4.6 itself notes that all false negatives are cases of "complete silence from all three views," which suggests that, at least for false negatives, there is no conflict between views that needs "reconciling." Removing code screening changes F1 by only 0.25 and 0.47 points, within the noise range, yet the abstract and conclusion still claim that all three modules contribute. The ablation is run only on the strongest model, GLM-5.3, and not on Qwen2.5-72B, which is the central deployment setting. The two selling points of "auditable evidence" and "governance separation" are not quantitatively evaluated (e.g., human verification of whether the evidence supports the verdicts, or the rate of hallucinated evidence), and governance findings are illustrated with only two cases. The local-deployment argument is also weak: the best-performing GLM-5.3 has about 744B parameters, the 7B model that is easy to deploy locally performs poorly, and the 72B model still reaches only 74% recall on MaliciousSkillBench.
-
Insufficient evaluation rigor and discussion of limitations. The knowledge base is distilled from confirmed cases, but the paper does not state whether these cases are disjoint from the evaluation samples and their source datasets. It also does not state whether prompts and rules were tuned on the evaluation sets, while baselines are run with their respective documented configurations and Sentry-LLM is a version adapted by the authors. In addition, the introduction promises to discuss limitations such as dynamic behavior and model dependence, but the paper has no dedicated limitations section.
Questions for authors
See the Weaknesses section above.
Dear Reviewers,
With the discussion period closing on November 18, please take a moment to read the authors’ responses. Please also indicate whether any further clarification or follow-up is needed.
Thank you for your time and contributions!
AC