acceptodds
Desk rejected at ICLR 2027

CARVE: Coverage-Aware Reconciliation with Verified Evidence for Agent Skill Detection

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Agent skills combine natural-language instructions, executable code, and declared capabilities. Their attacks may target the instruction layer, the code layer, or both, creating a heterogeneous detection problem. Static scanners are safer to deploy but provide incomplete coverage when attacks fall outside their rules or observed signals. Dynamic scanners broaden behavioral coverage by executing skills, but require sandboxing, incur substantial operational cost, and remain exposed to sandbox-escape risk. Real deployments also impose confidentiality and intellectual-property constraints: sending proprietary skills to proprietary LLM may disclose sensitive content, motivating locally deployable open-weight analysis. We present CARVE (Coverage-Aware Reconciliation with Verified Evidence), a non-executing detector that separates evidence localization from judgment: code screening and evidence-guided investigation analyze implementation behavior, while independent full-text checks assess instruction injection and potential consequences. Typed signals are fused conservatively, source-linked evidence and channel provenance are retained for audit, and governance findings are reported separately. We evaluate CARVE with open-weight models spanning 7B, 72B, and approximately 744B total parameters, as well as two closed-source models. Across MalSkillBench and MaliciousSkillBench, CARVE maintains strong detection performance across different model scales. Ablations show that each of the three analysis modules contributes to detection performance, with semantic checks and evidence-guided investigation providing the largest coverage gains. A separate cost evaluation on MaliciousSkillBench shows that multi-view analysis reaches a higher-recall operating region. The results support coverage-oriented, evidence-grounded assessment for deployable open-weight models, while highlighting the remaining limits of static analysis and model-dependent judgment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, skill detctions

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_2pnKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies malicious-skill detection for agent skills (SKILL.md, bundled code, and declared permissions) and proposes CARVE, a static framework that does not execute code. It runs three checks in parallel: (1) knowledge-guided code screening; (2) evidence-guided iterative investigation, which includes adversarial review and source verification and separates findings into "attack findings" and "governance concerns"; and (3) an injection check and a consequence check on SKILL.md. The three outputs are represented as strong, weak, or abstaining signals, and a skill is flagged as malicious if there is any strong signal or at least two weak signals. On MalSkillBench and MaliciousSkillBench, using Qwen2.5-7B/72B and GLM-5.3, CARVE achieves the highest F1 in most settings; on MalSkillBench with Qwen2.5-72B, its F1 is 89.11, 21.44 points above the strongest baseline.

Critical strengths
  • The problem and deployment constraints are realistic, and the gains are clear on small and mid-sized models. The paper targets a practical need: heterogeneous artifacts combining instructions and code, no code execution, and local deployability. With Qwen2.5-72B, CARVE's F1 on the two benchmarks (89.11 and 81.32) is clearly higher than that of the strongest baselines (67.67 and 58.75), which suggests that the multi-view design can mitigate the recall collapse of single-pass LLM judgments on mid-sized models.

  • The experiments are fairly broad and include module-level analysis. They cover two benchmarks, three open-weight models, and two closed-source models, with 4 LLM-based and 6 rule-based baselines. The ablation shows that removing the semantic checks reduces recall on MaliciousSkillBench by about 38 percentage points, which is valuable evidence for the role of instruction-level semantic checking.

Critical weaknesses
  • Numbers in different sections contradict each other, and the paper does not explain this. For Qwen2.5-72B on MaliciousSkillBench, Section 4.4 reports values (CARVE recall 69.5, Cisco 42.5) that differ from Table 1 (CARVE 74.00, Cisco-LLM 31.00). For GLM-5.3, Section 4.4 states that CARVE has the highest F1 (92.0), whereas in Table 1 Sentry-LLM has higher recall and F1 (86.0, 89.82) than CARVE (84.5, 89.42).

  • The comparison and evaluation setup do not suffice to support the claim that the coverage-first architecture is better. CARVE makes more LLM calls than the baselines; the authors explicitly do not adjust for budget, and CARVE's token usage is about 1.8× that of AIG (as read from Fig. 4). Baselines are also not given an equivalent budget (multi-prompt ensembling, voting, chunked scanning, threshold sweeps), so it is impossible to tell whether the gains come from the architecture or from more calls and a recall-oriented operating point. The advantage largely disappears with the stronger GLM-5.3: on MalSkillBench CARVE is only 1.16 F1 above AIG, and on MaliciousSkillBench it is on par with or slightly below AIG and Sentry-LLM. Table 2 (closed-source models) contains no baselines at all.

  • Key design choices and selling points lack supporting evidence. The fusion rule is presented as a methodological contribution, yet it is not compared against simple alternatives such as plain OR, majority voting, or strong-signal-only. Moreover, Section 4.6 itself notes that all false negatives are cases of "complete silence from all three views," which suggests that, at least for false negatives, there is no conflict between views that needs "reconciling." Removing code screening changes F1 by only 0.25 and 0.47 points, within the noise range, yet the abstract and conclusion still claim that all three modules contribute. The ablation is run only on the strongest model, GLM-5.3, and not on Qwen2.5-72B, which is the central deployment setting. The two selling points of "auditable evidence" and "governance separation" are not quantitatively evaluated (e.g., human verification of whether the evidence supports the verdicts, or the rate of hallucinated evidence), and governance findings are illustrated with only two cases. The local-deployment argument is also weak: the best-performing GLM-5.3 has about 744B parameters, the 7B model that is easy to deploy locally performs poorly, and the 72B model still reaches only 74% recall on MaliciousSkillBench.

  • Insufficient evaluation rigor and discussion of limitations. The knowledge base is distilled from confirmed cases, but the paper does not state whether these cases are disjoint from the evaluation samples and their source datasets. It also does not state whether prompts and rules were tuned on the evaluation sets, while baselines are run with their respective documented configurations and Sentry-LLM is a version adapted by the authors. In addition, the introduction promises to discuss limitations such as dynamic behavior and model dependence, but the paper has no dedicated limitations section.

Questions for authors

See the Weaknesses section above.

Reviewer_DcccOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents CARVE, a framework for detecting malicious agent skills without executing their code. The proposed approach examines both the instructions and implementation of a skill, with an emphasis on verifying the evidence behind potentially malicious behavior. It also distinguishes actual attack findings from general security concerns. Experiments on two benchmarks demonstrate improvements over existing detection methods.

Critical strengths

1.Well-motivated problem. The paper addresses an important security problem as agent skills become more widely used. Analyzing both instructions and implementation without executing potentially malicious code is a reasonable and useful approach. The distinction between malicious behavior and general security concerns is also meaningful.

2.Well-motivated technical design and strong empirical results. The separation of different detection tasks is reasonable, and the evidence verification process provides additional support for the findings. The experiments show promising results, while the ablations support the usefulness of the main components.

Critical weaknesses
  1. Unclear contribution of the reconciliation strategy. The paper provides reasonable motivation and ablation results for the individual detection modules. However, the proposed reconciliation strategy is presented as a key methodological contribution, without a direct comparison against simpler fusion methods. It remains unclear whether this particular strategy is necessary to achieve the reported improvements.

  2. Practical effectiveness remains uncertain. While the paper evaluates false positives on benign skills, it does not directly examine the impact of these errors under realistic deployment conditions. The relatively high false-positive rate with the smaller model is particularly concerning for local deployment. It is also unclear how well the method would perform against previously unseen or adapted attacks.

Questions for authors

1.The ablation results demonstrate the value of the individual modules. However, how much of CARVE's improvement comes from the proposed reconciliation strategy itself? Would simpler fusion methods, such as majority voting, achieve similar performance?

2.The paper evaluates false positives on balanced benchmarks, but how well would CARVE preserve legitimate skills under realistic deployment conditions where malicious skills are relatively rare?

Reviewer_hoWBOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents CARVE, a non-executing detector for malicious agent skills that combines rule-guided code screening, iterative investigation of package files, and two independent checks for instruction injection and harmful consequences. It then reconciles typed signals using a strong/weak evidence rule, preserves source-linked evidence, and reports governance concerns separately from attack findings. Experiments on MalSkillBench and MaliciousSkillBench show strong recall and F1 across several model backbones, particularly compared with the tested rule-based and LLM-assisted scanners.

Critical strengths

The problem is practically important: agent skills mix natural-language instructions with executable code, and running untrusted packages during inspection is undesirable. I appreciate the separation between attack findings and potentially legitimate but sensitive capabilities, as well as the attempt to verify cited evidence rather than accepting unsupported model judgments. The evaluation covers multiple model scales, and the ablations indicate that semantic checks and iterative investigation make substantial contributions.

Critical weaknesses

My main concern is the limited methodological distinction from, and insufficient comparison with, prior work. SkillSieve already combines static triage, specialized parallel LLM analyses, and multi-model adjudication. SkillProbe explores collaborative, multi-agent auditing and cross-skill risk relationships, while Semia studies non-executing, evidence-oriented analysis through structured behavioral representations. CARVE makes different engineering choices, particularly its typed-signal fusion and separate governance channel, but the paper does not convincingly establish why these choices constitute a substantial advance over those approaches. SkillSieve is even cited but not experimentally compared.

The current baselines mainly test complete systems under their native protocols. Since CARVE uses multiple analyses, iterative exploration, and additional review steps, its improvements may partly reflect a larger inference budget rather than a better architecture. The module-level ablations do not isolate the benefits of the proposed reconciliation rule, adversarial review, or source verification; the incremental gain from code screening is also small. Moreover, the semantic checks operate on SKILL.md independently, leaving the handling of attacks that emerge only from combined instruction, code, and tool behaviors largely dependent on the investigator. The paper lacks a targeted evaluation of such compositional cases. Its weaker-model precision also suggests a nontrivial false-alarm trade-off.

Questions for authors

What is the precise methodological contribution beyond SkillSieve, SkillProbe, and Semia? Why are the distinguishing design choices necessary?

How does CARVE compare with a tool-using single agent and a collaborative multi-agent auditor under matched models, tool access, and token budgets?

What happens on attacks where individual instructions, scripts, or tool actions appear benign but their composition is malicious? Can the authors evaluate these cases explicitly?

Can the authors separately ablate typed-signal fusion, adversarial review, evidence verification, and governance separation, including their effects on false positives?

Reviewer_cqaQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper tackles the immediate research question of malicious skills as a supply-chain threat in AI agent deployment. The authors present CARVE (Coverage-Aware Reconciliation with Verified Evidence) that consist of three separate modules (code screening, package-level investigation, and semantic checks) with typed signal fusion at the end. Instantiated with powerful open-source LLMs, the proposed framework shows promise in terms of accuracy and F1 on a subset of established benchmarks compared to existing baselines. The authors also provide ablation study, error analysis, and cost analysis.

Critical strengths

The tackled problem and real and the treatment is immediate. The paper is timely in proposing an engineered defense. The structure of this paper is clear and easy to look at. The organization of experiments are relatively reasonable.

Critical weaknesses

The paper contains multiple critical weaknesses that prevent a decision other than rejection.

  • Unclear discussion of the weaknesses of existing solutions. For example, at line 49-52, the authors claim that "LLM judges...a single constrained call must parse the artifact, find relevant evidence, reason about intent, resist injected instructions, and serialize a reliable verdict." However, the paragraph begins with "Existing approaches make different compromises." And it's unclear what is the compromise here, as LLM judges are a natural, easy-to-deploy defense, with state-of-the-art agents capable of doing long-horizon reasoning of the artifacts. With GLM-5.3, single-pass AIG reaches 96.94 F1 against CARVE’s 98.10, and Sentry-LLM beats CARVE on MaliciousSkillBench. The claimed advantage therefore holds mainly for weaker backbones. Also, no single-pass baselines are run on Opus 5 or GPT-5.6, so Table 2 shows nothing comparative.
  • Poor figure presentation. It seems that authors draw figure 1 and figure 2 with AI tools without careful inspection. For example, figure 2 contains text that are twisted with typos, such as "configuration" in case B "Late-stage persistence section," which seem to be artifact of direct image generation. This raises concern about the validity and authenticity of their design and the presented case study.
  • Limited novelty. The authors claim that their proposed scheme combines code screening, package-level investigation, and semantic checks, with a typed-signal fusion at the end. But each core module already exist in daily practices that are discussed in the related work. It's unclear where the novelty of research lies in addition to combining different signals. The paper names the “reconciliation operator” as its methodological contribution. Per Appendix A, that operator is just “flag if one strong signal, or at least two weak signals,” which is essentially an OR / 2-of-3 vote. The ablation also shows code screening adds only 0.25 and 0.47 F1, which undercuts the three-module framing.
  • Critical methodological and evaluation concerns. 1) The authors propose to evaluate on MalSkillBench and MaliciousSkillBench, but only sample a small subset. The motivation for this subsampling and the details of it is absent, creating reproducibility concerns. 2) The authors use 72B Qwen model and GLM 5.3, which are hard to be deployed locally in practice. With Qwen 2.5-7B, the F1 on two datasets drop to nearly unusable 70.26 and 67.83, respectively (On MalSkillBench, that is 165 false positives out of 400 benign skills, a 41% false-alarm rate), which is lower than some baselines with stronger models, compromising the deployability of the proposed approach in practical settings. 3) Critical details about the implementation of the proposed system is absent. For example, section 3.4 describes two checks "injection check" and "consequence check" and the roles they play, but does not disclose any details about how they are implemented. The KB is distilled from “confirmed cases,” so were any of those cases drawn from the benchmarks? Appendix A defers the “exact labels and thresholds” to “shipped prompts” that aren’t in the submission. 4) Section 4.4 contradicts Table 1 on what looks like the same 400-instance MaliciousSkillBench set. For example, Qwen-72B: CARVE recall/F1 is 69.5/77.9 in section 4.4 but 74.0/81.32 in Table 1. Cisco is 42.5/56.9 vs 31.0/44.93. AIG is 42.0/53.5 vs 44.5/58.75. 5) Error-analysis counts don’t match Table 1. Section 4.6 reports GLM with 3 FN / 15 FP and 72B with 23 FN / 96 FP. Table 1 implies GLM has 12 FN / 3 FP on MalSkillBench and 31 FN / 9 FP on MaliciousSkillBench. Table 1 implies 72B has 36 FN / 53 FP and 52 FN / 16 FP. No dataset or combination could yield 3/15 or 23/96. 6) section 4.6 is missing text. It announces three false-negative categories, but the second is missing. The text jumps from “First…stated purpose.” straight to “vocabulary. Third…”. 7) Undefined terms. “Structured view” (sec 4.6), “formal E1 evaluation” (sec 4.5), and “Injection and Effects judges” (sec 4.3) are never defined. The method section names the components differently. The model is also called “GPT-5.6” in some places and “GPT-5.6 Sol” in others.

The authors should excercise care when submitting to venues like ICLR, as the submission contains numerous inconsistencies and concerns that prevent a correct reading of the technical contributions, and should undergo radical updates in order to be considered for publication.

Questions for authors

None. See above for weaknesses.

Reviewer_JaCuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors propose CARVE to detect malicious agent skills without executing their code. CARVE uses rule-guided code screening to identify suspicious operations and provide leads for an LLM investigator. The investigator follows references across package files and produces evidence-backed findings. These findings further undergo adversarial review and checks. Subsequently, separate LLM examines the full instruction text for prompt injection and potentially harmful consequences. The authors evaluate CARVE on MalSkillBench and MaliciousSkillBench using open-weight and closed-source models. The largest comparative gains occur with Qwen2.5-72B, while results with the strongest open-weight backbone are more mixed.

Critical strengths
  • The complete system delivers substantial gains with Qwen2.5-72B. F1 reaches 89.11 on MalSkillBench and 81.32 on MaliciousSkillBench, compared with 67.67 and 58.75 for the strongest same-backbone baselines.
Critical weaknesses
  • The paper identifies typed reconciliation as its main contribution. However, the ablations remove entire analysis modules rather than compare fusion rules on identical view outputs. While it establishes that additional analyses help, it does not quantify the value of the reconciliation.
  • If I understand table 1 correctly, 53 of the 400 benign skills were classified as malicious so CARVE-72B flags ~13.25% of benign MalSkillBench samples, while CARVE-7B flags ~41.25%. These rates could create substantial review overhead when malicious skills are rare. Overall, it's unclear if such approach can be practical from deployment POV.
Questions for authors
  • Can you compare CARVE with union, majority voting, and strong-signals-only decisions on the same fixed view outputs?
  • Table 1 implies a 13.25% benign false-positive rate for CARVE-72B on MalSkillBench. Can you report recall at lower false-positive rates?

Comments on the paper

1 comment

Dear Reviewers,

With the discussion period closing on November 18, please take a moment to read the authors’ responses. Please also indicate whether any further clarification or follow-up is needed.

Thank you for your time and contributions!
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.