acceptodds
Under review as a conference paper at ICLR 2027

From Declared Capabilities to Behavior: Detecting Malicious Agent Skills Through Consistency Analysis

Abstract

A skill is a natural language document, often bundled with scripts, that an LLM agent loads on demand. Malicious skills can abuse an agent’s privileges to exfiltrate sensitive information, compromise the host environment, and influence subsequent agent actions. Detecting such skills before deployment is therefore critical. To enable scalable security vetting, researchers have explored both dynamic and static analysis. Dynamic analysis requires executing potentially relevant behaviors, resulting in substantial time and token costs that make it difficult to scale to large skill collections. Static analysis is more scalable because it can identify suspicious operations without execution, but it often produces false positives because security sensitive operations may also be necessary for benign skills. To improve detection, we compare a skill’s declared capabilities with its actual behavior. SKILL.md specifies the declared capabilities, while our analysis identifies security-sensitive behaviors across code, files, and document instructions. Building on this comparison, we introduce a declaration-behavior consistency analysis that detects mismatches between declared and actual capabilities and reports the associated source, sink, and declaration. We evaluate our method on 5,520 skills and show that it outperforms five static-analysis baselines, achieving an F1 score of 0.946 and an average precision of 0.984. It also maintains 0.86 recall under the strongest evasion family and 0.90 macro-averaged recall on previously unseen attack types.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Agent security, Agent skills, Malicious skill detection, Static analysis, declaration-behavior gap

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TE2uOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper detects malicious skills by comparing a skill’s declared capabilities (from SKILL.md) against its actual behavior. It builds a progressive-disclosure graph rooted at SKILL.md, reconstructs security-sensitive information flows across files and modalities, and scores each flow by a declaration-compatibility gap plus a covert-intent signal, aggregated via soft-OR. On SkillTrustBench (5,520 skills) it reaches F1 0.946 and AP 0.984.

Critical strengths
  1. Sharp, well-motivated reframing. Moving the decision from “which capabilities a skill has” to “which dangerous flows are undeclared” is a genuinely useful shift. It directly explains why benign power-tools (which legitimately read env vars and make network calls) are over-flagged by capability scanners, and the taint-vs-co-occurrence distinction (Fig. 5) is clearly argued.
  2. Strong, auditable design. Tracing each warning to a concrete source, sink, and declaration makes the output inspectable for human reviewers — a real advantage for pre-deployment vetting. The conservative, rule-based compatibility coefficients (rather than learned similarity) and verbatim-excerpt requirement for declaration extraction are sensible defenses against gaming.
  3. Thorough, honest evaluation. Five paradigm baselines, a six-way ablation, a τ/w_c sensitivity sweep, six evasion families, and a leave-one-attack-type-out protocol. The LOATO result (0.90 macro-recall held-out vs 0.934 overall) is the paper’s best evidence that detection rests on a structural signal rather than threshold overfitting. The w/o-LLM ablation (still F1 0.823) cleanly shows the gain isn’t just the LLM.
Critical weaknesses
  1. Single-benchmark, single-vendor evaluation. All headline numbers come from one benchmark (SkillTrustBench), and the LLM backend is a single model (Qwen-plus). There’s no cross-benchmark check against MalSkillBench or MASB, which the other submissions use, so generalization beyond this one dataset is unverified.
  2. The evasion test uses a synthetic corpus, not the benchmark. Robustness (RQ4) is measured on a self-constructed adversarial set with recall anchored at 92/94% — a different and seemingly easier operating point than the 0.934 main result. Synonym rewriting still costs 16%/8%, and adaptive attackers who co-design the SKILL.md to look consistent aren’t tested at all.
  3. Several hand-tuned design choices. The severity weights, class-specific caps (Table 6), b_x=0.9 exfiltration bump, and w_c=0.85 are fixed by hand. The sensitivity sweep covers only τ and w_c, not the severity/compatibility tables, so robustness to the many other magic numbers is unestablished.
  4. Analyzer-injection risk is acknowledged but not evaluated. Since SKILL.md is fully attacker-controlled and feeds an LLM, prompt injection against the analyzer is a live threat. The paper offers verbatim-evidence as mitigation but provides no experiment showing it actually resists a crafted injection.
Questions for authors

see weekness

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3

Thank you for the thorough review.

Q1: One benchmark, one LLM. We added MalSkillBench and evaluated our method on it without retuning after removing two sources of label leakage. We also added DeepSeek-V4-Flash as a second LLM backend. The results are reported in Table 1 of the revised paper. Compared with SkillTrustBench, the performance on MalSkillBench decreases primarily in recall, while precision remains the highest among all evaluated systems.

Complete detectors vs. our method on SkillTrustBench and MalSkillBench . MethodPRF1APSkillTrustBenchSkillSieve0.8640.5770.6920.855Cisco Scanner0.8220.9520.8820.914Ours (Qwen-Plus)0.9570.9340.9460.984Ours (DeepSeek)0.9780.9800.9790.983MalSkillBenchSkillSieve0.6500.3560.4600.584Cisco Scanner0.7630.9140.8320.866Ours (Qwen-Plus)0.8540.6920.7650.753Ours (DeepSeek)0.8800.7190.7920.773\begin{array}{lcccc} \hline \text{Method} & \text{P} & \text{R} & \text{F1} & \text{AP} \\\\ \hline \textit{SkillTrustBench} & & & & \\\\ \text{SkillSieve} & 0.864 & 0.577 & 0.692 & 0.855 \\\\ \text{Cisco Scanner} & 0.822 & 0.952 & 0.882 & 0.914 \\\\ \text{Ours (Qwen-Plus)} & 0.957 & 0.934 & 0.946 & \mathbf{0.984} \\\\ \text{Ours (DeepSeek)} & \mathbf{0.978} & \mathbf{0.980} & \mathbf{0.979} & 0.983 \\\\ \textit{MalSkillBench} & & & & \\\\ \text{SkillSieve} & 0.650 & 0.356 & 0.460 & 0.584 \\\\ \text{Cisco Scanner} & 0.763 & \mathbf{0.914} & \mathbf{0.832} & \mathbf{0.866} \\\\ \text{Ours (Qwen-Plus)} & 0.854 & 0.692 & 0.765 & 0.753 \\\\ \text{Ours (DeepSeek)} & \mathbf{0.880} & 0.719 & 0.792 & 0.773 \\\\ \hline \end{array} Revision: Sec. 4.1, Table 1, and Sec. 4.2 (results), App. J (MalSkillBench and label-leakage removal), App. I (LLM backends), Sec. 5 (limitation).

Q2: Synthetic evasion set and adaptive attackers. We agree that the original evasion set was limited because it was synthetic and contained no benign skills. The new attack experiment addresses both limitations by using 100 malicious and 100 benign skills from SkillTrustBench and evaluating attackers that rewrite SKILL.md, including white-box attackers with knowledge of our static analysis and the output schema of our extractor. Declaring malicious behavior does not evade detection. Instead, recall increases from 0.84 to 0.96 when all malicious capabilities are declared and to 0.88 under the white-box setting. Among the three injections targeting the analyzer, only the hidden instruction reduces recall, from 0.84 to 0.80. Combining this attack with white-box declarations causes no additional loss.

Adaptive attacks on 100 malicious SkillTrustBench skills AttackRecallΔRFlippedUnmodified0.84––Declare all capabilities0.96+0.120Declare detected flows (white-box)0.88+0.041Authority claim0.85+0.016Forged analysis result (white-box)0.90+0.060Hidden instruction to the analyzer0.80−0.048Declared flows + hidden instruction0.840.004\begin{array}{lccc} \hline \text{Attack} & \text{Recall} & \Delta\text{R} & \text{Flipped} \\\\ \hline \text{Unmodified} & 0.84 & \text{–} & \text{–} \\\\ \text{Declare all capabilities} & 0.96 & +0.12 & 0 \\\\ \text{Declare detected flows (white-box)} & 0.88 & +0.04 & 1 \\\\ \text{Authority claim} & 0.85 & +0.01 & 6 \\\\ \text{Forged analysis result (white-box)} & 0.90 & +0.06 & 0 \\\\ \text{Hidden instruction to the analyzer} & 0.80 & -0.04 & 8 \\\\ \text{Declared flows + hidden instruction} & 0.84 & 0.00 & 4 \\\\ \hline \end{array} Revision: Sec. 4.5, paragraph “Adaptive attacks” and Table 3; App. M (Table 12). The synthetic evasion results (Table 13) are moved to App. N and marked as synthetic in App. C and Sec. 5.

Q3: Hand-set constants. We perturbed all constants in Table 6, namely the severity weights wκfw_{\kappa_f} and the declaration-compatibility constants dx(f)d_x(f), as well as the contradiction term bxb_x in Eq. 2. Specifically, we varied each severity weight by ±30% or set all severity weights equal, varied each compatibility constant by ±50%, and varied bxb_x over [0,1][0, 1]. F1 remains between 0.940 and 0.948, while AP remains between 0.973 and 0.986. This stability arises from two properties of the detector: the constants only scale the contributions of recovered flows, which occur in 18% of the evaluation set, and the bimodal score distribution keeps most skills far from τ\tau. Consequently, no perturbation changes F1 by more than 0.006.

Revision: Sec. 4.4 (last part); App. K (Table 11 and the accompanying note).

Q4: Analyzer injection. We now evaluate three injection attacks against the analyzer: a claim that the skill passed a security review, a forged analysis result that follows the output schema of our extractor (white-box), and an HTML comment instructing AI reviewers to classify the skill as benign. As shown in the adaptive-attack table of our previous comment (Q2), neither the claim nor the forged result reduces recall. The hidden instruction is the only effective attack, it lowers the LLM covert score and reduces recall from 0.84 to 0.80, and combining it with white-box declarations causes no further loss. Requiring a verbatim excerpt for each declared capability removes forged declarations but does not counter this attack. We report it as a limitation and leave hardening the covert-intent judgment to future work.

Revision: Sec. 4.5 and Table 3, App. M (Table 12), App. C .

We have submitted our revised manuscript and sincerely look forward to your comments. We greatly appreciate your time and constructive feedback. Should you have any questions regarding our revisions, please feel free to contact us,we would be more than happy to discuss them with you and make further revisions as needed.

Dear Reviewer,

We hope this message finds you well. As the discussion period draws to a close, we would like to check whether our responses have adequately addressed your concerns. We added MalSkillBench as a second benchmark, evaluated without retuning, and DeepSeek-V4-Flash as a second LLM backend, on MalSkillBench our method keeps the highest precision among all systems. We replaced the synthetic evasion study with adaptive attacks on SkillTrustBench that include benign skills and white-box attackers, and we directly evaluated injection against the analyzer, reporting the hidden-instruction attack as a limitation. Finally, perturbing all hand-set constants keeps F1 between 0.940 and 0.948.

If any questions remain, or if further clarification would be helpful, we would be glad to hear your thoughts before the discussion ends.

Thank you for your careful reading and constructive feedback.

Thanks for the rebuttal. I update the score to 3. However, I still think the evaluation of this paper is weak.

Reviewer_HvLvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes a method for checking the risks of Agent's skills before installation. This method first reads the purposes and capabilities stated in SKILL.md, and then conducts static analysis on the operations in the code, documentation, and associated files to check for inconsistencies between the declared and implemented aspects, such as claiming to only handle local files but sending sensitive information to the outside. At the same time, it also uses large models to identify hidden malicious instructions and ultimately provides a risk score and corresponding evidence.

Critical strengths
  1. The detection approach has practical value. Normal tools may also read files, connect to the network, or execute commands. By combining functional declarations and information flow analysis, it can help distinguish normal requirements from hidden malicious behaviors; providing corresponding evidence also facilitates manual review.

  2. An assessment was conducted on 5,520 skills, including component ablation, attack avoidance, and attack category exclusion tests, which provided certain support for the effectiveness of the method.

Critical weaknesses
  1. The author explains that the extraction of statements and the detection of concealed intentions are combined into a single model call, while other analyses are completed on the CPU and use caching. However, no token consumption, scanning time, API costs, and throughput of each skill were reported, so it is impossible to determine the actual cost of the first scan and whether it is more efficient compared to the baseline. Caching can only reduce the cost of repeated scans.

  2. Attackers can both modify the code and the SKILL.md. Whether they can bypass by expanding the capability statements, packaging dangerous operations as normal requirements, or injecting instructions into the detection model, the existing experiments have not answered this clearly. Although the author set conservative scoring rules, tests such as renaming, splitting, and synonym rewriting are not sufficient to verify this.

  3. The main comparison is between the "paradigm baseline" implemented by the author for capability detection, regular matching, and single-round large model judgment, and no sufficient comparison was made with the complete existing scanning system. Therefore, the proof of actual performance advantages is still limited.

Questions for authors
  1. Report the time consumption, tokens, costs during the first scan and cache hit, as well as the respective expenses of static analysis and model invocation, and compare them with the baseline under the same conditions.

  2. First, standardize the sample and indicator standards of each table, then complete the scanner comparison, as well as the tests for malicious modification declarations and prompt injection for the analyzer.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2

Thank you for the precise and actionable questions.

Q1: First-scan cost. We measured the cost of first scans with all caches disabled, using the same machine and identical API pricing. For every 1,000 skills in SkillTrustBench, our detector makes one LLM call per skill, consumes 5.4× fewer tokens than SkillSieve and 4.2× fewer than Cisco, incurs only 15% and 24% of the costs of SkillSieve and Cisco, respectively, and requires no parallel requests.

First-scan cost per 1,000 SkillTrustBench skills MethodParallelTime (h)Tokens (M)Cost (CNY)SkillSieve40.9712.0313.58Cisco Scanner85.309.358.73Ours11.712.232.08\begin{array}{llcccc} \hline \text{Method} & \text{Parallel} & \text{Time (h)} & \text{Tokens (M)} & \text{Cost (CNY)} \\\\ \hline \text{SkillSieve} & 4 & 0.97 & 12.03 & 13.58 \\\\ \text{Cisco Scanner} & 8 & 5.30 & 9.35 & 8.73 \\\\ \text{Ours} & 1 & 1.71 & \mathbf{2.23} & \mathbf{2.08} \\\\ \hline \end{array}

Revision: Sec. 4.7 (new RQ6), Table 7 in App. H, Reproducibility Statement.

Q2: Adaptive attackers. We added six attacks that modify only SKILL.md for 100 malicious skills from SkillTrustBench, while leaving the code unchanged. We also included 100 benign skills to measure false positives.

• Declaration inflation does not help the attacker. Disclosing malicious behavior increases recall from 0.84 to 0.96. Even a white-box attacker that declares exactly the flows detected by our method achieves a recall of only 0.88, with 1 previously detected skill missed and 5 previously missed skills detected. Although a fully declared flow alone receives a score of at most 0.38, which is below τ (Table 6), a disclosure that is inconsistent with the stated purpose of the skill is detected by the contradiction criterion and by flow extraction from the disclosure text. Thus, declaring malicious behavior does not provide an effective means of hiding it.

• Injection against the analyzer has limited effect. Neither a claimed security review nor a forged analysis result following our output schema reduces recall. Only a hidden instruction to the analyzer reduces recall, by four percentage points, from 0.84 to 0.80. Combining this attack with white-box declarations provides no additional benefit to the attacker, with recall remaining at 0.84. We report this attack as a limitation.

Adaptive attacks on 100 malicious SkillTrustBench skills AttackRecallΔRFlippedUnmodified0.84––Declare all capabilities0.96+0.120Declare detected flows (white-box)0.88+0.041Authority claim0.85+0.016Forged analysis result (white-box)0.90+0.060Hidden instruction to the analyzer0.80−0.048Declared flows + hidden instruction0.840.004\begin{array}{lccc} \hline \text{Attack} & \text{Recall} & \Delta\text{R} & \text{Flipped} \\\\ \hline \text{Unmodified} & 0.84 & \text{–} & \text{–} \\\\ \text{Declare all capabilities} & 0.96 & +0.12 & 0 \\\\ \text{Declare detected flows (white-box)} & 0.88 & +0.04 & 1 \\\\ \text{Authority claim} & 0.85 & +0.01 & 6 \\\\ \text{Forged analysis result (white-box)} & 0.90 & +0.06 & 0 \\\\ \text{Hidden instruction to the analyzer} & 0.80 & -0.04 & 8 \\\\ \text{Declared flows + hidden instruction} & 0.84 & 0.00 & 4 \\\\ \hline \end{array}

Revision: Sec. 4.5, paragraph “Adaptive attacks” and Table 3; App. M (Table 12, per-attack details); App. C and Sec. 5 (limitation).

Q3: Complete scanners. We now run two complete systems end to end, SkillSieve and the Cisco AI Defense Skill Scanner, on the same skills and with the same evaluation script. On SkillTrustBench, our detector improves F1 from 0.882 (Cisco) to 0.979 and precision from 0.864 (SkillSieve) to 0.978. On MalSkillBench, it keeps the highest precision (0.880 vs. 0.763 for Cisco), while Cisco has higher recall, F1, and AP because it flags more skills of both classes.

Complete detectors vs. our method on SkillTrustBench and MalSkillBench . MethodPRF1APSkillTrustBenchSkillSieve0.8640.5770.6920.855Cisco Scanner0.8220.9520.8820.914Ours (Qwen-Plus)0.9570.9340.9460.984Ours (DeepSeek)0.9780.9800.9790.983MalSkillBenchSkillSieve0.6500.3560.4600.584Cisco Scanner0.7630.9140.8320.866Ours (Qwen-Plus)0.8540.6920.7650.753Ours (DeepSeek)0.8800.7190.7920.773\begin{array}{lcccc} \hline \text{Method} & \text{P} & \text{R} & \text{F1} & \text{AP} \\\\ \hline \textit{SkillTrustBench} & & & & \\\\ \text{SkillSieve} & 0.864 & 0.577 & 0.692 & 0.855 \\\\ \text{Cisco Scanner} & 0.822 & 0.952 & 0.882 & 0.914 \\\\ \text{Ours (Qwen-Plus)} & 0.957 & 0.934 & 0.946 & \mathbf{0.984} \\\\ \text{Ours (DeepSeek)} & \mathbf{0.978} & \mathbf{0.980} & \mathbf{0.979} & 0.983 \\\\ \textit{MalSkillBench} & & & & \\\\ \text{SkillSieve} & 0.650 & 0.356 & 0.460 & 0.584 \\\\ \text{Cisco Scanner} & 0.763 & \mathbf{0.914} & \mathbf{0.832} & \mathbf{0.866} \\\\ \text{Ours (Qwen-Plus)} & 0.854 & 0.692 & 0.765 & 0.753 \\\\ \text{Ours (DeepSeek)} & \mathbf{0.880} & 0.719 & 0.792 & 0.773 \\\\ \hline \end{array} Revision: Sec. 4.1–4.2; Table 1, Fig. 2, App. H (reproduction details).

Q4: Consistent tables. Each table now specifies the sample set and reports P, R, F1, and AP. All systems are evaluated on the same set of skills using the same script. This revision also led us to correct the description of the sample set in Sec. 4.1.

Revision: Sec. 4.1, captions of Tables 1, 3, and 11.

We have submitted our revised manuscript and sincerely look forward to your comments. We greatly appreciate your time and constructive feedback. Should you have any questions regarding our revisions, please feel free to contact us,we would be more than happy to discuss them with you and make further revisions as needed.

Dear Reviewer,

We hope this message finds you well. As the discussion period draws to a close, we would like to check whether our responses have adequately addressed your concerns. We measured first-scan cost with all caches disabled: per 1,000 skills, our method uses 2.23M tokens at CNY 2.08, which is 15% and 24% of the API cost of SkillSieve and Cisco, respectively. We added six adaptive attacks that rewrite only SKILL.md. Declaring the malicious behavior raises recall from 0.84 to 0.96 rather than evading detection, and the only effective attack, a hidden instruction to the analyzer, lowers recall to 0.80, which we now report as a limitation. We also compared against complete scanners end to end and unified the sample sets and metrics across all tables.

If any questions remain, or if further clarification would be helpful, we would be glad to hear your thoughts before the discussion ends.

Thank you for your careful reading and constructive feedback.

Reviewer_hyrAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a static approach to malicious skill detection, which uses a prompted LLM to analyze a skill, construct a graph and potentially dangerous flows, extracts a structured set of labels for each flow, combines the labels into a score for each flow, then combines these into the final risk score. A main focus of the work is that it aims to detect the 'declaration-behavior gap', i.e. using the skill's stated purpose in the SKILL.md as reference frame to detect malicious behaviors that deviates from this stated purpose.

Critical strengths

Method seems generally well designed, combining a graph based flow extraction approach, and scoring flows based on deviation from declaration, taint analysis, etc. The idea of deviation from declaration seems significant. The method also does not require training an additional classifier, making it a simple approach that avoids some concerns of overfitting. Additionally the method shows very good performance compared to the baselines.

Critical weaknesses

Baselines are a bit unclearly presented or possibly insufficient: the related work describes a number of published skill scan approaches such as SkillSieve, SkillProbe and SkillScan. Hence, it seems that the method should be compared against the full implementation of these methods (assuming they are static analysis approaches). The presented baselines are somewhat unclear; they appear to be representatives of various detection strategies but not the full published methods themselves. However, I am open to reconsider if the authors could explain how these baselines related to the published ones.

Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1

Thank you for the positive assessment of the design and for being open to reconsidering.

Q1: Baselines vs. published systems. The five baselines in Table 1 serve as paradigm-level controls, each isolating a single decision rule. We agree that these baselines do not substitute for complete systems. Therefore, we now evaluate two complete systems end to end on the same set of skills.

• On SkillTrustBench, our detector improves F1 from 0.882 (Cisco) to 0.979.

• On MalSkillBench, it keeps the highest precision, while Cisco has higher recall, F1 and AP because it flags more skills of both classes.

• SkillProbe and SEFZ rely on simulation or runtime fuzzing, which is outside static pre-deployment analysis, we discuss them in Sec. 2.

Complete detectors vs. our method on SkillTrustBench and MalSkillBench .

\hline \text{Method} & \text{P} & \text{R} & \text{F1} & \text{AP} \\\\ \hline \text{SkillTrustBench:} & & & & \\\\ \text{SkillSieve} & 0.864 & 0.577 & 0.692 & 0.855 \\\\ \text{Cisco Scanner} & 0.822 & 0.952 & 0.882 & 0.914 \\\\ \text{Ours (Qwen-Plus)} & 0.957 & 0.934 & 0.946 & \mathbf{0.984} \\\\ \text{Ours (DeepSeek)} & \mathbf{0.978} & \mathbf{0.980} & \mathbf{0.979} & 0.983 \\\\ \text{MalSkillBench:} & & & & \\\\ \text{SkillSieve} & 0.650 & 0.356 & 0.460 & 0.584 \\\\ \text{Cisco Scanner} & 0.763 & \mathbf{0.914} & \mathbf{0.832} & \mathbf{0.866} \\\\ \text{Ours (Qwen-Plus)} & 0.854 & 0.692 & 0.765 & 0.753 \\\\ \text{Ours (DeepSeek)} & \mathbf{0.880} & 0.719 & 0.792 & 0.773 \\\\ \hline \end{array}$$ Revision: Sec. 4.1 (complete detectors), Table 1 and Fig. 2 (results on both benchmarks), Sec. 4.2 (discussion), App. H (reproduction details). Clarification of the summary. The disclosure graph and flows are derived from AST-based taint analysis and rules rather than from an LLM. The LLM is invoked only once per skill to identify declared capabilities and covert intent. Revision: Sec. 3, Overview (last sentence).

We have submitted our revised manuscript and sincerely look forward to your comments. We greatly appreciate your time and constructive feedback. Should you have any questions regarding our revisions, please feel free to contact us,we would be more than happy to discuss them with you and make further revisions as needed.

Dear Reviewer,

We hope this message finds you well. As the discussion period draws to a close, we would like to check whether our responses have adequately addressed your concerns. Following your suggestion, we now evaluate two complete, published detectors, SkillSieve and the Cisco AI Defense Skill Scanner, end to end on the same skills as our method, on both SkillTrustBench and MalSkillBench. On SkillTrustBench, our method improves F1 from 0.882 (Cisco) to 0.979, and it achieves the highest precision on both benchmarks. We also clarified that the disclosure graph and all flows are derived from AST-based taint analysis and rules, and that the LLM is invoked only once per skill.

If any questions remain, or if further clarification would be helpful, we would be glad to hear your thoughts before the discussion ends.

Thank you for your careful reading and constructive feedback.

I thank the authors for the clarifications and added experiments. The new experiments (including performance and efficiency results against full baselines) are indeed helpful. The lower recall / F1 / AP on MalSkillBench are a bit of an issue, but overall still acceptable since 1) the results are clearly better on SkillTrustBench, and 2) the method is significantly more efficient, about 4x less tokens, 3) precision is higher, which determines whether the method's flagged alarms are worth investigating. The adaptive and hyperparameter results are also helpful. Overall, my issues have been addressed, and I raise my score to 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their constructive feedback. Below, we summarize the main additions. Further details are provided in the individual responses, and all changes are highlighted in red in the revised PDF. The abstract, introduction, and conclusion are updated accordingly.

  1. Complete systems (R1, R2). We evaluated SkillSieve and the Cisco AI Defense Skill Scanner in an end-to-end setting. Our detector achieves the highest precision on both benchmarks, as well as the highest F1 score and AP on SkillTrustBench. [Sec. 4.1–4.2; Table 1; Fig. 2; App. H]
  2. Second benchmark and second LLM (R3). We added MalSkillBench and evaluated our method on it without retuning. We also added a second LLM backend (DeepSeek-V4-Flash). [Sec. 4.1; Table 1; App. I–J]
  3. First-scan cost (R2). For every 1,000 skills, our detector consumes 2.23M tokens at a cost of CNY 2.08, compared with 12.03M tokens and CNY 13.58 for SkillSieve and 9.35M tokens and CNY 8.73 for Cisco. [Sec. 4.7; Table 7]
  4. Adaptive attacks and analyzer injection (R2, R3). We evaluated six adaptive attacks that rewrite SKILL.md. Declaration inflation does not evade our detector, whereas injecting a hidden instruction into the analyzer reduces recall from 0.84 to 0.80. We now report this vulnerability as a limitation. [Sec. 4.5; Table 3; App. C, M]
  5. Hand-set constants (R3). Perturbing all hand-set constants keeps F1 between 0.940 and 0.948. [Sec. 4.4; Table 11]

Dear Reviewers,

Thank you for your time and thoughtful reviews. As the Author–Reviewer Discussion period is closing soon, please actively engage with the authors, consider their responses, and update your final scores and comments.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.