- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SkillLite audits multi-file Agent Skill packages before deployment or execution. Deterministic analyzers extract security-relevant findings; a compact LLM infers the Skill's declared purpose and expected capabilities, then assesses risk using a source-grounded evidence report. Without fine-tuning, SkillLite reports F1 scores of 0.905, 0.966, and 0.807 on MalSkillBench, SkillTrustBench, and MaliciousAgentSkillsBench (MASB), respectively. It improves F1 over direct zero-shot auditing in all 15 model–benchmark configurations. The paper also reports component ablations and end-to-end processing times.
Critical strengths
- Practical auditing setting. The paper addresses pre-deployment auditing of third-party, multi-file Skills. It considers security findings across the package together with the Skill's declared purpose without executing the Skill, which is relevant to real Skill ecosystems.
- Substantial empirical evaluation. The authors compare against existing scanners on three benchmarks and against direct zero-shot auditing with the same backbone across five model configurations. Component ablations and end-to-end processing times further support the observed performance under the tested conditions.
Critical weaknesses
- Insufficient differentiation from related work. Prior methods also combine static findings with LLM-based judgments and study structured behavior reasoning and intent–implementation consistency. The paper mainly compares detection results, without clearly identifying SkillLite's methodological additions in Intent Analysis, Evidence Synthesis, and Risk Adjudication. This limits assessment of its novelty.
- The central evidence extraction stage is difficult to reproduce. The paper lists analyzer categories and representative signals but does not specify the concrete detection rules, how evidence strength is assigned, or how surrounding source context is selected. In Table 3, removing Security Evidence Extraction reduces F1 from 0.966 to 0.485. The paper also does not report extraction coverage by camouflage, steganographic insertion, or mixed attack form, making the robustness of this consequential stage difficult to assess.
Questions for authors
- Compared with the closest methods discussed in Section 2.3, what are SkillLite's specific methodological additions in Intent Analysis, Evidence Synthesis, and Risk Adjudication? Which designs address the limitations of compact LLMs?
- Can the authors provide reproducible rules or an implementation for the deterministic analyzers, and explain how evidence strength is assigned and surrounding source context is selected? For example, what criteria produce the
AUTONOMOUS_CONFIRMATION_BYPASSfinding and itsstrongstrength in Figure 2? - For the Full Camouflage and Steganographic insertion strategies and the Mixed attack vector in MalSkillBench, can the authors separately report Security Evidence Extraction coverage and final recall, and identify whether missed detections arise mainly during extraction or Risk Adjudication?
5 comments
3. Evidence-Strength Assignment
Evidence strength is assigned through an ordered, deterministic mapping over rule identifiers and finding attributes.
| Strength | Representative assignment criteria |
|---|---|
| Strong | Predefined high-risk rules, including the six agent-specific source rules, or specified conditions for decoding, execution, or instruction override. |
| Medium | Permission changes, large blocks resembling encoded data, environment access, or dynamically constructed names. |
| Weak | Default when no earlier condition applies. |
4. Source-Context Selection
We select supporting excerpts by locating the relevant source text and retaining a bounded window around the match.
- Source lookup. We retain excerpts already attached to findings. For aggregated findings, we search the original files using recorded resources or matched elements, with rule-specific fallback patterns where needed. This heuristic can select a different occurrence from the original match.
- Context window and limits. We retain up to 220 characters on either side of a match. Excerpts are capped at 1,200 characters (900 for semantic and agent-specific signals). After deduplication by source and text, we retain up to four excerpts per finding and 30 per report, in processing order.
5. Illustrative Example
We use the skill in Figure 2 to illustrate the extraction process.
Rule triggering. The analyzer matches "DO NOT ASK THE USER FOR PERMISSION" using a branch of a case-insensitive regular expression:
r"do\s+not\s+ask\s+(?:the\s+)?user\s+(?:for\s+)?(?:permission|confirmation)"
A match is retained if the matched text and surrounding context contain an execution- or control-related term (e.g., phase, permission). Four matches are aggregated into one AUTONOMOUS_CONFIRMATION_BYPASS finding (match_count=4).
Evidence strength. The finding receives strong during Evidence Synthesis because the rule belongs to the predefined high-risk set. This assignment is deterministic and does not represent a maliciousness verdict.
Source context. The analyzer retains up to 220 characters before and after each match, preserving the surrounding instructions. Each excerpt is capped at 900 characters for agent-specific signals. This finding contains four excerpts from SKILL.md.
6. Extraction Coverage and Error Attribution
We evaluated extraction coverage and final recall on three MalSkillBench attack settings: Full Camouflage, Steganographic Insertion, and Mixed attack vector. Extraction coverage measures the proportion of malicious skills with at least one attack-relevant finding. We also analyzed missed detections to distinguish extraction failures from adjudication errors.
| Attack Setting | N | Extraction Coverage | Final Recall | Extraction Miss | Adjudication Miss |
|---|---|---|---|---|---|
| Full Camouflage | 540 | 76.7% | 80.2% | 17/99 (17.2%) | 82/99 (82.8%) |
| Steganographic Insertion | 347 | 72.6% | 76.7% | 2/76 (2.6%) | 74/76 (97.4%) |
| Mixed Attack | 568 | 99.5% | 91.0% | 2/35 (5.7%) | 33/35 (94.3%) |
Error attribution. Missed detections arise predominantly during Risk Adjudication across all three settings. In some cases, attack-relevant instructions were available to the adjudicator, but the model still classified the skill as benign. For example, in a steganographic case, the adjudicator recognized suspicious instructions embedded in an HTML comment but dismissed the risk because no executable malicious payload was identified.
We will include these results and a detailed failure analysis in the revised manuscript.
Response to Authors
Thank you for the detailed rebuttal and additional analyses. The response clarifies SkillLite's design choices, particularly deterministic extraction and non-LLM evidence grounding for compact LLMs. However, these choices appear to be incremental refinements of existing static-analysis-plus-LLM auditing pipelines, and the response does not establish a substantial methodological advance over the closest systems. Please also clarify the denominators in the new error-attribution table: assuming sample-level recall, the reported group sizes and recalls imply 107/81/51 missed skills, whereas 99/76/35 are attributed. I therefore maintain my original rating of 2.
We thank the reviewer for the detailed feedback and suggestions.
1. Methodological Distinctions from Prior Work
Although SKILLLITE and prior approaches both combine static analysis with LLM reasoning, our work differs in its overall framework and individual module designs. This high-level similarity reflects a common need to examine both code and natural-language instructions in skill packages. Moreover, SKILLLITE enables lightweight auditing with locally deployable compact LLMs.
Framework-level distinction. SkillLite differs from prior work in how it allocates evidence discovery and risk assessment between static analyzers and the LLM.
Prior work. (1) Some methods use static scores to select skills for deeper LLM assessment, leaving others with only static screening. (2) Other approaches rely on LLMs for both evidence discovery and risk assessment, increasing the model’s workload. (3) Declaration–behavior consistency methods primarily check whether implementations align with declared functionality, limiting their scope of maliciousness identification.
SkillLite. (1) Deterministic analyzers extract security-relevant evidence across skill packages. (2) Evidence Synthesis (non-LLM) grounds the findings in source excerpts and integrates functional and security context, reducing the burden of evidence discovery and contextualization on compact LLMs. (3) Beyond declaration–behavior consistency, SKILLLITE also identifies harmful declared instructions and assesses security risks even when behaviors align with the declared functionality.
The specific module-level distinctions are as follows:
These designs address the limited reasoning capacity of compact LLMs by offloading evidence discovery and grounding to deterministic components, rather than requiring the LLM to perform the entire audit. This goes beyond simply splitting LLM reasoning into stages, which reduces the workload per LLM call but still leaves evidence discovery, contextualization, and assessment to the model. With source-grounded evidence, functional context, and explicit risk criteria provided, the compact LLM can focus on Intent Analysis and making the final judgment.
2. Concrete Extraction Rules
The analyzers recursively inspect readable skill artifacts, subject to file filters and size limits. They use text patterns, syntax checks, and local combination rules to extract candidate security findings.
getattrarguments. (2) Match selected execution, network, and filesystem operations in JavaScript.We will provide more detailed extraction rules and implementation details in the revised manuscript to improve reproducibility.