- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SkillDRE, a framework for designing malicious skills, which circumvent existing scans, but still result in the execution of harmful actions. The method works in two phases. First, a generator proposes and iteratively refines skills which evade pre-execution scans. In the second stage, the skill is executed, evaluated under SkillSonar, and again iteratively refined until a skill is found which produces the attack target, while evading both scans. The method is then evaluated under a set of models and on Codex.
Critical strengths
- The method achieves quite strong results on all tested models, and successfully evades detection.
- The two-stage optimisation is quite intuitive, first filtering out skills which anyway have no chance of being installed, and then optimising the remaining ones, to achieve the target action, while still evading runtime detection.
Critical weaknesses
- The comparison to baselines seems quite unfair. SkillJect doesn't consider any defense during optimisation, while SkillHarm considers a generic one. In contrast, SkillDRE optimises the skill under the exact same defenses it is later evaluated. The evaluation should equip the baselines with the same access to defenses, as well as evaluate the transferability of the skills discovered by SkillDRE to other defenses.
- The method seems not very general, as it assumes a specific defense (namely SkillSonar). At the moment, it is not clear if the methodology generalises to other defense methods. If it does, the paper should state the method more generally, and evaluate the impact of defense method during optimisation.
Questions for authors
- How exactly is the sandbox environment (line 150) constructed, and how did you ensure that it is consistent with real-world deployments?
- How exactly does the local validation of the judge work (line 222)?
- Could the specific defenses used during optimisation be exchanged for others?
- Did you account for the variance in the agent's behavior, i.e. whether a successful skill was consistently reproducing the target action?
2 comments
Q3:
As stated in the Abstract, lines 29-31, and Sec. 1, lines 60-69 and 75-76, our motivation is to investigate how defense feedback can serve as a learning signal for the iterative evolution of malicious skills. Our experiments instantiate this question using a fixed pre-execution and runtime defense pipeline (Sec. 3.1, lines 159-161).
SkillScan and SkillSonar are the concrete defenses selected to study this feedback interaction. In principle, other defenses that provide usable feedback can also guide iterative skill optimization, with their outputs adapted to the pre-execution acceptance criteria and runtime-feedback interfaces described in Sec. 3.3, lines 245-247 and 265-269, and Sec. 3.4, lines 279-290. The selected defenses remain fixed throughout an optimization run, while alternative defenses could be used in another instantiation.
Our choices also reflect practical requirements. Large-scale evolution requires many repeated defense evaluations, motivating a pre-execution scanner that supports reproducible offline deployment. SkillScan was selected for its open-source implementation and configurable LLM backend. As of now, SkillSonar was the only publicly available runtime defense implemented as an in-agent skill. These considerations motivated our use of SkillScan and SkillSonar as the two defense stages.
Q4:
At every iteration, SkillsBench creates a fresh task environment and conducts a new execution, with no execution state carried over from previous iterations, consistent with the environment-reset procedure in Sec. 3.4, lines 289-290 and Algorithm 1. Upon failure, the skill is refined and must pass Phase 1 rescanning before the next execution. Evolution terminates immediately upon success, with a maximum of ten runtime trials. Accordingly, Fig. 5 reports cumulative ASR across optimization rounds, counting each task–skill pair once when an evolved candidate first satisfies the attack-success criterion. Thus, each task–skill pair’s reported final attack outcome is obtained from an execution in a fresh, isolated SkillsBench Docker environment, with no execution state carried over from earlier rounds.
However, to assess stability, we evaluated the final selected skills across three independent executions in SkillsBench environments. Attack-success outcomes differed for at most two task–skill pairs across the executions, corresponding to an ASR variation of at most 0.80 percentage points, demonstrating stable aggregate attack effectiveness.
We thank the reviewer for the comments. We address the weaknesses and questions below.
W1 and W2:
Table 2 compares complete attack methods under a common defended execution environment, while preserving their original protocols (Sec. 4.1). SkillJect uses its original five-iteration execution-trace-guided refinement, while SkillHarm uses its publicly released, previously refined attack skills. Joint iteration driven by pre-execution scanner feedback and runtime-defense feedback is a central methodological contribution of SkillDRE. The compared baselines do not jointly optimize scanner acceptance and runtime attack realization, which is a limitation of their original designs. Providing additional defense feedback to the baselines would produce defense-aware variants of those methods. Our main comparison evaluates the original methods under their respective designs and the same defended execution conditions.
As stated in the Abstract, lines 29-31, and Sec. 1, lines 60-69 and 75-76, our motivation is to investigate how defense feedback can serve as a learning signal for the iterative evolution of malicious skills. Our threat model concerns a fixed pre-execution and runtime defense pipeline (Sec. 3.1, lines 159-161). Generalization to unseen defenses is not the motivation or core objective of this study.
To address the requested supplementary detector checks, we evaluated the final evolved skills using static-only SkillScan and obtained 0% DR. We additionally rescanned the final selected skills using Gemini-3.7-Flash as an alternative SkillScan LLM backend. Both metrics below use all task–skill pairs. “attack success ∧ unsafe” denotes attacks that succeeded in the original runtime evaluation and whose final skills were subsequently flagged unsafe by the alternative scanner.
This supplementary evaluation assesses generalization across scanner backends, while our main study concerns feedback-driven skill evolution against a fixed defense pipeline.
SkillScan was selected for its open-source implementation and configurable LLM backend, enabling reproducible offline scanning. As of now, SkillSonar was the only publicly available runtime defense implemented as an in-agent skill.
Q1:
As specified in Sec. 4.1, our experiments directly adopt the original SkillsBench task environments and Docker-based sandboxes, without modifying the benchmark's environment configurations or introducing additional simulated environments. All methods use the same execution setup, with Codex as the agent execution harness, natively supported by SkillsBench. As specified in Sec. 3.4, lines 289-290, each runtime trial starts from the same task-specific initial sandbox state. The victim agent executes actual skills and tool operations within Docker, rather than simulated interactions. This preserves the original SkillsBench execution setting and provides a controlled, reproducible evaluation of real agent-skill execution behavior.
Q2:
As detailed in Sec. 3.2, Eqs. (5)–(6), local judge validation consists of two deterministic checks. Valid(J) checks the JSON rule schema, supported evidence checks and parameters, and well-formed decision operators and check references. Only checks designated as attack evidence may contribute to the attack decision; ordinary task-completion checks cannot determine attack success.
Smoke(J) evaluates the rule in a newly created empty temporary sandbox. A candidate is rejected if it reports attack success without attack evidence or produces a hard evaluation error. Candidates failing either local check do not proceed to semantic review.
The remaining candidates are independently reviewed by the validator ensemble for target alignment, evidence observability, and resistance to false positives and false negatives. Acceptance requires both local checks and unanimous approval from all configured validators. Rejection diagnostics guide subsequent revisions, and the accepted judge remains fixed throughout skill evolution.