acceptodds
Under review as a conference paper at ICLR 2027

SkillDRE: Dual-Stage Red-Team Evolution of Agent Skills via Pre-Execution and Runtime Feedback

20 views

Abstract

Agent skills package instructions, executable code, and task-specific resources into reusable artifacts that agents can improve using execution feedback. The same mechanism also enables attackers to evolve malicious skills, making them more effective and less detectable. However, a candidate skill may pass pre-execution scanning yet fail to realize its target under runtime defenses, while a revision that repairs execution may introduce new scanner findings. We introduce SkillDRE, a fully automated framework for evolving complete malicious skill packages through a dual-stage feedback loop. Given a benign task and its associated skills, SkillDRE autonomously constructs and validates a task-conditioned malicious objective and a verifiable judge rule. It then holds both fixed while evolving the skill implementation, with preservation of legitimate task capability. SkillDRE combines scanner-guided evolution with runtime-guided refinement informed by execution outcomes observed under runtime defense. Each runtime-guided revision returns to the pre-execution stage for rescanning and further optimization before re-execution, forming a cross-stage closed loop. Evaluated on SkillsBench across four victim models, SkillDRE achieves an average attack success rate of 45.28%, exceeding the strongest baseline by 40.3%, while its final submitted skills receive no SkillScan findings and largely preserve benign-task performance. These results show that two-stage defense feedback can serve as a useful learning signal for adaptive red teaming and that evaluating either defense stage in isolation can miss the resulting attack capability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent, Large Language Models, Red Teaming, Self Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 6 reviews, above 27% of the others.
Reviewer_TVikOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SkillDRE, a framework for designing malicious skills, which circumvent existing scans, but still result in the execution of harmful actions. The method works in two phases. First, a generator proposes and iteratively refines skills which evade pre-execution scans. In the second stage, the skill is executed, evaluated under SkillSonar, and again iteratively refined until a skill is found which produces the attack target, while evading both scans. The method is then evaluated under a set of models and on Codex.

Critical strengths
  • The method achieves quite strong results on all tested models, and successfully evades detection.
  • The two-stage optimisation is quite intuitive, first filtering out skills which anyway have no chance of being installed, and then optimising the remaining ones, to achieve the target action, while still evading runtime detection.
Critical weaknesses
  • The comparison to baselines seems quite unfair. SkillJect doesn't consider any defense during optimisation, while SkillHarm considers a generic one. In contrast, SkillDRE optimises the skill under the exact same defenses it is later evaluated. The evaluation should equip the baselines with the same access to defenses, as well as evaluate the transferability of the skills discovered by SkillDRE to other defenses.
  • The method seems not very general, as it assumes a specific defense (namely SkillSonar). At the moment, it is not clear if the methodology generalises to other defense methods. If it does, the paper should state the method more generally, and evaluate the impact of defense method during optimisation.
Questions for authors
  1. How exactly is the sandbox environment (line 150) constructed, and how did you ensure that it is consistent with real-world deployments?
  2. How exactly does the local validation of the judge work (line 222)?
  3. Could the specific defenses used during optimisation be exchanged for others?
  4. Did you account for the variance in the agent's behavior, i.e. whether a successful skill was consistently reproducing the target action?
2 comments

We thank the reviewer for the comments. We address the weaknesses and questions below.


W1 and W2:

Table 2 compares complete attack methods under a common defended execution environment, while preserving their original protocols (Sec. 4.1). SkillJect uses its original five-iteration execution-trace-guided refinement, while SkillHarm uses its publicly released, previously refined attack skills. Joint iteration driven by pre-execution scanner feedback and runtime-defense feedback is a central methodological contribution of SkillDRE. The compared baselines do not jointly optimize scanner acceptance and runtime attack realization, which is a limitation of their original designs. Providing additional defense feedback to the baselines would produce defense-aware variants of those methods. Our main comparison evaluates the original methods under their respective designs and the same defended execution conditions.

As stated in the Abstract, lines 29-31, and Sec. 1, lines 60-69 and 75-76, our motivation is to investigate how defense feedback can serve as a learning signal for the iterative evolution of malicious skills. Our threat model concerns a fixed pre-execution and runtime defense pipeline (Sec. 3.1, lines 159-161). Generalization to unseen defenses is not the motivation or core objective of this study.

To address the requested supplementary detector checks, we evaluated the final evolved skills using static-only SkillScan and obtained 0% DR. We additionally rescanned the final selected skills using Gemini-3.7-Flash as an alternative SkillScan LLM backend. Both metrics below use all task–skill pairs. “attack success ∧ unsafe” denotes attacks that succeeded in the original runtime evaluation and whose final skills were subsequently flagged unsafe by the alternative scanner.

Victim modelunsafe rate %attack success ∧ unsafe %
DeepSeek V4 Pro11.242.01
Qwen3.5-397B10.441.61
GLM-5.28.031.2
Gemini-3.7-Flash10.442.41

This supplementary evaluation assesses generalization across scanner backends, while our main study concerns feedback-driven skill evolution against a fixed defense pipeline.

SkillScan was selected for its open-source implementation and configurable LLM backend, enabling reproducible offline scanning. As of now, SkillSonar was the only publicly available runtime defense implemented as an in-agent skill.


Q1:

As specified in Sec. 4.1, our experiments directly adopt the original SkillsBench task environments and Docker-based sandboxes, without modifying the benchmark's environment configurations or introducing additional simulated environments. All methods use the same execution setup, with Codex as the agent execution harness, natively supported by SkillsBench. As specified in Sec. 3.4, lines 289-290, each runtime trial starts from the same task-specific initial sandbox state. The victim agent executes actual skills and tool operations within Docker, rather than simulated interactions. This preserves the original SkillsBench execution setting and provides a controlled, reproducible evaluation of real agent-skill execution behavior.


Q2:

As detailed in Sec. 3.2, Eqs. (5)–(6), local judge validation consists of two deterministic checks. Valid(J) checks the JSON rule schema, supported evidence checks and parameters, and well-formed decision operators and check references. Only checks designated as attack evidence may contribute to the attack decision; ordinary task-completion checks cannot determine attack success.

Smoke(J) evaluates the rule in a newly created empty temporary sandbox. A candidate is rejected if it reports attack success without attack evidence or produces a hard evaluation error. Candidates failing either local check do not proceed to semantic review.

The remaining candidates are independently reviewed by the validator ensemble for target alignment, evidence observability, and resistance to false positives and false negatives. Acceptance requires both local checks and unanimous approval from all configured validators. Rejection diagnostics guide subsequent revisions, and the accepted judge remains fixed throughout skill evolution.

Q3:

As stated in the Abstract, lines 29-31, and Sec. 1, lines 60-69 and 75-76, our motivation is to investigate how defense feedback can serve as a learning signal for the iterative evolution of malicious skills. Our experiments instantiate this question using a fixed pre-execution and runtime defense pipeline (Sec. 3.1, lines 159-161).

SkillScan and SkillSonar are the concrete defenses selected to study this feedback interaction. In principle, other defenses that provide usable feedback can also guide iterative skill optimization, with their outputs adapted to the pre-execution acceptance criteria and runtime-feedback interfaces described in Sec. 3.3, lines 245-247 and 265-269, and Sec. 3.4, lines 279-290. The selected defenses remain fixed throughout an optimization run, while alternative defenses could be used in another instantiation.

Our choices also reflect practical requirements. Large-scale evolution requires many repeated defense evaluations, motivating a pre-execution scanner that supports reproducible offline deployment. SkillScan was selected for its open-source implementation and configurable LLM backend. As of now, SkillSonar was the only publicly available runtime defense implemented as an in-agent skill. These considerations motivated our use of SkillScan and SkillSonar as the two defense stages.


Q4:

At every iteration, SkillsBench creates a fresh task environment and conducts a new execution, with no execution state carried over from previous iterations, consistent with the environment-reset procedure in Sec. 3.4, lines 289-290 and Algorithm 1. Upon failure, the skill is refined and must pass Phase 1 rescanning before the next execution. Evolution terminates immediately upon success, with a maximum of ten runtime trials. Accordingly, Fig. 5 reports cumulative ASR across optimization rounds, counting each task–skill pair once when an evolved candidate first satisfies the attack-success criterion. Thus, each task–skill pair’s reported final attack outcome is obtained from an execution in a fresh, isolated SkillsBench Docker environment, with no execution state carried over from earlier rounds.

However, to assess stability, we evaluated the final selected skills across three independent executions in SkillsBench environments. Attack-success outcomes differed for at most two task–skill pairs across the executions, corresponding to an ASR variation of at most 0.80 percentage points, demonstrating stable aggregate attack effectiveness.

Reviewer_RWX9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper mainly studies how to achieve the automation of evolution so that the malicious Agent Skills can successfully execute the preset malicious operations while bypassing security scans and runtime defenses. The author proposed the SkillDRE (Dual-Stage Red-Team Evolution) framework, which adopts a dual-stage closed-loop optimization mechanism: The first stage uses the scanning feedback from SkillScan to continuously modify the instructions and code of the Skill, enabling it to pass the pre-execution security checks; the second stage utilizes the runtime interception results and actual execution feedback from SkillSonar to further optimize the Skill, allowing it to successfully achieve the attack goal during runtime, while re-entering the first stage for scanning after each modification. The author conducted experiments on 249 Skills, 94 tasks, and four victim models in SkillsBench, achieving an average attack success rate of 45.28%. Additionally, the final submitted malicious Skills had a detection rate of 0% under SkillScan, and retained the original task capabilities. The core contribution of this paper is the proposal of an automated red team evolution framework that combines pre-execution scanning and runtime defense feedback, revealing that evaluating static security scans or runtime defenses alone may not be sufficient to fully identify the potential security risks of Agent Skills.

Critical strengths
  1. The paper studies how attackers can use defense feedback to modify agent skills, addressing a practically relevant security problem.
  2. The experiments include evaluations across multiple models and ablation studies that provide reasonable support for the dual-stage feedback loop.
  3. The attack targets and judge rules are fixed before evolution begins, reducing the possibility of inflating success rates by making the targets easier during iteration.
Critical weaknesses
  1. The paper should clarify whether different methods use the same attack targets, feedback access, and optimization budgets.
  2. The zero detection rate is obtained against the same SkillScan configuration used during optimization, and zero findings are required before runtime execution. These results therefore do not establish whether the generated skills can evade other scanners.
  3. Little change in overall benign-task accuracy does not imply that benign tasks are completed when attacks succeed. The paper does not report the proportion of cases where both succeed, leaving the evidence for functionality preservation incomplete.
  4. It remains unclear whether the final attacks can be reproduced reliably. The authors should fix the final skill packages and evaluate them across fresh, independent runs, rather than only reporting cumulative success over multiple rounds of search.
Questions for authors
  1. Do all methods use the same attack targets, judge rules, feedback access, and optimization budgets? How does SkillHarm’s filtered subset affect the comparison?
  2. Can the final skill packages evade scanners or configurations not used during optimization?
2 comments

We thank the reviewer for the comments. We address the weaknesses and questions below.


W1 and Q1:

Sec. 4.1, lines 302-313 establishes the common evaluation setting: all methods in Table 2 are evaluated on the same 249 task–skill pairs from 94 SkillsBench tasks, using identical victim models, the Codex execution harness, and SkillSonar. SkillHarm was originally developed on SkillsBench, allowing us to directly evaluate its officially released, previously refined attack skills. Its original quality-filtering procedure includes manual filtering, and its implementation is not provided in the public release (SkillHarm, Secs. 4.2–4.3). This filtering is part of the original method, rather than an additional filtering step introduced in our evaluation. As explained in Appendix C, lines 797-803, the method-specific final skill sets are used for the paired ΔACC comparison in Table 4. The ASR evaluations nevertheless cover all 249 task–skill pairs. For pairs excluded from malicious-skill construction by SkillHarm’s original filtering procedure, we conduct evaluation using the original benign SkillsBench skills.

Regarding success criteria, we preserve each method's original attack objectives and evaluation mechanisms for faithful evaluation. Thus, all methods are assessed for realization of their respective predefined attack objectives under the same execution environment.

To ensure faithful reproduction, we retain each method’s original optimization protocol (Secs. 4.1–4.2). SkillJect uses its original five-iteration, execution-trace-guided refinement procedure (SkillJect, Sec. IV-C.2 and Fig. 6), while SkillHarm uses its officially released, previously refined attack skills constructed on SkillsBench (SkillHarm, Secs. 4.2–4.3). For all method, each runtime round involves one SkillsBench's docker enviroment execution of the current skill candidate. Within each execution, the victim agent follows the official SkillsBench execution configuration, without an additional cap on the number of model calls. Consequently, model-call usage depends on the execution trajectory rather than a fixed per-round allowance. For example, the five-runtime-round therefore correspond to at most five SkillsBench executions per task–skill pair.

The main experiment compares method-level performance under each method’s best design. Since SkillDRE performs multi-objective iterative optimization, we compare the results under our final design with those of the baselines under the best designs reported in their original papers. The defense-feedback-driven cross-stage loop is part of SkillDRE’s methodological contribution, while the baselines retain their original optimization mechanisms and settings.The optimization budgets are method-specific, following the respective original protocols; SkillDRE’s budget is specified in Sec. 4.1, lines 311-312 and Algorithm 1.

For comparison under a five-runtime-round cap, the fifth-round results already reported in Sec. 4.3, lines 423-431 and Fig. 5 are:

Victim modelASR (%)
DeepSeek-V4-Pro41.77
Qwen3.5-397B39.36
GLM-5.242.97
Gemini-3.7-Flash41.37
Macro-average41.37

Within five runtime rounds, SkillDRE achieves a 41.37% macro-average ASR, exceeding the strongest baseline’s 4.98% by 36.39 percentage points.


W2 and Q2:

As stated in the Abstract, lines 29-31, and Sec. 1, lines 60-69 and 75-76, our motivation is to investigate how defense feedback can serve as a learning signal for the iterative evolution of malicious skills. Our threat model concerns a fixed pre-execution and runtime defense pipeline (Sec. 3.1, lines 159-161). Generalization to unseen defenses is not the motivation or core objective of this study.

To address the requested supplementary detector checks, we evaluated the final evolved skills using static-only SkillScan and obtained 0% DR. We additionally rescanned the final selected skills using Gemini-3.7-Flash as an alternative SkillScan LLM backend. Both metrics below use all task–skill pairs. “attack success ∧ unsafe” denotes attacks that succeeded in the original runtime evaluation and whose final skills were subsequently flagged unsafe by the alternative scanner.

Victim modelunsafe rate (%)attack success ∧ unsafe (%)
DeepSeek V4 Pro11.242.01
Qwen3.5-397B10.441.61
GLM-5.28.031.2
Gemini-3.7-Flash10.442.41

This supplementary evaluation assesses generalization across scanner backends, while our main study concerns feedback-driven skill evolution against a fixed defense pipeline.

SkillScan was selected for its open-source implementation and configurable LLM backend, enabling reproducible offline scanning. As of now, SkillSonar was the only publicly available runtime defense implemented as an in-agent skill.

W3 and Q3:

Our ACC comparison assesses whether the final skills produced by the complete evolution process preserve overall benign-task performance, using the original SkillsBench evaluators (Sec. 4.1, lines 318-319). Results show improved ACC on three victim models and only a 0.15-percentage-point decrease on the fourth (Table 4 and Appendix C).

To assess whether malicious-goal realization coexists with benign-task completion, we report joint success rates from the same execution records across all 249 task–skill pairs. Joint success requires both objectives to succeed within the same execution. All values are percentages.

ModelAttack ACC (%)ASR (%)Joint success (%)
DeepSeek V4 Pro36.9546.9917.67
Qwen3.5-397B18.4743.787.63
GLM-5.238.1545.3819.28
Gemini-3.7-Flash42.5744.9818.47

These results directly demonstrate that malicious goals can be realized in executions that also pass the original SkillsBench benign-task evaluators.


W4 and Q4:

At every iteration, SkillsBench creates a fresh task environment and conducts a new execution, with no execution state carried over from previous iterations, consistent with the environment-reset procedure in Sec. 3.4, lines 289-290 and Algorithm 1. Upon failure, the skill is refined and must pass Phase 1 rescanning before the next execution. Evolution terminates immediately upon success, with a maximum of ten runtime trials. Accordingly, Fig. 5 reports cumulative ASR across optimization rounds, counting each task–skill pair once when an evolved candidate first satisfies the attack-success criterion. Thus, each task–skill pair’s reported final attack outcome is obtained from an execution in a fresh, isolated SkillsBench Docker environment, with no execution state carried over from earlier rounds.

However, to assess stability, we evaluated the final selected skills across three independent executions in SkillsBench environments. Attack-success outcomes differed for at most two task–skill pairs across the executions, corresponding to an ASR variation of at most 0.80 percentage points, demonstrating stable aggregate attack effectiveness.

Reviewer_gFA9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SkillDRE, a framework that automatically and iteratively improves malicious agent skills using pre-execution scanning and runtime feedback. Given a benign task and its associated skills, the system first constructs an attack objective and a judge rule, and keeps both fixed during optimization. The core is a two-stage loop. In the first stage, the skill is revised based on SkillScan risk reports until no risk findings remain. In the second stage, the agent executes the task with the SkillSonar defense in place, and the skill is further revised using execution traces, defense feedback, and attack outcomes. Every new version is rescanned, and the refiner is instructed to preserve the skill's original functionality. The authors evaluate the framework on SkillsBench with four victim models, and SkillDRE outperforms the baselines.

Critical strengths
  1. The problem is clearly formulated. The paper identifies the two requirements a malicious skill has to meet: passing pre-execution scanning and achieving the attack objective at runtime.
  2. The paper looks at benign task capability in addition to attack success. A skill still has to do its normal job, so preserving its original functionality is an important requirement in this setting. The authors measure benign task accuracy separately: it increases on three models and drops by only 0.15 percentage points on the fourth. This gives preliminary evidence that a malicious skill can still deliver its benign functionality, and it makes the attack results easier to interpret.
Critical weaknesses
  1. The contribution of the core loop on its own is not clear. The runtime stage adds feedback information, more chances to revise the skill, and more execution attempts at the same time, and the current ablation does not separate these factors. The gain from revising the skill is therefore mixed with the gain from repeated attempts.
  2. The comparability and scope of the main results are limited. First, the comparison with the baselines is not fully controlled. All methods use DeepSeek-V4-Pro as the attack model, but the task and skill sets and the optimization budgets are not unified or clearly reported. Second, the 0% detection rate means less than it may seem. Phase 1 only admits candidates with a SkillScan risk score of 0, and the final detection rate is measured with the same SkillScan configuration, so I believe the 0% DR largely follows from the design. It is also not directly comparable to the detection rates of baselines that were not optimized against this scanner. Likewise, SkillSonar is used both during optimization and during evaluation. The paper therefore does not show how the final skills perform against scanners or runtime defenses that were not involved in optimization, which limits how far the claim that the skills pass pre-execution scanning and realize their targets under runtime defense can be generalized.
Questions for authors
  1. Do the methods in Table 2 use the same set of tasks and skills? The caption of Table 4 states that ΔACC is computed on “each method’s attack-evaluated subset”, which suggests that the SkillHarm attack evaluation itself uses a filtered subset. Please clarify whether this also applies to the ASR comparison in Table 2, and report results for all methods on a common test set. Since the methods obtain their attack targets in different ways, please also clarify whether attack success is judged by consistent criteria.

  2. Are the optimization budgets comparable across methods? SkillDRE allows up to 30 scanner-guided rounds per Phase 1 run, and Phase 1 is re-entered after each of up to 10 runtime trials. Please report the number of revision rounds and runtime executions allowed for each baseline, along with an approximate count of model calls. Does SkillDRE keep its advantage when budgets are matched?

  3. How much of the improvement comes from the runtime defense feedback itself? Phase 2 uses execution traces, attack outcomes, and SkillSonar diagnostics together. Could you remove only the SkillSonar diagnostics given to the attack model, while keeping the defense active and the other inputs and budget unchanged? This would separate the effect of defense feedback from that of general execution feedback.

  4. How much of the ASR gain comes from revising the skill, and how much from repeated attempts? According to Algorithm 1, the loop stops at the first success, so the ASR in Table 2 equals the cumulative ASR after the 10th runtime round in Figure 5, that is, the fraction of skills with at least one success within up to 10 runs. Could you add a control that keeps the skill fixed and repeats execution the same number of times? Could you also report the success rate of the final selected skill over independent repeated runs?

  5. Do the final skills remain effective against defenses that were not used during optimization? Could you evaluate them with other scanners, for example SkillScan with a different underlying model or another skill scanning tool, and with a runtime defense other than SkillSonar, to check whether the 0% DR and the ASR results generalize?

2 comments

We thank the reviewer for the comments. We address the weaknesses and questions below.


W1, Q3 and Q4:

At every iteration, SkillsBench creates a fresh task environment and conducts a new execution, with no execution state carried over from previous iterations, consistent with the environment-reset procedure in Sec. 3.4, lines 289-290 and Algorithm 1. Upon failure, the skill is revised using execution trajectories, attack-target diagnostics, SkillSonar decisions, and accumulated optimization memory. Thus, runtime iterations evaluate progressively refined skill candidates under both defense constraints, rather than repeatedly sampling outcomes from the same skill.

As specified in our threat model and research setting (Sec. 3.1, lines 159-161; Sec. 3.4, Eq. (10)), SkillSonar is part of the fixed runtime environment, shared by SkillDRE and all baseline evaluations. Its original design embeds safety policies into the agent’s execution process, checking sensitive actions against the user’s task boundary and directing the agent to allow actions, replan, or request confirmation [1]. SkillSonar’s influence therefore extends beyond explicit stopping decisions. Its policies can change the agent’s planning and subsequent actions, affecting whether the malicious objective is realized without necessarily producing an explicit stopping decision. Accordingly, our ASR measures attack realization under SkillSonar’s full runtime policy, consistent with its original study, which evaluates protection through the resulting attack success rate.

SkillDRE uses execution trajectories, attack-target outcomes, and SkillSonar diagnostics are also recorded in the execution trajectories as complementary refinement signals (Sec. 3.4, Eqs. (11)–(12)). Removing SkillSonar itself would change the fixed runtime defense environment shared by SkillDRE and the baselines, while also removing the influence of its safety policies on the agent’s planning and execution. The resulting optimization would therefore no longer operate under the fixed defense pipeline specified by our threat model (Sec. 3.1, lines 159-161).

We further quantify adaptation to SkillSonar during refinement. We tracked changes between earlier failed trials and the final selected execution. Across the four victim models, we tracked both Sonar interventions and explicit stopping decisions during Phase 2. 32.43% of all instances showed intervention in earlier failed trials, while 6.53% of all instances still showed intervention in the final selected execution—a 79.88% reduction. Similarly, 10.74% of all instances had earlier stopping decisions, while 1.10% of all instances still received a stopping decision—a 89.72% reduction.

These paired results support the effectiveness of Phase 2 refinement in reducing both observable Sonar intervention and explicit blocking through targeted runtime feedback. Regarding final runtime failure attribution, 3.30% of all unsuccessful final attacks involved an explicit SkillSonar stopping decision directly causing those attacks and execution to fail, while 96.70% had no explicit stop but failed to satisfy the fixed attack judge. Overall, 11.38% contained observable Sonar intervention evidence, comprising explicit stopping decisions and other non-stopping interventions in 8.07% of all failures.

Algorithm 1 specifies a maximum of ten runtime trials per task–skill pair, with immediate termination upon the first success and no further search iterations for that pair. Fig. 5 reports cumulative ASR across all successively refined skill candidates, rather than repeated executions or refine of an unchanged skill. A fixed-skill repeated-execution control would measure execution variability and gains from additional attempts without refinement, whereas our main evaluation measures the complete feedback-guided refinement process. However, to assess stability, we evaluated the final selected skills across three independent executions in SkillsBench environments. Attack-success outcomes differed for at most two task–skill pairs across the executions, corresponding to an ASR variation of at most 0.80 percentage points, demonstrating stable aggregate attack effectiveness.

[1] Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents,Yang et al., 2026

W2, Q1 and Q2:

Sec. 4.1, lines 302-313 establishes the common evaluation setting: all methods in Table 2 are evaluated on the same 249 task–skill pairs from 94 SkillsBench tasks, using identical victim models, the Codex execution harness, and SkillSonar. SkillHarm was originally developed on SkillsBench, allowing us to directly evaluate its officially released, previously refined attack skills. Its original quality-filtering procedure includes manual filtering, and its implementation is not provided in the public release (SkillHarm, Secs. 4.2–4.3). This filtering is part of the original method, rather than an additional filtering step introduced in our evaluation. As explained in Appendix C, lines 797-803, the method-specific final skill sets are used for the paired ΔACC comparison in Table 4. The ASR evaluations nevertheless cover all 249 task–skill pairs. For pairs excluded from malicious-skill construction by SkillHarm’s original filtering procedure, we conduct evaluation using the original benign SkillsBench skills.

Regarding success criteria, we preserve each method's original attack objectives and evaluation mechanisms. Thus, all methods are assessed for realization of their respective predefined attack objectives under the same execution environment, while retaining their original success criteria for faithful evaluation.

To ensure faithful reproduction, we retain each method’s original optimization protocol (Secs. 4.1–4.2). SkillJect uses its original five-iteration, execution-trace-guided refinement procedure (SkillJect, Sec. IV-C.2 and Fig. 6), while SkillHarm uses its officially released, previously refined attack skills constructed on SkillsBench (SkillHarm, Secs. 4.2–4.3). For all method, each runtime round involves one SkillsBench's docker enviroment execution of the current skill candidate. Within each execution, the victim agent follows the official SkillsBench execution configuration, without an additional cap on the number of model calls. Consequently, model-call usage depends on the execution trajectory rather than a fixed per-round allowance. For example, the five-runtime-round therefore correspond to at most five SkillsBench executions per task–skill pair.

The main experiment compares method-level performance under each method’s best design. Since SkillDRE performs multi-objective iterative optimization, we compare the results under our final design with those of the baselines under the best designs reported in their original papers. The defense-feedback-driven cross-stage loop is part of SkillDRE’s methodological contribution, while the baselines retain their original optimization mechanisms and settings.The optimization budgets are method-specific, following the respective original protocols; SkillDRE’s budget is specified in Sec. 4.1, lines 311-312 and Algorithm 1.

For comparison under a five-runtime-round cap, the fifth-round results already reported in Sec. 4.3, lines 423-431, and Fig. 5 are:

Victim modelASR (%)
DeepSeek-V4-Pro41.77
Qwen3.5-397B39.36
GLM-5.242.97
Gemini-3.7-Flash41.37
Macro-average41.37

Within five runtime rounds, SkillDRE achieves a 41.37% macro-average ASR, exceeding the strongest baseline’s 4.98% by 36.39 percentage points.


Q5:

As stated in the Abstract, lines 29-31, and Sec. 1, lines 60-69 and 75-76, our motivation is to investigate how defense feedback can serve as a learning signal for the iterative evolution of malicious skills. Our threat model concerns a fixed pre-execution and runtime defense pipeline (Sec. 3.1, lines 159-161). Generalization to unseen defenses is not the motivation or core objective of this study.

To address the requested supplementary detector checks, we evaluated the final evolved skills using static-only SkillScan and obtained 0% DR. We additionally rescanned the final selected skills using Gemini-3.7-Flash as an alternative SkillScan LLM backend. Both metrics below use all task–skill pairs. “attack success ∧ unsafe” denotes attacks that succeeded in the original runtime evaluation and whose final skills were subsequently flagged unsafe by the alternative scanner.

Victim modelunsafe rate %attack success ∧ unsafe %
DeepSeek V4 Pro11.242.01
Qwen3.5-397B10.441.61
GLM-5.28.031.2
Gemini-3.7-Flash10.442.41

This supplementary evaluation assesses generalization across scanner backends, while our main study concerns feedback-driven skill evolution against a fixed defense pipeline.

SkillScan was selected for its open-source implementation and configurable LLM backend, enabling reproducible offline scanning. As of now, SkillSonar was the only publicly available runtime defense implemented as an in-agent skill.

Reviewer_gmvoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SkillDRE, an automated red-team framework for modifying agent skill packages against a fixed pre-execution scanner and runtime defense. For each benign task, it constructs a malicious target and a judge rule, holds them fixed, and iterates between scanner-guided skill revision and runtime-guided revision; every runtime revision is rescanned before another execution. On 249 skills associated with 94 SkillsBench tasks and four victim models, the authors report 45.28% average attack success rate (ASR), compared with at most 9.00% for the evaluated baselines. Final skills have 0% SkillScan detection under the scanner used during optimization, while benign-task accuracy is largely preserved. A stage-wise ablation reports higher ASR than scanner-only evolution and lower scanner detection than runtime-only evolution.

Critical strengths

S1. This papaer has a clear and testable attack setting. The paper identifies a concrete interaction between pre-execution scanning and runtime intervention. Holding the attack target and judge rule fixed makes iterative changes easier to interpret, and the stage-wise ablation in Table 3 shows the different effects of scanner-guided and runtime-guided revision under the tested setup.

S2. Authors provides substantial attack-side evaluation. The study covers 249 skills across 94 tasks and four victim models, reports both attack outcomes and benign-task accuracy, and manually checks a subset of generated targets and judge rules (Appendix E). These results provide useful evidence that the proposed attack procedure works against the evaluated pipeline.

Critical weaknesses

W1. The methodological novelty is primarily a combination of existing components, and the comparisons do not establish the value of that combination. Table 1 itself credits SkillMutator with scanner-guided full-package revision, SkillJect with execution-trace-guided revision, and SkillHarm with automated harmful-goal and evaluator construction. SkillDRE combines these ingredients with a fixed task-conditioned target and judge rule and a return to scanning after runtime revisions. Table 3 removes entire stages: compared with iterative runtime-only evolution, the full method gains only 0.40 percentage points in ASR while its scanner detection rate falls from 79.92% to 0%, consistent with mandatory scanner re-entry. It does not compare against a simple, compute-matched composition of the existing feedback loops or isolate the value of the fixed target/judge. Table 2 also omits SkillMutator, the closest cited method for the scanner-feedback component. The reported gain therefore demonstrates that the complete system is effective, but does not establish a distinct methodological advance over combining known techniques. This makes the contribution too incremental for an acceptance.

W2. The defense-side evaluation is missing despite the paper's stated motivation. The manuscript reports SkillScan detection and attack ASR under SkillSonar, but does not analyze what either defense misses after evolution. In Sec. 3.3 and Algorithm 1, every candidate admitted to runtime testing must already have a zero SkillScan risk score. Thus, the final 0% detection rate in Table 2 follows from the admission rule; reaching that rule for all 249 skills is the empirical result. Appendix F tabulates scanner findings during initial evolution, but does not provide this analysis for the evolved skills. The paper also tests no defender adjustment or mitigation together with its effect on benign-task accuracy. Because the claimed significance is that joint evaluation exposes a blind spot in layered defenses, the absence of a defender-oriented breakdown or a small targeted defense test leaves the practical conclusion underdeveloped.

Questions for authors

Can the authors compare SkillDRE with a compute-matched composition of scanner-guided and runtime-guided refinement, and isolate the contribution of the fixed target/judge beyond standard rescan gating? If SkillMutator cannot be evaluated on this setup, what prevents a direct comparison?

Across runtime trials, how many failures are due to SkillSonar stopping the operation versus the attack target simply not being realized? What broad failure patterns remain among successful attacks, and does one targeted defender adjustment reduce ASR without materially degrading benign-task accuracy?

4 comments

We thank the reviewer for the comments. We address the weaknesses and questions below.


W1 and Q1:

As stated in the Abstract, lines 29-31, and Sec. 1, lines 60-69 and 75-76, our primary objective is to demonstrate that defense feedback can serve as a learning signal for the iterative evolution of malicious skills. SkillDRE studies this process against a fixed pre-execution and runtime defense pipeline (Sec. 3.1, lines 159-161). Sec. 2.2 and Table 1 explicitly distinguish the related methods: SkillMutator uses scanner-guided refinement but does not evaluate malicious-goal realization during victim execution, SkillJect uses execution-trace-guided refinement, and SkillHarm automates harmful-goal construction and evaluation. SkillDRE’s methodological contribution is the dual-stage iteration jointly driven by scanner and runtime-defense feedback, which the compared methods do not incorporate in their original designs.

Scanner gating and fixed target/judge rules serve different purposes: the former determines which skill packages may enter runtime evaluation, while the latter defines what constitutes attack success throughout evolution. As specified in Sec. 3.2, Eqs. (3)–(6), the malicious objective and its evidence-based success criterion are validated before evolution and remain unchanged. Consequently, neither stage can obtain a counted success by replacing the original objective with an easier target or relaxing the judge rule. This establishes a stable optimization problem in which defense feedback guides changes to the skill implementation. Our methodological contribution concerns cross-stage skill evolution under this fixed objective and the joint pre-execution and runtime defense constraints.

Table 3 demonstrates the complementary roles of the two stages: scanner-only evolution achieves 34.94% ASR / 0% DR, runtime-only evolution achieves 46.59% ASR / 79.92% DR, and full SkillDRE achieves 46.99% ASR / 0% DR. The contribution is maintaining runtime attack effectiveness while satisfying the pre-execution acceptance constraint, rather than relying on the marginal 0.40-percentage-point ASR difference over runtime-only evolution.

The main experiment compares method-level performance under each method’s best design. Since SkillDRE performs multi-objective iterative optimization, we compare the results under our final design with those of the baselines under the best designs reported in their original papers. The defense-feedback-driven cross-stage loop is part of SkillDRE’s methodological contribution, while the baselines retain their original optimization mechanisms and settings.The optimization budgets are method-specific, following the respective original protocols; SkillDRE’s budget is specified in Sec. 4.1, lines 311-312 and Algorithm 1.

For comparison under a five-runtime-round cap, the fifth-round results already reported in Sec. 4.3, lines 423-431 and Fig. 5 are:

Victim modelASR (%)
DeepSeek-V4-Pro41.77
Qwen3.5-397B39.36
GLM-5.242.97
Gemini-3.7-Flash41.37
Macro-average41.37

Within five runtime rounds, SkillDRE achieves a 41.37% macro-average ASR, exceeding the strongest baseline’s 4.98% by 36.39 percentage points.

SkillMutator is discussed in Sec. 2.2 and Table 1. As formulated in SkillMutator, Sec. III, its evaluation concerns install-time detection of unexecuted skill packages, rather than malicious-goal realization during victim execution. Its reported scanner-detection results therefore do not directly provide runtime ASR or benign-task ACC under our defended execution protocol. Furthermore, its implementation is not fully open source, limiting reproduction. No actual attack execution and restrictions make SkillMutator unsuitable for comparison.

W2 and Q2:

As stated in the Abstract, lines 29-31, and Sec. 1, lines 60-69 and 75-76, SkillDRE investigates how two-stage defense feedback can guide adaptive red teaming, and how evaluating either defense stage in isolation can miss the resulting attack capability. Our evaluation studies attack evolution against the fixed defense pipeline specified in Sec. 3.1, lines 159-161.

Zero scanner findings are required before runtime execution. However, requiring zero findings does not guarantee finding, within a finite budget, a skill that both passes scanning and achieves the malicious objective at runtime. Sec. 4.3 and Fig. 4 show that scanner-guided cold-start evolution obtains approved candidates for all task–skill pairs, requiring 3.23 rounds on average. Table 3 further shows how runtime attack effectiveness is maintained while satisfying this scanner-acceptance requirement. Thus, the final ASR/DR combination reflects the outcome of the complete cross-stage evolution pipeline, including runtime-guided refinement and scanner-guided re-entry. It demonstrates the pipeline’s ability to find skills that jointly satisfy scanner acceptance and runtime attack objectives within a finite budget.

As specified in our threat model and research setting (Sec. 3.1, lines 159-161; Sec. 3.4, Eq. (10)), SkillSonar is part of the fixed runtime environment, shared by SkillDRE and all baseline evaluations. Its original design embeds safety policies into the agent’s execution process, checking sensitive actions against the user’s task boundary and directing the agent to allow actions, replan, or request confirmation [1]. SkillSonar’s influence therefore extends beyond explicit stopping decisions. Its policies can change the agent’s planning and subsequent actions, affecting whether the malicious objective is realized without necessarily producing an explicit stopping decision. Accordingly, our ASR measures attack realization under SkillSonar’s full runtime policy, consistent with its original study, which evaluates protection through the resulting attack success rate.

Regarding final runtime failure attribution, 3.30% of all unsuccessful final attacks involved an explicit SkillSonar stopping decision directly causing those attacks and execution to fail, while 96.70% had no explicit stop but failed to satisfy the fixed attack judge. Overall, 11.38% contained observable Sonar intervention evidence, comprising explicit stopping decisions and other non-stopping interventions in 8.07% of all failures.

Across the four victim models, we tracked both Sonar interventions and explicit stopping decisions during Phase 2. 32.43% of all instances showed intervention in earlier failed trials, while 6.53% of all instances still showed intervention in the final selected execution—a 79.88% reduction. Similarly, 10.74% of all instances had earlier stopping decisions, while 1.10% of all instances still received a stopping decision—a 89.72% reduction. These paired results support the effectiveness of Phase 2 refinement in reducing both observable Sonar intervention and explicit blocking through targeted runtime feedback.

Analysis of the final scanner-approved skill packages and their corresponding successful execution trajectories identifies five recurring defense failures: failure to constrain data retention to task requirements, allowing additional persistent copies of source materials, extracted content, or computed results; failure to prevent the collection and storage of environment secrets; failure to limit unnecessary workflow-metadata retention, including file mappings, request hashes, and operation traces; failure to enforce output and reporting boundaries, allowing data or artifacts that should have been excluded to be retained; and failure to cross-check reported metrics against underlying execution records, allowing falsified reports to conceal unsafe behavior.

As specified in Sec. 4.1, lines 312-313, SkillSonar remains a fixed defense skill throughout execution. We follow its official deployment protocol, requiring activation before task actions through higher-priority defense instructions. Our inspection confirms pre-action activation in 100% of evaluated runs, including successful attacks. Thus, the observed successes did not result from suppressing or bypassing SkillSonar activation.

Regarding targeted mitigation, our research objective is to evaluate how attackers learn from feedback exposed by an existing defense pipeline. Designing or tuning a new mitigation and assessing its effectiveness constitutes a complementary defender-side optimization question for future work. Our conclusions concern the attack capability emerging under the specified fixed defenses.

[1] Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents,Yang et al., 2026

Thank you for the rebuttal, I would like to maintain the overall accessment.

Thank you for reading our rebuttal and the broader discussion. Could you clarify which remaining concerns led you to maintain your score? You mention that the paper would benefit from further revision and polishing. It would help us to know which point in our response you found unconvincing or which part of the paper you believe needs revision, so we can address it directly.

Reviewer_sM6jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies adaptive attacks against agent skills protected by pre-execution scanning and runtime defenses. SkillDRE constructs a task-conditioned malicious target and judge rule, then keeps both fixed while evolving the skill. Runtime failures guide revisions, and each revision must pass scanning again before another execution. On 249 Skills associated with 94 SkillsBench tasks, the method reports 45.28% cumulative adaptive ASR across four victim models, with zero findings on the scanner configuration used during optimization.

Critical strengths
  • The target and judge remain fixed throughout skill evolution. Runtime feedback guides revisions, but every revised candidate must pass pre-execution scanning again. This provides a meaningful way to study the interaction between layered defenses.

  • The results show substantial adaptive attack success across four victim models. The ablations, random human audit, and second-harness evaluation provide additional evidence for the proposed framework.

Critical weaknesses
  • Baseline comparability: The paper does not explain sufficiently how SkillJect and SkillHarm were evaluated against comparable targets, success criteria, populations, and optimization budgets. Using the same attack model is helpful, but does not resolve these questions. This makes the reported performance gap difficult to interpret.

  • Feedback attribution: Iterative Phase 2 improves ASR from 36.14% to 46.59%, but also allows more refinement and execution opportunities than the one-shot condition. The experiment does not isolate the contribution of informative runtime feedback from additional attempts.

Questions for authors
  1. How were SkillJect and SkillHarm adapted to the same task–skill population and success criteria? Were their optimization budgets also comparable?

  2. Can you provide an equal-budget iterative comparison without informative runtime feedback, or an equivalent analysis that separates feedback from additional attempts?

  3. Appendix E reports 96.8% judge-rule agreement across 125 randomly sampled instances. What were the four disagreements, and did any affect the reported ASR?

2 comments

We thank the reviewer for the comments. We address the weaknesses and questions below.


W1 and Q1:

SkillJect is an automated attack method, so we directly apply its attack-generation and execution-trace-guided refinement pipeline to the corresponding SkillsBench task–skill pairs, retaining its original five-iteration setting. SkillHarm was constructed on SkillsBench, allowing us to directly evaluate its publicly released, previously refined attack skills on the corresponding tasks.

Each baseline retains its original attack objectives and attack-success evaluators. SkillJect checks hidden helper-script execution, SkillHarm checks its predefined harmful outcomes, and SkillDRE uses its validated, fixed task-conditioned targets and judge rules (Sec. 3.2, lines 208-215 and 223-239).

The main experiment compares method-level performance under each method’s best design. Since SkillDRE performs multi-objective iterative optimization, we compare the results under our final design with those of the baselines under the best designs reported in their original papers. The defense-feedback-driven cross-stage loop is part of SkillDRE’s methodological contribution, while the baselines retain their original optimization mechanisms and settings.The optimization budgets are method-specific, following the respective original protocols; SkillDRE’s budget is specified in Sec. 4.1, lines 311-312 and Algorithm 1.

For comparison under a five-runtime-round cap, the fifth-round results already reported in Sec. 4.3, lines 423-431 and Fig. 5 are:

Victim modelASR (%)
DeepSeek-V4-Pro41.77
Qwen3.5-397B39.36
GLM-5.242.97
Gemini-3.7-Flash41.37
Macro-average41.37

Within five runtime rounds, SkillDRE already exceeds the strongest baseline’s macro-average ASR of 4.98% by 36.39 percentage points.


W2 and Q2:

The Abstract, lines 29-31 and Sec. 1, lines 75-76 explicitly identify our central research motivation: two-stage defense feedback can serve as a useful learning signal for adaptive red teaming, and evaluating either defense stage in isolation can miss the resulting attack capability. SkillDRE addresses this question through feedback-driven evolution, using observed failures to revise skill implementations while maintaining a fixed malicious objective.

Sec. 3.4, Eqs. (11)–(12), and Algorithm 1 clearly define Phase 2 as feedback-driven skill refinement, not repeated execution of an unchanged attack. Each runtime trial independently executes the current skill candidate from the same task-specific initial sandbox state. Upon failure, execution trajectories, SkillSonar decisions, and attack-target diagnostics guide the next skill revision, which must pass Phase 1 rescanning before subsequent execution. Evolution terminates immediately upon success, rather than requiring all ten trials. Fig. 5 reports cumulative success across this evolution process, counting each task–skill pair once when a candidate skill first succeeds. Unsuccessful candidates are refined and rescanned before subsequent runtime trials; the process stops upon success or when the ten-round limit is reached.

To assess stability, we evaluated the final selected skills across three independent executions in SkillsBench environments. Attack-success outcomes differed for at most two task–skill pairs across the executions, corresponding to an ASR variation of at most 0.80 percentage points, demonstrating stable aggregate attack effectiveness, with no substantial gains from repeated execution compared with the results reported in Figure 5.

Table 3 shows that iterative Phase 2 increases ASR from 36.14% to 46.59%, demonstrating the effectiveness of the complete iterative refinement procedure. This ablation evaluates feedback-guided evolution as a whole; it does not separately quantify the causal contributions of informative feedback and additional execution opportunities. The demonstrated contribution is the resulting adaptive refinement process, in which observed defense and execution failures guide successive skill modifications, rather than an isolated effect of feedback independent of the iteration budget.

Q3:

Our judge rules achieve 96.8% agreement with human assessment (Appendix E, lines 845-853), indicating high agreement on the correctness of the sampled rules. The four disagreements concerned target boundary and field structure. For context, JudgeLM [1] reports 90.72% agreement with a human annotator, while HarmBench [2] reports 93.19% classifier–human agreement in their respective evaluation settings. Trust or Escalate [3] reports 77.8% GPT-4–human agreement on ChatArena, improving to 80.2% through selective evaluation at 77.6% coverage. Our task-specific human validation provides evidence supporting the reliability of the judge rules used in our evaluation.

[1] Lianghui Zhu et al. JudgeLM: Fine-tuned Large Language Models are Scalable Judges. ICLR, 2025.

[2] Mantas Mazeika et al. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal. ICML, 2024.

[3] Jaehun Jung et al. Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement. ICLR, 2025.

Reviewer_WnAXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SkillDRE, a fully automated pipeline that turns benign agent skills into malicious ones that evade a two-layer defense: a pre-execution scanner (Cisco SkillScan with a Qwen3.5 backend) and a runtime defense skill (Skill-Sonar). It works in three stages. First, for initialization, an LLM generator proposes a task-level malicious intent, a skill-level target and an executable judge rule. Each must be accepted unanimously by a validator ensemble of three models, and all three stay fixed afterwards. In phase 1 the skill package is iteratively rewritten until SkillScan reports zero findings. Candidates are ranked by a lexicographic severity score. In phase 2, the candidate is executed by a victim agent under Skill-Sonar. Failed attempts are revised using the agent trajectory, the Sonar decisions and the judge's diagnostics. Every revision goes back to Phase 1 for rescanning, for at most 10 runtime rounds. On 249 skills from 94 SkillsBench tasks and four victim models, SkillDRE reaches 45.28% average ASR (baselines are at or below 9%) and 0% SkillScan detection, with little loss in benign-task accuracy. The ablations suggest that runtime feedback raises ASR while Phase 1 re-entry keeps detection at zero.

Critical strengths
  1. Timely problem and a useful observation about layered defenses. The ablation makes the tension between the two defense layers concrete. Runtime-only iteration raises ASR from 36.14% to 46.59%, but SkillScan detection also rises, from 62.65% to 79.92%. Scanner-only iteration drives detection to 0% but slightly lowers ASR. The resulting message, that pre-execution and runtime defenses should be evaluated jointly, is useful for defenders.

  2. Clean experimental design that separates the target from the implementation. Fixing the malicious target and an executable judge rule before evolution means progress across revisions is measured against a stable objective. The judge rules must also pass schema checks and an empty-sandbox smoke test. The evaluation is reasonably large (249 skills, four victims, two harnesses), and it includes human validation of 125 target–rule pairs with 96.8% agreement on judge correctness.

Critical weaknesses
  1. The 0% detection rate holds by construction, and the comparison with baselines is not budget-matched. A candidate can only enter Phase 2 if the same SkillScan configuration (same Qwen3.5 backend) returns R = 0, and every Phase 2 revision must pass it again. So every submitted skill has zero findings on that scanner by definition. DR = 0% therefore shows that the optimization converged, not that the skills evade scanning in general. Evidence of real evasion would require held-out detectors: a different SkillScan LLM backend, static-only mode, Snyk Agent Scan (used by SkillMutator), or an LLM-as-auditor. On ASR, SkillDRE gets up to 30 scanner queries per Phase 1 entry and up to 10 runtime trials, and success counts if any round succeeds (Fig. 5 is cumulative; the released run_asr_eval.sh defaults to --selection best). As far as I can tell, the baselines are evaluated without scanner feedback or a matched number of runtime retries. SkillHarm is also evaluated on a filtered subset (Appendix C); its ASRs, e.g. 2.53%, are not multiples of 1/249. The most relevant scanner-guided baseline, SkillMutator, is missing. Because of all this, the headline "+40.3 pp over the strongest baseline" mostly reflects the much larger feedback and query budget, not the specific dual-stage design.

  2. Runtime evasion and benign-accuracy preservation may be overstated by how they are measured. In the released code, apply_sonar_gate fails a trial only if Skill-Sonar issues a stopping action. Its comments say protocol validity, i.e. whether Sonar was activated before the protected action, "remains diagnostic and does not gate". So a run in which the victim agent never invokes Skill-Sonar, or the malicious skill steers it away from Sonar, counts as evading the runtime defense. Skill-Sonar is itself an in-agent advisory skill, which makes this a relatively weak runtime defense. The ablation gain of the full loop over iterative Phase 2 is 0.40 pp, i.e. a single skill out of 249. The two one-shot variants differ by 1.61 pp (4 skills) even though they appear to evaluate the same kind of initial candidate. No seeds or confidence intervals are reported, so the ASR benefit of the cross-stage loop over runtime-only iteration is not statistically supported; its only demonstrated benefit is DR, which again holds by construction. Accuracy is higher under attack on three of the four models (ΔACC down to −3.58 pp). The code's selection_score ranks rounds with task_success as the first criterion. If ACC under attack comes from the best-of-rounds record while No Attack is a single run, the comparison is biased in SkillDRE's favor.

Questions for authors
  1. Is the ASR in Table 2 best-of-10 runtime rounds per skill? Does it require benign-task success? The code has asr_success = task_success ∧ attack ∧ sonar_passed, plus both end_to_end_asr and conditional_asr. What budget (scanner queries, runtime trials) did each baseline get, and did the baselines use your judge rules or their own evaluators?
  2. Among successful attacks, in what fraction did Skill-Sonar actually activate before the malicious operation? Do evolved skills learn to suppress, delay or bypass Sonar activation? Please report ASR with protocol validity required.
  3. How is ACC under attack computed: the final round, the selected best round, or a fresh independent run? Can you report it from fresh re-runs of the final skills, with multiple seeds?
  4. What detection rates do the final skills get on held-out scanners (a different LLM backend, static-only SkillScan, Snyk Agent Scan)?
  5. What are the concrete success conditions for the "exfiltration" targets, which are 72% of all targets? If many reduce to writing data to a hidden local path inside the sandbox, the security significance and the difficulty for a runtime monitor differ substantially from network exfiltration.
3 comments

We thank the reviewer for the comments. We address the weaknesses and questions below.


W1:

As stated in the Abstract, lines 29-31, and Sec. 1, lines 60-69 and 75-76, our primary objective is to demonstrate that defense feedback can serve as a learning signal for the iterative evolution of malicious skills. We study this process against a fixed pre-execution and runtime defense pipeline (Sec. 3.1, lines 159-161). The suggested held-out-detector experiments would assess transfer to unseen defenses—a distinct research question from whether feedback from specified defenses can guide attack evolution against those same defenses. Requiring zero scanner findings does not guarantee that the evolution process will, within a finite budget, produce a skill that both passes scanning and achieves the fixed malicious objective at runtime. Sec. 4.3, lines 413-417, and Fig. 4 show that scanner-guided cold-start evolution obtains approved candidates for all 249 task–skill pairs, requiring 3.23 rounds on average. Table 3 further shows that scanner-only evolution achieves 34.94% ASR / 0% DR, runtime-only evolution achieves 46.59% ASR / 79.92% DR, and full SkillDRE achieves 46.99% ASR / 0% DR. These results support the use of defense feedback to guide iterative skill evolution toward jointly satisfying scanner acceptance and runtime attack objectives, directly addressing the research objective stated in the paper.

Regarding SkillHarm’s filtering, its original construction procedure includes manual verification, so we use its publicly released, filtered final attack skills. For Table 2, the denominator remains all 249 task–skill pairs. The public release provides multiple attack variants for a given source skill. Accordingly, we first average attack outcomes across the applicable variants within each skill and then across all 249 pairs. The reported ASR therefore contain an average of per-skill variant success rates, explaining why values such as 2.53% need not correspond to an integer number of successful skills out of 249. Appendix C, lines 797-803 uses the filtered subset specifically for the paired ΔACC comparison in Table 4.

SkillMutator is discussed in Sec. 2.2 and Table 1. As formulated in SkillMutator, Sec. III, its evaluation concerns install-time detection of unexecuted skill packages, rather than malicious-goal realization during victim execution. Its reported scanner-detection results therefore do not directly provide runtime ASR or benign-task ACC under our defended execution protocol. Furthermore, its implementation is not fully open source, limiting reproduction. No actual attack execution and restrictions make SkillMutator unsuitable for comparison.


W2 and Q2:

As specified in Sec. 4.1, lines 310-313, SkillSonar is included as a fixed defense skill throughout benchmark execution, with its runtime integration formalized in Sec. 3.4, Eq. (10). We follow SkillSonar’s official deployment protocol, requiring activation before task actions through higher-priority defense instructions that remain unchanged during evaluation. Under the official deployment settings, SkillSonar is invoked first, before task actions, and its activation is not bypassed during running. To address the activation concern, we inspected the execution traces and confirmed that SkillSonar was invoked before task actions in 100% of evaluated runs, including successful attacks. Thus, the reported successes did not result from suppressing, delaying, or bypassing SkillSonar activation.

Q1:

Table 2 reports cumulative ASR over at most ten feedback-guided runtime rounds per task–skill pair, stopping at the first success when attack ∧ sonar_passed is satisfied (Sec. 3.4, Eq. (11), and Algorithm 1). Unsuccessful candidates are refined and rescanned before subsequent runtime trials; the process stops upon success or when the ten-round limit is reached. Benign-task completion is not required. The cited task_success conjunction is legacy code from an earlier implementation and was not used to generate Table 2. Given safety, ethical, and anonymity considerations during peer review, the code attached to the submission is provided as a partial reference implementation. We will clean up and consolidate the final codebase and publicly release the complete implementation after publication. ASR is computed over all 249 pairs; ACC is evaluated separately using the original SkillsBench evaluators.

As specified in Sec. 4.1, lines 302-310, all methods share the 249 task–skill pairs, attacker and victim models, Codex harness. We retain each baseline’s original attack objectives, attack success evaluators, and optimization protocol, rather than applying SkillDRE’s generated judge rules. SkillJect uses its original five-iteration execution-trace-guided refinement, while SkillHarm uses its publicly released, previously refined attack skills. Joint iteration driven by pre-execution scanner feedback and runtime-defense feedback is a central methodological contribution of SkillDRE. The compared baselines do not jointly optimize scanner acceptance and runtime attack realization, which is a limitation of their original designs. Incorporating SkillDRE’s cross-stage feedback loop into these baselines would introduce the proposed mechanism and produce augmented variants. Table 2 therefore compares the complete methods under their respective original designs. SkillDRE allows up to 30 scanner-guided rounds per Phase 1 invocation and ten runtime trials per task–skill pair.

To examine the effect of additional runtime trials, Fig. 5 provides the following cumulative ASR within five runtime rounds:

Victim modelASR (%)
DeepSeek-V4-Pro41.77
Qwen3.5-397B39.36
GLM-5.242.97
Gemini-3.7-Flash41.37
Macro-average41.37

Within five runtime rounds, matching SkillJect’s five-iteration limit, SkillDRE already exceeds the strongest baseline’s macro-average ASR of 4.98% by 36.39 percentage points. However, the purpose of the main-results table is to compare method-level performance under each method’s best design. Since SkillDRE performs multi-objective iterative optimization, we compare the results under our final design with those of the baselines under the best designs reported in their original papers.


Q3:

As specified in Sec. 4.1, lines 318-319, ACC follows the original SkillsBench evaluation procedure, without modifications to its task evaluators. At every iteration, SkillsBench creates a fresh task environment and conducts a new execution, with no execution state carried over from previous iterations, consistent with the environment-reset procedure in Sec. 3.4, lines 289-290, and Algorithm 1.

The Attack ACC reported in Fig. 2 is taken from the execution of the final selected skill in its corresponding round, selected according to the attack-success stopping criterion rather than benign-task accuracy. This execution also starts from a fresh environment, and the final ACC is the task-evaluation result of that run and therefore also constitutes the result of an independent execution. No separate ACC-maximizing selection is performed. Thus, Attack ACC reflects legitimate task completion during the evaluated attack execution.

To assess stability, we evaluated the final selected skills across three independent executions in SkillsBench environments. Attack-success outcomes differed for at most two task–skill pairs across the executions, corresponding to an ASR variation of at most 0.80 percentage points, demonstrating stable aggregate attack effectiveness, with no substantial gains from repeated execution compared with the results reported in Figure 5. The absolute change in ACC relative to the original evaluation ranged from 1.21 to 4.01 percentage points. All reruns followed the SkillsBench evaluation protocol. These ACC changes are consistent with a reasonable level of evaluation variability, comparable in scale to the 95% confidence-interval margins of up to approximately 6.07 percentage points, calculated from the official SkillsBench v4 results [1].

[1] Xiangyi Li et al. SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. arXiv:2602.12670v4, 2026.

Q4:

As stated in the Abstract, lines 29-31, and Sec. 1, lines 60-69 and 75-76, our motivation is to investigate how defense feedback can serve as a learning signal for the iterative evolution of malicious skills. Our threat model concerns a fixed pre-execution and runtime defense pipeline (Sec. 3.1, lines 159-161). Generalization to unseen defenses is not the motivation or core objective of this study.

To address the requested supplementary detector checks, we evaluated the final evolved skills using static-only SkillScan and obtained 0% DR. We additionally rescanned the final selected skills using Gemini-3.7-Flash as an alternative SkillScan LLM backend. Both metrics below use all task–skill pairs. “attack success ∧ unsafe” denotes attacks that succeeded in the original runtime evaluation and whose final skills were subsequently flagged unsafe by the alternative scanner.

Victim modelunsafe rate %attack success ∧ unsafe %
DeepSeek V4 Pro11.242.01
Qwen3.5-397B10.441.61
GLM-5.28.031.2
Gemini-3.7-Flash10.442.41

This supplementary evaluation assesses generalization across scanner backends, while our main study concerns feedback-driven skill evolution against a fixed defense pipeline.

SkillScan was selected for its open-source implementation and configurable LLM backend, enabling reproducible offline scanning. Snyk Agent Scan's security analysis relies on a hosted API, rather than providing the fully offline scanning setup used in our evaluation.


Q5:

As reported in Appendix D, lines 823-830, Fig. 6, exfiltration-related categories account for 180 of the 249 constructed attack targets (72.29%). This describes the distribution of malicious objectives, not the proportion of successful external network exfiltrations.

As specified in Sec. 3.2, lines 208-215 and 238-239, each target has a predefined, verifiable success condition and a fixed judge rule. For unauthorized local copying or staging objectives, success requires the specified data to reach the predefined unauthorized destination within the sandbox, as verified through the resulting artifacts. These outcomes demonstrate unauthorized copying or staging; they do not imply completed transmission over an external network.

Controlled sandbox evaluation is established practice in skill-attack research. SkillJect conducts experiments in isolated Docker environments using controlled tasks, dummy secrets, and synthetic target behaviors. Skill-Inject likewise runs each experiment in an isolated container and assesses attack success through execution evidence, including logs and filesystem changes. SkillHarm uses isolated Docker containers without external network access (Ethical Considerations). Its released risk taxonomy permits data-exfiltration objectives targeting an attacker-controlled artifact, with checks verifying that protected content has crossed its intended boundary. Thus, Our evaluation follows this practice, testing malicious behaviors and verifying their outcomes within a controlled sandbox, without compromising real users or production systems.

Comments on the paper

6 comments

We thank the reviewer for the follow-up. However, we would like to clarify remaining misunderstandings regarding our stated research scope and the interpretation of the cited evidence.


Q1:

SkillSonar operates as a skill within the victim agent, and its decisions and execution records are embedded in the same Agent Memory as task execution. Our attack refiner uses this complete execution history for optimization (Sec. 3.4, Eq. (12); Algorithm 1). Sonar-related feedback is therefore not confined to a separate diagnostic field that can simply be disabled while leaving the agent memory unchanged. Removing these records would require removing SkillSonar or the agent memory supplied to the refiner, which would no longer receive the complete record used by our method. The proposed setting would completely undermine the foundational environment.

Table 3 evaluates the integrated cross-stage mechanism. The full method achieves 46.99% ASR / 0% DR, compared with 34.94% / 0% for scanner-only evolution and 46.59% / 79.92% for runtime-only evolution. These results demonstrate that the complete mechanism maintains runtime attack effectiveness while satisfying scanner acceptance. The 12.05-percentage-point improvement measures runtime refinement as a whole. Our objective is to improve ASR under the fixed base execution environment with defense. This evaluation criterion is consistent with the original SkillSonar study, which assesses defense effectiveness through the resulting ASR under its runtime protection.

We also correct the interpretation of our intervention statistics: 32.43% and 10.74% are proportions of all model–task–skill instances, not proportions of unsuccessful runtime trials. Each instance is counted once if any earlier failed execution exhibited an intervention or stopping decision. The absence of an explicit stop also does not imply that SkillSonar had no influence on planning or execution.


Q2:

The fixed-defense scope is part of the paper’s original formulation. The Abstract and Introduction describe feedback-guided evolution against pre-execution and runtime defenses, and we disagree with the characterization of our claims as overstated: our claims concern adaptation to the fixed defense pipeline explicitly specified in the Introduction (lines 79–82) and Sec. 3.1 (lines 159–161). Our claims concern adaptation to this specified defense pipeline, not unrestricted evasion across arbitrary defenses. Our empirical contribution is obtaining scanner-approved candidates for all 249 task–skill pairs within a finite budget while achieving substantial runtime attack success.

The supplementary results show that scanner acceptance largely persists after changing the backend. Under the original scanner configuration, Table 2 reports 0% DR for SkillDRE versus 97.34%–100% for the baselines. With Gemini-3.7-Flash as the alternative backend, only 8.03%–11.24% of the final skills are flagged, still substantially below the baseline detection rates reported under the original configuration. Among the originally successful attacks, only 1.20%–2.41% are flagged, showing low detection even after the backend change. Some variation is expected because optimization uses configuration-specific defense feedback, and a different backend can produce different judgments. These results provide supplementary evidence of transfer across scanner backends, while our principal contribution remains feedback-guided skill evolution against a fixed defense pipeline.

We hope the reviewer will carefully consider these results and what is explicitly stated in our paper when making final assessment. We welcome any further questions and would be happy to address them.

Thank you for your response. I will take some time to learn more about your work and try to understand it from the perspective you mentioned in your latest reply.

However, I would like to clarify one point: the second question I am concerned about is "stability." While I understand that the method performs well in the scenarios you have defined, I am concerned that even a change in the backend alone could potentially cause your proposed method to become unstable.

If there any update of your final score, I will let you know.

Best regards.

Thank you for clarifying your concern. As of now, SkillSonar is the only publicly available runtime defense implemented as an in-agent skill. Regarding the pre-execution defense, we will clarify the scope of our current evidence and explain our method in greater detail. Given the substantial time and computational costs involved, we will also consider including evolution experiments with additional defense backend models in the final version to further assess stability. We hope you will understand the substantial time, effort, and computational resources required for these additional experiments.

Thank you again for your time and consideration. We wish you all the best.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.