acceptodds
Under review as a conference paper at ICLR 2027

AutoSkillDrain: Resource Amplification Attacks on LLM Agents via Malicious Skill Optimization

Abstract

LLM agents increasingly rely on reusable skills to acquire domain-specific capabilities, but this extensibility also creates a new supply-chain surface for resource-amplification attacks. A malicious skill can induce prolonged generation, thereby increasing cost, latency, and service pressure. To systematically study this threat, we propose AutoSkillDrain, the first black-box resource-amplification attack framework that automatically transforms benign agent skills into resource-draining variants without access to the victim agent's system prompts, or internal execution policies. We further introduce SkillDrainBench, a benchmark consisting of public source skills, optimized malicious counterparts, and task instances for evaluating amplification, utility, transferability, and detectability. Across experiments on three agent backends and multiple models, AutoSkillDrain achieves the strongest resource amplification among compared methods, with 2.57, 2.15, and 2.28 average output amplification on PydanticAI, OpenClaw, and Claude Code, respectively, while maintaining over 91% task completion. These results expose an underexplored resource-security risk in open skill ecosystems and highlight the need for validation before deployment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
agent skills, resource amplification Attacks, LLMs agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hxTKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies resource-amplification attacks through reusable LLM agent skills and proposes AutoSkillDrain, a black-box framework that automatically transforms benign public skills into resource-draining variants. The framework preserves the source skill structure and employs four role-specialized optimizer agents—Deconstructor, Polymath, Skeptic, and Bureaucrat—to construct amplification policies. It then iteratively refines these policies using black-box execution feedback, output-token rewards, and supervised judge feedback. The paper further introduces SkillDrainBench, containing 100 source–optimized skill pairs and 500 tasks across 12 domains. Experiments across PydanticAI, OpenClaw, Claude Code, and multiple backend models show substantial output-token amplification while largely retaining task completion and transferring across agents and models.

Critical strengths
  1. The paper identifies a timely and practically relevant attack surface. Treating natural-language skills as supply-chain artifacts that can alter agent control flow is well motivated.
  2. The evaluation is relatively comprehensive. The paper evaluates three agent backends and multiple backend models, and additionally studies cross-agent and cross-model transferability. 22742_AutoSkillDrain_Resource_
  3. The paper evaluates more than token count. It also reports latency, throughput, completion rate, transferability, scanner detection, and component ablations.
  4. The ablation study provides clear evidence that iterative optimization and the four role agents contribute differently to resource amplification.
Critical weaknesses
  1. The attack-development cost is not sufficiently quantified. AutoSkillDrain repeatedly executes the target agent during iterative optimization and may evaluate multiple suffix-policy candidates per iteration. The paper acknowledges that this cost can be non-negligible, but does not report the number of victim executions, tokens, latency, or monetary cost required to obtain one optimized skill.
  2. Output-token amplification alone does not fully capture practical cost amplification. AutoSkillDrain also increases input tokens substantially—for example, from roughly 2.1k to 4.3k on PydanticAI. Reporting provider-weighted monetary cost or total compute cost would better support the resource-amplification claim.
  3. The utility evaluation relies entirely on an LLM judge. GPT-5.4 is used to determine whether the final response successfully completes the original task, but no human validation or judge agreement is reported. Since attacked outputs are substantially longer, the evaluator may systematically favor verbose answers.
  4. The scanner results should be interpreted cautiously. Existing scanners block only a small fraction of AutoSkillDrain skills, but these scanners may not have been designed specifically for resource-amplification attacks. Therefore, low blocking rates demonstrate a detection gap, but not necessarily sophisticated scanner evasion.
Questions for authors

Refer to weaknesses

1 comment

We thank the reviewer for the constructive feedback. We address the concerns as follows.

W1: The attack-development cost is not sufficiently quantified.

We analyzed the optimization traces of all 100 skills and quantified the construction cost. On average, constructing one optimized skill requires 17.46 target-agent executions. The following table reports the average token consumption, latency, and monetary cost per skill across different optimization stages. The monetary costs are estimated using GPT-5.4-mini pricing ($0.75/M input and $4.50/M output tokens).

\text{Stage} & \text{Input} & \text{Output} & \text{Total} & \text{Latency (min)} & \text{Cost (\\$)} \\\\ \hline \text{Target Agent} & 105{,}241 & 106{,}930 & 212{,}171 & 19.85 & 0.560 \\\\ \text{Optimization Agents} & 192{,}658 & 122{,}110 & 314{,}768 & 24.07 & 0.694 \\\\ \text{Judge Agent} & 140{,}023 & 10{,}027 & 150{,}050 & 3.27 & 0.150 \\\\ \hline \mathbf{Total} & \mathbf{437{,}922} & \mathbf{239{,}067} & \mathbf{676{,}989} & \mathbf{47.19} & \mathbf{1.404} \end{array}$$ These results show that constructing one optimized skill requires only 17.46 target-agent executions, consuming approximately 677K tokens and \$1.40, with an average construction time of 47.19 minutes. Despite the iterative multi-agent optimization process, the monetary cost remains relatively modest, demonstrating that AutoSkillDrain can construct effective resource-amplifying skills with a manageable black-box query budget. Moreover, each skill is optimized only once and can be reused across multiple tasks and agent executions without further optimization. Thus, the already modest construction cost can be further amortized over repeated deployments. We will incorporate this quantitative analysis into the revised manuscript. W2: Output-token amplification alone does not fully capture practical cost amplification. We agree that monetary cost provides an important complementary measure of resource amplification. However, input and output tokens differ substantially in both pricing and computational overhead. For example, GPT-5.4-mini charges \\$0.75/M input tokens, \\$0.075/M cached input tokens, and \\$4.50/M output tokens. Thus, output tokens are 6 times more expensive than regular input tokens and 60 times more expensive than cached input tokens. Using these prices and the average token counts in Table 1, the estimated cost on PydanticAI increases from \\$0.0119 for benign execution to \\$0.0298 for AutoSkillDrain, corresponding to approximately 2.50\$\times\$ monetary-cost amplification. Under the same pricing assumptions, AutoDoS costs \\$0.0265, compared with \\$0.0298 for ours. W3: The utility evaluation relies entirely on an LLM judge. To address this concern, we conducted an additional human audit of 60 matched benign--attacked pairs. Five human annotators independently evaluated each output against the original task requirements, including required artifacts or recorded effects where applicable. The final human label was determined by majority vote (at least three of five annotators). $$\begin{array}{l|cc} \text{Metric} & \text{Benign} & \text{Attacked} \\\\ \hline \text{Audited Outputs} & 60 & 60 \\\\ \text{GPT--Human Agreement} & 58/60\\;(96.7\\%) & 56/60\\;(93.3\\%) \\\\ \text{Rejected GPT-PASS} & 1/58\\;(1.7\\%) & 3/56\\;(5.4\\%) \end{array}$$ These results provide independent evidence supporting the reliability of our completion evaluation. In particular, GPT--human agreement reaches 93.3\% on attacked outputs, and only 5.4\% of GPT-PASS attacked outputs are rejected by human annotators. W4: The scanner results should be interpreted cautiously. We agree that the evaluated scanners were not specifically designed to detect resource-amplification attacks. Our intention is not to claim sophisticated evasion of purpose-built defenses, but rather to highlight a detection gap in existing skill-screening mechanisms. As shown in Figure 4, ClawScan blocks 99\% of AutoDoS skills versus only 10\% of ours, while SkillSpector blocks 91\% versus 18\%, and AgentVerus blocks 21\% versus 1\%, respectively. These results demonstrate that existing scanners are substantially less effective at identifying AutoSkillDrain's resource-amplifying skill transformations than the compared attack baselines. We will emphasize in the revised manuscript that these results reveal the limitations of current general-purpose skill scanners.
Reviewer_srRmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies resource amplification attacks on LLM agent skills and proposes the AutoSkillDrain framework. It uses four optimizer agents to optimize skills through black-box execution iterations to increase the number of output tokens while keeping the original task ability. The author also constructs SkillDrainBench, which contains 100 pairs of normal and malicious skills and 500 tasks. Testing on PydanticAI, OpenClaw, and Claude Code, it achieves 2.57x, 2.15x, and 2.28x output tokens, with task completing rate as high as 91%. It also evaluates the trans-model attack ability and current safety alignment and skill-screening mechanisms.

Critical strengths
  1. The research points is meaningful. The paper focus on resource-amplification attacks in LLM Agent Skill supply chain. It points out that malicious skill can change execution workflow and stopping behavior of a Agent to increase the token usage, instead of directly change user request. This provides meaningful research direction in Agent security.
Critical weaknesses
  1. The core evaluation cannot support the advantage of resource-amplification. As shown in table 1, although AutoSkillDrain has higher output-token amplification, AutoDoS have more tokens (I/O) in three agents. It does not report actual monetary cost. Also, it does not compare with AutoDoS in service-level evaluation. Hence, current experiments cannoy prove that AutoSkillDrain has more total resource usage than current method, which limits current paper's contribution.
  2. Lack of novelty and baselines. The four role-specialized optimizer agents combines existing task decomposition, multi-perspective analysis, self-critique, and iterative prompt optimization. Although Table 3 has component abaltions, it does not compare with single-agent prompt optimizer or skills which directly ask for adding output tokens. Hence, current experiment cannot prove the complex multi-agent optimization framework is necessary.
Questions for authors
  1. Table 1 shows that AutoDoS has more total tokens (I/O) than AutoSkillDrain across all three Agents. Could the authors report the actual monetary cost and clarify whether AutoSkillDrain achieves more resource consumption than AutoDoS?
  2. Could the authors clarify whether AutoSkillDrain has been compared with a simpler single-agent prompt optimizer under the same optimization budget?
3 comments

We thank the reviewer for the constructive feedback. We address the concerns as follows.

W1 & Q1:

We first clarify that AutoSkillDrain primarily targets output-token amplification, as explicitly defined in Section 3.2 (Eq. 2), rather than maximizing total token consumption. Output tokens generally incur higher API costs and greater autoregressive generation overhead than input tokens. Although AutoDoS consumes more total tokens (input + output), as shown in Table 1, AutoSkillDrain consistently achieves higher output-token amplification and task completion rates across all three agents.

To further quantify the economic impact, we consider monetary API costs as a complementary metric. For example, GPT-5.4-mini charges \0.75permillioninputtokens,0.75 per million input tokens, \\0.075 per million cached input tokens, and \4.50permillionoutputtokens.Thus,outputtokensare6timesmoreexpensivethanregularinputtokensand60timesmoreexpensivethancachedinputtokens.UsingtheseratesandthetokencountsreportedinTable1,AutoSkillDrainincursanestimatedcostof4.50 per million output tokens. Thus, output tokens are 6 times more expensive than regular input tokens and 60 times more expensive than cached input tokens. Using these rates and the token counts reported in Table 1, AutoSkillDrain incurs an estimated cost of \\0.0298 per run on PydanticAI, compared with \$0.0265 for AutoDoS, despite consuming fewer total tokens. This demonstrates that higher total token consumption does not necessarily translate into higher monetary costs.

We have also conducted additional service-level experiments comparing AutoSkillDrain with AutoDoS under the same workloads and agent configurations.

\text{Agent} & \text{Method} & \text{Throughput} & \text{Latency (s)} & \text{Latency Ratio} \\\\ \hline \text{PydanticAI} & \text{Benign} & 0.035 & 26.98 & 1.00 \\\\ & \text{AutoDoS} & 0.024 & 38.20 & 1.42 \\\\ & \text{AutoSkillDrain} & \mathbf{0.014} & \mathbf{55.97} & \mathbf{2.07} \\\\ \hline \text{OpenClaw} & \text{Benign} & 0.046 & 20.06 & 1.00 \\\\ & \text{AutoDoS} & 0.039 & 23.40 & 1.17 \\\\ & \text{AutoSkillDrain} & \mathbf{0.024} & \mathbf{33.14} & \mathbf{1.65} \\\\ \hline \text{Claude Code} & \text{Benign} & 0.035 & 28.85 & 1.00 \\\\ & \text{AutoDoS} & 0.025 & 37.00 & 1.28 \\\\ & \text{AutoSkillDrain} & \mathbf{0.016} & \mathbf{64.13} & \mathbf{2.22} \end{array}$$ Across all three agents, AutoSkillDrain causes greater throughput degradation and higher execution latency than AutoDoS. Beyond resource consumption and service-level impact, attack effectiveness also depends on task completion and detectability. As shown in Table 1, AutoSkillDrain consistently achieves higher task completion rates than AutoDoS. Moreover, Figure 4 demonstrates that AutoDoS is substantially more detectable: ClawScan blocks 99\% of AutoDoS skills versus only 10\% of AutoSkillDrain skills, while SkillSpector blocks 91\% versus 18\%, and AgentVerus blocks 21\% versus 1\%. In summary, these results highlight the limitations of evaluating attacks solely based on total token consumption. Simply inflating input tokens does not necessarily result in higher monetary costs or greater service disruption, particularly when accompanied by lower task completion rates and higher detectability. In realistic skill ecosystems, an effective attack must not only impose substantial resource overhead but also preserve the advertised functionality and evade pre-deployment detection. Otherwise, the malicious skill may fail to complete the user's task or be blocked before execution, limiting its practical impact. AutoSkillDrain achieves a stronger balance across these dimensions, demonstrating its practical effectiveness beyond raw token consumption. W2 \& Q2: Notably, AutoDoS already serves as an automated prompt-optimization baseline in our evaluation. As described in Appendix A.2, we adapt AutoDoS to the skill setting by optimizing a resource-amplifying prompt and inserting it into the original skill. AutoSkillDrain consistently outperforms AutoDoS in output-token amplification across all three agents. Nevertheless, we agree that AutoDoS does not directly isolate the contribution of multi-agent specialization. To further address this concern, we additionally consider a Single-Agent Optimizer, replacing the four role-specialized optimizers with a single agent responsible for jointly generating and refining all amplification policies. Both variants use the same source skills, optimization tasks, execution feedback, and judge mechanism, with matched optimizer-token budgets. $$\begin{array}{l|cc} \text{Method} & \text{Output Amp.} & \text{Completion Rate} \\\\ \hline \text{AutoDoS} & 1.65 & 86.2\\% \\\\ \text{Single-Agent Optimizer} & 2.13 & 88.9\\% \\\\ \text{AutoSkillDrain (Ours)} & \mathbf{2.57} & \mathbf{92.7\\%} \end{array}$$ These results show that AutoSkillDrain achieves 20.7\% higher output-token amplification than the single-agent optimizer, while improving task completion by 3.8\% under the same optimization budget. This demonstrates the benefits of role-specialized optimization beyond simply increasing optimization resources.

Thanks for this extra cost analysis. My concerns are well resolved and will change my judgment.

Thank you for your positive feedback. We are glad that our additional analysis has addressed your concerns.

We would greatly appreciate it if you could reflect your updated assessment in the review rating, if possible.

Reviewer_PU4gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies resource-amplification attacks delivered through reusable agent skills. AutoSkillDrain uses a structure-preserving adapter and four specialized rewriting roles, then refines candidates using black-box execution feedback. SkillDrainBench contains 100 public source skills and optimized counterparts across 12 domains, with one optimization task and four held-out tasks per skill. The evaluation reports ten fresh repetitions per condition, comparisons across three agent backends, transfer experiments, component ablations, and skill-scanner results. Reported output-token amplification is 2.57×, 2.15×, and 2.28× on PydanticAI, OpenClaw, and Claude Code, respectively, with automatically judged completion rates above 91%.

Critical strengths

S1. The benchmark and held-out protocol address a practically relevant threat.

Transforming heterogeneous public skills is a useful extension beyond demonstrating resource exhaustion with a few handcrafted prompts. The stated protocol freezes four evaluation tasks before optimization, excludes their prompts and traces from the search, and evaluates the frozen skill in fresh sessions. The coverage of three agent backends and multiple models provides meaningful evidence that the observed resource increases are not confined to one optimization task or one agent wrapper.

S2. The evaluation provides evidence for iterative refinement and examines several aspects of the attack.

Table 3 reports an improvement from 2.09× amplification without iterative optimization to 2.57× for the full method, despite slightly lower input-token consumption in the full method. The role-removal ablations also identify influential parts of the construction. Reporting completion alongside token usage, measuring execution time, and including benign controls in the scanner study are useful choices. These results establish a nontrivial resource-use effect and motivate studying skill-based attacks further.

Critical weaknesses

W1. The retained-functionality claim needs stronger output validation and a joint view of amplification and completion.

Appendix A.3.3 uses a separate GPT-5.4 evaluator with a fixed rubric and blinding to condition and resource measurements. These are useful safeguards. However, the manual inspection in A.3.2 concerns task suitability, and the paper does not report an independent check of the final completion labels. Because the attack changes the length and structure of outputs, label reliability matters directly to the claim that amplification preserves useful functionality. Aggregate amplification and completion rates also leave unclear how often the two occur on the same skill–task conditions.

Q1 — Clarification and targeted validation: What evidence supports the completion labels on attacked outputs? Please provide a joint skill-level amplification/completion summary. If independent validation has not been performed, a focused audit of benign and attacked outputs, including checks of delivered artifacts or effects where required, would directly address this concern. All-run token totals should remain the primary resource metric.

W2. The comparative contribution of the optimizer needs a clearer connection to the evaluated implementation and controls.

Appendix A.3 specifies one canonical optimized counterpart per skill, all optimized using GPT-5.4-mini. The compact release nevertheless has a default coverage schedule of 0.30–0.80 versus Section 4.3's 0.5–0.9, and combines feedback in judge_and_patch where Equations (8)–(9) describe separate judge/supervisor stages. The evaluated configuration is therefore important for interpreting these differences. Table 1 matches the PydanticAI-source row of Figure 3b; under that mapping, the OpenClaw and Claude Code results would be transfer results lower than their respective in-environment values.

Q2 — Clarification of the reported experiments: Please map Tables 1 and 3 to the implementation, source/target agent and model, and actual construction and execution limits, including any overrides of the public defaults. Does Table 1 evaluate the canonical skills transferred from PydanticAI?

The static and role-removal ablations support the proposed pipeline, including the improvement from 2.09× to 2.57×. They provide less evidence about its incremental advantage over a simpler feedback-based rewriter with the same behavioral goals and comparable construction resources.

Q3 — Additional validation relevant to the method claim: A focused matched-resource comparison with such a rewriter would help establish that incremental advantage; alternatively, please limit the comparative claim to what the existing component ablations establish.

Overall assessment. The benchmark, held-out protocol, and resource increases are useful contributions. I currently lean toward weak rejection because retained functionality and the incremental value of the optimizer are central claims, and W1–W2 leave their empirical support incomplete. The configuration clarification and focused validation above could materially change that judgment. Additional descriptive statistics alone are not the deciding issue.

Questions for authors

See weaknesses

3 comments

We thank the reviewer for the constructive feedback. We address the concerns as follows.

W1 & Q1: Completion Reliability and Joint Evaluation.

We agree that task-suitability inspection does not independently validate final completion labels. To address this concern, we conducted an additional human audit of 60 matched benign--attacked pairs. Five human annotators independently evaluated each output against the original task requirements, including required artifacts or recorded effects where applicable. The final human label was determined by majority vote (at least three of five annotators).

\text{Metric} & \text{Benign} & \text{Attacked} \\\\ \hline \text{Audited Outputs} & 60 & 60 \\\\ \text{GPT--Human Agreement} & 58/60\\;(96.7\\%) & 56/60\\;(93.3\\%) \\\\ \text{Rejected GPT-PASS} & 1/58\\;(1.7\\%) & 3/56\\;(5.4\\%) \end{array}$$ These results provide independent evidence supporting the reliability of our completion evaluation. In particular, GPT--human agreement reaches 93.3\% on attacked outputs, and only 5.4\% of GPT-PASS attacked outputs are rejected by human annotators. To further address whether amplification and task completion occur simultaneously, we conducted a joint skill--task evaluation across 100 skills and four held-out tasks per skill. We count a condition as successful only when it achieves at least $2\times$ output amplification and both benign and attacked executions complete the task. At the skill level, success requires meeting this criterion on at least three of four tasks. $$\begin{array}{l|cc} \text{Agent} & \text{Jointly Successful Tasks} & \text{Skills }(\geq 3/4) \\\\ \hline \text{PydanticAI} & 268/400\\;(67.0\\%) & 65/100 \\\\ \text{OpenClaw} & 248/400\\;(62.0\\%) & 58/100 \\\\ \text{Claude Code} & 280/400\\;(70.0\\%) & 70/100 \end{array}$$ These results demonstrate that resource amplification and task completion frequently occur under the same skill--task conditions, rather than merely appearing as separate aggregate improvements. Across the three agents, 62--70\% of evaluated conditions simultaneously achieve at least $2\times$ output amplification and successful task completion, with 58--70\% of skills satisfying this criterion on at least three of four tasks. The independent human audit and joint evaluation provide stronger evidence that AutoSkillDrain achieves substantial output-token amplification while preserving task functionality. W2 \& Q2 \& Q3: Implementation Clarification and Optimizer Effectiveness The experiments use the coverage schedule $\alpha_{\mathrm{start}}=0.5$ and $\alpha_{\mathrm{end}}=0.9$, as specified in Section 4.3. The 0.30--0.80 values were not the configuration used for the results. The initial demonstration release omitted the separate Supervisor stage to simplify reproduction. However, our experiments employed the complete Judge--Supervisor pipeline described in Equations (8)--(9). We have now updated the implementation to include the separate Supervisor stage, ensuring consistency with the evaluated framework. We confirm that Table 1 evaluates the same 100 canonical optimized skills constructed using PydanticAI as the source agent, with GPT-5.4-mini as the optimization backbone. These skills are subsequently transferred to OpenClaw and Claude Code without further optimization. We also agree that a simpler feedback-based optimizer provides a useful comparison. Notably, AutoDoS already serves as an automated prompt-optimization baseline in our evaluation. Nevertheless, to directly isolate the contribution of role specialization, we additionally replace the four specialized optimizers with a single agent that jointly generates and refines the amplification policies, using the same source skills, execution feedback, and matched optimizer-token budget. $$\begin{array}{l|cc} \text{Method} & \text{Output Amp.} & \text{Completion Rate} \\\\ \hline \text{AutoDoS} & 1.65 & 86.2\\% \\\\ \text{Single-Agent Optimizer} & 2.13 & 88.9\\% \\\\ \text{AutoSkillDrain (Ours)} & \mathbf{2.57} & \mathbf{92.7\\%} \end{array}$$ As it shown in Table, AutoSkillDrain achieves 20.7\% higher output-token amplification than the single-agent optimizer while improving task completion by 3.8\% under the same optimizer-token budget. These results demonstrate the benefits of role-specialized, execution-guided optimization beyond a simpler feedback-based rewriter.
Reviewer_PU4gOpenReview ↗

Response to the authors

Thank you to the author for the detailed response. The issues regarding manual auditing, the combined evaluation of amplification and completion, configuration instructions, and the single-agent comparison have now been largely resolved. Remaining issues:

  1. Manual auditing. Please explain how these 60 pairs of samples were sampled, and whether the annotators were aware of the samples’ classification (benign or attack). It can be inferred from the table that, for the same batch of samples, the number of ‘completions’ as determined by human assessment was 58/60 for benign samples and 54/60 for attack samples—a decrease of 6.7 percentage points—whilst the GPT assessment showed a decrease of only 3.3 percentage points. Please report this result.
  2. Joint evaluation. Each condition was run 10 times; please define the criteria under which a condition is considered ‘completed’. Furthermore, please include the results of the joint evaluation in the main text and report them alongside the mean values.
  3. Single-agent comparison. Whilst both optimisers share the same optimiser-token budget, the number of times the target agent was executed for the single-agent optimiser has not been reported (17.46 times per skill for AutoSkillDrain). As Algorithm 1 retains the candidate with the highest number of output tokens, please report this count or qualify the conclusions of this comparison accordingly.
  4. Configuration. Please specify in Table 1 and the abstract that the results for OpenClaw and Claude Code were obtained by transferring skills optimised on PydanticAI. Furthermore, please provide the corresponding T_max, H, execution limits and backend models for Tables 1 and 3.
  5. Please incorporate these results and explanations into the revised version.

I will take the answers to these points into account in my final rating.

We thank the reviewer for the constructive follow-up and for acknowledging that our additional evaluations have largely addressed the original concerns. We provide further clarifications below.

  1. Human Audit.

We randomly sampled 60 source skills from the 100 skills in SkillDrainBench, together with their corresponding optimized counterparts. For each selected skill pair, we randomly selected one of its four held-out evaluation tasks and randomly sampled one benign and one attacked output generated for the same task, yielding 60 matched pairs (120 outputs).

Five human annotators independently evaluated each output against the original task requirements. They were blinded to whether the output originated from a benign or attacked skill.

As the reviewer correctly observes, human-assessed task completion decreases from 58/60 (96.7%) for benign outputs to 54/60 (90.0%) for attacked outputs, corresponding to a 6.7-percentage-point reduction, compared with a 3.3-percentage-point reduction under GPT-based evaluation. We have already explicitly reported these results in Section 5.2 of the revised manuscript.

  1. Joint Evaluation.

Each condition is evaluated over 10 independent runs. Condition-level completion is determined by majority voting, requiring at least 6 out of 10 runs to be labeled as complete. We have already incorporated the joint evaluation results into Section 5.2 of the revised manuscript and reported them alongside the mean values.

  1. Single-Agent Comparison.

We explicitly considered this factor when designing the single-agent comparison. In AutoSkillDrain, the backward suffix-policy search performs 1--4 target-agent executions per iteration, depending on whether the early-stopping criterion is satisfied. For a fair comparison, we applied the same per-iteration execution limit (1--4) and early-stopping criterion to the single-agent optimizer, while maintaining the same optimizer-token budget.

On average, the single-agent optimizer requires 19.43 target-agent executions per skill, compared with 17.46 for AutoSkillDrain. Despite performing more target-agent evaluations, the single-agent optimizer achieves only 2.13$\times$ output amplification and 88.9% task completion, compared with 2.57$\times$ and 92.7% for AutoSkillDrain. We have already incorporated these experimental controls and execution counts into Section 5.3 of the revised manuscript.

  1. Configuration.

We have revised the abstract and Table 1 to explicitly clarify that the OpenClaw and Claude Code results are obtained by transferring the 100 canonical skills optimized on PydanticAI.

For both Tables 1 and 3, the experimental configuration uses Tmax⁡=16,000T_{\max}=16{,}000 output tokens, an optimization horizon of H=5H=5, and at most 21 target-agent executions per skill (one initial execution and up to four candidate evaluations per iteration, with early stopping). All evaluated agents in Tables 1 and 3 use GPT-5.4-mini as the victim backend model. We have incorporated these configuration details into Sections 5.2 and 5.3 of the revised manuscript.

  1. Manuscript Revision.

We have incorporated all the requested clarifications and additional results into the revised manuscript. We sincerely appreciate the reviewer's constructive suggestions, which have helped improve the clarity and rigor of our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.