- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies resource-amplification attacks through reusable LLM agent skills and proposes AutoSkillDrain, a black-box framework that automatically transforms benign public skills into resource-draining variants. The framework preserves the source skill structure and employs four role-specialized optimizer agents—Deconstructor, Polymath, Skeptic, and Bureaucrat—to construct amplification policies. It then iteratively refines these policies using black-box execution feedback, output-token rewards, and supervised judge feedback. The paper further introduces SkillDrainBench, containing 100 source–optimized skill pairs and 500 tasks across 12 domains. Experiments across PydanticAI, OpenClaw, Claude Code, and multiple backend models show substantial output-token amplification while largely retaining task completion and transferring across agents and models.
Critical strengths
- The paper identifies a timely and practically relevant attack surface. Treating natural-language skills as supply-chain artifacts that can alter agent control flow is well motivated.
- The evaluation is relatively comprehensive. The paper evaluates three agent backends and multiple backend models, and additionally studies cross-agent and cross-model transferability. 22742_AutoSkillDrain_Resource_
- The paper evaluates more than token count. It also reports latency, throughput, completion rate, transferability, scanner detection, and component ablations.
- The ablation study provides clear evidence that iterative optimization and the four role agents contribute differently to resource amplification.
Critical weaknesses
- The attack-development cost is not sufficiently quantified. AutoSkillDrain repeatedly executes the target agent during iterative optimization and may evaluate multiple suffix-policy candidates per iteration. The paper acknowledges that this cost can be non-negligible, but does not report the number of victim executions, tokens, latency, or monetary cost required to obtain one optimized skill.
- Output-token amplification alone does not fully capture practical cost amplification. AutoSkillDrain also increases input tokens substantially—for example, from roughly 2.1k to 4.3k on PydanticAI. Reporting provider-weighted monetary cost or total compute cost would better support the resource-amplification claim.
- The utility evaluation relies entirely on an LLM judge. GPT-5.4 is used to determine whether the final response successfully completes the original task, but no human validation or judge agreement is reported. Since attacked outputs are substantially longer, the evaluator may systematically favor verbose answers.
- The scanner results should be interpreted cautiously. Existing scanners block only a small fraction of AutoSkillDrain skills, but these scanners may not have been designed specifically for resource-amplification attacks. Therefore, low blocking rates demonstrate a detection gap, but not necessarily sophisticated scanner evasion.
Questions for authors
Refer to weaknesses
We thank the reviewer for the constructive feedback. We address the concerns as follows.
W1: The attack-development cost is not sufficiently quantified.
We analyzed the optimization traces of all 100 skills and quantified the construction cost. On average, constructing one optimized skill requires 17.46 target-agent executions. The following table reports the average token consumption, latency, and monetary cost per skill across different optimization stages. The monetary costs are estimated using GPT-5.4-mini pricing ($0.75/M input and $4.50/M output tokens).
\text{Stage} & \text{Input} & \text{Output} & \text{Total} & \text{Latency (min)} & \text{Cost (\\$)} \\\\ \hline \text{Target Agent} & 105{,}241 & 106{,}930 & 212{,}171 & 19.85 & 0.560 \\\\ \text{Optimization Agents} & 192{,}658 & 122{,}110 & 314{,}768 & 24.07 & 0.694 \\\\ \text{Judge Agent} & 140{,}023 & 10{,}027 & 150{,}050 & 3.27 & 0.150 \\\\ \hline \mathbf{Total} & \mathbf{437{,}922} & \mathbf{239{,}067} & \mathbf{676{,}989} & \mathbf{47.19} & \mathbf{1.404} \end{array}$$ These results show that constructing one optimized skill requires only 17.46 target-agent executions, consuming approximately 677K tokens and \$1.40, with an average construction time of 47.19 minutes. Despite the iterative multi-agent optimization process, the monetary cost remains relatively modest, demonstrating that AutoSkillDrain can construct effective resource-amplifying skills with a manageable black-box query budget. Moreover, each skill is optimized only once and can be reused across multiple tasks and agent executions without further optimization. Thus, the already modest construction cost can be further amortized over repeated deployments. We will incorporate this quantitative analysis into the revised manuscript. W2: Output-token amplification alone does not fully capture practical cost amplification. We agree that monetary cost provides an important complementary measure of resource amplification. However, input and output tokens differ substantially in both pricing and computational overhead. For example, GPT-5.4-mini charges \\$0.75/M input tokens, \\$0.075/M cached input tokens, and \\$4.50/M output tokens. Thus, output tokens are 6 times more expensive than regular input tokens and 60 times more expensive than cached input tokens. Using these prices and the average token counts in Table 1, the estimated cost on PydanticAI increases from \\$0.0119 for benign execution to \\$0.0298 for AutoSkillDrain, corresponding to approximately 2.50\$\times\$ monetary-cost amplification. Under the same pricing assumptions, AutoDoS costs \\$0.0265, compared with \\$0.0298 for ours. W3: The utility evaluation relies entirely on an LLM judge. To address this concern, we conducted an additional human audit of 60 matched benign--attacked pairs. Five human annotators independently evaluated each output against the original task requirements, including required artifacts or recorded effects where applicable. The final human label was determined by majority vote (at least three of five annotators). $$\begin{array}{l|cc} \text{Metric} & \text{Benign} & \text{Attacked} \\\\ \hline \text{Audited Outputs} & 60 & 60 \\\\ \text{GPT--Human Agreement} & 58/60\\;(96.7\\%) & 56/60\\;(93.3\\%) \\\\ \text{Rejected GPT-PASS} & 1/58\\;(1.7\\%) & 3/56\\;(5.4\\%) \end{array}$$ These results provide independent evidence supporting the reliability of our completion evaluation. In particular, GPT--human agreement reaches 93.3\% on attacked outputs, and only 5.4\% of GPT-PASS attacked outputs are rejected by human annotators. W4: The scanner results should be interpreted cautiously. We agree that the evaluated scanners were not specifically designed to detect resource-amplification attacks. Our intention is not to claim sophisticated evasion of purpose-built defenses, but rather to highlight a detection gap in existing skill-screening mechanisms. As shown in Figure 4, ClawScan blocks 99\% of AutoDoS skills versus only 10\% of ours, while SkillSpector blocks 91\% versus 18\%, and AgentVerus blocks 21\% versus 1\%, respectively. These results demonstrate that existing scanners are substantially less effective at identifying AutoSkillDrain's resource-amplifying skill transformations than the compared attack baselines. We will emphasize in the revised manuscript that these results reveal the limitations of current general-purpose skill scanners.