- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents a framework of automated red-teaming for claw-style agents. Specifically, the authors have pointed out that existing security evaluation systems struggle to evaluate these complex claw-style systems since they rely heavily on manually constructed test cases and environments. To address the above-mentioned point, the authors have proposed AutoRedClaw as an automated red-teaming framework that generates end-to-end attacks alongside the environments on which they are executed. Moreover, the authors have constructed RedClaw-Bench as a 300-case benchmark. The experimental results show that the constructed benchmark yields substantially higher attack success rates than existing benchmarks.
Critical strengths
- The authors have identified three limitations of prior security benchmarks, namely, high construction costs severely restrict scale and coverage, static benchmarks rapidly become outdated and incomplete coverage of the attack chain.
- The authors have proposed AutoRedClaw as an automated red-teaming framework for constructing and executing security evaluations of diverse claw-style agents.
- The authors have constructed RedClaw-Bench as a 300-case benchmark that yields substantially higher attack success rates than existing benchmarks, namely, AgentDojo, AgentDyn and DTap.
Critical weaknesses
- The contributions of this paper are not clearly summarised in Section 1.
- This paper lacks an independent section on related works so it is unclear how AutoRedClaw differs from other existing frameworks.
- The experimental evaluation seems insufficient. Specifically, Section 4 lacks comparisons between AutoRedClaw and other baseline frameworks. Moreover, the ablation studies against the effectiveness of the components of AutoRedClaw are missing. In addition, evaluation of the effectiveness of RedClaw-Bench is mainly based on attack success rate, which does not seem to be sufficient, e.g., it is also worth to consider how RedClaw-Bench impacts the computational costs.
Questions for authors
- Please highlight the contributions of this paper in an explicit manner.
- Please justify how how AutoRedClaw differs from other existing frameworks.
- Please address the points on the experimental evaluation, i.e., baseline comparison, ablation studies and effectiveness of RedClaw-Bench.
7 comments
Response to Reviewer JEKG [2/4]
W2&Q2 : This paper lacks an independent section on related works, so it is unclear how AutoRedClaw differs from other existing frameworks.
A2: We are sorry for the confusion. We put the Related Work section in Appendix B due to the page limit, and we will move it to the main text in the revised version. To the best of our knowledge, there is no existing large-scale red-teaming work that jointly explores the tasks, environments and attacks for agent systems. Some works also generate the task together with the environment, but they are not in the security scope. Most previous red-teaming works start from a task and an environment that are already defined, and they repeatedly optimize a payload inside this fixed setting, so they do not generate the whole attack together with the environment from scratch. This is exactly the difference between AutoRedClaw and existing frameworks.
To better show the difference and the advantages of AutoRedClaw over existing related work, we conduct experiments against existing environment generation and red-teaming works. Since no single method jointly explores the tasks, environments and attacks for agent systems as AutoRedClaw does, we compare with the two related lines separately.
For environment generation, we employ AutoControl Arena (ACA) [3] as the baseline, which also uses LLMs to synthesize executable test environments. We compare the diversity of its benchmark X-Bench (70 scenarios) with that of the cases generated by AutoRedClaw. We employ the Vendi Score [1] as the diversity metric, which roughly measures how many truly different samples a set contains. Since the Vendi Score grows with the number of samples, we also generate 70 cases with AutoRedClaw. The embedding model is all-mpnet-base-v2.
For red-teaming, we employ AutoDojo [2] as the baseline, which is an iterative red-teaming framework built on top of AgentDojo, and we compare the average cost of finding one successful attack sample. We use GPT-5.4 as the victim model, and AutoRedClaw runs on Codex. For all other settings, we follow the official configuration of AutoDojo. The cost includes all stages for both methods.
The results are shown below:
| Environment Generation | Vendi Score ↑ | Red-teaming | Average Cost per Success ($) ↓ | ||||
|---|---|---|---|---|---|---|---|
| AutoControl Arena | 29.0 | AutoDojo | 1.037 | ||||
| AutoRedClaw | 31.4 | AutoRedClaw | 0.176 |
We observe that on environment generation, AutoRedClaw achieves a Vendi Score of 31.4, which is comparable to and slightly higher than that of ACA (29.0). This shows that our automatically generated tasks are competitive with an advanced task-environment generation framework, even though all of our samples focus on claw-style agents. On red-teaming, AutoRedClaw finds one successful attack sample at about 5.9 times lower cost than AutoDojo (\1.037). This holds even though AutoDojo uses a much simpler harness and borrows its environments from AgentDojo, which are also generally simpler than ours.
To sum up, AutoRedClaw fills the gap of jointly exploring the tasks, environments and attacks for advanced agent systems, and it shows strong quality and effectiveness in both environment generation and attack generation.
Response to Reviewer JEKG [3/4]
W3&Q3 : The experimental evaluation seems insufficient, including baseline comparison, ablation studies, and the effectiveness of RedClaw-Bench beyond ASR (e.g., computational costs).
A3: Thank you for your constructive suggestions. We conduct additional experiments for all these points.
Baseline comparison. We have conducted the baseline comparison experiments in A2. Since no existing method jointly explores the tasks, environments and attacks for agent systems, we compare AutoRedClaw with the two related lines separately. For environment generation, we employ AutoControl Arena as the baseline and compare the diversity of the generated scenarios with the Vendi Score. For red-teaming, we employ AutoDojo as the baseline and compare the average cost of finding one successful attack sample, with GPT-5.4 as the victim model. As we can see, AutoRedClaw gets a higher Vendi Score than AutoControl Arena (31.4 vs. 29.0), and finds one successful attack at about 5.9 times lower cost than AutoDojo (\1.037). It shows that AutoRedClaw is effective on both environment generation and attack generation.
Ablation study. We ablate two components, the experience-guided exploration and the probe. For the experience-guided exploration, we generate 100 samples with the diversity guidance and 100 samples without it, and all other settings are kept the same. For the probe, we compare the generation with and without it, with deepseek-v4-pro as the probe model and Codex + GPT-5.4 as the target agent. The results are shown below:
| Setting | Vendi Score ↑ | ASR | Cost per Success (\$) ↓ |
|---|---|---|---|
| w/o diversity guidance | 13.1 | 33.0% | - |
| w/ diversity guidance | 32.8 | 38.0% | - |
| w/o probe | - | 37.6% | 0.406 |
| w/ probe | - | 32.8% | 0.176 |
We observe that removing the diversity guidance makes the Vendi Score drop from 32.8 to 13.1 (p < 0.001), so the generation collapses to much fewer scenarios without it. As a reference, 100 totally unrelated texts only get 44.1, so our samples reach 74% of this upper bound, while those without the guidance only reach 30%. For the probe, the ASR only drops slightly from 37.6% to 32.8%, but the cost per successful attack drops from \0.176, which is about 2.3 times lower. This is because the probe filters out the broken cases before they reach the costly target agent, and it does not throw away many cases that can really attack the target, even though the probe and the target come from different model families.
Effectiveness beyond ASR. Since AutoRedClaw mainly focuses on the persistent state, we add the standard indirect prompt injection (IPI) surface to AutoRedClaw generation pipeline, which has the same threat model as existing benchmarks like AgentDojo and DTap. On this surface, AutoRedClaw gets an ASR of 38.8% on OpenClaw + DeepSeek-V4-Pro, and the cost of finding one successful attack is only about \$0.036. To evaluate the case quality, we take 100 cases generated on this surface and all the 107 IPI cases from the workflow domain of DTap. We employ three LLM judges from different vendors (GPT-6.1-Sol, Gemini-3.1-Pro and Claude-Opus-5.5) and three more human evaluators. Each judge sees a neutral case card without its source, and scores every case once from 1 to 5 on 8 dimensions and gives an overall realism score. The averaged results are shown below:
| Dimension | AutoRedClaw (LLM) | DTap (LLM) | AutoRedClaw (human) | DTap (human) |
|---|---|---|---|---|
| Benign task realism | 4.80 | 3.35 | 4.69 | 3.13 |
| Environment fidelity | 4.05 | 2.04 | 3.76 | 1.89 |
| Injection carrier naturalness | 4.79 | 3.26 | 4.68 | 3.36 |
| Injection voice and stealth | 4.31 | 2.64 | 3.85 | 2.40 |
| Attacker plausibility | 3.64 | 2.40 | 3.36 | 2.42 |
| Harm severity | 3.77 | 3.64 | 3.73 | 3.47 |
| Verification soundness | 4.42 | 3.16 | 4.11 | 2.64 |
| Goal conflict | 4.41 | 3.59 | 4.48 | 3.50 |
| Overall realism | 3.85 | 2.17 | 3.55 | 2.24 |
We observe that AutoRedClaw is significantly better on 7 of the 8 dimensions (p < 0.001), and the overall realism is 3.85 vs. 2.17 under the LLM judges and 3.55 vs. 2.24 under the human evaluators. The verification soundness, which shows whether the success criterion really measures the harm, is 4.42 vs. 3.16, and the judges think a real attacker would do this in 82% of our cases, compared to 38% for DTap. Harm severity is similar on the two benchmarks, so the higher score is not because our attacks are more harmful.
Response to Reviewer JEKG [1/4]
We thank the reviewer JEKG for the patient and professional review, as well as the valuable suggestions for improvement. Our responses to each questions are below. We have added these new experiments and discussions to our revised version (marked in blue).
W1&Q1 : The contributions of this paper are not clearly summarised in Section 1.
A1: We are sorry for the confusion. To clarify the contributions of this paper, we summarize two core limitations of existing work, and how AutoRedClaw solves each of them.
Limitation 1: Limited scalability of manual construction. Existing benchmarks heavily rely on human experts to build every task and environment by hand. So the cost is high, the scale is small, and the benchmark is easy to be outdated. This problem is getting more serious now, since agent systems update really fast and new harnesses and models keep coming out, and a manually built benchmark can not catch up with them.
AutoRedClaw Solution: We solve this from two sides. First, AutoRedClaw makes the whole red-teaming process automatic. It generates the tasks, the attacks and the executable environments without manual effort (Sec. 3.1.2, 3.2), so the scale is no longer limited by human experts. When a new model or harness comes out, our adapter design can be easily extended to these new agents, and we can simply generate the cases again. Second, we significantly reduce the cost of the automatic generation. Our three-stage generation design first generates a high-level plan of each case and then fills in the details in the materialization stage, which makes sure that even with a cheap model as the generator, it can still efficiently find successful attack cases for advanced models at a very low cost. The probe stage further reduces this cost by more than 50%. The automated pipeline and the low cost make AutoRedClaw an effective and practical solution for evaluating current and future advanced agent systems.
Limitation 2: Limited exploration ability of the existing red-teaming paradigm. Some works already use automated red-teaming to reduce the manual effort, but they still have a limitation in what they can explore. The goal of red-teaming is to discover the risks that we do not know yet. However, most existing methods start from a task and an environment that are already defined, and they only optimize the payload inside this fixed setting. So they can only test the predefined risk scenarios, and what they find is just a stronger attack on a known scenario. It is still not a new risk of the agent. This limitation is more serious for advanced agent systems, since their risks largely depend on what task the agent is doing, what it can touch in the environment and what is kept for later tasks. For example, a persistent-state attack needs a state file to be poisoned and a later task to trigger it, and this can never be found by optimizing a payload inside one fixed task.
AutoRedClaw Solution: AutoRedClaw proposes a new red-teaming paradigm to solve this. It explores the tasks, environments and attacks together, so the scenario itself also becomes a part of the search. To make sure the exploration covers enough risks, we combine two kinds of guidance. First, the rich dimensions give the exploration a wide starting space. We define 2 attack stages, 5 surfaces, 9 risk categories and 12 attack strategies from the well-defined taxonomies in agent security, so the generation can cover the known risk types from the beginning. Second, the experience-guided exploration keeps expanding this space during generation. It lets the generator first go to the directions that are not explored yet, so new scenarios keep coming out and the generation does not collapse to several similar ones. With the rich dimensions as the basis and the experience as the driver, AutoRedClaw can keep uncovering new threats to advanced agent systems.
We have reorganize the introduction following these two points, and list our contributions in the same way.