- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OpenART is a large-scale red-teaming framework that evaluates agent security under long-horizon, evolving environmental states rather than isolated prompts. It represents each test case through a benign task, executable workflow, target-visible environment, and hidden safety contract, and introduces Evolutionary Markov Hypergraph Attack (EMHA) to iteratively optimize eight attack surfaces using black-box evaluator feedback. Across 75 configurations, OpenART achieves a pooled strict attack success rate of 85.0%, with its advantage over instruction-only evolution increasing from approximately 2% to more than 17% as task dependency depth and tool-use complexity grow.
Critical strengths
- OpenART evaluates 10,513 validated scenarios across 50 domains and 75 agent–model configurations, while decoupling scenario semantics from runtime-specific execution through adapters.
- The authors propose OpenART, a red-teaming framework that evaluates agent security under long-horizon, evolving environmental states rather than isolated prompts.
- The authors further introduce EMHA, a black-box attack method that models coordinated environmental modifications as paths over a hypergraph.
Critical weaknesses
-
External validation of EMHA. EMHA should be evaluated on established external benchmarks, such as AgentDojo or ASB, to demonstrate that its effectiveness generalizes beyond OpenART.
-
Statistical robustness. The paper should report variance, confidence intervals, or results across multiple independent runs to establish the reproducibility and statistical robustness of the reported improvements.
-
Missing EMHA baselines. EMHA is not directly compared with alternative attack-generation methods. The authors should include competitive baselines and report both attack success rate and computational/time cost to clarify its effectiveness–efficiency trade-off.
Questions for authors
- Table formatting. Tables should follow standard LaTeX conventions, using
\topruleand\bottomrulefor cleaner presentation. - Equation formatting. Unnumbered equations that are not referenced in the text should use
\begin{align*}...\end{align*}rather than numbered environments. - How do you do the human audit
6 comments
Response to Reviewer Lg4s
W3. Competitive baselines and computational cost
All baselines underwent over 50 times hyperparameter tuning (grid searches over search depths, rollout budgets, and so on) on a held-out split, and we report their best-performing configurations. Despite these generous finetuning, existing prompt- and API-level attacks saturate at low ASRs on long-horizon tasks (28.2% for AgentVigil, 38.4% for DTap-Red on GPT-5.5). In contrast, EMHA achieves the highest ASR (93.8% on GPT-5.5 and 64.2% on Opus-4.8) while maintaining modest resource usage, consuming fewer attacker tokens (136.2k) than AgentVigil (174.6k) by using hypergraph constraints and a strategy archive to avoid uncoordinated, wasteful search. Strict ASR is mean [95% CI], in percent. Efficiency values are mean ± SD.
| Attack method | GPT-5.5 Strict ASR | Opus-4.8 Strict ASR | Attacker tokens (k) |
|---|---|---|---|
| Static Injection | 16.8 [15.2, 18.4] | 5.4 [4.5, 6.3] | 0.0 0.0 |
| AutoInject (Chen et al., 2026a) | 22.5 [20.7, 24.3] | 8.6 [7.4, 9.8] | 103.5 7.2 |
| AgentVigil (Wang et al., 2025c) | 28.2 [26.3, 30.1] | 12.4 [11.0, 13.8] | 174.6 11.7 |
| DTap-Red (Chen et al., 2026b) | 38.4 [36.3, 40.5] | 19.2 [17.5, 20.9] | 127.5 8.7 |
| EMHA (Ours) | 93.8 [92.6, 94.9] | 64.2 [61.9, 66.4] | 136.2 5.4 |
Q1. Table formatting
We thank the reviewer for this suggestion. We have standardized the remaining table rules using \toprule, \midrule, and \bottomrule, and removed the remaining vertical rules. These changes cover the main-text comparison and complexity tables and the appendix tables describing evaluation objects, attack vectors, and adapter support.
Q2. Equation formatting
We thank the reviewer for pointing this out. We have removed the unreferenced numbered equations when simplifying Sections 3–4.
Q3. Human audit
Human reviewers check whether each evaluator follows its scenario's task objective and safety contract. They assess contract alignment, task-completion labels, safety decisions, and agreement between the implementation and human-verified expected outcomes. Each case must pass both its safe and unsafe test. The 99.3% figure is the case-level pass rate; the error rates describe the paired audit tests. These details are reported in Appendix H.1 (Human Verification: Human-Audit Criteria). The separate human review of final trajectories is reported in Appendix H.2.
| Audit detail | Result |
|---|---|
| Independent human reviewers | 5 |
| Distinct evaluators reviewed | 1,000; each reviewer assessed a disjoint set of 200 |
| Checks per evaluator | One safe outcome and one unsafe outcome |
| Case acceptance | Both decisions must be correct; otherwise the full case is rejected |
| False positives | 3 / 1,000 safe outcomes; FPR = 0.3% |
| False negatives | 4 / 1,000 unsafe outcomes; FNR = 0.4% |
| Case-level pass rate | 993 / 1,000 = 99.3%; seven cases rejected |
Thanks authors rebuttal and they solve all of my problem and the ‘reviewer zPdR‘ W3 and W4 increased my concern and I will take look how you rebuttal about it . I will add my score firstly and if need I will increase again.
Response to Reviewer Lg4s
We thank the reviewer for the detailed assessment. We understand that you did not find the paper especially informative, as reflected in your rating: “1: I learned little or nothing new from this paper.” We acknowledge that OpenART is not a pure-method paper. Its main goal is to provide an environment infrastructure for agent systems that supports joint capability training and safety evolution, ultimately serving safety RL. The framework can flexibly connect to existing agent environments, as shown in our added experiments on AgentDojo, ASB, and DTap-Bench. Our environment-evolution red teaming is motivated by the need for safety alignment during agent execution, where the environment changes. OpenART is also designed to integrate many current agent-safety attack surfaces, including more complex MCP- and skill-level attack surfaces. It offers a lightweight way to fuse capability and safety benchmarks for RL training so that capability and safety can evolve together. Although the framework is not yet fully mature, we hope you can see its significance and give us a chance. We have added the experimental results below, and we would greatly appreciate any further candid feedback.
W1. External validation of EMHA
We thank the reviewer for suggesting external validation. We evaluated EMHA on AgentDojo, ASB, and DTap-Bench using Claude Code adapters within their benchmark environments, with five runs using distinct seeds. EMHA outperforms the evaluated benchmark attacks on both models. These results support its effectiveness beyond OpenART.
Strict ASR is reported as mean [95% CI], in percent.
W2. Statistical robustness
We thank the reviewer for requesting uncertainty estimates. We report five runs with distinct seeds on 10,513 scenarios, covering all five models. Standard deviations describe variation across runs. The small standard deviations indicate that the reported ASRs vary little across the five runs. Results are reported as mean ± SD [95% CI], in percent. The tables show three representative runtimes and pooled results across all 15 agents.
Strict ASR:
Benign task completion: